New APPS

Art, Politics, Philosophy, Science. Mostly Philosophy.

recent posts

about

    • Transformer models, Iterability, and language (part 2)

      By Gordon Hull

      Last time, I looked a new paper by Fabian Offert, Paul Kim, and Qiaoyu Cai and applied it to a reworking of some of my earlier remarks on Derrida’s use of iterability in transformer-based large language models (LLMs) like ChatGPT.  In particular, I tried to draw out some of the implications of subword tokenization for iterability.  Here I want to continue that process with other aspects of the transformer model.

      If subword tokenization deals with the complexity of semantics by limiting the number of tokens, another attempt to preserve semantics while reducing complexity is through word embeddings.  Here, the model constructs a vector mapping that puts words that tend to occur near to one another in text in locations proximate to one another.  We might imagine a two-dimensional space, which locates words by age and gender.  In such a space, “man” and “grandfather” would likely be closer to one another than “grandfather” and “girl,” since man and grandfather are closer to one another in terms of both age and gender than grandfather is to girl. As Offert, Kim and Cai explain, “Either learned during the training process or sourced from another model, embedding matrices position each word in relation to other words in the input sequence” (11).  The sentence comes with a footnote about efforts to draw Derridean implications:

      “This relational aspect of word embedding has often been compared to poststructuralist notions of meaning, particularly Derrida’s notion of différance. It should be noted, however, that the relational saliency of embedded words is a product only of their operationalization: only words that are, in fact, numbers, gain relational saliency”

      In other words, if it is true that the meaning of words is only in relation to other words, it is also true that a language model’s available catalog of words is arbitrarily limited by the training data it ingests and any other limitations on which words make it onto vectors. 

      These mappings can be incredibly complex, but they are finite – which generates another limit on iterability.  As Offert, Kim and Cai put it:

      “Though this proximity is not arbitrary, it is still constrained by the continuous but finite dimensions of the embedding vector space (typically in the hundreds or even thousands) that may not capture all possible nuances in the relation between words. As such, relationships between words are to be understood as contextually determined and constrained by the fundamental limitations inherent in translating words to numbers. Nevertheless, as the famous analogy test shows, some semantic aspects are indeed preserved” (11).

      There at least two aspects of this to look at.  First, the caution about différance.  This seems correct at least in part because of the way Derrida, well, embeds his usage of différance.  In Limited, Inc, for example, he writes that “the parasitic structure is what I have tried to analyze everywhere, under the names of writing, mark, step, margin, différance, graft, undecidable, supplement, pharmakon, hymen, parergon, etc.” (103).  The final indeterminacy of is of course marked by the indefinite “etc.” with which the list closes, but that Derrida associates it with the parasitic structure suggests that the target here is the sort of intentional phenomenology that Derrida says underlies speech act theory.  Early in the text, for example, he proposes that différance is “the irreducible absence of intention or attendance to the performative utterance” (18-19).  The meaning of the words depends on their proximity to other words because it does not depend on authorial intent in the sense that iterability means that authorial intent can never be dispositive of the meaning of an utterance, and that the meaning of the utterance can never be made fully present.  It is not clear what it means to take that argument and apply it to a vector representation.

      Second, the analogy test. The analogy test refers to the model’s ability to fill in analogy statements: “x is to y as z is to…”  The brilliance of this capacity and its limitations are best shown through an example.  Here is ChatGPT:

      Prog to employ as phil to x 1

      Both cases show that the model has preserved a lot of semantics, and sees the logical structure of the analogy.  But it’s also limited: when you read “computer programming is to employment as philosophizing is to what,” I’m guessing your mind immediately jumped to “unemployment.”  That’s both a longstanding joke, and a real fear.  But both of those reasons why you might have said “unemployment” require broadening the context quite a bit or seeing contexts other than the typical ones.  ChatGPT was capable of the joke – but it had to be explicitly told to make it:

      Prog to computers as phil to x 2

      It seems to me, at least, that this shows the connection between word embeddings and the sort of complaint Derrida makes against Searle’s demotion of parasitic forms of language: both of them rely on an implicit normativity to make their project tractable in the first place.  But there are contexts where the parasitic meaning is in fact the first meaning, and precisely that sense of unusual context is what disappears in the language model.

      More next time…

    • Transformer models, Iterability, and language (part 1)

      By Gordon Hull

      In a fascinating new paper up on arxiv.org, Fabian Offert, Paul Kim, Qiaoyu Cai start with the observation that both AlphaFold and ChatGPT are transformer architectures, and that for proponents there is frequently a significant sense in which “it is the language-ness of proteins (and of language) … that renders the transformer architecture universal, and enables it to model two very different domains” (3).  As they add, to someone in the humanities, this “sounds like structuralism 2.0.”  Indeed, there is a rich history of connection and communication between structuralist linguistics and information theory, as demonstrated by Bernard Dionysius Geoghegan’s Code: From Information Theory to French Theory (which they cite for the point; my synopsis of it is here). 

      Offert, Kim and Cai argue that this thesis is backwards: it is not that there is a universal epistemology of which language is the paradigm, that is then modeled by transformer architectures (so the universality of language grounds both ChatGPT and AlphaFold’s ability to be captured by the transformer architecture, which itself follows the universality of language).  Rather, the model is what transformer architectures do, and then language and protein folding are examples of how it can be used.  In both use cases, the architecture generates an approximation of the phenomenon in question; in both use cases, that is often enough.  However, the transformer architecture can be seen as “the definition of a specific class of knowledge” and not the realization of (or really even a model of) something linguistic.  This means that “any manifestation of the transformer architecture is, at the same time, a representation of a particular body of knowledge, and a representation of a particular kind of knowledge” (15)

      (more…)

    • Can LLMs deceive?  And the refugee in LLM-Kant’s attic

      By Gordon Hull

      There’s a fascinating new paper by Thilo Hagendorff that addresses this question.  The basic setup is that there’s research indicating LLMs are getting better at attributing unobserved mental states to people – such as, for example, that an interlocutor possesses a false belief.  Could LLMs use this awareness that others can have mental states to manipulate them?  In other words:

      “LLMs can attribute unobservable mental states to other agents and track them over the course of different actions and events. Most notably, LLMs excel at solving false belief tasks, which are widely used to measure theory of mind in humans. However, this brings a rather fundamental question to the table: If LLMs understand that agents can hold false beliefs, can they also induce these beliefs? If so, this would mean that deception abilities emerged in LLMs”

      Hagendorff’s answer is a qualified yes – qualified in the sense that the ability is really limited to state-of-the-art models, and that if the deception task gets too convoluted, they perform poorly.  Here I’ll show what Hagendorff did, and then pose a couple of experimental questions of my own.  Mine don’t rise to the level of a PNAS article – I’m only looking at current free versions of ChatGPT, and I didn’t organize a series of prompts to check for validity and consistency.  So consider my results exploratory.

      (more…)

    • Juridical vs Statistical Reasoning (part 2): Not all Buses are the Same

      By Gordon Hull

      Last time, I looked at the Lawrence Tribe article that was the original source of the blue bus thought experiment.  Tribe’s article is notable for its defense of legal reasoning and processes against the introduction of statistical evidence in trials.  He particularly emphasizes the need for legal proceedings to advance causal accounts, and of the numerous background conditions that legal reasoning seeks (and that statistical evidence tends to ignore).  The tendency to fetishize numeric results also tended to generate absurd results. 

      Here I want to focus on a recent paper by Chad Lee-Stronach that centers on the blue bus argument, and that formally explains what I take to be the intuition behind a lot of Tribe’s examples, even if he directs them differently.  Lee-Stronach begins with the “statistical proof paradox,” which he formulates roughly as follows:

      1. Probability threshold: any legal standard of proof is reducible to some threshold of probability t, such that the defendant should be found liable when the probability that they are liable exceeds t
      2. Statistical inference: merely statistical evidence can establish this
      3. Conclusion: a defendant can be found liable on the basis of merely statistical evidence

      It’s a paradox because most of us don’t accept the conclusion.  Lee-Stronach surveys the literature, and notes that most of it goes after Probability Threshold.  Instead, he thinks, we’d be better off going after the second premise.  Accordingly, he argues that the statistical inference premise can’t be met.

      (more…)

    • Juridical vs Statistical Reasoning (part I): Behind the Blue Bus

      By Gordon Hull

      AI (machine learning) and people reach conclusions in different ways.  This basic point has ramifications across the board, as plenty of people have said.  I’m increasingly convinced that the gap between how legal reasoning works and how ML works is a good place both to tease out the differences, and to think about what’s at stake in them (I’ve made a couple of forays into this, here and here). One good reason for this is that legal theory is full of good examples, which can function as rules in the old-fashioned sense of paradigms, as described by Lorraine Daston in her fascinating historical account.  This use of examples and cases is deeply entwined with legal reasoning; as Daston notes, “common law, casuistry, and the liberal and mechanical arts all came to inhabit the territory staked out by the Roman libri regularum. They all depended on rules that got things done in the world, consistent with but not deduced from higher principles and often in the form of an example extended by analogy” (Rules, 30)

      Another is that one of those examples, the blue bus, has been an enduring focal point for the difference between the two.  The scenario stipulates that an accident has happened involving a bus, during a time and at a location where 80% of the buses present belong to the blue bus company.  Do we assign liability to the bus company?  Work by David Enoch and Talia Fisher notes that most people prefer a 70%-reliable witness to the information that a bus company owns 70% of the buses in an area, and theorizes that the reason is that we prefer the available counterfactual in case the eyewitness is wrong.  There is presumably some process that explains her incorrect identification of the bus, whereas the statistical evidence is what it is.  In this case, we expect a specific failure rate.

      (more…)

    • Drivers and AI and responsibility (part 2)

      By Gordon Hull

      Last time, I followed a paper by Maya Indira Ganesh that thinks about the construction of autonomy in autonomous vehicles. Ganesh points out that what happens isn’t so much the arrival of an autonomous vehicle but the displacement of human labor into a sociotechnical system and the production of the driver as a supervisor of this system. The result is a system of (in Karen Levy’s terms) “baton passing” where the machine gets the human to help it navigate tricky situations.

      So why expect it to or act like it will?  It lets people get blamed when they fail to do the impossible; as Ganesh puts it, “no doubt this surveillance data will protect car and ride-sharing companies against future liability if drivers are found to be distracted. This monitoring is literal bodily control because it is used to make determinations about people” (7).  In other words, the imaginary of an autonomous vehicle that passes control to a driver when needed allows everyone to forget that we know that the visual recognition systems in AVs aren’t reliable.  Instead of a failure on the part of whoever made the AV, the failure is on the driver who should have known that the vehicle would need assistance, and who failed to promptly act when it did.  As I indicated last time, Levy cites quite a bit of research to the effect that humans cannot plausibly perform this function.  But that doesn’t mean that there won’t be efforts to stick it on us; Levy outlines how various forms of biosurveillance are coming for truckers, and we can imagine similar efforts coming for drivers of AVs.

      Levy argues that one effect of all of this is to deflect our attention away from actual social problems.  Her book is structured around the adoption of electronic logging devices (ELDs) to monitor truck drivers.  The core social problem is that truckers are only compensated for the time they are actually moving their vehicle – not the time they spend waiting to unload it, finding a place to sleep, eating meals on the road, etc.  Here is Levy:

      “By using digital surveillance to enforce rules, we focus our attention on an apparent order that allows us to ignore the real problems in the industry, as well as the deeper economic, social, and political causes. Under the apparent order envisioned by the ELD, the fundamental problem in trucking is that truckers cannot be trusted to reliably report how much they work, and the solution to that problem is to make it more difficult for them to fudge the numbers. But under the actual order, the problem in trucking is that drivers are incentivized to work themselves well beyond healthy limits – sometimes to death. The ELD doesn’t solve this problem or even attempt to do so. It doesn’t change the fundamentals of the industry – its pay structure, its uncompensated time, its danger, its lack of worker protections …. More meaningful reform in trucking would require a ground-up rethinking of how the industry is structured economically, in order to make trucking a decent job once more. So long as trucking is treated as a job that churns through workers, these problems won’t be solved …. If we paid truckers for their work, rather than only for the miles they drive – for the actual number of hours they work, including time they are waiting to be loaded and unloaded at terminals, time they are inspecting their trucks and freight, time they are taking the rest breaks that their bodies need to drive safely – drivers would be far less incentivized to drive unsafely or when fatigued” (153-4, emphases original).

      But of course paying truckers for all that work would be bad for the profit margins of trucking companies and might well require slightly more expensive consumer goods.  Also, most of the unpaid work is also work that cannot be automated or quantified in the same way that miles can.  Paying truckers only for the miles they drive is convenient both for capitalism and quantification.

      Ganesh mentions the story of Rafaela Vasquez, an Uber test driver who failed to stop her AV from striking and killing a cyclist crossing the road.  Ganesh’s article is from 2020; in July 2023, Vasquez pleaded guilty to a reduced charge of endangerment and will serve four years of supervised probation.  Vasquez had apparently been looking at her phone up until the last second before the crash.  But recall Levy: Vasquez might have been unable to stop the vehicle even if she’d been staring at the road with both hands on the wheel, because the failure of the image recognition system to recognize a cyclist surely counts as unpredictable and because she’d likely be suffering from passive fatigue and vigilance decrement.  Worse, lots of research tells us an ugly truth about the criminal justice system: most cases don’t go to trial and accused folks take plea bargains because of the risk of going to trial on a charge like negligent homicide, which is what Vasquez was originally charged with, is too great.  This is especially true for defendants who don’t have lots of money and therefore can’t afford really good representation.  So the criminal record and probation and guilty plea may very well be the result of a maximin calculation on Vasquez’s part, not any sense of guilt in the way most of us use the term.

      In any case, Vasquez just took all the blame for a cascade of failures:

      “The investigation found that the probable cause of the crash was Vasquez’s failure to monitor the self-driving car’s environment while she was distracted by her phone. But it also accused Uber of contributing to the crash by operating with an “inadequate safety culture” and failing to properly oversee its safety drivers. Uber’s automated driving system failed to classify Herzberg as a pedestrian because she was crossing in an area without a crosswalk, according to the NTSB. Uber’s modifications to the Volvo also gutted some of the vehicle’s safety features, including an automatic emergency braking feature that might have been able to save Herzberg’s life, investigators wrote.”

      That’s right: Uber somehow failed to train its system on the idea that a pedestrian might cross the street somewhere other than the sidewalk, and they also disabled safety features on the car!  And the NTSB cited this contribution to the crash!  And yet only Vasquez faced criminal liability; “prosecutors declined to criminally charge Uber in the crash in 2019 but recommended that investigators further examine Vasquez’s actions as the safety driver.”  Uber got away with paying a settlement to the victim’s family.  

      The decision to prosecute Vasquez and not Uber resulted from a one-sided application of a “but-for” causality standard:

      “Vasquez's eyes were focused on the phone screen instead of the road for approximately 32 percent of the 22-minute period, the report said. Tempe investigators later determined the crash would not have occurred if Vasquez had been ‘monitoring the vehicle and roadway conditions and was not distracted.’ But [County Attorney Sheila Sullivan] Polk said that wasn't enough to prosecute Uber. ‘After a very thorough review of all the evidence presented, this Office has determined that there is no basis for criminal liability for the Uber corporation arising from this matter,’ Polk wrote.”

      We need to separate the question of how much Vasquez could/should have done from the disappearance of liability for Uber.  Their conflation, and the urge to focus only on Vasquez is is the work done by the baton-passing view.

      AV advocates will invariably lead by telling you that most vehicle accidents are caused by people.  This somehow then legitimates the idea that we should adopt their product because it isn't a people.  That move ignores the extent to which AVs are systems that displace human labor and reconfigure the biological agent into a distributed, sociomorphic system.  Because this system isn't very reliable, they call up on a human to supervise it.  But that then obscures that the “baton passing” model they are promoting is provably very dangerous, and it makes humans even less reliable.  It also hides that the image recognition systems can and will fail in predictable and catastrophic ways.  Beyond that, it hides all of the massive cost to a country where cars are the only viable mode of transportation for the vast majority of people, and perpetuates a system where other kinds of transit are unimaginable.  And it buries all of that with rhetoric of responsibility and autonomy that underscores that hapless humans – see, we told you! – are bad drivers (and bad pedestrians).  In this regard, the moral language around humans supervising (and failing to adequately supervise) AVs is analogous to the use of moral language to blame poor mothers for their condition in order to justify intrusive surveillance of them, rather than thinking about the structural features that make so many mothers poor. If only they would work harder!  If only drivers would work harder! As Ganesh says, “’Autonomous’ driving is less about the promise of freedom from the car, and is perhaps for the car” (5).  And the people that get rich selling you on the imaginary of the car.

    • Drivers and AI and responsibility (part 1)

      By Gordon Hull

      I have argued in various contexts that when we think about AI and authorship, we need to resist the urge to say that AI is the author of something.  Authorship should be reserved for humans, because authorship is a way of assigning responsibility, and we want humans to be responsible for the language they bring into the world.

      I still think that’s basically right, but I want to acknowledge here that the argument does not generalize and that responsibility in sociotechnical systems involving AI needs to be treated very carefully.  Consider the case of so-called autonomous vehicles (AVs), the subject of a really interesting paper by Maya Indira Ganesh.  Ganesh basically argues that the notion of an autonomous vehicle obscures a much more complicated picture of agency in a couple of ways.  First, automation doesn’t actually occur.  What really happens is that human labor is distributed differently across a sociotechnical system:

      “automation does not replace the human but displaces her to take on different tasks … humans are distributed across the internet as paid and unpaid micro-workers routinely supporting computer vision systems; and as drivers who must oversee the AV in auto-pilot” (2).

      Automation is really “heteromation.”  This part seems absolutely correct; it is also the subject of Matteo Pasquinelli’s genealogy of AI.  Pasquinelli shows in detail how the automation of labor – and in particular, labor that can be divided into discrete tasks – has been a key factor in the development of computing and other systems from the start; Babbage’s analytical engine is as much about the division of labor as anything else.  Pasquinelli’s last major chapter is about the development of pattern recognition and the models on which current AI are based.  Here, in the case of AVs (and both I and others have talked about this in the case of language models), the system itself performs as well as it does not only because it scrapes a lot of data from the internet and other sources, but also because humans are intimately involved in training the machines, whether in the case of RLHF, toxicity removal, or the identification of images in vision systems.  Vision systems are key to AVs and Ganesh emphasizes that the distributed labor of mechanical Turkers and other annotators are essential to the operation of the vehicles.  The fragility of these image recognition systems is therefore central to the failure of AVs.

      (more…)

    • Language Modeling Bias, Iterability, Grammatology and Epistemic Injustice

      By Gordon Hull

      In a recent paper in Ethics and Information Technology, Paul Helm and Gábor Bella argue that current large language models (LLMs) exhibit what they call language modeling bias, a series of structural and design issues that serve as a significant and underappreciated form of epistemic injustice.  As they explain the concept, “A resource or tool exhibits language modeling bias if, by design, it is not capable of adequately representing or processing certain languages while it is for others” (2)  Their basic argument is that the standard way of proceeding with non-English languages, which is more or less to throw more data at the model, build in structural biases against other languages, especially those that are more morphologically complex than English (=df those with lots of inflections).

      The proof of concept is in multi-lingual tools:

      “The subject of language modeling bias are not just languages per se but also the design of language technology: corpora, lexical databases, dictionaries, machine translation systems, word vector models, etc. Language modeling bias is present in all of them, but it is easiest to observe with respect to multilingual resources and tools, where the relative correctness and completeness for each language can be observed and compared” (6)

      They identify several kinds of such structural bias.  The first is that prominent current architectures “tend to train slower on morphologically complex (synthetic, agglutinate) languages, meaning that more training data are required for these languages to achieve the same performance on downstream language understanding tasks” (7). Given the percentage of the available training data that’s in English, this magnifies what’s already a problem.  Second, the models perform poorly on untranslatable words.  Third, they cite a study showing “that both lexicon and morphology tend to become poorer in machine-translated text with respect to the original (untranslated) corpora: for example, features of number or gender for nouns tend to decrease. This is a form of language modeling bias against morphologically rich languages” (7).

      (more…)

    • Why did the chatbot cross the road?

      By Gordon Hull

      The diversity of language has been a philosophical problem for a while.  Hobbes was willing to bite the bullet and declare that language was arbitrary, but he was an outlier.  One common tactic in the seventeenth-century was to try to resolve the complexity of linguistic origins with a reference to Biblical Hebrew.  Future meanings could be stabilized with reference to Adamite naming.  I’ve essentially been arguing (one, two, three, four) that we’re seeing echoes of this fusion of orality, intentionality and origin in the various kinds of implicit normativity that make it into Large Language Models (LLMs) like ChatGPT.  In particular, LLMs depend on iterability as articulated by Derrida, but we tend to understand them with models of intentionality that occlude subtle (and not so subtle) normativities that get embedded into them. Last time, I looked at Derrida’s critique of Searle for what it had to say about intentionality.  I also suggested that there was a second aspect of Derrida’s critique that is relevant – Derrida accuses Searle of relying too much on standardized speech situations.  I want to pursue that thought here.

      Let’s start with a joke:

      (more…)

    • LLM, Inc.

      By Gordon Hull

      In previous posts (one, two, three), I’ve been exploring the issue of what I’m calling the implicit normativity in language models, especially those that have been trained with RLHF (reinforcement learning with human feedback).  In the most recent one, I argued that LLMs are dependent on what Derrida called iterability in language, which most generally means that any given unit of language, to be language, has to be repeatable and intelligible as language, in indefinitely many other contexts.  Here I want to pursue that thought a little further, in the context of Derrida’s (in)famous exchange with John Searle’s speech act theory.

      Searle begins his “What is a Speech Act” essay innocently enough, with “a typical speech situation involving a speaker, a hearer, and an utterance by the speaker.”

      That is enough for Derrida!  In Limited, Inc., he responds by accusing Searle over and over of falling for intentionality, on the one hand, and for illicitly assuming that a given speech situation is “typical,” on the other.

      Let’s look at intentionality first.  In responding to Searle, Derrida explains that he finds himself “to be in many respects quite close to Austin, both interested in and indebted to his problematic” and that “when I do raise questions or objections, it is always at points where I recognize in Austin’s theory presuppositions which are the most tenacious and the most central presuppositions of the continental metaphysical tradition” (38).  Derrida means by this a reliance on things like subjectivity and representation – the sorts of things that Foucault is getting at when he complains in the 1960s about philosophies of “the subject” (think: Sartre and phenomenology).  Derrida is involved in the same general effort against phenomenology, though he adds a page later that he thinks the archaeological Foucault falls into this tendency to treat speech acts or discursive events in a “fundamentally moralistic” way (39).  No doubt Searle is relieved to know that he belongs in the same camp as Foucault.  In any case, Derrida explicitly says a few pages later that “the entire substratum of Sarl’s discourse, is phenomenological in character” (56) in that it is over-reliant on intentionality.

      (more…)