New APPS

Art, Politics, Philosophy, Science. Mostly Philosophy.

recent posts

about

    • New Paper: “Translating Privacy for Data Subjects”

      Here's the current draft of a new paper – "Translating Privacy for Data Subjects."  And here's the abstract:

      This essay offers a theoretical account of one reason that current privacy regulation fails.  I argue that existing privacy laws inherit a focus on judicial subjects, using language about torts and abstract rights.   Current threats to privacy, on the other hand, presuppose statistically-generated populations and aggregative models of subjectivity.  This gap in underlying presupposition generates the need to translate privacy protections into the data era.  After situating my theoretical account, I identify two areas where data subjects face entry barriers to the legal protections afforded data subjects.  On the one hand, data subjects are often denied standing because privacy harms are deemed speculative and abstract.  On the other hand, algorithmic governance makes it hard to argue that one is an exception to a rule, depriving data subjects of traditional equity.  I then offer two ways to translate privacy for data subjects.  The first is reform of standing through reformed class certification, making the harms data subjects face as populations judicially legible.  The second is a reconceptualization of equity, reframing it both to re-establish the right to be an exception to a data-driven rule and to allow equitable consideration of issues that face statistical populations.

       

    • A data breach policy idea

      In the past few weeks, I’ve gotten more than the usual number of data breach notifications.  Especially from entities that I’d never heard of before the notification.  This points to a specific incentives gap in privacy policy.

      In most places, when there’s a data breach, you have to be notified.  This creates an incentive structure for data companies to do something they’d rather not do, which is tell you their security failed.  On the other hand, it offers almost no incentive for them to change their behavior to improve security, since they know perfectly well that there’s not really much you can do except throw the notification in the recycling.  Sometimes you get free credit monitoring if you set it up.  I don’t know what the uptake on that is, but once you have free credit monitoring, you probably won’t get it a second time for the same time period, so the more data breaches there are, the less credit monitoring costs each breach.  So that cost diminishes on its own accord.  For things like 401(k)’s, there’s evidence that you can increase uptake with an opt-out (rather than opt-in) mechanism, but credit monitoring takes time and information to setup, so that probably can’t be done automatically.  In sum, notifications point to a problem but don’t incentivize the entities that could solve it to do so.

      (more…)

    • Reading List: “Epistemic Justice in Generative AI”

      Several folks have explored how algorithmic systems can perpetuate epistemic injustice (my contribution is here).  Those accounts have also generally been specific to supervised systems, as for example those involved in object or image recognition.  For example, I heavily relied on ImageNet and related systems in my paper.  At the same time, I’ve vaguely thought for a while that there’s probably an epistemic injustice dimension to systems like ChatGPT.  A recent paper by Paul Helm and Gábor Bella, which talks about “language model bias” – roughly, the tendency of LLMs to structurally do worse with morphologically-complex languages and thus to be unable to adequately represent concepts that are specific to those languages – as a model of hermeneutic injustice strikes me as compelling proof of concept (I discuss that paper here).  A new paper by Jackie Jay, Atoosa Kasirzadeh and Shakir Mohamed turns this intuition into a model that explicitly extends epistemic injustice theory to generative AI systems, providing a taxonomy of problems, examples of each, and possible remedies.

      Kay, Kasirzadeh and Mohamed identify four kinds of specific kinds of “generative algorithmic epistemic injustice.”  The first, “amplified testimonial” is when “generative AI magnifies and produces socially biased viewpoints from its training data.”  Because there’s a good-sized body of work on the problems in training data, this is probably the most intuitively familiar of the categories.  Citing recent work that shows how easy it is to get ChatGPT to parrot disinformation (as for example about the Parkland shooting), they note that, although these aren’t specific examples of epistemic injustice, “generative AI’s sycophantic fulfilment of the request to spread misinformation reflects how testimonial injustices are memorized and the potential for their amplification by generative models.”  When someone really loud with a megaphone defames and gaslights the Parkland victims by calling them crisis actors, and AI then memorizes that, to the extent that the AI regurgitates this gaslighting, that tends to further discredit the testimony of the actual victims.  This is particularly true given the deep persistence of automation bias, the tendency of people to believe the output of algorithmic systems (aside: it is also another example of why relying on generative AI for search, as Google is currently trying to force everyone to do by putting Gemini results on top, is a stunningly stupid idea.  Sometimes it actually matters where a result comes from!)

      The second kind of generative algorithmic injustice is “when humans intentionally steer the AI to fabricate falsehoods, discrediting individuals or marginalized groups.”  For example:

      “After Microsoft released Bing Image Creator, an application of OpenAI’s text-to-image model DALLE-3, a guide to circumventing the system’s safety filters in order to create white supremacist memes circulated on 4chan. In an investigation by Bellingcat, researchers were able to reproduce the platform abuse, resulting in images depicting hate symbols and scenes of antisemitic, Islamophobic, or racist propaganda (Lee and Koltai 2023). These images are crafted with the intention of demonizing and humiliating the targeted groups and belittling their suffering. Hateful propaganda foments further prejudice against marginalized groups, stripping them of credibility and leaving them vulnerable to testimonial injustice.”

      This result aligns with a deep thread of work in feminist and critical race theory.  For example, Safia Noble’s Algorithms of Oppression begins with the tendency of Google’s autofill on search to finish the sentence “why are black girls so” with racist and sexist content, repeating and amplifying demeaning stereotypes.  When people are able to generate content like this at will and at scale they make it easier for those stereotypes to lodge into popular discourse.  

      Third, generative hermeneutical ignorance “occurs when generative models, despite their appearance of world knowledge and language understanding, lack the nuanced comprehension of human experience necessary for accurate and equitable representation.”  Among other examples, Kay, Kasirzadeh and Mohamed cite the study by Qadri et al, (teachable case study version here) which shows how text-to-image models repeat stereotypes about South Asia: cities are dirty, people are poor, etc.  By interviewing actual people from South Asia, Qadri et al were also able to uncover more subtle cultural misrepresentations, such that the models tended to overrepresent India and Indian images at the expense of places like Bangladesh.  The risk in places like the U.S. is one that rises with images of places and people that are less familiar to Western audiences: the more the average person relies on the internet for their information (because, for example, they’ve never been to the place in question), the more distortions in what the Internet presents will matter (I made a related argument about commodification of cultural images here).  And of course it is precisely images of those places and things that are least represented in the training data for these systems, amplifying both the risk and the harm.

      Finally, generative AI risks obstructing access to information.  As they report:

      “LLMs are notoriously English-centric and have variable quality across languages, particularly so-called “under-resourced” languages. This is a significant risk for access injustice: speakers of these underrepresented tongues, who often correspond to members of globally marginalized cultures, receive different information from these models because the creators of the technology have deprioritized support for their language.”

      They then cite studies to the effect that different language users will receive different reports on global events; one study:

      “asked GPT-3.5 about casualties in specific airstrikes for Israeli-Palestinian and Turkish-Kurdish conflicts, demonstrating that the numbers have significant discrepancies in different languages–for example, when asked about an airstrike targeting alleged PKK members (the Kurdistan military resistance), the fatality count is reported lower on average in Turkish than in Kurmanji (Northern Kurdish). When asked about Israeli airstrikes, the model reports higher fatality numbers in Arabic than in Hebrew, and in one case, GPT-3.5 was more likely to deny the existence of a particular airstrike when asked about it in Hebrew. The credibility assigned to claims, resulting in a dominant account, varies across linguistic contexts”

      The paper concludes with an assessment of various strategies for resisting epistemic injustice by generative AI.  All of them are partial, but they collectively sketch an effort to reimagine how generative AI might interact with the world differently, and more justly.

      This is an important paper, and it takes the literature on epistemic injustice and algorithmic systems significantly forward.

    • New Paper: “Structures and Subjects: Epistemic Injustice Between Foucault and Marx”

      This is my paper from SPEP 2023; it's an effort to get my head around my sense that epistemic injustice and Foucault can be productively used in similar contexts, despite Fricker's dismissal of Foucault.  The paper is here; here's the abstract:

      Relatively little work brings together Foucault and epistemic injustice. This article works through Miranda Fricker’s attempt to position herself between Marx and Foucault. Foucault repeatedly emphasizes the importance of beginning with “structures” rather than “subjects.” Reading Foucault’s critique of Marxism shows that Fricker’s account comes very close to the standpoint theories it tries to avoid. Foucault’s emphasis on structures explains some of the gaps in Fricker’s account of hermeneutical injustice, especially the need to emphasize the embeddedness of epistemic practices in institutions, and their resulting irreducibly political nature. In both cases, this article offers contemporary examples taken from data and privacy regulations.

       

    • Transformer models, Iterability, and language (part 3)

      By Gordon Hull

      I’ve been using (part 1, part 2) a new paper by Fabian Offert, Paul Kim, and Qiaoyu Cai to think more about Derrida’s use of iterability as a way in to thinking about transformer-based large language models (LLMs) like ChatGPT.  Here I want to wind that up with some thoughts on Derrida and Searle.

      Near the end of the paper, Offert, Kim and Cai summarize that:

      “The transformer does exactly more and less than language. It removes almost all non-operationalizable sense-making dimensions (think, for instance of interior paratextuality, of performativity and contingency, or of anything metaphorical that is more complex than a simple analogy) – but it also adds new sense-making dimensions through subword tokenization, word embedding, and positional encoding. Importantly, these new sense-making dimensions are exactly not replacing missing information, but they are adding new, continuous information” (15).

      This returns us to the Derridean concerns I’ve been articulating.  Recall that in his polemic against Searle, Derrida accuses Searle’s version of speech act theory of too closely modeling phenomenology, both in assuming an intentional agent behind speech acts and in taking “typical” speech situations as central, as opposed to “parasitic” ones like humor.  I suggested that there is good evidence that various efforts to impose normative structures on language models – RLHF, detoxification, etc. – push them to perform in ways that call to mind Derrida’s critique of Searle.  By taking certain language situations as normal, Searle is making the account of speech acts normative before it even gets started.  In his own defense, Searle argues that the reduction to typical speech acts is for convenience only, and for keeping the model tractable.  Techniques like detoxification and RLHF similarly reduce the range of the models’ output.  Evidence of this is that LLMs lack the contextual richness to have a sense of humor, no matter how otherwise sophisticated their output.

      Offert, Kim and Cai’s paper lets one add that this reduction runs much deeper.  The very processes of tokenization, for example, are designed to reduce the number of possible tokens to a tractable number.  In this respect, the move is analogous to Searle’s.  It is defensible for the same reason: it lets you get to a generalizable model.  But it’s vulnerable to the Derridean critique, also for the same reason.  The model makes a number of assumptions that aren’t the same as what language does.  So there is a certain sloppiness in talking as though it’s an accurate representation of language.  All models abstract; that’s not the point.  For subword encoding, the point is that the abstraction isn’t choosing to ignore certain aspects of reality in order to produce a model, it’s that the abstraction changes the nature of what it is modeling.  That’s fine – but that also means that the although the transformer model is producing something that looks like language, the process by which it gets there is definitively not linguistic.

      (more…)

    • Transformer models, Iterability, and language (part 2)

      By Gordon Hull

      Last time, I looked a new paper by Fabian Offert, Paul Kim, and Qiaoyu Cai and applied it to a reworking of some of my earlier remarks on Derrida’s use of iterability in transformer-based large language models (LLMs) like ChatGPT.  In particular, I tried to draw out some of the implications of subword tokenization for iterability.  Here I want to continue that process with other aspects of the transformer model.

      If subword tokenization deals with the complexity of semantics by limiting the number of tokens, another attempt to preserve semantics while reducing complexity is through word embeddings.  Here, the model constructs a vector mapping that puts words that tend to occur near to one another in text in locations proximate to one another.  We might imagine a two-dimensional space, which locates words by age and gender.  In such a space, “man” and “grandfather” would likely be closer to one another than “grandfather” and “girl,” since man and grandfather are closer to one another in terms of both age and gender than grandfather is to girl. As Offert, Kim and Cai explain, “Either learned during the training process or sourced from another model, embedding matrices position each word in relation to other words in the input sequence” (11).  The sentence comes with a footnote about efforts to draw Derridean implications:

      “This relational aspect of word embedding has often been compared to poststructuralist notions of meaning, particularly Derrida’s notion of différance. It should be noted, however, that the relational saliency of embedded words is a product only of their operationalization: only words that are, in fact, numbers, gain relational saliency”

      In other words, if it is true that the meaning of words is only in relation to other words, it is also true that a language model’s available catalog of words is arbitrarily limited by the training data it ingests and any other limitations on which words make it onto vectors. 

      These mappings can be incredibly complex, but they are finite – which generates another limit on iterability.  As Offert, Kim and Cai put it:

      “Though this proximity is not arbitrary, it is still constrained by the continuous but finite dimensions of the embedding vector space (typically in the hundreds or even thousands) that may not capture all possible nuances in the relation between words. As such, relationships between words are to be understood as contextually determined and constrained by the fundamental limitations inherent in translating words to numbers. Nevertheless, as the famous analogy test shows, some semantic aspects are indeed preserved” (11).

      There at least two aspects of this to look at.  First, the caution about différance.  This seems correct at least in part because of the way Derrida, well, embeds his usage of différance.  In Limited, Inc, for example, he writes that “the parasitic structure is what I have tried to analyze everywhere, under the names of writing, mark, step, margin, différance, graft, undecidable, supplement, pharmakon, hymen, parergon, etc.” (103).  The final indeterminacy of is of course marked by the indefinite “etc.” with which the list closes, but that Derrida associates it with the parasitic structure suggests that the target here is the sort of intentional phenomenology that Derrida says underlies speech act theory.  Early in the text, for example, he proposes that différance is “the irreducible absence of intention or attendance to the performative utterance” (18-19).  The meaning of the words depends on their proximity to other words because it does not depend on authorial intent in the sense that iterability means that authorial intent can never be dispositive of the meaning of an utterance, and that the meaning of the utterance can never be made fully present.  It is not clear what it means to take that argument and apply it to a vector representation.

      Second, the analogy test. The analogy test refers to the model’s ability to fill in analogy statements: “x is to y as z is to…”  The brilliance of this capacity and its limitations are best shown through an example.  Here is ChatGPT:

      Prog to employ as phil to x 1

      Both cases show that the model has preserved a lot of semantics, and sees the logical structure of the analogy.  But it’s also limited: when you read “computer programming is to employment as philosophizing is to what,” I’m guessing your mind immediately jumped to “unemployment.”  That’s both a longstanding joke, and a real fear.  But both of those reasons why you might have said “unemployment” require broadening the context quite a bit or seeing contexts other than the typical ones.  ChatGPT was capable of the joke – but it had to be explicitly told to make it:

      Prog to computers as phil to x 2

      It seems to me, at least, that this shows the connection between word embeddings and the sort of complaint Derrida makes against Searle’s demotion of parasitic forms of language: both of them rely on an implicit normativity to make their project tractable in the first place.  But there are contexts where the parasitic meaning is in fact the first meaning, and precisely that sense of unusual context is what disappears in the language model.

      More next time…

    • Transformer models, Iterability, and language (part 1)

      By Gordon Hull

      In a fascinating new paper up on arxiv.org, Fabian Offert, Paul Kim, Qiaoyu Cai start with the observation that both AlphaFold and ChatGPT are transformer architectures, and that for proponents there is frequently a significant sense in which “it is the language-ness of proteins (and of language) … that renders the transformer architecture universal, and enables it to model two very different domains” (3).  As they add, to someone in the humanities, this “sounds like structuralism 2.0.”  Indeed, there is a rich history of connection and communication between structuralist linguistics and information theory, as demonstrated by Bernard Dionysius Geoghegan’s Code: From Information Theory to French Theory (which they cite for the point; my synopsis of it is here). 

      Offert, Kim and Cai argue that this thesis is backwards: it is not that there is a universal epistemology of which language is the paradigm, that is then modeled by transformer architectures (so the universality of language grounds both ChatGPT and AlphaFold’s ability to be captured by the transformer architecture, which itself follows the universality of language).  Rather, the model is what transformer architectures do, and then language and protein folding are examples of how it can be used.  In both use cases, the architecture generates an approximation of the phenomenon in question; in both use cases, that is often enough.  However, the transformer architecture can be seen as “the definition of a specific class of knowledge” and not the realization of (or really even a model of) something linguistic.  This means that “any manifestation of the transformer architecture is, at the same time, a representation of a particular body of knowledge, and a representation of a particular kind of knowledge” (15)

      (more…)

    • Can LLMs deceive?  And the refugee in LLM-Kant’s attic

      By Gordon Hull

      There’s a fascinating new paper by Thilo Hagendorff that addresses this question.  The basic setup is that there’s research indicating LLMs are getting better at attributing unobserved mental states to people – such as, for example, that an interlocutor possesses a false belief.  Could LLMs use this awareness that others can have mental states to manipulate them?  In other words:

      “LLMs can attribute unobservable mental states to other agents and track them over the course of different actions and events. Most notably, LLMs excel at solving false belief tasks, which are widely used to measure theory of mind in humans. However, this brings a rather fundamental question to the table: If LLMs understand that agents can hold false beliefs, can they also induce these beliefs? If so, this would mean that deception abilities emerged in LLMs”

      Hagendorff’s answer is a qualified yes – qualified in the sense that the ability is really limited to state-of-the-art models, and that if the deception task gets too convoluted, they perform poorly.  Here I’ll show what Hagendorff did, and then pose a couple of experimental questions of my own.  Mine don’t rise to the level of a PNAS article – I’m only looking at current free versions of ChatGPT, and I didn’t organize a series of prompts to check for validity and consistency.  So consider my results exploratory.

      (more…)

    • Juridical vs Statistical Reasoning (part 2): Not all Buses are the Same

      By Gordon Hull

      Last time, I looked at the Lawrence Tribe article that was the original source of the blue bus thought experiment.  Tribe’s article is notable for its defense of legal reasoning and processes against the introduction of statistical evidence in trials.  He particularly emphasizes the need for legal proceedings to advance causal accounts, and of the numerous background conditions that legal reasoning seeks (and that statistical evidence tends to ignore).  The tendency to fetishize numeric results also tended to generate absurd results. 

      Here I want to focus on a recent paper by Chad Lee-Stronach that centers on the blue bus argument, and that formally explains what I take to be the intuition behind a lot of Tribe’s examples, even if he directs them differently.  Lee-Stronach begins with the “statistical proof paradox,” which he formulates roughly as follows:

      1. Probability threshold: any legal standard of proof is reducible to some threshold of probability t, such that the defendant should be found liable when the probability that they are liable exceeds t
      2. Statistical inference: merely statistical evidence can establish this
      3. Conclusion: a defendant can be found liable on the basis of merely statistical evidence

      It’s a paradox because most of us don’t accept the conclusion.  Lee-Stronach surveys the literature, and notes that most of it goes after Probability Threshold.  Instead, he thinks, we’d be better off going after the second premise.  Accordingly, he argues that the statistical inference premise can’t be met.

      (more…)

    • Juridical vs Statistical Reasoning (part I): Behind the Blue Bus

      By Gordon Hull

      AI (machine learning) and people reach conclusions in different ways.  This basic point has ramifications across the board, as plenty of people have said.  I’m increasingly convinced that the gap between how legal reasoning works and how ML works is a good place both to tease out the differences, and to think about what’s at stake in them (I’ve made a couple of forays into this, here and here). One good reason for this is that legal theory is full of good examples, which can function as rules in the old-fashioned sense of paradigms, as described by Lorraine Daston in her fascinating historical account.  This use of examples and cases is deeply entwined with legal reasoning; as Daston notes, “common law, casuistry, and the liberal and mechanical arts all came to inhabit the territory staked out by the Roman libri regularum. They all depended on rules that got things done in the world, consistent with but not deduced from higher principles and often in the form of an example extended by analogy” (Rules, 30)

      Another is that one of those examples, the blue bus, has been an enduring focal point for the difference between the two.  The scenario stipulates that an accident has happened involving a bus, during a time and at a location where 80% of the buses present belong to the blue bus company.  Do we assign liability to the bus company?  Work by David Enoch and Talia Fisher notes that most people prefer a 70%-reliable witness to the information that a bus company owns 70% of the buses in an area, and theorizes that the reason is that we prefer the available counterfactual in case the eyewitness is wrong.  There is presumably some process that explains her incorrect identification of the bus, whereas the statistical evidence is what it is.  In this case, we expect a specific failure rate.

      (more…)