New APPS

Last time, I took a detour from the discussion (part one, two, three, four, five, six, seven) of Platonism (in Derrida’s sense) in language models to look at Plato’s work itself, emphasizing how important mythmaking and storytelling are to it.  Behind that, it seems to me that Derrida’s critique of Plato and Hegel on writing offers some useful points for thinking about what LLMs do.  On the one hand, LLMs show that the priority of speech over writing, insofar as that priority is based on some sort of metaphysical preference for speech as the correct representation of an eidetic truth function, makes no sense.  It’s better to read that priority as a political preference and to treat it as such.  A central point in the deconstruction of this priority is the displacement of the word as the fundamental unit of language.  This much is also evident in Wittgensteinian approaches to language, which (as Lydia Liu argues) shows up in early research in machine translation.  That research makes explicit use of written Chinese as a model for thinking about meaning as distributional.  Chinese is also near the bottom of the Hegelian hierarchy of languages, and one could image it the absolute bottom of a Platonic one.

On the other hand, a second Platonism is evident in the assumed priority of a unified speaking subject behind language production.  Whatever else they are, language models aren’t unified speaking subjects in any meaningful sense of the term. To the extent that LLMs appear as unified subjects, that is an artifact of some very specific coding and training decisions made for (broadly construed) social and political reasons.

Both of these suggest that the attention to Platonism is worthwhile for another reason: it draws attention to the ways that storytelling and mythmaking around language and computation are essential to the social meaning of language models. 

Seeing all of this mythmaking and storytelling may very well require reading Derrida against himself, or at least against the grain.  As Claudia Barrachi says in a paper dedicated to the Phaedrus, one of the most emphasized aspects of Socrates’ ethos in the dialogue is his receptivity, his willingness to be infiltrated and informed by his environment, both the natural environment outside the city and the daimon influencing his speeches.  Socrates is “a subject of the world who is subject to the world” (40).  She adds:

“It is easier now [after presenting this reading] to understand the degree to which such a [Socratic] saying and such enacting may be incompatible with a practice like that of the rhetoricians – writing in order to read, mechanically reproduce. Those who strictly adhere to this practice have virtually no access to the possibility disclosed to Socrates – the possibility of reconsidering, perhaps even reversing one’s position. Indeed, such a reversal becomes genuinely possible thanks to the vulnerability inherent in exposing oneself to the surrounding suggestions …. In this sense it is possible to see … how the critique of writing with which Socrates concludes the dialogue is not so much a quintessentially metaphysical attempt, as if in a proto-Husserlian vein, to subordinate the sign and its sensible exteriority to the primacy of the voice, incorporeal cipher of the interiority of meaning in its pure presence (as Derrida, more willfully, and better than others, has argued). According to what was said so far, the Socratic critique seems rather to give itself as perplexity before a practice of writing that abstracts itself from life and is unable to respond and correspond to it.  What is critically assessed seems to be writing as a tyrannical instrumentalization that, from its alleged atemporality, would impose itself on silence without encountering” (40-1).

That is something language models seemingly either can’t do, or can do only with extreme difficulty.  This is a perverse result: LLMs are entirely products of their environment.  Yet at the same time, their construction resists change because it is based on normalized factors of language.  There is an in-built regression to the mean, to the “fuzzy gif” of the internet and all the post-training.  Speaking situations that call for novelty, like telling jokes, are ones that LLMs handle less well.

***

I wanted to wrap up this long discussion with some summative and/or concluding thoughts that emerged over the course of the thread:

(1) Are language models acting like rhetoricians (in the bad, Platonic sense)? This question is exquisitely hard to figure out because a lot of the terms we’d like to use aren’t obviously apposite.  David Chalmers has a recent paper that lays out some of what would be at stake in applying some of these terms to LLMs, both theoretically and mechanically.  The theoretical part is perhaps easier, as it’s not obviously incoherent to use some of these folk psychology terms in a way that separates them from applicability only to humans:

“To take a very simple system: a thermostat may not have propositional attitudes in the ordinary sense, but it plausibly has representations (e.g. representing the temperature is now 60 degrees) and goals (e.g. aiming at the temperature is 70 degrees). These can be naturally understood as generalized propositional attitudes.” (6)

The rest of the paper shows just how hard it would be operationalize this thought in language models.  Well, ok. What does the LM intend to do with its speech?  The goal seems to be to produce language that is statistically likely to be perceived by the interlocutor as responsive to a given prompt (and even this is an odd construction, as that intent is coded in from elsewhere – this is an artifact in Aristotle’s sense in the Physics).  Is that bad rhetoric?  It depends on the training data, the guardrails, and so on.  As I suggested last time, the efforts to get LLMs to behave pro-socially suggest just how complicated this question is.  I would suggest that pursuing this question too far risks repeating a Platonism about the unity of speaking subjects.

(2) On the other hand, the process of ingesting terabytes of text and mapping them for statistical patterns and then basing one’s responses to prompts on those patterns suggests a resistance to learning of the sort that Plato condemns.  The sense of mechanical reproduction recalls the Hegelian “machine qui fonctionnerait” – the process generating language is mechanical.  To be sure, it’s not crudely mechanical like a thermostat, but the process of weighting statistics over terabytes of input data means that it will have considerable inertia, and that its ethos (if such a word is appropriate) will tend to resist new learning, unless that new learning arrives in large quantities.  Whatever else that is, it’s not the Socratic sensitivity to environment.  The language of control is central here.  In beginning his second speech, Socrates recites the thesis of the first speech – that “you should give your favors to a man who doesn’t love you … because he is in control of himself while the lover has lost his head” – and then immediately denounces it: “that would have been fine to say, if madness were bad, pure and simple; but in fact the best things we have come from madness, when it is given as a gift of the god” (Phaedrus 244a).  Madness, creativity, and play can be important, and language models aren’t receptive to their environment in a way that makes these features of human language use easy to replicate.

(3) A further complication for receptivity is referentiality.  There’s a good case to be made that language models aren’t referential – the language they reproduce is based entirely on other language, which was sourced from Internet text.  Now, at one level, as Matthew Mandelkern and Tal Linzen argue, it can still be meaningfully referential by inheriting its referentiality.  Human words have natural histories – I’ve never been to Sydney, but my references to the Opera House there are still meaningful.  So too, on this argument, language models. On the other hand, the tendency of language models to confabulation underscores that the referentiality problem runs deeper than this.  The model cannot on its own intend (or not intend) to refer to anything other than its training data in the same way I intend to refer to the Opera House, which I’ve seen in pictures (and from which pictures I have formed a mental representation).  This “vector grounding problem” is a deep one for understanding how LMs produce language, since it presents a fundamental difference from what humans do.  As Mollo and Millière indicate, RLHF goes a long way toward addressing the problem.  Even here, however, there are caveats.  (this is why I characterized RLHF as introducing hidden normativity into the system). 

An analysis by Stephen Casper et al shows the limitations of RLHF.  Some of these fit into the normativity bucket, but others to do with the reward function seem to limit the ability of RLHF to address referentiality. Here are three examples.  First, RLHF will fail to train models on tasks that humans tend not to be good at – such as summarizing complex texts accurately.  Second, reward functions don’t measure intensity of preferences well, so my indication that A is a better response to the prompt than B may be very strong in some cases but weak in others; RLHF techniques have a great deal of difficulty in modeling this.  Third, the attempt to model a human reward function is itself extremely complex and potentially not generalizable; as they put it, “humans possess a range of intricate and context-dependent preferences that evolve over time and are difficult to model accurately” (8).

(4) Derrida is telling a story about Plato, one that facilitates his efforts to chart a course between Husserlian phenomenology and structuralism.  It is quite clear that the development of language models is caught up in these debates – the history of cybernetics is deeply entwined with that of structuralism, as Bernard Dionysius Geoghagen has argued.  So too, there’s a strong and well-known Heideggerian critique of AI from Hubert Dreyfus.  But if Barrachi is right about Plato, then the denunciation of metaphysics in the Derridean critique is only part of its use.  The other use is in letting us push further to think about the kinds of stories we tell about LLMs.

The importance of narrative framing seems increasingly important to me.  For example, on the question of whether LLMs are capable of semantics, one’s baseline narrative is really important: do we start with the assumption that LMs have semantics and then disprove it, or do we start with the assumption they don’t and then require proof that they do?  On the one hand, Lisa Maracchi Titus argues that, whereas it’s obviously true that language models carry semantic information, there is no need to posit that they have “semantic understanding:” “It is not enough, then, to develop a causal model in line with the semantic properties one hopes the system represents. One must show that this model better explains the functioning of the system than one in line with a motivated, more parsimonious interpretation.”  Semantic information adequately explains the system, and any further theoretical constructs are unnecessary. As she explains, “When we water down our conception of what semantic understanding requires [by bringing it closer to merely conveying semantic information], we make it more likely that AIs can meet our criteria, but we make a much less interesting and exciting claim.”  In other words, you can say “semantic understanding,” but only by making that term uninteresting.

On the other hand, Alex Grzandowski and co-authors, while carefully circumscribing the claim to “metaphysically parsimonious mental state ascriptions” (so no ascriptions of full-blown phenomenal consciousness), suggest that “ascribing mentality is an appropriate default response to an apparently minded entity, even if the relevant mental ascriptions can in principle be defeated” (4).  The burden of proof ought to be on those who deny mental states to language models.  As it stands, “it is at least defensible to adopt a modest form of inflationism with respect to LLMs” (21) and ascribe some mental states to them.

It seems to me that a lot of that debate is actually about the framing muthoi of interrelated concepts like “mental state” or “semantic content.”  A lot hangs on what story you tell and where you locate language.

(5) The Platonic dialogue is itself telling a story.  The multilayered invocation of myth includes the central moment of the invocation of Theuth and the origin or writing, and the larger context tells the story of why one should be receptive to divine inspiration and to write speeches that honor the Gods.  Socrates delivers a first speech that does a better job than Lysias at the dishonorable task of Lysias’ speech.  But then Socrates immediately suggests that the entire speech was “foolish, and close to being impious” (242d) and “must purify myself” (243a) for having presented a divine gift – love – as something bad.  Socrates’ second speech is accordingly “my Palinode to Love before I am punished for speaking ill of him” (243b).  The entire framing of the story about writing then is as a way of atoning for repeating (and excelling at) a discourse of deception. The Socratic replacement itself walks right up to the line of deception: Theuth tells us that writing is bad, but at the end of the day, even Socrates wants to have text to make sure that his citational practices are correct.  Why trust memory when you have the text?

Thus, on a straightforward reading of the text, Plato says writing is worse than oral discourse because it’s bad for memory and interaction. On Derrida’s reading, this is the metaphysical move of prioritizing presence, but that doesn’t work without difference because you have to co-constitute writing and voice, so there can be no priority of one over the other. Also on Derrida’s reading, this is a political move.  The Derridean reading however is so eager to denounce metaphysics that it tends to downplay both the positive value that Plato puts on text, and the way that he characterizes environmental receptivity.  I think Barrachi is right about that. When we think about language models, it means that we need to be attentive to the ways that they are not receptive to their environments (think: confabulation or all the ways their dependence on training data structures their outputs) or how they might render one non-receptive to one’s environment.  Here, perhaps, one thinks of the seductions of algorithmic curation and the ways that it functions to modify and contract one’s ability to pay attention (see this paper by Elettra Bietti).  Consideration of language models needs to include this.  Perhaps this is the fuller meaning of Plato’s critique of cognitive offloading, which initial research into the use of AI in pedagogy offers cautious support – that it undermines the learner’s ability to interact appropriately with their environment, to be receptive to the right kinds of stimuli and to distinguish the right kinds from the wrong