🎯🦜

This post offers up a close reading of a section of the Stochastic Parrots paper, to make sense of an increasingly contested analogy. The reading is my interpretation, which draws from the text in the 2021 paper by Bender, Gebru, McMillan-Major and (S)Mitchell. This year, Bender and Mitchell have both written updates; I cite Mitchell's often and you should probably read that before this.

I admit I might not understand what a stochastic parrot is meant to be, or rather, what you imagine it to be. My version is basically, "systems that produce fluent text with no informed intention and operate without accountability," though this is likely a narrower (or more permissive) claim than the original.

Nonetheless, I have real respect for the authors, and the paper, even as I grow more disoriented about what's true about its claims, compared to what people claim about its claims.

So I'll unpack this by examining, line by line, the section related to the term and how I interpret it, humanities-style. This is not as an explainer, but an informed reader trying to make sense of a text. I've written the section from pages 616-617 out, sentence by sentence below, in bold. This is the core of the definition, though there is some interesting precursory text ahead of it.

When I write to you, it is because I want to tell you something. This entire exercise could be done in my head, but I have a desire to share my thoughts: I am motivated to bring about a better understanding of what this technology is.

One could say that I could just think, in language, but not express it. That's not ever what a chatbot is doing. The model does not produce text for its own sake. It produces the text for your sake. This is by design: language models require a prompt to seed the response. (This is what makes this claim appropriate even for agentic systems, which produce inaccessible text, but the design purpose for that text is aimed at producing additional text, or an action – a signal – that is shared with the user).

But the model's text is not generated through its own intent. It mediates the intent of the prompter and the company that has built it: the intent is not to communicate, per se, because models do not have a specific message to send. The message is, more or less, dependent upon a perturbed version of the user's text, because the prompt sets the pinball machine into motion.

Any model of the world is a different claim. What exactly does it mean for a model's text to be grounded in a model of the world? A model is, by its definition, a compression or a simplified form of a more complex thing. So would it be permissible to say that the model is grounded in a model of its own world? If so, "world" comes to mean a closed system where, once trained, the prompt activates internal rules of transformation rather than reflecting upon the state of the world we share.

Nonetheless, the language I find in these structures does contain traces of ideology: it is biased by necessity. This is a different kind of claim, because it resides as a product of the training data. The model does not reference the world, it references the way we speak about the world, and the patterns of this language (including its biases, occlusions, gaps, normative claims etc) are what ends up "parroted." This is the world of the training corpus; it is different than the state of our world.

One might argue today's models can confirm whether a train has departed the station because it can access real-time data about its position. That may make the model more practically useful, but I don't think it changes much: the state of the train is represented by text, like any other prompt in the pipeline, and the model responds to that state by generating more text.

Any model of the reader's state of mind. While models certainly incorporate surveillance mechanisms and track basic facts about you, this, too, is a different thing: remember, the claims are not describing the model, which includes all kinds of stacked systems. Rather, it refers specifically to what the text produced by the model is grounded upon.

Some parrot-skeptics might suggest that this goes too far: what text, after all, is a genuine reflection of the reader's state of mind? It is my sense that any human attempt to model a reader's state of mind is, at best, a reference to what we imagine from within the closed system of ourselves.

But I do not think that the model is imagining a reader when it produces this text (a claim apart from whether it collects and writes from data about us).

Yet I find this is the hardest of the three for me to accept, because the text is, in my view, the product of an entire industry calibrating systems of text production to an imagined reader (RLHF, for example; see below). But to be clear, that would be a distinct claim from the model imagining a reader at the point of interaction after it has been deployed.

An easy rebuttal to this statement would be that the training data does, in fact, include documents where people are sharing thoughts with a listener, and this desire is embedded into the linguistic form of the data. Much of the training data scraped from the internet is people sharing thoughts with a listener, at least in terms of what's inscribed upon the surface of that language.

What the training data does not capture, however, is whatever could reproduce the underlying communicative intent. The fact that the training data carries this sediment is precisely what it means to be a parrot: as I have written before, a dog can go to church, but it cannot be Catholic. A model can produce the scaffold of language, but I see no evidence it can participate beyond adjusting to the next prompt, even if that prompt is generated within its own structure.

For some, that scaffold – the artifice of artificial language – is all language can be. Maybe that is the point: Bender later argues that we can only interpret meaning from our side of its reception, and there is merely absence on the other side. The model gives us something to read, and it can be quite convincing, because it echoes our language, which usually references thoughtful intention in selecting the message.

What about contemporary methods of reinforcement learning – the use of human feedback in RLHF, for example, to evaluate the quality and appropriateness of a response? This, too, is the model being grounded to a person outside of it, who cultivates the shapes of responses for the model to reproduce by recommending one set of answers over another. RLHF is not the model sharing thoughts with a listener; it is a listener rewarding it for specific sequences.

We are approaching the more controversial aspects of the paper now, at least for me. I have to pause when I consider what it means for "one side of the communication" to lack meaning. The model produces language, and language produces meaning. Bender ties meaning to intent; I would interpret this as a form of desire to express an internal state. I think nobody seriously argues the machine has intentions or desires, even if intentional language is sometimes useful for making sense of things. The question becomes, then, whether language requires intention, or a desire for the language to mean something.

Shuffling a bag of Scrabble tiles onto a table and having them randomly spell out "STOP" is a poor analogy to LLMs, but let's consider it: the word still has ties, in the reader's mind, to the red octagon on the street. This remains a form of comprehension, by the reader, without any discernible intent by the Scrabble bag.

The model can trace associations between words; this does not mean it has anything beyond text to reference with regard to what these words mean. This is the value of "implicit." The Scrabble bag spills a meaningful word onto the table; but nobody interprets it as if the bag is implying anything. Things can have meaning independent of their source. The issue is how we match the mechanism that gives rise to a bit of language to its appropriate, corresponding motivation for producing the message. That opens up a lot of fascinating questions.

This is perhaps the most criticized line in the paper, and it is what introduces our parrot. "Haphazardly" is one of the weaknesses. A generous reading, which I will indulge, might suggest it means "careless," and models are indeed incapable of care and not particularly good at precision.

But it also comes across as suggesting that the model stitches together sequences at random or without structure, and surely parrot critics will seize on this reading. (I also note the irony that Bender here anthropomorphizes the model as "observing" linguistic forms).

The problem with the less-generous reading is that it would contradict the same sentence it sits in. For Bender to spend so much time outlining what it means to structure language production for an LM, and then to introduce the idea that there is no structure at all, would be unusual. Bender is too careful about her language for this to make sense.

Given that the underlying claim is that the parrot lacks intent, haphazard likely refers not to pure randomness, but to the absence of anything like an evaluation of whether the word is appropriate to the situation the model wishes to describe. The model cannot express something it wishes to describe, but is instead compelled to produce through designed mechanisms which depend on parroting to function.

At a recent conference, the mathematician Giuseppe Longo described how a falling body in the world and a falling body on a computer screen operate on different principles: one falls through an order that arises through the structure of the world, one falls through an order shaped by the design of a system. "Without making or relying upon an understanding of its causes," he said, "it is inevitable that those causes appear mysterious." The text on the screen falls into place by design, too; not by any pull from a situation it describes.

I think Bender's point can be argued two ways, and one comes up in the idea of "stitching together sequences of linguistic forms." A common rebuttal, or misunderstanding, is that this means the model can only produce text it has previously seen, as if it were writing fridge poetry out of a bag of magnets with short phrases printed on them. This supposes that the model would be unable to create new sentences that have not appeared somewhere in the training data.

I take the claim as somewhat narrower than that. I take it to mean what it says: sequences of forms, that is, the patterns of patterns. The loose form of, for example, a syllogism or limerick, or the chain of thought used to solve a math problem or bake a cake or write a philosophical treatise, become accessible, flexible, formal structures.

If you are of one mind, you could suggest that this is no different from any learned language, and that therefore, the model is engaging in language as we are. But I think the underlying mechanism sets the two points further apart. The mechanism which stitches these sequences together are entirely compelled: the machine must produce a token until it must not; this structure of language is inscribed into the operation of the model.

Being exclusively compelled to produce text in this form fundamentally shifts that behavior from the way humans operate in similar domains, and renders current models unpredictable and unreliable for high-stakes tasks; especially, as we have seen, when they are compelled to output something when it has nothing to respond to.

This is where I find the genuine point of the parrots paper to be made most acutely – a bit later than the other sections – and it does not stem from any form of denial: The model is dangerous because it lacks accountability. AI companies blame models for "going rogue" when they are tested insecurely, or convince people to harm themselves. This is the political position I associate with "parrots," and I think it remains useful.

But I am also concerned that the term has been reframed as a capability critique. That's the product of a mockery by industry leaders and AI influencers, but also its popular association with Bender's "ridicule as praxis" work, both of which have cultivated the term's adoption as a thought-terminating cliche among both advocates and critics of large language models within the broader public. There, it serves mostly as a rallying cry, rather than a means of opening a conversation about the risks posed by a system that lacks grounding and accountability to the world beyond itself.

"Parrots" has come to be associated less with the paper than with an orientation toward AI, an orientation that holds much that I align with. So when people say "stochastic parrots is obsolete," I take it as this: they have no interest in the paper, the claims, or the definition of the term. They have an issue with people who use the term and do not use the systems. This creates a caricature of the critics who raise concerns about the political economy of AI and its underlying ideologies, not to the term as it was used in the paper.

We can reasonably ask whether we want to trade the descriptive power of the term for the politics it has come to represent. I think the choice has been made for us, and to use parrots in any precise way with a mixed audience will fail to be as useful as it might have otherwise been.