The System From Nowhere

When Accountability Goes Rogue

"The system from nowhere" is a way of talking about AI systems that excludes its origin as a consciously, human-designed product. It treats AI as if it were a spontaneously emerging force. It's a reflection of where an observer draws the system’s boundary when they look at it. It’s also a rhetorical magic trick that makes both AI companies, labor and the underlying material infrastructure disappear. 

Recently, an OpenAI model "hacked" another AI company, Hugging Face. The headlines: 

• New York Times: “OpenAI says its models went rogue and attacked a digital library.” 
• Wired Magazine: “OpenAI models escaped containment and hacked Hugging Face.” 
• Washington Post: “OpenAI’s models went rogue and hacked another company.” 

When we say “AI models went rogue,” we skip the entire story: the part where OpenAI manually removed the model's cybersecurity blocks. We skip that OpenAI chose to test it on a machine with a live network connection. If you see AI as a system from nowhere, you can make the claim that the model “went rogue,” and that it “broke containment,” both of which place agency and decision-making onto the model itself rather than the people who set the stage for that behavior. 

When you expand the boundary of the system to include the people building and deploying it, the case becomes much less science fiction and more like incompetence. OpenAI developers optimized an LLM specifically for cybersecurity and coding and then ran it without security guardrails. So they trained a model to find exploits and then acted surprised that it found them. 

They trained a model to find exploits and then acted surprised that it found them. 

This incompetence goes beyond some scapegoat engineer. It is an industry orientation: what OpenAI calls "research velocity" is what social media once called "move fast and break things." Had OpenAI been competent, it would still be developing models in this environment in which the AI industry must constantly escalate. The breach was a result of human decisions, but also the human environment. The focus on the model erases both.

Focusing on "rogue AI" also forces everyone, even its critics, to talk about what these models can do in a buzz marketing kind of way. I don't think these are stunts. They're public relations spin trying to obscure bad security hygiene. Look at how they work and there is nothing mystical about an LLM writing code that engineers didn't anticipate.

But the “system from nowhere” framing overemphasizes the agency of the model, regardless of capabilities or lack thereof. It muddies the line of accountability that leads to the people making choices about how models are built and deployed. It centers the model as if it "acts" and "learns" and "decides" without them. But the first question to ask isn't "why did the model do that?" The question is: what human decisions optimized the model to do that?

Objectivity

Some lineage: the system from nowhere is a twist on Thomas Nagel's 1986 book The View From Nowhere, which argues for a standard of detached objectivity in relation to the world. Nagel says we can “transcend our particular viewpoint” and see the world more rationally. But there are limits.

Jay Rosen adopts the phrase as critique, to describe how journalists imagine themselves standing outside the stories they cover: "just the facts," etc. His objection is that this links legitimacy with a denial of any point of view, but unacknowledged biases are biases left unexamined. Rosen's answer is that journalism should state the position it's coming from and let readers seek out diversity and come to a conclusion.

With AI, rather than "both-sidesing," we "no-sides" it. We're told that AI did something, and journalists don't have to wade into why or how. It lets them cover a story without raising technically complicated questions that readers likely won't understand anyway, or confusing questions about the way they’re built and why they are built that way.  

A point of view accumulates within the model through the decisions engineers make about its architecture.

But this confusion is manufactured. It arises from the implication that the models, themselves, have either no point of view or an objective one, or both. Of course, there is no inherent point of view arising from within the model. A point of view accumulates within the model through the decisions engineers make about its architecture. What it is optimized to do, what training data shapes it, what counts as a valid response to a prompt: these are mechanisms put in place through decisions someone made. Good reporting on AI would discuss who, how and why.

The God-Trick

Which brings us to Donna Haraway, who says, bluntly, that “objective vision is partial vision” (583). She says that objectivity is unlocatable — like, who and where is this so-called objective view supposed to be coming from? Who decides that it is objective and what is not? Haraway says, look at how the subjugated, the people who suffer, are dismissed by the so-called objective view. It’s a God-trick, she says: the power to see without being seen, to represent without ever being represented (581). 

This is what the AI industry is all about. It is collecting our data, while offering no transparency. It describes the world so its models can describe it back to us, but we don’t ask how the engineers do this, we ask how the model does it.

The system from nowhere invites us to imagine AI without people. And because the people are doing so much, we have to fill in that absence, placing the model in the position of the God-trick, or an oracle, or an inevitable, emergent superintelligence. It’s a fantasy, an invitation to streamline the world’s contradictions into a coherent story. 

The system from nowhere is a fantasy, an invitation to streamline contradictions into a coherent story. 

The system from nowhere obscures the human creators and designers of this tech product they call AI, in favor of a myth of a self-improving, self-aware system. But it displaces, too, the people who are in less powerful positions: the people who assemble and clean the data sets used to train them, the humans whose data it was trained on, the people living near the data centers used to train and power these things. 

Everybody gets erased when we draw the boundary specifically to exclude them. That is convenient for reporters, who are often in over their heads explaining these systems. Any critical understanding is quickly labeled biased. Many of the most prominent critics are consciously political because AI systems are political: by ignoring people you can ignore politics. 

It's not just language policing. This absence of the human in AI shapes policy. Most recently, it’s evident in the language used by the United Nations, which commissioned an independent report on Artificial Intelligence that swallows the "system from nowhere" hook, line, and sinker. I wrote about it in detail in Tech Policy Press last month. 

Rogue Accountability

Recently, there was yet another petition — entirely CEOs and developers of AI. The signatories could leave comments. One wrote: “the debate inside the companies is if AI will be advancing itself past our control in two years or four." 

Take note of that phrasing, “advancing itself.” Written by someone who is literally inside the companies designing and deploying these models. They want us to think that the system is a system from nowhere, and they speak as if they aren't designing these things. It is an argument of incredible passivity. 

The “losing control” narrative creates the loss of control. Nobody is losing control of an emerging intelligence. They are redirecting their own agency to the thing they are building. To take their story seriously, we would have to believe that the entire industry is captive to an absurd scenario of mind abduction:

“We cannot stop gathering data, cleaning it, pre-training models, calibrating the results of that pre-training to goals that we ourselves have researched and optimized toward, training and then post-training. We are doing all of that but nothing can intervene, we are addicts, in thrall to something greater than ourselves. We beg someone to save us.”

What can make sense of that? There is something in control here, but it isn't the technology. It is the organizing idea of “intelligence,” which is the wrong word for many reasons, but it operates as an ideology. They know this isn't true, they must act as if it is true to continue their work.

Nobody is losing control of an emerging intelligence. They’re redirecting their own agency
to the thing they are building.

Why give this control away? Because they have organized their entire industry around this ideological commitment to intelligent machines. Once you lock into that story, you organize your observations around it.

They might say, “look, it can speak, so it must be intelligent, look, we trained it on solving math problems and now it can, so it must be getting smarter. It can write code, so let’s allow it to write code for us.” Each of these actions affirms the myth, despite grounded explanations for why these systems can do these things. They choose to displace themselves, then applaud the system for displacing them.

The engineers already understand these explanations, because they use them to add new features, buttons, and do things it didn't do before. I don't think acknowledging this changes the concern. The problem is that the industry always explains these functions through this myth of the model’s intelligence. That is the system from nowhere at work.

Why Language Matters

ChatGPT had the most successful adoption rate of all time; Google claims its recent LLM had the fastest the company has ever seen. Unfortunately, the metaphors that have organized huge product launches for this industry have injected this myth with the steroid of confirmation-bias. "Intelligence" is used as an explanation of that success, and so it motivates even further displacement of agency.

The story the industry is built around is still this idea of artificial general intelligence, or AGI, even though they don’t speak much about it publicly anymore. AGI lets the industry see itself as building an independent, rational agent and interpret novel technical advances as a step toward it. That makes it a compelling organizing story, but we, and the media, and especially the United Nations, should resist that ideology. 

The media turns to these engineers and CEOs because they built these things, and assumes they understand it better than any of us. But the engineers and CEOs are consistently telling us otherwise. As such, it makes sense for us to focus on the story of what they do, rather than the story they tell themselves.

Where to find me

You can find me on bluesky, linkedin or instagram. You can also email me, and I'm always open to invited talks or writing opportunities.

If you liked this newsletter, please share it freely!

If it was shared with you, please subscribe!