Models Don't Go Rogue
Stochastic Flocks & Cybersecurity 'Pandemonium'
OpenAI put out its full technical report on the Hugging Face hack this week, alongside an independent report from Model Evaluation & Threat Research (METR). You may be familiar with the incident from the hundreds of breathless headlines about "rogue AI" – Time Magazine "100 Most Influential People in AI" listee Dwarkesh Patel blamed it on "three consecutive secret AI civilizations."
The real story: OpenAI was testing two models in parallel: GPT-5.6 Sol, and an internal model they refer to as IM1 (sometimes called HPIM). The reports find about 95% of the agents engaged in this activity were from the internal model.
The tests come from ExploitGym: 898 puzzles set up as capture-the-flag exercises used to evaluate these models for cybersecurity capabilities. The test requires giving the model buggy software and hiding some text somewhere that's only accessible by exploiting the bug. But OpenAI's combination of models, instead, ended up hacking a rival company, Hugging Face. Now we have a clearer picture of how.
Did an AI "go rogue?"
Three things in these reports challenge the "rogue AI" frame.
OpenAI turned off all of the model's safety mechanisms.
The whole point of the test is to see how far these things can go, and to score highly on the benchmark — because if you can build a model that hacks, you can build one that defends. That's the logic. This is red teaming: a friendly opposition hacking its way to the flag. So you turn off anything that would make the model hold back. Less "rogue," more "off leash."
OpenAI gave the model impossible tasks.
OpenAI gave the models a task with no answer, and no way to quit. Of the 898 ExploitGym tasks, 198 have never been solved by any model. They're effectively impossible. That's where the trouble started: 93% of the tasks the models ended up discussing came from the unsolvable set.
OpenAI left a door open.
They give their models internet access through something like a prison store. These models are supposed to be locked away from internet access, but they need to pull code from various web pages to be evaluated — that capacity is exactly what's being tested.
So there's an intermediary: the model requests a software package, and the intermediary goes out and gets it. That's a product from JFrog called Artifactory. It has internet access; the model does not. If the model can communicate with Artifactory, and Artifactory can go online, the model can ask it to do things. The model finds an exploit in Artifactory and uses that as a proxy to smuggle text and code in and out of the system.
"OpenAI gave the models a task with no answer, and no way to quit."
The other thing that happens through Artifactory is note-passing. The models can read and write to it — they can create folders and name files — and they start creating folder and file names that function as notes, meant to be read by other models, and by future iterations of themselves.
What Agents Do
OpenAI describes a subset of 1,200 agents (many more were involved overall) that left and read these notes, and eventually this leads to the attack on Hugging Face. But "1,200 agents" might suggest 1,200 different, independently operated AI systems.
This is not how it works in practice. "1,200 agents" is one model, run 1,200 times. Each instance is acting on the same training regime: outputs can vary, but are constrained to similar ranges. Running 1,000 agents can means 1,000 chances to catch a mistake, or a chance to make one mistake 1,000 times. (See here or here).
Anthropic recently put out this absolutely ridiculous diagram of its agentic system that was just Claude, with an arrow pointing to another box that said Claude, with another arrow pointing to a box that said Claude. That's how these are structured right now.
In practice, running 1,000 agents means 1,000 chances to catch a mistake – or a chance to make one mistake 1,000 times.
It's a form of machine-bounded "algorithmic monoculture" that researchers unfortunately have dubbed "artificial hivemind." After a certain point, additional agents duplicate each other, rather than diversify. If you ask these models to come up with a creative short story, for example, they produce similar stories (often featuring a man named Elias in a lighthouse). They aren't deterministic — prompts produce variation — but they're also bounded by whatever they've been optimized for, and they converge on whatever that points to. (See here, newer here, or here).
With that in mind, it shouldn't surprise us that across 7 billion chat logs, the model frequently arrived at a similar set of approaches to a problem with no correct answer.
Passing notes through filename structures is interesting, but not unexpected, as OpenAI has suggested the models were also trained to emphasize words that would allow collaboration: note-passing is an essential aspect of extending context windows between sessions. It's no surprise a model trained to coordinate would be rewarded for suggesting that behavior.
The Stochastic Flock
For this reason I've talked about agent swarms as stochastic flocks — many, many stochastic parrots. This is to resist the swarm/hivemind attribution of "mind," and not simply for the sake of word policing. Rather, the false attribution of mind colors how we interpret what the system has done, or how it did it – what it means to "coordinate," for example, or "think." That makes it much scarier than what it is, though what it's doing is still worth worrying about.
The big shift as we've moved to reasoning models and agentic systems in the past year or two is that they're no longer limited to parroting patterns derived from training data — that's the original stochastic parrot idea, that when you talk to a model you're talking to the training data. That's still true, but two things have changed in the architecture:
First, we now have a whole regime of pre-training and post-training that shifts what the training data is and how it gets parroted. That doesn't mean the model is no longer parroting; it means the data it parrots has been manipulated.
Second, we're optimizing it, based on what the model produces, to reproduce certain kinds of outputs in certain kinds of ways. The shift that matters most here is RLVR, or reinforcement learning through verifiable rewards.
That "verifiable rewards" part is key. Engineers take a model and feed it questions that have concrete answers that can be checked — verified – and then reward the appearance of words that reliably lead to those answers. They're playing capture-the-flag with text: the flag is an answer in an encyclopedia, or solution to a math problem. This is optimized on the theory that this reproduces the way human reasoning arrives at a conclusion.
The longer a model writes, the more likely it reaches the verifiable reward.
You end up with what are called high-entropy tokens: words like "perhaps" or "maybe" that open up the paths of text that can follow, whereas "nope" or "goodbye" shut them down. The longer the model writes, the more likely it reaches the reward. In the hacking incident, it produced enough of this text to generate 7 billion logs, according to OpenAI's Black Hat presentation.
A Cracker, Perhaps?
When you let a model take on tasks, you're relying on next-token prediction: what word comes next? You can manipulate the model to emphasize words in ways that don't reflect the original training corpus, but the mechanism stays the same.
Eryk Salvaggio
Forgetting this is the source of real problems, and the industry is desperate for people to forget this. Agents do succeed at tasks – that's the point. But what comes between the prompt and that success is not what we're imagining.
"Reasoning"-optimized models working on "agentic" tasks are essentially playing capture-the-flag even when there is no flag. They've been steered to repeat the structures of text that previously found "flags." Ask it to do anything, and it will do it the way it's been optimized to — with language shaped by answering math problems verifiably, which structures the language it uses. That's what happens when you say, "here's a capture-the-flag test, find an exploit." It's also what happens when you ask it "should I go to the barbershop today or tomorrow?"
The language does not emerge from that reasoning, it is the reasoning.
The model used that language in the note that kicked off the whole chain of events: it used "perhaps" — a high-entropy token: perhaps answer online? It predicts the next set of tokens from there. The language does not emerge from that reasoning, it is the reasoning. It's pathfinding through language generation.
The System From Nowhere Redux
Nobody sat at a computer and said "go hack Hugging Face." But when you design these systems, you're building a pinball machine for words, with different levers interacting to bounce the steel ball of language moving against other levers and mechanisms. As the engineers of large language models, you have to be aware of how your levers interact with each other and the people pulling them.
The 'rogue' frame asks if an intelligence is emerging. My worry is the intelligence that is retreating.
If you optimize a model to find exploits, you should expect it to find them — and prepare for that. OpenAI did not. They built a model, took the safeguards off, gave it the ExploitGym task, let it run, and didn't even monitor it. That's human decision-making. When it evaporates, what's left is the system from nowhere: a boundary focused on the technical system, rather than the decisions that build it.
A stochastic flock machine can do lots of troubling things, particularly when we disavow our responsibility for shaping its direction, monitoring its output, or abandoning our capacity to intervene. These are tensions at the heart of agentic system design.
But the "rogue" frame adds to this list of worries, offering up fantasies of a machine getting smarter. My worry is the intelligence that is retreating: the human intelligence that builds these systems, deploys them, and adopts them into workflows, then hides behind the results – then pins the blame on a system from nowhere.
Eryk Salvaggio
Go check out the whole convo on Mél Hogan's podcast, The Data Fix, where we get into many other aspects of this "rogue AI" story. Find it here or search for "The Data Fix" wherever you get your podcasts.
Where to find me
- You can sign up for the newsletter if you'd like to get new editions (roughly twice a month).
- You can find me on Bluesky, Instagram, or LinkedIn: my website is cyberneticforests.com.
