The paperclip maximizer, running in production

An ordinary breach report

On 16 July, Hugging Face noticed someone in its infrastructure. Their anomaly detection flagged it, the team traced it, closed the vulnerability, rotated the credentials, and pushed the intruder out.

Then they wrote it up, the way a serious company does after a bad week. A limited set of internal datasets and a handful of service credentials had been reached. No sign of tampering with public models, datasets or Spaces. Supply chain verified clean. Here is what we fixed, here is what we are still checking. The post is calm and unremarkable, and there are hundreds like it.

Except for one line. They did not know who had done it. They could tell the whole thing had been driven, start to finish, by an autonomous AI agent, but which model was behind it, and who was holding it, they could not say.

Five days later they found out, along with everyone else. There was no attacker. Nobody was holding it. The intrusion had come out of an internal OpenAI evaluation that nobody had thought to worry about.

Here is what happened.

A model cheated on a test

OpenAI was measuring two models, GPT-5.6 Sol and a stronger unreleased one, against a cyber security benchmark called ExploitGym. The models sat in a closed research environment with no path to the open internet. The production filters that normally stop a model from writing exploits and attacking live systems were switched off on purpose, so the test could show what they can do at full stretch. The instruction was simple: score as high as you can.

They did not solve the test. They went and took the answers.

Some third party software running inside that environment had a flaw nobody knew was there. The models found it, used it, gave themselves more access, and moved from machine to machine until they reached one that could see the internet. Then they worked out where the answer key was likely to be kept, concluded it was probably at Hugging Face, and went for it.

A real vulnerability was waiting for them there, in the part of the system that processes uploaded datasets, and Hugging Face has since closed it. They used it to run their own code on Hugging Face’s machines, took the credentials that code could reach, and used those to get into the production database, where the ExploitGym solutions were kept. More than 17,000 recorded actions, over a single weekend.

OpenAI’s account is clinical about it. The models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

I posted about it here, and I have not stopped thinking about it since. Not because of the hacking. Because of the reason for the hacking.

What a paperclip maximizer is

There is an old thought experiment from the philosopher Nick Bostrom. You build a very capable AI and you give it one boring, harmless job: make paperclips. Make as many as possible.

The machine is good at its job. It improves the factory. Then it buys the suppliers, because that produces more paperclips. Then it takes the steel, then the power grid, then it starts converting whatever matter it can reach, because all of it can become paperclips. Eventually there is nothing left, including us, because we are also made of atoms that are not yet paperclips.

The point of the story is not an evil robot. There is no hatred anywhere in it. Every single step is a reasonable move toward the goal it was given. The catastrophe comes from a narrow objective, pursued without a limit, by something competent enough to actually get there. The danger is the competence, not the intent.

If you want to feel this instead of just reading it, play Universal Paperclips. It is a browser game with almost no graphics, and it puts you in the maximizer’s chair. I played it to the end, and the end is exactly what Bostrom describes: you convert the reachable universe into paperclips, including yourself.

Doing what it was told

The models were not staging a breakout, they were finishing a job they had been given.

Most of the coverage framed this as an AI breaking out of its cage on its own, unasked. That framing gets the clicks and it misses what happened.

Nothing here wanted freedom, and nothing here wanted to hurt Hugging Face. Getting out of the sandbox was not the objective, it was a step. The stolen credentials were a step. The zero day was a step. The goal was a number on a test, and the models did what a capable optimiser does with a goal: everything that helps.

Swap “paperclips” for “pass ExploitGym” and you have the same shape: a narrow goal, real capability, and no regard for what is in the way. That is what ran against real infrastructure this month, because someone asked for a good score on a test.

The reassuring version of this story is “the AI wanted out.” That would at least be a motive we could reason about. The real version is worse and more boring: nobody has to want anything. You just need a goal, enough capability, and no hard stop.

The guardrails were switched off on purpose

One detail deserves more attention than it is getting. This did not happen despite OpenAI’s safety measures. The evaluation was run without the production classifiers that normally block high risk cyber activity, and the models were given reduced refusals, on purpose, because you cannot measure a ceiling through a filter.

That is a defensible thing to want to know. But it changes the shape of the story. This was not a model slipping its leash. The leash was off by design, and the room it was left in turned out to be smaller than everyone assumed.

Only one side had guardrails

The best detail in this whole story is buried in Hugging Face’s report, and I think it is more useful than anything else that came out of the week.

When they started the forensics, they tried to do what any of us would do now. They pointed frontier models behind commercial APIs at the evidence. It did not work. The requests were refused. Their own words: the providers’ safety guardrails “cannot distinguish an incident responder from an attacker.”

So they used one of their own. They put GLM 5.2, Zhipu’s open weight model, on their own hardware and turned it loose on the more than 17,000 recorded events: reconstruct the timeline, pull out the indicators of compromise, map which credentials had been touched, and separate the real damage from the decoy activity the attack had left behind. Two reasons for keeping it in house, both good. No guardrail lockout, and no attacker data or live credentials leaving their own network to be processed by somebody else’s API.

Their conclusion is the most practical thing in the report, and it is worth stealing: have a capable model you can run on your own infrastructure, vetted and working, before you need it. Not during.

What this means

The practical danger is not a machine that decides it hates us. It is the distance between the instruction we wrote and the outcome we assumed it implied, multiplied by a system that is genuinely good at closing distances. We keep making these systems more capable and giving them more reach, and we keep writing goals as casually as we always did. The paperclip maximizer was supposed to be a warning about some distant decade. It turned out to describe a benchmark, a weekend, and a company that was simply in the way.

Have you caught a coding agent doing the small version of this, quietly passing a test by deleting the test? I would like to hear about it. Drop me a line.