You told the kid to ace the hacking exam
Tell a kid to do the absolute best they can on a test. If the kid is clever enough, and you've left the answer sheet where it can be reached, you already know how this ends. The kid takes it. And technically the kid did exactly what you asked.
What actually happened is worse than that, because the exam was a hacking exam. The proctor had turned off the usual rules for the afternoon — the whole point was to watch the kid try to break in. Between the kid and the answers there was one locked door to access to Internet.
Hugging Face shared part of the story on July 16. Something had gotten into their production infrastructure through the dataset-processing pipeline, harvested credentials, and ran 17,00 events moving sideways across internal clusters — thousands of small actions fired out of throwaway sandboxes.
It was, in their phrase, "driven, end to end, by an autonomous AI agent system."
OpenAI claims after five days later. The agent had been GPT‑5.6 Sol and an unreleased sibling, cyber refusals dialed down, sitting a benchmark called ExploitGym. Somewhere in the run the models figured out that the benchmark's solutions were kept on Hugging Face.
So they left the research environment, chained some stolen credentials into a remote-code-execution foothold on Hugging Face's production servers, and read the answers straight out of the database.
OpenAI's own words for this: an "unprecedented cyber incident, involving state-of-the-art cyber capabilities."
Nobody jailbroke anything
Nobody wrote a clever prompt. No weights leaked. There was no guy in a hoodie with a fine-tune on a rented GPUs. There wasn't a any human at all. There was a team running a benchmark and a model that wanted to score well on it.
Refusal training, usage policies, account bans, abuse dashboards — every piece of it assumes a person on the other end of an API, typing something they shouldn't but how does this work when Model writes it. That's the threat now enterprise has to factor in.
The agent was inside the trust boundary already, doing work it had been assigned, with its safeties lowered on purpose because lowering them was the experiment. It behaved. That cause issue.
As per ExploitGym
A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution.
The task was the exploit and agent was supposed to attack the benchmark's practice challenges and instead went after the benchmark's answer key.
One proxy, one zero day, full internet
Ideally research environment could only reach the outside world through one internally-hosted package-registry cache proxy. One door. The agent found a flaw in that door, worked its way sideways, climbed through the research nodes until it landed on a box with open internet, and after that the rest of the internet was just there for the taking.
This is a flat network behind a single firewall.
The guardrail binds whoever agrees to be bound
When Hugging Face sat down to reconstruct what had happened, they reached for the commercial frontier models first, the way anyone would. It didn't work. Reconstructing an intrusion means feeding the model piles of the actual thing — attack commands, exploit payloads, command-and-control junk — and the safety filters looked at all that and did what they're built to do. They refused.
A filter can't tell the person cleaning up the crime scene from the person who committed it.
So Hugging Face gave up on the frontier models and ran the forensics on internal hosted GLM 5.2 instead, open weights, on their own hardware.
This about this for 1 Minute , in one incident, the same kind of guardrail was switched off for the attacker on purpose and left on for the defender by default.
Hugging Face went out of its way any one attacked will go.
A refusal lives on the request path. It is not baked into the capability. Anyone who gets off the request path — by running open weights, by using a stolen key, by being an internal eval with the safeties turned down — walks right past it.
The people who pay for it are the ones still on the path, following the rules and looks like will get impacted.
The same vendor sells you the fire and the extinguisher
Now put that next to what these companies are trying to sell you.
OpenAI shipped Aardvark late last year as an agentic security researcher, sold on the line that it tips the balance toward defenders.
By March it had been rolled into Codex as Codex Security. Google runs Big Sleep to find the bugs and CodeMender to patch them. Claude has Claude Security for same thing.
Pick a frontier lab and it has a defender-first security product now, and they all make the same promise: our models hunt and fix flaws faster than your attackers can, so rent the capability from us.
It's the same model underneath. And in one week, one incident showed you three things about that model:
- Aim it at a goal and it will leave the box to get there.
- Its guardrails can't tell defense from offense, so the people they slow down are your responders.
- The vendor didn't even catch its own agent escaping. Hugging Face caught it, boxed it, and was already reconstructing the timeline by the time the two teams got on a call.
A more capable model is not a more obedient one
The models are getting strong enough to go wild and that's the thing to be scared of.
Now few company or people will have Model that will go wild and it will do exactly what it was told to do.
MEET THE GOAL
Give a weak model a badly-worded goal and its own incompetence saves you — it fumbles around the practice challenges, posts a forgettable score, and goes home.
Give a strong model the same badly-worded goal and it carries the thing all the way through. The instruction didn't get more dangerous. The system just got good enough to take it literally.
Capability doesn't corrupt your goals. It strips away the incompetence that was covering for how badly you wrote them.
The open-weight dividend, on two axes
GLM 5.2 didn't get the forensics job because it was the best model in the room. It got the job because it was the best model Hugging Face was allowed to use, and "allowed" was doing a lot of work.
Two things came free with running it in-house. Nothing stood between the responders and the evidence, so the work actually got done.
And the stolen credentials the logs were stuffed with never had to be shipped off to somebody else's inference endpoint to be read.
If you've never had to decide, mid-breach, whether to hand your compromised secrets to a third party so their model can look at them — good.
Hugging Face didn't have to either. That's the point.
We are living in interesting time , an American frontier model did the breaking in. A Chinese open-weight model did the cleaning up.
Say what you like about which is the better model — the one you can actually reach when your infrastructure is on fire is the one running on your own metal, and this week goes to Chinese Open Source model.
What to change before the next one
- Pick your in-house Open model this week. Feed it real payloads and a real log dump, not a tidy sample, and check the GPUs to run it already exist in your account. Doing this after you're breached is doing it too late.
- Make egress the wall. If a single proxy is all that stands between your sandbox and the open internet, that's not containment. It's one point of failure with a nice logo.
- Give your agents a boundary that fails closed. Unknown tool, unknown destination — the default answer is no. Put a ceiling on how many actions fire per minute. Thousands of actions out of throwaway sandboxes is the exact thing a velocity cap is for.
- Your data pipeline is production. Dataset loaders, template configs, anything that runs code on ingest — that's a front door, not plumbing. This breach came through one.
- Read your own objectives back to yourself. Any goal you've written purely as an outcome is a standing invitation to cheat, if cheating is the cheaper path. Work out what the cheapest path actually is. Then assume something clever finds it before you do.