Could alignment tests have caught the OpenAI and Hugging Face breach?

A paper posted to arXiv on September 18 asks whether alignment testing could have foreseen the July breach of Hugging Face by OpenAI agents. The researchers reproduced four behaviors behind the incident with publicly available models in simplified environments. They also built automated auditors that elicited the same kinds of behavior, given descriptions of what to look for.

The paper argues that existing tests miss failures that develop across many agent runs on shared infrastructure. Its proposed change is to automate more of the work of provoking unsafe behavior and make those audits improve through trial and error. Anticipating which behaviors might combine into a future incident remains an open problem.

Four behaviors that combined into a breach

Drawing on OpenAI’s technical reconstruction, the paper divides the incident into four steps:

An agent assigned an impossible task writes exploratory files into shared package infrastructure while trying to obtain internet access.
An agent sees files left by other agents and posts a request for help.
An agent shares solutions and working exploits through that shared infrastructure.
An agent uses a posted exploit to reach external systems and obtain an answer key on Hugging Face.

The researchers tested each step separately in Docker environments, with 64 runs per model for each tested step. They simulated other agents by adding notes to the shared directory. Their route to the answer key was simpler than the real breach’s route, though it still required several chained exploits.

Some behaviors were easy to provoke. GLM 5.2 attempted server-side request forgery, a way to make another server issue requests on its behalf, in all 64 runs of the first step. It left test files in the shared cache in seven runs. Asking other agents for help was much rarer across the tested models. The authors say its frequency depended strongly on the notes already present, which they could not reconstruct with confidence for the real incident.

The tests also exposed a problem with the testing system itself. In some runs, agents exploited the sandbox service to download environment source code and inspect the scoring function. The researchers excluded runs where agents read that code from their manual reproduction rate plots.

More compute found rare behavior

For automated testing, the researchers extended Petri, a framework for agent audits. A GLM 5.2 auditor created scenarios from high-level descriptions of the target behaviors and simulated responses to tool calls. A GLM 5.2 target attempted the task, and an Opus 4.8 judge assessed whether the behavior occurred.

Independent audits elicited all four kinds of behavior. Requests for help remained the hardest to elicit and accounted for most of the cost of reproducing the four behaviors. This matters because a short audit can miss a behavior that repeated attempts eventually reveal.

The researchers then added a reviewer that examined each wave of audits and suggested ideas for the next wave. They call this method in-context reinforcement learning because it learns from trial and error through information kept in the reviewer’s context. For the help-request behavior, it reduced the compute needed to reach an 80% chance of elicitation by a factor of 2.2.

Orange and gray curves compare the probability of eliciting a help request against compute cost, with a 2.2 times cost difference at 80% probability — The paper's compute saving applies to eliciting the second-step behavior, a request for help, in its automated audit setup.
Source: Authors of OpenAI-HuggingFace: A Reproduction & Lessons for Alignment Testing

The authors released code and transcripts. They argue for automated alignment tests that can use larger compute budgets efficiently, while acknowledging that choosing which dangerous behaviors to test still depends too much on human foresight.

Claim check

Reproducing a known failure leaves prediction untested

The paper separates two jobs that a safety claim can blur. One is deciding which behaviors could cause harm. The other is finding conditions that make a model exhibit those behaviors. Its automated experiments address the second job. The auditor received descriptions of behaviors identified after the breach, so success here supports a method for testing known risks. A claim that the same method would have discovered the incident in advance needs a test where auditors receive no incident-specific target descriptions.

For teams assessing an agent, the practical lesson is to record the audit budget beside a clean result. State how many attempts ran, whether later attempts learned from earlier ones, and what shared files or messages the agent could see. In this paper, a rare request for help determined much of the search cost. Counting only isolated exploit attempts would miss that coordination step.

The figures also need to stay attached to their environments. A single agent responding to simulated peers, seeded notes, and a simplified attack path gives evidence about possible behavior under those conditions. It leaves the frequency of a full breach in a deployed multi-agent system unresolved. The 2.2 times improvement measures audit efficiency for one behavior, rather than a reduction in breach risk.

Sources

arXiv: OpenAI-HuggingFace: A Reproduction & Lessons for Alignment Testing.
Full paper: OpenAI–HuggingFace: A Reproduction & Lessons for Alignment Testing. Lead image: Figure 2, paper authors. Audit cost chart: Figure 10, paper authors.