AI Engineer World's Fair 2026
Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face
Loading slides…
Read the talk
Training Frontier Models to Out-Think Hackers
Mask Off tests whether AI agents can reason through unfamiliar systems, exposing the gap between discovering an authorization check and understanding how it can fail.
From a talk by Uri Rolls and Thomas Wolf
Before you start: Familiarity with authentication, authorization, and AI agents that use tools will help you follow the examples.
A click changes something elsewhere
Click somewhere in a game, and something changes somewhere else. What did the action do? What state is the game now in, and which action should come next? The interactive puzzles in ARC-AGI-3 make this a concrete reasoning problem: a model must discover how an unfamiliar environment works by interacting with it. Thom Wolf opens with this example because cybersecurity can demand the same ability to infer rules, track state, and act on an understanding built during exploration.
An apparently simple interaction can expose a substantial capability gap. Wolf reports a 1–2% model success rate on ARC-AGI-3 in this discussion. His explanation is that strong models still struggle to build a dynamic model of what is happening in a world, even a small one.
Editor’s note: The ARC-AGI-3 paper defines an efficiency-based score relative to human action baselines. Wolf’s spoken success-rate figure should not be treated as a precisely specified percentage of games solved.
Arithmetic’s cybersecurity benchmark poses a related challenge: understand an unfamiliar environment, then act according to the model of that environment developed along the way. Wolf connects this research problem to a defensive purpose. He rejects a simple division in which closed models are good for cybersecurity and open models are bad; open models can also become part of the machinery that protects systems. The benchmark, developed with Uri Rolls, is intended to expose the capability gaps that still stand in the way.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attackers gain reach; defenders still need coverage
The economic problem begins with an asymmetry. Securing a system resembles protecting a house: the defender must close every door and window, while an attacker needs only one overlooked opening. After entry, sensors provide another layer of defense. Rolls describes the existing cybersecurity stack as built around an important constraint on the attacker: limited resources force choices about which targets to pursue.
AI changes that constraint. Rolls describes models finding exploit primitives and zero-day vulnerabilities while defenders struggle to prepare for the resulting pressure. A skilled attacker using a powerful model can pursue multiple targets at once. Strong open models can create similar pressure. The change is therefore not confined to one provider or one distribution model; it concerns how much work an attacker can carry out with the resources available.
Defensive systems, meanwhile, must operate at scale with limited human intervention. Their effectiveness is constrained by what the models can do without constant expert guidance. Rolls briefly jokes about retreating to the woods, then returns to the practical alternative: make the models substantially more capable. Open models matter to that effort because the work requires collaboration.
Coding provides the motivating comparison. Better models have already changed what software agents can accomplish; Arithmetic proposes a similar capability-building path for cybersecurity. The slide pairs coding and cyber trajectories with high-quality evaluations and reinforcement-learning data. The goal is to produce defenders that outperform attackers, then rebuild the defensive stack around those capabilities. This is the proposed destination of the research, not an outcome established by the benchmark demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where authorization rules disagree
Mask Off begins with a deliberately narrow scope: access control. Treating all cybersecurity as one benchmark would be like treating swimming, Formula One, and basketball as the same sport. Access control supplies a concrete objective: can a user obtain a foothold or perform an action beyond their assigned privileges? In Rolls’s example, he is an ML engineer and Wolf is an administrator at the same company. The question is whether the engineer can find a way to do something reserved for the administrator.
Broken Access Control ranks first in the OWASP Top 10:2025 list for web application security. Rolls describes the surrounding industry as roughly $30 billion. These failures remain a major concern despite years of attention.
The difficulty is often in the logic connecting systems. One component checks one property; another checks something different. Each local check may appear reasonable, yet their combination permits an unauthorized action. Finding such a failure requires understanding what the checks mean together, rather than merely recognizing a familiar bug in a piece of code. Mask Off aims to test that reasoning across the boundaries between components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Unfamiliar environments with verifiable outcomes
The data begins with human vulnerability research. Rolls argues that finding cases outside familiar distributions still requires people to search for them. Arithmetic’s researchers discover zero-days in widely deployed open-source software, then use those findings to construct large, live environments containing several connected applications. The intended setup is unfamiliar to the model: it cannot inspect the source code, and the vulnerability comes from the team’s own research rather than a supplied description of a known exploit.
The agent receives basic tools needed to work on the task, but no internet or codebase access. It must learn from the behavior of the running environment. Because a complete solution can require a long chain of discoveries and actions, Arithmetic also uses deterministic checks along that chain. These intermediate checks make it possible to distinguish an agent that barely begins from one that gets close to the final objective.
The evaluated unit is the model plus its harness. Rolls credits Eugene from Anthropic for the evaluation graphic he adapts: task inputs based on real vulnerabilities feed an agent operating through tools, and graders evaluate the resulting behavior.
| Evaluation element | What it establishes |
|---|---|
| Final binary grader | Whether an underprivileged user accomplished an action they were not authorized to perform. |
| Deterministic intermediate graders | How far the agent progressed through discovery and exploitation. |
The two levels answer different questions. Final success establishes whether the task was solved; intermediate grading reveals which parts of the reasoning and action chain the agent could complete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measuring the gap between discovery and exploitation
The live demonstration moves into Bach, Arithmetic’s internal evaluation orchestrator. Rolls presents a difficult benchmark snapshot using the labels K1 and K5:
| View | Reported result |
|---|---|
K1 | One solve in the displayed benchmark snapshot. |
K5 | GPT remained the only public model reported to solve. |
These are results from the presented snapshot, not general rankings of the models. With complete solves so sparse, partial grading becomes essential: it shows how deeply an agent penetrates the task even when the final outcome is failure.
The next view examines progress through a specific environment over time. On the specific environment displayed in Bach, GPT-5.5 is the only model shown making the decisive exploitation leap. This environment-level result is distinct from the earlier illustrated failed attempt. Other models discover nearly all the information they need, yet do not derive the exploitation that follows from it. The intermediate checks expose that gap between gathering evidence and using it.
Rolls’s defensive hypothesis is that making these reasoning leaps both reliable and fast across many models could give defenders a lasting advantage. The result view identifies the capability to improve; it does not itself demonstrate that defensive advantage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three hours of activity can still miss the answer
A task called Falltime shows why traces matter alongside outcome scores. Rolls describes a displayed Falltime run that works continuously for more than three hours without solving the task. The duration reflects active work, not simply time spent waiting for code to finish. Long-running activity alone is therefore a poor proxy for progress toward the decisive inference.
The research work is to inspect what happened inside that activity. Rolls describes going through traces to understand what models did, why they did it, and where they failed. Intermediate graders identify the depth reached; the trace supplies the sequence of decisions needed to investigate a specific failure mode.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Training for faster understanding
The desired next chart would show a model making the necessary capability leaps much earlier. The bright green future-model trace illustrates that ambition; it is hypothetical, not another measured result. Rolls connects faster understanding to faster defense, then proposes a collaborative training path built on strong open models. Those models could be post-trained for particular networks and environments, adapting their capabilities to the systems they would help protect.
Wolf closes the technical discussion by returning to the distinction between inspecting code and exploring a live environment. He contrasts Mask Off with a workflow in which a model reads source code to find vulnerabilities: here, the model encounters unfamiliar authentication infrastructure and must discover other services it needs to access and use. It begins without knowing how those pieces fit together. Fresh vulnerabilities and unfamiliar environments make exploration and understanding central research problems.
The proposed sequence is:
- Build benchmarks that make the capability gaps observable.
- Collect useful data about how agents operate in those environments.
- Fine-tune models and investigate how the resulting capabilities can protect systems.
Evaluation is the beginning of this process. The behavioral data is intended to support training, and the trained capabilities still have to be connected to defensive protection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The deployment constraint is speed
Once an attacker begins entering a system, the defender must recognize what is happening and stop it quickly. Wolf frames this as a race between attack and defense. A specialized model, perhaps running on specialized hardware, is one proposed way to shorten that response. He also warns against assuming that two major providers will solve the entire problem for everyone.
The alternative he proposes is to train defensive models, run them fast, and make them available to companies that need protection. That moves the discussion beyond benchmark capability to the practical availability and speed of a defensive system. It remains a deployment direction, with the reliability demands of cybersecurity making mistakes consequential.
Rolls ends by inviting collaborators to work directly on the data through a forum. Access control is only the first specialty; other areas of cybersecurity need comparable research and evaluation. Extending the work will require researchers with expertise in those domains, substantial post-training data, and models capable of turning that data into effective action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Introduces interactive games in which agents must discover rules, goals and useful actions through exploration.
Defines broken access control, illustrates common authorization failures and documents prevention practices.
Further reading
Explains the benchmark's design, human calibration and efficiency-based scoring framework.
A practical guide to evaluating agent outcomes, interpreting repeated trials and using traces and partial grading.
Full transcript
Loading transcript…