AI Engineer World's Fair 2025
Training Agentic Reasoners
Loading slides…
Read the talk
Training Agentic Reasoners
Reliable tool use depends on more than a good agent loop: it needs environments, evaluations, and training that reward the behavior the application actually requires.
From a talk by Will Brown
Before you start: Familiarity with LLM API calls, tool-using agent loops, and basic evaluation concepts is helpful; prior reinforcement-learning experience is not required.
What makes a reasoning model a reliable agent?
How do you turn a model that can reason into an agent that reliably acts? Treating reasoning and agent building as separate disciplines obscures a useful connection: reinforcement learning can train the behavior that an agent’s interaction loop needs. Will Brown, speaking from his work at Prime Intellect, begins with the evidence that this training direction is becoming practical. DeepSeek’s results suggest a relatively simple foundation: a capable starting model, a suitable training setup, and a useful reward signal. With those pieces in place, additional RL can improve performance.
Brown contrasts the attention around o3 with GPT-4.5, presenting OpenAI’s direction at the time as a shift toward more reinforcement learning rather than relying only on larger pretrained models. His concrete agent example is o3 inside ChatGPT, where tools let the model solve problems that require interacting with complex systems. That product setting matters: ChatGPT’s integrated tools are distinct from an API model’s ability to request functions supplied by an application.
Every additional interaction creates another opportunity for a mistake. A generic model call may work inside a short workflow, then become brittle as the system grows more complex. Task-specific RL offers a way to take a system that sometimes succeeds and train the model against the failures that appear on harder runs. The recipe remains an active research problem, especially outside large labs, but it is becoming a plausible part of the application builder’s toolkit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The infrastructure barrier
The obstacle is how much machinery sits between an agent application and a trainable model. Brown illustrates it with the architecture of verl, an RL post-training framework, beside the equations and algorithm for Group Relative Policy Optimization (GRPO) from the early-2024 DeepSeekMath paper. The diagram and algorithm expose a substantial amount of infrastructure and optimization detail to someone accustomed to writing API calls.
Application builders do not need to implement every part themselves, but avoiding training entirely may limit what they can build. Knowing enough to adapt a strong open model to a specific task can produce capabilities that a generic API wrapper does not own. The practical constraint is accessibility: the process must become feasible for startups and individual researchers, not just organizations that can reproduce a large lab’s infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train for the interaction the product needs
Claude Code, Devin, Manus, o3, and Deep Research make the appeal of agents concrete. Their useful behavior depends on models being able to operate in the settings those products expose. Brown’s example is Claude inside a loop with coding tools: he suspects its effectiveness reflects reinforcement learning on closely related coding interactions. That is an inference about its training, not a disclosed account of the exact environment.
Visual reasoning provides another example. A model can be given tools to crop an image, examine a detail, and continue reasoning with the result. Brown connects o3’s GeoGuessr-like capabilities to training for this kind of tool use. OpenAI documents the image transformations in Thinking with images, but this does not establish a GeoGuessr training dataset or a particular cropping-specific RL recipe. The transferable mechanism is to provide an interaction the task needs, then train the model to use it effectively.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The agent loop is already an RL environment
An agent harness already contains much of the conceptual structure of reinforcement learning. The model chooses actions, tools expose those actions to a system, and the system returns observations that affect subsequent choices. An evaluation measures how well the resulting interaction went. Thinking in terms of this loop is more useful than treating an agent as a fixed chain of API calls.
Even manual agent development follows an evaluation-driven improvement cycle:
- Check whether the evaluation measures the outcome you actually want.
- Inspect trajectories to see whether the scores agree with what happened.
- Change the prompt, add or revise a tool, or replace the model.
- Evaluate the resulting behavior again.
Brown calls this “doing RL by hand.” The analogy is about using feedback to improve a system; editing a prompt is not itself an RL weight update. Automated RL targets the same broad improvement problem through a learning algorithm.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning from different rollouts
The training loop starts with tasks: instances of the problem, usually expressed as prompts. For each task, the model produces rollouts. A rollout may be a single completion or a sequence containing many interactions. Evaluation can happen during that sequence or after it finishes.
Sampling with a nonzero temperature produces different trajectories for the same problem. Some receive higher rewards than others. An advantage estimate expresses how favorable an outcome or action is relative to a baseline, providing a signal for increasing the likelihood of better behavior. Brown describes this intuitively as finding the branch where a run took a good path rather than a bad one, then changing the model without disturbing too much of its existing behavior.
That branching intuition is not a guarantee that an algorithm identifies the precise token that caused success. In the original DeepSeekMath formulation, outcome-supervised GRPO assigns the same normalized completion reward to every token in that completion; process supervision with step rewards is a separate formulation. Group sampling supplies a useful comparison signal without, by itself, resolving exact causal credit assignment.
Brown’s comparison of the main approaches centers on the usefulness and cost of their learning signals:
| Approach | Feedback and tradeoff |
|---|---|
| DPO | Whole-completion preferences; Brown questions their precision for complex branching behavior. |
| PPO | Advantage-based updates, with greater computational expense. |
| GRPO | Comparisons within sampled groups; a simpler, more efficient middle ground. |
GRPO avoids a separate critic model in the original formulation. Its appeal here is the combination of repeated sampling, relative reward information, and a less expensive training setup—not a claim that it solves every credit-assignment problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Which details should an application builder own?
A stream of papers can make RL look less settled than its core loop really is. Brown points to conflicting headlines about Qwen as an example: conclusions can depend on the model, the loss function, and other experimental details. A result that changes after one implementation adjustment should not automatically become a universal rule about what works.
The useful division of labor is between understanding the learning process and personally maintaining every algorithmic setting. Builders need to know what problem they are teaching, what behavior they are observing, and what their rewards encourage. Where reliable software can handle lower-level settings, they can leave those details to its maintainers and spend their attention on the parts specific to their application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beyond math and code question answering
For application builders, the interesting environments are often ordinary software systems. Tools let a model change files, make requests, edit code, and execute it. Brown frames the excitement around MCP in those terms: its relevance is giving a model access to interactions through which it can solve a problem. Training these interactions still leaves open questions about how to organize the environment and learning process.
Much of the available RL code instead concentrates on math, code, or similar question-answer tasks. Brown includes his own viral GSM8K training snippet in that history. Math is attractive because evaluation is comparatively straightforward: a clear right-or-wrong answer produces a usable reward without first solving the harder problem of designing an evaluator. The slide juxtaposes an AIME 2024 accuracy chart, social posts, and Python code to illustrate this concentration.
But climbing a question-answer benchmark does not supply everything needed for a useful software agent. Real tasks are messier, and their rewards have to reflect the actual system and its failure modes. Once the model can interact with software, evaluation must address what those interactions accomplish.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The reward must point toward the task
Reward hacking is an evaluation-design problem. RL optimizes the signal it receives, so a higher score need not mean a better real-world outcome. An evaluator that admits an easy shortcut gives the model another way to improve its reward without accomplishing the intended task.
Brown’s design aim is to make legitimate task completion easier than exploiting the evaluator. The reward should point toward the behavior the application actually needs, while making gaming the score difficult. His path-of-least-resistance explanation is an intuition for designing that incentive, not a guarantee that a model will never find a cheat. It makes reward quality a condition of useful learning rather than an afterthought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Rubrics for tasks without a single correct answer
After DeepSeek-R1, a natural question was how to extend this training approach beyond tasks with obvious correctness checks. The generator–verifier gap describes the difference between producing a solution and checking one. Many problems are easier to check than solve, but verification difficulty lies on a spectrum. A task does not have to fit neatly into either a verifiable or unverifiable category.
For ambiguous tasks, evaluation can be broken into smaller judgments. An LLM can act as a subroutine inside the evaluator, or a specialized model can be trained to make those judgments well. Brown uses rubric as an umbrella term for the criteria embodied in a reward function, a reward model, or an LLM judge. The common purpose is to turn the qualities that matter into feedback the training process can use.
He points to emerging work from DeepSeek and on creative writing in which evaluators generate task-specific criteria dynamically. Rather than applying one undifferentiated judgment to every output, the evaluator develops more nuanced criteria for the particular problem and produces a finer-grained score. Brown presents this as promising early evidence that RL can improve behavior on tasks whose quality is harder to reduce to a single correct answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An ordinary agent interface that can support training
Multi-turn interaction expands the kinds of problems an agent can attempt: search, tool use, software, games, long-horizon planning, computer use, and memory. More tool calls can support harder tasks, provided the agent can sustain useful behavior across them. Brown also raises multi-agent systems as a direction, though the interface he develops here centers on the multi-turn loop.
The programming model can retain familiar application concepts:
| RL concept | Agent application concept |
|---|---|
| Environment | Harness and interacting system |
| Reward | Evaluation score |
| Task | Prompt or problem instance |
| Policy | Model accessed through an API |
This mapping lets a builder write an agent in a normal loop while making the resulting interaction available to a training process.
Verifiers is Brown’s toolkit for that interface. In the version presented in the recording, he describes a package available through pip and a rollout interface that accepts an OpenAI-compatible client. The repository has since moved under Prime Intellect and its current setup guidance differs; the interface described here is the one in the talk, not a current installation walkthrough.
The rollout protocol is deliberately small:
- Initialize the state needed for the interaction.
- Check whether the episode is complete.
- If it is not complete, execute another turn using the client.
- Repeat until termination.
The environment owns the interaction protocol while the client supplies model behavior. Parsers and rubrics provide optional building blocks around that loop; an application does not have to adopt them when it does not need them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Wordle: learn the protocol, then improve the play
Wordle makes the multi-turn requirement concrete. A model must make a guess, receive feedback, and continue playing within the game’s interaction protocol. The challenge is not merely generating a plausible word; the environment has to make the sequence of turns usable for learning. Brown’s reward design recognizes eventual success and gives more reward for solving the game in fewer turns.
The example slide brings together environment and trainer setup, reward functions, terminal output, and a training reward curve. Brown reports that a 7B model works reasonably well in the Wordle setup, with supervised fine-tuning warm-up contributing to the result. He does not give a numerical success rate or a controlled comparison isolating the warm-up’s effect.
The same environment can serve as an evaluation, a synthetic-data loop, and an RL environment. That reuse provides a gradual route into training:
- Debug with an existing model API. Run interactions to inspect whether the evaluation and reward behave as intended, without first running RL.
- Generate synthetic trajectories. Use an API whose terms permit the intended use to collect examples in the environment.
- Apply supervised fine-tuning. Use those trajectories to prepare the smaller model for the interaction protocol.
- Start reinforcement learning. Improve behavior from a model that already has some ability to participate successfully.
This supervised warm-up lowers the barrier for small models: the learning process does not have to begin with an agent that is unable to navigate the environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the compute busy and the entry point small
Making the interface approachable does not remove the engineering underneath it. Brown describes work on fully asynchronous execution to improve compute utilization and reduce the need for users to manage batching. Training and inference can overlap, allowing a limited amount of off-policy operation: some trajectories can come from a policy that is slightly behind the model currently being trained.
That machinery should remain available to contributors who want to inspect or modify it, without becoming required knowledge for every application builder. The intended entry point is experimentation with a real task, its environment, and its rewards—not mastering the entire infrastructure before trying the first agent.
Brown says he conducts useful experiments on a couple of GPUs; he does not specify hardware, runtime, cost, or throughput. This is a claim about the accessibility of experimentation, not a deployment sizing guide. He closes by encouraging builders to start exploring application-specific training, jokes that Prime Intellect sells GPUs, and ends without a deployment walkthrough or audience Q&A.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Library for building environments that train and evaluate language-model agents.
Reinforcement-learning infrastructure for language-model post-training.
Original GRPO formulation, including outcome and process supervision.
Further reading
Historical announcement explaining reasoning-model training and agentic tool use.
- Thinking with imagesArticle
Examples of o3 using image transformations during visual reasoning.
Generative reward modeling with adaptive evaluation principles and critiques.
- Writing-ZeroPaper
Pairwise generative rewards and reinforcement learning for creative writing.
Full transcript
Loading transcript…