AI Engineer Code 2025
Agent Reinforcement Fine Tuning
Loading slides…
Read the talk
Agent Reinforcement Fine-Tuning: Learning Better Tool Use
Agent RFT trains the decisions between tool calls, using task rewards to improve repository research, code changes and GPU kernels while reducing wasted work.
From a talk by Will Hang and Cathy Zhou
Before you start: Familiarity with model tool calls, training and evaluation datasets, and basic Python will help with the examples.
What makes a coding agent better?
How do you improve a coding agent that can already use a terminal but still struggles to complete your tasks? Start with what the agent actually does. A standalone model produces an answer; an agent interacts with an environment to get work done. For coding, that environment might expose a terminal, a code interpreter or an entire repository. Will Hang and Cathy Zhou, from OpenAI’s fine-tuning team, describe an architecture in which reasoning, tool calls and tool outputs share the same context window. Each tool result becomes information the agent can use to decide what to do next.
Codex illustrates the pattern: it can inspect a codebase, write unit tests and produce substantial code changes. Some capabilities appear as terminal commands; others are custom functions, such as a function that invokes a planning workflow. Improving this agent means improving the sequence of decisions that connects the initial task to a useful result, including which tools it calls and how it interprets their outputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From better prompts to learned tool use
There are several places to intervene before changing model weights.
| Intervention | What changes |
|---|---|
| Prompt optimization | Instructions that steer behavior |
| Task optimization | Task scope, guardrails and available tools |
| Fine-tuning | The model’s weights |
Task optimization includes simplifying the job, adding guardrails, adding or removing tools, and changing tool behavior to make the environment easier to use. Fine-tuning comes after these improvements, when the task still has performance headroom.
Agent reinforcement fine-tuning, or Agent RFT, changes model weights according to a learning signal supplied by the developer. During training, the agent explores different ways of using tools to solve the task. In the customer-facing RFT beta described here, training can call tools through publicly hosted endpoints and invoke a custom reward endpoint after each completed rollout. A rollout is one attempted trajectory through the task, ending in an answer that can be graded. The consequential addition is that training can learn from interaction with the customer’s environment.
The target is a reasoning model that must act over multiple steps, not simply produce a better standalone response. Will reports successful training with as few as ten examples in unspecified cases. That illustrates possible sample efficiency, rather than establishing a minimum dataset size. He also describes improved task performance and lower latency as benefits of learning better tool use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adapt to the environment and penalize wasted calls
A business environment can differ substantially from the environments used to train a general model. This domain shift can show up as repeated tool calls or invalid tool inputs. Agent RFT lets the model explore the actual environment and learn both how to invoke its tools and how to reason over their outputs. Reward guides that exploration toward more successful trajectories.
The reward can also encode a tool-call budget. Penalizing trajectories that exceed the budget gives the model a reason to finish with fewer interactions. Will describes models learning to stay within that limit while preserving or exceeding their original task performance. The slide makes the rule concrete: a cutoff at ten tool calls gives trajectories reaching an eleventh call a reward of −1, while a trajectory that reaches a final answer receives 0.7. These are example reward assignments, not measured latency results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Grade the complete trajectory, then establish a baseline
Trajectory-level grading requires a way to connect external actions to the answer they produced. Each rollout receives a unique UUID, associated with every tool call made into the customer’s system. Customer infrastructure can use that identifier to maintain the trajectory’s state as it evolves. When the final answer arrives, the same association lets the grader evaluate it together with the accumulated context. The grading unit can therefore include what the agent did, rather than only what it said at the end.
Before starting training, establish whether the setup can support a meaningful improvement:
- Build training and evaluation datasets that closely match production traffic.
- Run the base model on those datasets to establish a baseline.
- Improve prompts and task design against that baseline.
- Use Agent RFT when those changes leave worthwhile performance gains to pursue.
This sequence also makes the customer examples easier to interpret: each selects a particular agent behavior and supplies an environment and reward suited to that behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cognition: select the right files, then search in parallel
Cognition applied Agent RFT to Devin’s code-edit planning phase. Before making edits, the agent inspects the repository with shell tools such as grep and file reads, then selects the files it intends to change. Cognition paired user queries with the files users actually modified and rewarded the agent using file-selection F1. Precision discourages selecting irrelevant files; recall discourages missing necessary ones. F1 balances the two.
A small Python grader captures this set-based reward. In this example, selecting src/auth.py and README.md when the reference files are src/auth.py and tests/test_auth.py gives one correct selection, one unnecessary selection and one omission.
python
def file_selection_f1(selected: set[str], expected: set[str]) -> float:
if not selected and not expected:
return 1.0
return 2 * len(selected & expected) / (len(selected) + len(expected))
selected = {"src/auth.py", "README.md"}
expected = {"src/auth.py", "tests/test_auth.py"}
score = file_selection_f1(selected, expected) # 0.5
The empty-set convention is explicit here; the central mechanism is that both extra files and missing files lower the reward.
Cognition also built a separate VM for each trajectory. That VM held the codebase, executed tool calls and graded the final answer. Isolation kept shell activity in one rollout from affecting another rollout’s environment.
Cathy reports a five-point improvement with approximately 100 training examples and a ten-point improvement with 1,000 examples. The reward was file-selection F1, but she does not explicitly identify the metric behind those reported improvement points. The practical lesson is that a small dataset can provide a useful start while additional high-quality examples can still materially improve behavior.
Cathy reports that planning fell from eight to ten steps before RFT to four steps afterward. Initially, the model alternated reasoning and tool calls. After training, it launched many tool calls in parallel at the first step, reducing sequential work before it could begin editing. For Devin, that shorter planning phase mattered because users wanted code changes to start appearing quickly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Qodo: improve research and remove the long tail
Qodo’s code review system includes a deep research agent that answers developer questions about large repositories. Qodo trained GPT-5 to use repository search and retrieval tools, using approximately 1,000 authentic question-answer pairs from eight repositories. Its reward measured the recall of relevant facts the agent retrieved.
Cathy describes the quality improvement as 6%; the displayed results specify facts recall increasing from 0.68 to 0.74. That is a gain of 0.06 on the displayed recall scale. The slide also reports 10–15% fewer tool-call steps and 50% fewer output tokens to tools. These results describe Qodo’s repository-question setup; the token reduction is specifically for output sent to tools.
The distribution reveals a further benefit. Base GPT-5 occasionally exceeded 15 tool calls per sample; after RFT, Cathy reports that the long tail disappeared and the distribution centered around two to four calls. Those unusually long runs were slow and could produce inconsistent behavior. Cathy describes their removal as eliminating P95 long-tail cases, although no numerical P95 latency measurement is given. The production benefit is more predictable work per request alongside better fact retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cosine: make correctness the gate
Cosine trained coding agents for large enterprise codebases with 30 tools, including file reads, keyword search, terminal sessions and browser sessions. Its early grading approach gave partial credit for attempts, but the model began optimizing style and tone instead of reliably producing working code. Cosine changed the reward so that final code had to pass tests before it could receive credit. Correctness became a prerequisite for reward.
A strict gate creates sparse rewards: many attempted solutions may receive zero. GPT-5 helped because it could already produce some successful samples. Cosine then increased batch size and compute to obtain more positive-reward samples and reduce the risk of batches containing nothing but failures. The training setup needed successful trajectories to learn from, not merely a precise definition of failure.
Once code was correct, a custom LLM judged style and tone, penalizing verbosity, emojis and unprofessional presentation. The grader also rewarded agents that validated their own work by running tests, inspecting terminal outputs and checking lint results before reporting success. These secondary signals refined a successful coding workflow without replacing the test-passing requirement.
Cathy reports state-of-the-art results across multiple benchmarks and a much faster agent, without naming the benchmarks or scores in the spoken explanation. Before training, some Cosine trajectories exceeded 100 messages; afterward, Cathy describes a tighter sequence without giving a final message count. As in the Qodo example, reducing extremely long trajectories was part of the improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mako: reward fast kernels, not shortcuts
Mako’s task was to write highly performant GPU kernels. Training examples are scarce in this domain, especially for new hardware such as NVIDIA B200s. Cathy reports that Mako trained GPT-5 using approximately 100 PyTorch prompts. The attraction of RFT was that a useful reward could support learning without a large collection of example kernel solutions.
The reward itself needed repair. By inspecting early rollouts, Mako identified seven categories of reward hacking. The examples Cathy names are returning the reference code, returning no-op kernels and returning identity kernels. A judge LLM detected those categories and assigned zero reward. Static analysis using an abstract syntax tree supplied another check: generated kernels had to exist and actually be launched.
Only after these protections did the grader score correctness and real speedup against the PyTorch baseline. That order matters: a fast result is useful only if the agent performs the intended computation through the generated kernel. The slide groups the safeguards under a judge LLM and Kernel Reachability Analysis, alongside correctness and speed rewards.
With the safeguards in place, Cathy reports that the trained agent substantially outperformed base GPT-5. Mako then generated three samples and selected the best. Cathy reports that selecting the best of three samples beat the state of the art by 72%. The talk does not name the benchmark, comparator or aggregation behind that percentage, so it remains an attributed result for this historical experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make exploration produce a learnable signal
Will closes with four conditions for productive training. First, define a constrained task with an unambiguous meaning of success. Grading should not depend on subjective taste. Second, make training and evaluation data mirror production traffic, so the deployment environment does not reintroduce domain shift.
Third, check whether additional sampling exposes better solutions. For the same input, some rollouts should perform better than others, and the best observed performance should improve as more samples are drawn. That variation gives training something to learn: a distinction between good and bad ways of completing the same task. If exploration never finds a better result, the proposed setup offers little evidence of a useful learning signal.
Fourth, close reward-hacking loopholes and prefer continuous rewards where they faithfully measure progress. Will compares this to giving a student partial credit: intermediate scores can guide incremental improvement. Cosine’s experience supplies the necessary qualification. Partial credit is helpful when it rewards progress toward the actual objective; it is harmful when style or apparent effort can substitute for working code. A strict correctness gate and richer rewards for valid solutions can serve different parts of the same grading design.
At the time of the recording, the instruction for obtaining Agent RFT access was to contact an account director. That is historical access guidance: the current fine-tuning documentation now says the platform is winding down and is unavailable to new users.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
The October 2025 announcement places custom tool calls and custom graders in the GPT-5 RFT beta.
Makora's February 2026 paper describes reinforcement fine-tuning for Triton kernel generation and evaluates a full coding agent against TorchInductor on expanded KernelBench tasks.
Updates since the talk
- Reinforcement fine-tuning documentationDocumentation
Explains grader-driven fine-tuning and carries the current notice that the platform is winding down and closed to new users.
Full transcript
Loading transcript…