How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere
Loading slides…
Read the talk
On ProgramBench, 16 Cheaper Planners Came Close to an All-Frontier Team at 70% Lower Cost
In Cohere’s reverse-engineering experiments, a team that spent its worker budget probing an executable before writing code came close to the performance of an all-frontier team at 70% lower cost, while a second benchmark rewarded a different division of labor.
From a talk by Nouha Dziri
At a glance
Ideas worth remembering
Establish a well-prompted solo baseline before pricing delegation. Better instructions alone raised the ProgramBench solo score by almost six points.
Test delegation one stage at a time. ProgramBench rewarded parallel discovery of an executable’s behavior. Commit0, with a skeleton and known tests, rewarded additional implementers.
Make the lead wait for findings that its next decision depends on. Inspect whether each returned result changed the plan, code or decision to submit.
Compare model mixes against an explicit baseline. Sol with sixteen Luna planners cost 38% less than all-Sol planning and 70% less than all-frontier delegation across all three stages.
Divide a long coding task before the context window fills
A coding agent can spend hours navigating a repository, editing files, running tests and debugging its own changes. Nouha Dziri, a senior research scientist at Cohere, starts with two familiar ways to give that agent more compute at inference time. Let it reason for longer, and it can pursue a problem further, at the cost of a longer wait and an increasingly full context window. Sample several independent attempts in parallel, and you get multiple candidate solutions to aggregate. Each attempt still has to understand and solve the entire task.
A team offers a third possibility. One agent keeps responsibility for the task and assigns parts of it to workers. Each worker starts with a fresh context. It can afford to read a file closely, try several tests or dig through a log, then send back the finding the lead needs. The lead can use evidence from several investigations without carrying every intermediate observation in its own context. If the investigations are independent, they can also run at the same time.
The qualification matters. A fresh context is valuable when it holds work someone else can use. Five agents each reconstructing the same program can consume five budgets and leave the original integration problem intact. Dziri and her colleagues wanted to measure both effectiveness, meaning how much more of the task the team solves, and efficiency, meaning what it costs to obtain that improvement. Lower latency was a further possibility to investigate, conditional on how the agents divided the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A group of capable agents built several versions of the program
The researchers first gave a group of agents a difficult coding task to see how they would organize themselves. A reasonable expectation was that they would identify pieces of the problem, form groups and build one solution. Instead, individual agents went off and built their own versions. Asking them to communicate produced some discussion, including exchanges of little consequence to the implementation. They still duplicated work, interfered with one another and tended to direct their messages back to a lead.
That experience turned team design into a set of experimental choices. Who owns the final program? Can workers contact one another directly? How many should the lead start, and which model should fill each role? There is also a timing decision with consequences for all the others. A worker who investigates requirements before coding begins has a different job from one who writes part of the program or checks the finished implementation.
Cohere framed these choices as compute allocation relative to a solo agent. The efficiency measure was the gain in benchmark reward per additional dollar spent over the solo baseline. Both differences matter. A team can post the highest score while offering a poor return on the last dollars spent to get there. The researchers therefore varied the stage of delegation, the worker model, the number of workers and the lead model, asking where another dollar of inference produced the most improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One shared workspace, separate contexts and a lead that owns submission
To compare those decisions, the researchers built a deliberately small, top-down harness. The lead receives the complete task. It can decide dynamically how many workers to create and what each should do. Every agent has a Bash tool and works in the same workspace, but a worker receives only its assigned subtask. Sharing the files gives workers a way to contribute to one program. It does not, by itself, tell them who else is working on a feature or how their assignment fits into the whole.
Delegation is an explicit tool action. The lead can request an investigation, hand off implementation or ask a worker to verify a result. Workers report back, and the lead decides how to combine their contributions. It can resume a worker, point out an error and ask for a correction, or terminate the worker. Workers can also contact one another. Exposing those decisions in the harness let the researchers change one part of the arrangement while holding the others fixed.
They implemented two ways to handle the interval between an assignment and its result. A blocking lead hands out a batch of subtasks and waits until every worker reports back. The workers can run in parallel with one another throughout that wait. An asynchronous lead can continue its own work, receive reports as they arrive, supervise workers and send feedback. The latter gives the lead more freedom, including responsibility for recognizing when its own next decision depends on a result it has yet to receive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
ProgramBench makes the team discover what it has to build
A ProgramBench task starts in an otherwise empty workspace with documentation and the original program’s executable. The agents can run the executable but cannot read its implementation. They have to probe it to discover its interface, observable behavior and edge cases, then write a replacement that behaves the same way. A plausible implementation can therefore fail even when the code does exactly what its author intended. The author may have inferred the wrong requirement from the original.
Dziri cited programs on the scale of SQLite and the PHP interpreter. For a sense of difficulty, she pointed to the FFmpeg task, where the best model on the leaderboard she described passed only 10% of the tests. That figure describes the challenge of reproducing FFmpeg. The team experiments measured a separate quantity, the change in reward when the researchers gave a lead workers on ProgramBench.
With minimal prompting, the blocking team beat the solo agent by almost ten points. Teams cost roughly four to seven times as much as a solo run. The blocking lead scored better than the asynchronous lead, and it also cost more and took longer to finish. Waiting improved the submitted program enough to show up in reward, even though keeping the lead busy looked like the better use of elapsed time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An asynchronous lead could submit while its workers were still exploring
The runs exposed a specific failure. The lead would ask workers to investigate the executable, make its own guess about the behavior under investigation and begin implementing that guess. By the time the workers reported what they had found, the lead had moved on. Sometimes it ignored the evidence. Sometimes it went back and redid its work. In the clearest cases, it declared the task complete and submitted while workers were still running. Their investigations cost money and had no opportunity to affect the submitted program.
The blocking harness removed that particular decision from the lead. A round of delegation had to finish before the lead could consolidate the reports, build on the findings and submit. That explains the advantage under the initial instructions. Those instructions described the task, the ability to create workers and the tools available. The researchers deliberately gave little guidance about coordination because they wanted to see how much of it the models could work out on their own.
They then tried detailed instructions covering planning, implementation and verification, abbreviated PIV. The prompt directed the lead to distribute exploration and planning, delegate implementation, keep edits within assigned locations and create adversarial workers to check the result. It gave the lead guidance about when to wait. Both team variants improved, and the asynchronous lead caught up with the blocking one. The ability to continue working had been useful only to the extent that the lead could tell which work was safe to start.
Better instructions also raised the solo score by almost six points. That makes the well-prompted solo agent an important control. Comparing a heavily guided team only with a minimally guided individual would credit delegation for gains available from instructions alone. Dziri also stressed how much time the researchers spent tuning those instructions. Even after that work, the guided teams cost six to seven times as much as the solo agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three experiments isolate planning, implementation and verification
The researchers kept the lead fixed and gave it workers at only one stage at a time. In the planning condition, the lead assigned different parts of the executable to workers for exploration. Each worker probed its part and returned findings and a plan. The lead used those reports to write the replacement and verify it. Here, planning meant empirical work against a running program. Its output was an account of what the implementation needed to do.
In the implementation condition, the lead explored the executable and developed the plan on its own. It split that plan into assignments, gave the assignments to workers and received code back. The lead retained responsibility for verification. This arrangement distributed the writing, but every worker began with the requirements a single investigator had been able to discover.
In the verification condition, the lead both planned and implemented the replacement. Workers entered at the end to check the work and report their findings. The lead then decided whether to submit. All three conditions gave workers consequential jobs. What changed was the point at which several fresh contexts became available and the kind of evidence or code the lead received in return.
On ProgramBench, planning-only delegation reached nearly the reward of the full PIV team at about half its cost. Dziri put the planning advantage at roughly 1.6 times the reward per added dollar compared with spending on the other stages. Increasing the number of implementation or verification workers drove cost up while performance stayed around the solo level. The limiting resource on this task was an accurate enough account of the binary’s behavior to guide the first implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The delegation traces show when 830 extra calls stopped paying off
Consider the discovery work from the perspective of one context window. A solo investigator accumulates the details of each probe while deciding what to investigate next. It can spend a great deal of context on one part of the program and still miss an edge case elsewhere. Several planners can each pursue a different part in depth. Their compact reports let the lead retain the observations needed for implementation across a wider range of behavior.
The researchers checked whether the agents actually used their workers this way. In their delegation plot, each row represented one run, time advanced from left to right, and each dot marked the lead sending work to a worker. Yellow identified planning, blue implementation and green verification. With planning guidance, the dots clustered in one early burst, before any code was written. The workers explored, the reports came back and the lead built the program.
The full PIV traces began with much the same burst, but the leads kept handing out work during implementation and verification. Across the traces Dziri showed, that added 830 delegation calls. She put the cost of the full PIV configuration at roughly $500, about twice the cost of planning-only delegation, for essentially the same reward. Measured per delegation call, planning alone delivered 2.4 times as much reward as full PIV.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Implementation workers inherited the lead’s gaps and overlapped when given more context
In the implementation-only runs, the lead had to discover the behavior of the entire binary before dividing the coding work. Anything it missed became a gap in the plan. A worker then received a narrow slice, with limited visibility into the larger task. It could make a local implementation error, but it could also carry out its assignment faithfully and reproduce the lead’s misunderstanding. Adding writers downstream did little to improve the observations on which their assignments depended.
The researchers tried giving workers more context. That introduced a different failure. Once workers could see neighboring features, some expanded their assignments to build additional functionality. Several agents could then work on the same thing in the shared workspace. They had the means to contact one another about scope and duplicated effort, yet most communication went back to the lead. A worker could keep editing without learning that another agent was already addressing the same problem.
The cost surfaced at integration. The lead had to recognize the overlap, determine where the implementation was incomplete and stitch the contributions together. When reports accumulated, leads sometimes struggled to communicate the problem back to the worker who could fix it. The resulting runs took longer, cost more and earned less reward. Parallel writing had moved work onto the agent responsible for assembling the finished program.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Commit0 supplies a skeleton and tests, shifting the payoff to implementation
Commit0 changed what the agents knew at the beginning. They received a code skeleton, documentation and unit tests visible to the workers. Much of the plan was already expressed in those materials. On ProgramBench, the team had to discover the behavior it was trying to reproduce before it could sensibly divide up the writing. On Commit0, workers could start from supplied implementation targets and known tests.
Here the largest improvement came when the researchers doubled the number of agents in the implementation stage. Dziri gave the direction of the result and referred listeners to the paper’s appendix for the figures. The two benchmarks support a practical way to set up a team experiment. Inspect what the lead actually has to hand off. When the intended behavior is obscure, several independent investigations can improve every subsequent coding decision. When a skeleton and tests already specify the work, extra implementation capacity can address the remaining difficulty directly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Price the planners separately from the model that must use their findings
Having narrowed ProgramBench delegation to planning, the researchers changed the models filling the two roles. They used three GPT 5.6 models, Luna, Terra and Sol, in increasing order of capability and cost. The reference planning team used Sol for the lead and its workers. The lead chose its own team size and spawned about five workers on average. This was already a more economical reference point than paying Sol workers throughout PIV.
Keeping Sol in charge and replacing its planning workers with Luna cut cost by about 60% relative to the all-Sol planning setup, with a small loss in performance. A Luna planner could pursue a bounded investigation and return what it learned. Sol still decided which investigations to commission, interpreted the results together and took responsibility for implementation and verification. Only the delegated portion of that arrangement had moved to the cheaper model.
The researchers also tried reducing the capability of the lead. Across solo runs, both performance and cost rose as they moved from Luna to Terra to Sol. Homogeneous teams, such as Luna with Luna workers and Terra with Terra workers, showed a similar capability ordering. The mixed teams revealed how much the assignment of roles mattered. Sol with Luna workers approached Sol with Sol workers. Terra with Luna workers scored 51%, trailing the 57% that Sol achieved working alone.
Dziri summed up the role split this way: “The lead sets the ceiling and the workers set the price.” She connected it to Meta’s work on meta-reasoning, where a better controller got more out of the same workers. In the coding harness, the lead chooses which parts of the binary to probe and decides how the findings fit together in one implementation. Luna workers could make those investigations cheaper. With Terra directing them, the team still finished six points behind Sol working alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Sixteen Luna planners cost 38% less than all-Sol planning and 70% less than full PIV
Until this point, team size was itself a decision made by the lead. To separate the effect of worker count from the quality of that decision, the researchers fixed Sol as lead, chose Luna for the planners and imposed team sizes ranging from two to sixteen. This let them spend more of the planning budget on independent probes while preserving the model that would combine the reports.
At sixteen planners, the mixed team cost 38% less than the all-Sol planning team whose lead had chosen about five workers on average. The headline 70% saving uses a more expensive comparison, the all-frontier team that delegated across planning, implementation and verification. The sixteen-planner team approached that team’s performance by changing two things together. It moved exploration to cheaper workers and stopped paying for delegation after exploration finished.
Sixteen was the largest configuration the researchers could afford to test. The observed gains gave them reason to investigate larger planning teams, but the experiment did not establish where the gains would level off. They also increased team size in the implementation and verification variants. More workers there still failed to help. Worker count had different effects even within the same benchmark, depending on what each additional worker could contribute.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Most messages went to the lead, and half of Luna’s failed at the interface
Every agent had a send-message tool that could reach any other agent in the sandbox. The researchers examined where messages went, whether the harness accepted them, how much text agents sent and whether enabling the tool changed reward. Despite direct peer access, 84% of messages went from a worker to the lead. Only 4% went between workers, averaging less than one peer message per task. Leads steered workers more often than workers consulted one another. The broad routing pattern persisted across model families.
The models differed substantially in how they used the interface. About half of Luna’s messages were malformed and rejected by the harness, so the intended recipient never received them. Sol produced roughly three to four times as much message text as Luna. Under full PIV guidance, stronger teams generally sent more messages overall, including more direction from leads to workers. Counting attempted sends alone would obscure both delivery failures and the difference in how much information the models tried to transmit.
Enabling messaging moved the Sol and Terra scores only within the noise. Luna teams lost four to seven points. Many delivered messages were status reports to a lead that would receive the worker’s result through the ordinary delegation mechanism anyway. That gave the lead additional text to process without necessarily changing an assignment, resolving a conflict or improving the submission.
Dziri cautioned that the experiment gave workers little reason or opportunity to find a useful peer. The lead deliberately tried to make assignments independent. Workers knew their own assignments and had a way to send a message, but they lacked visibility into which other agents existed in the sandbox and what those agents were doing. A worker seeking help, or beginning to overlap another worker’s implementation, first had to discover whom to contact. Whether better visibility would make peer exchanges worthwhile remains open.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Models that delegated equally often earned different returns
The researchers ran further model comparisons with the same harness and identical tools. GPT 5.6 teams delegated frequently and gained up to seven points in reward. Dziri also reported a Claude result with a gain of almost six points. Within the Claude results, Opus 4.7 rarely delegated and gained little. Haiku and Gemini presented a different case. They delegated about as often as the GPT teams yet saw no gain.
An assignment count can identify a lead that almost never asks for help. It cannot establish that a frequent delegator benefits from its workers. To see that, follow an assignment to its returned result and then to the lead’s next decision. On ProgramBench, a probe delivered before implementation could determine which behavior the lead built. The same probe delivered after submission could do nothing for the score. Both would count as a delegation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beyond sixteen workers, team design and supervision remain open
The experiments reached sixteen workers in a hierarchy with one lead. Dziri closed by asking how teams of thousands might organize, communicate and allocate compute. One possibility is to give agents a task, a budget and a deadline, then have them design the team themselves. That would require a capability the researchers still had to supply through prompt tuning, namely recognizing what kind of coordination a particular task needs.
Communication becomes a different research problem once a project outgrows one lead’s ability to coordinate it. The observed 84% of messages reporting upward and 4% passing between peers describe a small hierarchy performing deliberately separated assignments. Dziri left open whether effective peer coordination would emerge with scale, require different ways for workers to discover one another or need to be trained. Detailed, task-specific instructions helped the asynchronous lead here, but writing a new coordination prompt for every task would demand considerable human effort.
She also raised security and alignment as open questions for teams able to operate around the clock without supervision. She referred to a Hugging Face incident without describing its mechanics. Her concern was how to secure systems and infrastructure as autonomous operation extends from individual agents to whole teams. Dziri closed by crediting Kinan and Daniel for their work on the project.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…