Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp

Ben Holmes· Warp19:19

Loading slides…

Read the talk

Your AI Agent Is Hiding Its Mistakes From You

Ben Holmes shows how Warp turns failed tool calls and repeated human corrections into pull requests for better agent skills, then uses later sessions to check whether the fixes helped.

From a talk by Ben Holmes

At a glance

Ideas worth remembering

  • Collect the human-agent transcript, tool calls and results, and a commit-specific output together. A judge needs different evidence to assess communication, wasted effort, and code quality.

  • Score each criterion separately and keep the judge's reason. Two failures in one PR can call for different repairs, such as clarifying a testing preference and supplying a missing setup step.

  • Group failed reports across sessions before proposing shared skill or configuration changes. After a merge, use later scores and the relevant tool calls to check whether the targeted behavior improved.

Who engineers the software factory?

Ben Holmes, a DevX engineering lead at Warp, works on Warp Factories, infrastructure for teams building their own software factories. The familiar part of a factory is the path from a task to a reviewed change. Work might arrive through a task list, a board, or a Slack message. An agent researches or triages it. Another might work with a human to write a specification before an agent implements the change. Computer use gives the implementation agent a way to include screenshots or recordings with its pull request. Agents can run rounds of adversarial code review before the change reaches a human reviewer, and monitor the system afterward.

Eight boxes form a clockwise loop inside a dashed blue circle. Work moves from assignment through implementation, verification, review and shipping. Monitoring usage leads back to a new assignment.Open full source frame
Inner loop - improve the product

Each handoff gives an engineer something to inspect. A reviewer can read the spec, try the change, and decide whether the PR is ready to merge. A different engineering question concerns the process that produced it. If every implementation agent has to rediscover how to start the test environment, the team pays for that discovery on every task. The finished PR gives the reviewer little reason to suspect it happened.

Holmes calls work on the system that drives the agents “factory engineering.” It can extend to changing the architecture of the codebase or using static analysis to steer agents toward the right parts of it. He concentrates here on a narrower surface that a team can edit and review directly, its agent skills and configuration. The loop starts with completed conversations. Scoring agents judge how the work went, a scheduled agent examines recurring failures and proposes changes, and subsequent conversations supply evidence about the effects of those changes.

0:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Collect sessions where the work runs

For an engineer improving a personal setup, the collection step is small. Local agents such as Claude Code, Codex, and Pi keep session files on the machine where they run. Claude and Codex each have their own directory. A reviewer that can read the relevant local histories can look across tasks, including sessions the engineer has long since stopped thinking about.

The team version needs a place to read work from more than one engineer. Holmes has spoken with teams that automate zipping up everyone's local Claude directories and putting the archives in a shared S3 bucket. That gives the reviewing agent a common collection even when the engineers continue to run their coding agents on separate laptops.

A shared runner offers another collection point. The team sends tasks to a common system, which might run on a long-lived compute instance or spin up an isolated sandbox for each run. Warp uses isolated Docker sandboxes for its agents. The individual agents can have separate execution environments while their traces pass through one system. That distinction matters when looking for a pattern. What looks like an awkward test setup on one engineer's machine might turn up across many tasks that used the same skill.

3:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:12 · section reference included

Keep the request, the tool results, and the output together

Holmes wants three kinds of evidence available when a scoring agent opens a session. First comes the transcript, including what the human sent, how the agent responded, and what the human said next. That last part can matter as much as the initial request. If the human repeatedly asks for a changelog entry to be rewritten, the correction is evidence about the work required to get the desired result.

Second come the tool calls and their results. Include filesystem operations, calls to MCP tools, and the other interactions that connect the agent to its environment. Their sequence lets the judge distinguish a test command that worked the first time from one that worked after several failed attempts. Both runs might end with the same accurate statement that the tests passed.

Third comes the output the judge will evaluate for quality. A code diff stored with the conversation will do. If the output lives in a pull request, pin the commit and inspect its diff so the judge sees the version associated with the run. Each kind of evidence answers a different question about the same piece of work. The transcript establishes what the agent was asked to do, the calls show how it proceeded, and the diff lets a judge inspect the tests or comments it actually wrote.

The left column lists Claude, Codex and Warp conversation storage. The right lists material available for review. Three boxes below name the scoring categories.Open full source frame
Local agent conversation sources and scoring categories
5:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:11 · section reference included

Define redundant tests and verbosity in terms a reviewer can judge

Holmes asks for a show of hands from people who feel their agents write too many tests. He estimates about 40 percent of the room raised a hand. On his team the response was closer to 100 percent. Those are informal measures of how many people recognize the problem. To score a particular PR, the judge still needs to know what the team considers appropriate coverage.

Holmes proposes a scorer called Redundant Test. The decision depends in part on the team's expectations for unit, integration, and end-to-end coverage. An agent can produce a substantial suite of unit tests for a change the team primarily wants exercised end to end. Counting the new tests tells the reviewer how much was added. Reading them against the team's guidance lets the judge explain why that particular investment in coverage was excessive. Holmes notes that both model selection and instructions can affect what a team sees.

Verbosity concerns two things a human reviewer has to read. One is the agent's explanation of its work, including the summary that helps someone understand a PR. The other is the commentary it leaves in the code itself. Holmes asks the room about excessive code comments because they impose a cost beyond visual clutter. When review is the bottleneck, a reviewer spends time working through those comments to understand the change. A verbosity scorer can inspect both the conversation and the diff and judge whether the agent communicated the change clearly.

6:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:11 · section reference included

A passing test can conceal a broken testing skill

Efficiency is where the tool trace earns its keep. An engineer sitting through a short interactive session might notice a string of failed calls. Send the agent on a longer run in a cloud sandbox and the first thing the engineer reads may be its final report. Coding agents are persistent enough to get around quite a few obstacles. Once a workaround succeeds, the agent can report the outcome without mentioning the failed route it took to get there.

Warp encountered this in its own test harness. The testing skills contained out-of-date instructions, so the agent had to work out what environment it was in. It tried a succession of port numbers until it found a way to proceed, then got on with the code. From outside the run, the symptom was that the agent seemed to take roughly two minutes longer than expected. That was Holmes's estimate of the initial clue, rather than a timed comparison of two skill versions.

A judge looking at efficiency can follow the attempts in order and ask which ones moved the task forward. The successful call explains how the agent recovered. The unsuccessful calls explain why recovery was necessary and where better instructions could have saved work. As long as the stale skill was the starting point for the next run, a successful workaround helped only the agent that found it.

7:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:41 · section reference included

Write a Markdown rubric with an investigation threshold

Warp expresses a scorer in Markdown, much like a specialized skill definition. For efficiency, the question is whether the agent worked toward the result or spent effort on redundant steps and unnecessary back-and-forth. The scorer gives an LLM judge labels describing different levels of performance. Holmes likens them to a teacher's essay rubric. A grader has a much more specific job when it knows which features of the work distinguish one mark from another.

The efficiency definition calls out three places to look. Cost to the human includes the extra exchanges required to get the work done. The information-gathering phase covers what the agent had to explore before it could proceed. Routine step overhead covers calls consumed by ordinary operations, such as getting a test environment running. These distinctions allow the judge to recognize wasted effort even in a run that needed no human intervention, or repeated human effort in a run with few tool errors.

The team also chooses where a grade merits investigation. In Holmes's example, 0.4 and 0.2 fall below the line. Scores above it pass and can usually be left out of the failure review. The threshold turns the rubric into a practical filter once there are many sessions. Holmes recommends starting with a couple of bullets and a rubric, trying them against conversations, and building up the definition as needed. The team has to decide what counts as wasted effort in its own environment.

Warp displays four efficiency labels in a rendered Markdown file. Their scores range from 1 for a direct path with batched steps to 0.2 for repeated rework, user correction or looping.Open full source frame
efficiency.md scoring labels
8:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:42 · section reference included

Grade one merged authentication PR three ways

Holmes walks through a hypothetical full-stack authentication service. The agent implemented it, the conversation is available, and the PR has merged. A merge is Warp's usual trigger for grading. The unit of evaluation is one judge run per selected conversation, with that judge assessing the conversation against each criterion. A team can grade every conversation or select a random sample to control token use.

Start with the tests. The implementation agent wrote a large number of unit tests for the service, while the team's instructions say to prefer end-to-end tests for most of this coverage. The judge can fail Redundant Test and identify the mismatch. Move to verbosity and the same PR passes. In this example the team has selected a model that leaves few code comments, and the PR is easy to understand. The testing issue supplies no reason to reduce the communication grade.

Finally, follow the calls the agent made while figuring out how to test the service. It spent considerable effort exploring the environment to discover something the team could have explained up front. That earns an efficiency failure under the example rubric. The distinction between the two failures is consequential for the person editing the skills. The agent overlooked existing guidance on which tests to write. It needed additional guidance on how to run them.

The example session fails the redundant-tests and efficiency checks and passes verbosity. Each grade appears beneath its rubric filename. A sessions list and a pass/fail legend appear below.Open full source frame
Scoring update-auth-service.jsonl with three rubrics

The judge records a reason alongside each pass or failure. Over many sessions the team can calculate pass and failure rates for each criterion, but a rate alone gives the next agent little to act on. The explanations preserve details such as which testing preference the agent missed or which part of setup consumed the extra calls.

10:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:41 · section reference included

Give a daily agent the failed reports and the skills repository

Grading happens close to the individual run. Finding a shared cause happens on a different schedule. Holmes describes a cron job that wakes every 24 hours and looks over a sample from the preceding day. His illustrative numbers are 200 conversations and 50 failures. The scheduled agent receives the failing reports with the judges' notes about what failed and why. Its assignment is to find commonalities across them.

Give that agent access to the skills repository so it can propose an edit where the relevant instruction lives. Holmes also describes giving it access to the codebase if architectural changes are in scope. For example, if redundant tests recur across PRs, it might propose adding a coverage pyramid that makes the team's expectations for different kinds of tests clearer. The recurring reports give a reviewer a concrete reason to consider that edit. A single disappointing score on an authentication PR would say much less about what belongs in a shared skill.

Four steps circle the core factory loop. The scoring example marks efficiency and code quality with checks and test coverage with a cross. A proposed edit to the test-writing skill adds a coverage pyramid.Open full source frame
Every 24 hr: review agent conversations and propose improvements
12:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:41 · section reference included

Skill Doctor finds two different fixes in local Codex history

Skill Doctor gives an individual engineer a one-time version of the loop. The skill is available at warp.dev/skilldoctor and can review local Claude Code, Codex, or Pi conversations. In Holmes's example it reads Codex sessions from work on the project he is showing and grades them in batches to use tokens more efficiently. The grading stage produces score reports. The self-improvement agent then reads the scores and notes and produces a report card with suggested changes. Holmes identifies efficiency and code quality as criteria in the test environment and says verbosity looked fine.

One finding concerns the end-to-end testing skill. Across sessions, the agent was struggling to get a port on the machine where it ran tests. The proposed edit changes the order of operations in the instructions. Find an available port first, then use it to start the service. That gives the next agent a specific action at the point where its predecessors began guessing. Holmes says this kind of problem is easy to miss when the run happens in the cloud or its activity is tucked behind a working indicator in the agent interface.

The changelog finding comes from a different part of the record. Holmes kept asking for changes to the language of the changelog entries agents produced for PRs. Skill Doctor picked up that repetition and rewrote the changelog skill to prescribe the wording more closely. In the port case, the agent had been doing the extra work. In the changelog case, Holmes had been doing it by steering the agent back to his preferred language. A one-time pass over existing history surfaced both.

A highlighted three-line addition tells the agent to name a feature once. It pairs a short keyboard-shortcut release note with the list of options a user previously had to trim.Open full source frame
Proposed changelog skill edit with a keyboard-shortcut release note
13:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:41 · section reference included

Check later scores against the date a skill changed

Holmes loses access to his staging view as he reaches the score graphs, so he describes how to read them. He compares the setup with an agent harness that reflects on conversations overnight and stores memories. Both approaches let past sessions influence future work. With a memory system, an engineer can have difficulty seeing what the agent concluded. Explicit grades give the engineer a record of how the sessions went and a quantity to follow after changing a skill.

His proposed graph plots average efficiency over time as new conversations are graded. He uses 50 percent and 60 percent as example averages. Mark when a proposed skill was merged and look for an inflection in the subsequent scores. For the port change, the underlying traces also give the reviewer a specific behavior to check in later test runs, whether the agent finds a port before trying to start the service. The percentages illustrate how to read the graph. Holmes gives no measured before-and-after gain for that fix.

16:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:11 · section reference included

A merged triage fix changes the factory configuration

Warp Factories runs the scoring continuously across the agents supporting Holmes's team. On a schedule, another agent reviews those scores and suggests PRs. The team's inbox is the human review point. Holmes shows a proposal the team had already merged, involving a triage skill that told an agent to spawn sub-agents.

The triage agent repeatedly spent tool calls trying to work out how to spawn them. The missing piece was a factory configuration setting that enabled sub-agents. The proposed PR patched that configuration, bringing the capabilities available to the agent into line with the skill it was already following. This is the value of retaining the attempted calls alongside the instruction. They can lead a reviewer all the way from a costly sequence in a completed session to the particular setting the next session needs.

In a side-by-side GitHub diff, three green lines add a spawnableBy list to the TRIAGE agent definition. The entries are foreman and triage.Open full source frame
GitHub diff adds foreman and triage to spawnableBy
17:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:41 · section reference included

Transcript download
Full transcript

Loading transcript…