Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic

Loading slides…

Read the talk

Your Old Prompts Are Making Claude Worse

On a roughly 50-ticket support benchmark, Anthropic’s Lance Martin shows how tuning model, effort and prompt cut cost by 90% while improving the score. He also shows how Claude revised an API skill that scored 66% and how long-running agents can recover from a sandbox failure.

From a talk by Lance Martin

At a glance

Ideas worth remembering

  • During a model migration, score the inherited prompt on the new model before auditing it. That intermediate run separates the effect of the upgrade from the cost and quality effects of removing obsolete instructions.

  • Inspect consecutive agent requests at the first byte where their prefixes diverge. Put shared tools and instructions first, project context next, then session state and growing message history. Verify the result with the cache hit rate.

  • Give the hill climber training failures, keep test examples out of its editing loop, and measure both splits each round. For cost work, set a baseline quality floor and search model, effort and prompt together.

  • Before trusting an open-ended grader, apply its rubric yourself, compare grades across models, and repeat the grading run. Use separate binary judgments or a pairwise choice where those decisions are easier to make consistently.

  • For open-ended artifacts, try judging pairs. Use several absolute grades when a human can explain adjacent grades and inexpensive model judges reproduce the distinction.

  • Give a hill climber a budget and held-out cases. When progress stalls, group the misses and inspect the questions and graders before paying for more rounds.

  • Compare model and effort as a pair on the same workload. Start with a capable model at low effort and measure cost per task against the existing quality floor.

  • For long-running agents, preserve a queryable session history. Choose whether the eval should grade the trajectory or the final outcome, then promote consequential production failures into regression tests.

An application can sit inside the cost–quality frontier

Before paying for a stronger model or accepting a lower score to save money, Lance Martin asks where the application sits on its own evaluation. Picture cost on the horizontal axis and quality on the vertical axis. For a given task, the best available choices trace a Pareto frontier. At a point on that frontier, improving one measure entails giving something up on the other. Martin sketches a family of choices with Fable at the expensive, high-performing end, followed by Opus, Sonnet and Haiku.

The model name alone tells an engineer very little about whether an application reaches that curve. An agent might pay repeatedly to process identical instructions. Its prompt might also make Claude spend tokens on a verification routine that gets it no closer to the right answer. An application with either problem has room to move left on the chart, and one with confusing instructions may be able to move up at the same time. Martin, a member of technical staff at Anthropic, has seen customers make both gains through small configuration changes. His first example concerns a piece of configuration that often survives several model upgrades intact.

0:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Audit the instructions an earlier model needed

Prompts accumulate debt much as code does. Several model generations ago, engineers routinely asked a model to “think step by step,” gave it a scratchpad, and specified how to use that space to reason. They added a verification instruction such as “double check your work” to get it to revisit an answer, or a general booster such as “be maximally thorough.” Each intervention addressed a recognizable behavior. Together they could become a standing procedure imposed on every request, regardless of its difficulty.

Frontier Claude models already choose when to reason and naturally verify their work, in Martin’s experience. An inherited demand for thoroughness can therefore change the amount of work the model does even when its first answer would have been adequate. Contradictory rules have a different effect. A weaker instruction follower could quietly ignore one rule, producing an answer that made the prompt appear serviceable. A stronger instruction follower tries to honor both. The upgrade exposes an inconsistency that was present all along.

Martin describes two routes to the prompt audit in Claude Code. Use the checkup skill’s prompt audit for Claude Code’s own configuration, including its prompts and skills. Use the bundled Claude API skill’s prompt audit for application code that calls Anthropic’s APIs. Both reach the same underlying audit. The distinction is where the engineer is working. From the working directory, the audit can inspect skills, tools and prompts for these patterns and remove them. That scope matters if the application prompt looks clean but a tool’s instructions reintroduce the same verification ritual.

Two stacked boxes show prompt-audit commands in large monospace text. The upper command starts with /checkup and the lower with /claude-api.Open full source frame
Claude Code prompt-audit commands
1:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:31 · section reference included

Separate the gain from a model upgrade from the gain from a prompt edit

Martin built a synthetic customer-support evaluation with about 50 tickets and simulated a migration from Opus 4.8 to Opus 5.5. He introduced the prompt anti-patterns individually and measured the results. The intermediate point in his comparison averages across those experiments. This makes it possible to ask two questions about the same queue of tickets. How does the new model perform with the inherited habits still in place? What changes when the audit removes them?

The migration alone moved the point to the left and slightly upward. Opus 5.5 cost less and scored a little better even with the extra instructions. Running the prompt audit after the migration yielded a further reduction in cost and another improvement in score. Martin attributes the token savings to removing boosters and verification rituals that induced Opus 5.5 to do excessive work. Resolving instructions that confuse the stronger model also gives it a better chance of answering correctly.

A diagram shows a legacy prompt shortened by an audit. Beside it, arrows connect three points on a chart: Opus 4.8 legacy at 2.45¢ and 88.1% accuracy, Opus 5.5 unaudited at 2.02¢ and 90.0%, and Opus 5.5 audited at 1.83¢ and 92.0%.Open full source frame
Cost per ticket and accuracy before and after a prompt audit

For an engineer reproducing the comparison, the intermediate measurement is worth keeping. A before-and-after test that changes the model and cleans up the prompt in one move gives a single result for two decisions. Holding the ticket set fixed and measuring the inherited prompt on the new model first reveals how much the application was leaving on the table after the upgrade. Martin reports the direction of the two gains here. The percentage savings he gives later come from separate experiments.

3:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:49 · section reference included

Order an agent’s context by how long each part stays the same

Consider two successive requests in an agent session. Between them, Claude may have made a single tool call. The new information is small, yet the next request carries much of the same context as the last one. A prompt-cache hit lets the application reuse that work at lower cost and with less latency. Martin puts the read price at roughly a tenth of the base token rate, with the qualification that pricing varies by model. Over many turns, the proportion of repeated input that actually hits the cache matters to the economics of the whole session.

The cache requires a strict prefix match. A byte that changes early in the prompt breaks the match at that position. Thousands of identical tokens after the change cannot extend the matching prefix. An engineer can therefore have two requests that contain almost exactly the same information and still get little reuse. Inspecting which material repeats is only half the diagnosis. Its position in the assembled request determines how much the cache can recognize.

Two ten-position request diagrams compare cache reads. The top row reuses positions 1–8. Changing the time from 9:41am to 9:42am at position 4 leaves only positions 1–3 cached in the bottom row.Open full source frame
Caching is a strict prefix match

Martin arranges the input in four layers. System instructions and tool definitions shared across sessions belong at the beginning. Project material such as CLAUDE.md and project memory comes next. Session state follows, and messages accumulate at the end with each turn. On a project switch, the shared instructions and tools can still match. Within a project, a new session can reuse the project material too. Appending the next message gives the existing conversation a chance to match all the way through the previous turn.

The Claude Console exposes the application’s prompt-cache hit rate. The Claude Code team tracks that number closely, and Martin recommends checking it for applications built with the API. A diagnostic in Claude Code can inspect an API project, identify where its prompt construction breaks reuse, and suggest a better order. It also looks at prompts and effort settings. For a cache investigation, the important output is a concrete opportunity to move changing material behind a reusable prefix, followed by a hit rate that shows whether the change worked.

5:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:30 · section reference included

Default benchmark harnesses left up to 72% in cache savings

Martin and colleagues on Anthropic’s applied AI team ran the diagnostic against several internal benchmarks, using the default harnesses that issued the model requests. Some harnesses made poor use of the prompt cache. The team found opportunities to reduce cost by up to 72% with no meaningful change in measured performance. The upper end of that result belonged to those particular harnesses. An application already getting long prefix matches has a different starting point.

Bars show cost per run falling across four benchmarks, from a 24.3% reduction on SWE-bench Verified to 72.9% on tau2-bench (retail). Beside each bar, a dot and whiskers show the score change and its 95% confidence interval.Open full source frame
cost-optimize can reduce cost by fixing prompt caching

This experiment isolates a different expense from the support-ticket prompt audit. Removing a verification instruction changes the work Claude decides to do and can affect its answer. Making better use of the cache addresses the price of processing material the agent was going to receive again anyway. That explains why the benchmark score could hold steady while the cost moved substantially. It also explains why the default harness deserves inspection before anyone interprets an expensive benchmark run as evidence that the model itself is too costly.

7:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:41 · section reference included

Check that a higher evaluation score is attainable

Martin’s next step depends on Claude’s capacity for self-correction. Given an evaluation, Claude can examine failures, change part of the application and try again. Martin calls that loop hill climbing. Before letting it spend tokens on successive attempts, he checks whether the evaluation gives it a meaningful hill to climb. His diagnostic picture plots tokens per attempt, a measure of how much thinking the model does, against the resulting score. Separate curves represent a frontier, mid-sized and small model, roughly Opus, Sonnet and Haiku in his example.

He looks for four properties. First, stepping up to a more capable model should generally improve the score. Second, allowing more thinking should generally improve performance within a model. Third, the strongest model should still have some solvable failures, leaving headroom for a better configuration. Fourth, repeated runs under the same conditions should yield similar scores. A large difference between two successive runs means less if the same configuration routinely produces swings of that size.

Three curves rise and flatten from low to high effort. The frontier model scores highest at each effort level, followed by the mid-size and smaller models. Short error bars mark each point, and all three curves stay below the perfect-score line.Open full source frame
Model score versus action tokens per attempt

The headroom check requires reading actual cases. An evaluation at 80% appears to offer another 20 percentage points. If its remaining questions are ambiguous, a model can keep producing reasonable answers without getting credit. Even a widely used evaluation can contain enough such tasks to be effectively saturated. An autonomous optimizer can burn through a substantial budget searching for a prompt that will recover points the evaluation gives it no reliable way to earn.

The build-eval workflow in the Claude API skill starts with a problem, what the engineer wants to measure, and raw data such as traces from the team’s existing tracing system. The skill analyzes the data, builds an evaluation, and reviews the cases and grading criteria with the engineer. During Q&A, Martin describes a change he has proposed: let the engineer inspect the raw traces Claude has flagged before it turns them into evaluation cases.

9:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:02 · section reference included

Keep the test cases out of Claude’s editing loop

Once the evaluation behaves sensibly across models, thinking budgets and repeated runs, Claude can start proposing changes autonomously. The risk is that it learns what earns points on this particular collection of examples. Suppose one benchmark task needs optical character recognition. An optimizer permitted to change the harness might add an OCR tool and solve the case. If production users never submit that kind of problem, the new tool improves the benchmark while doing nothing for them. A prompt can achieve the same spurious gain more quietly by picking up phrasing distinctive to the examples.

The Claude API skill’s hill-climb workflow splits the evaluation into training and test sets. Claude inspects the training failures and uses those cases to decide what to change. The runner measures performance on both sets each round, while the test examples stay out of Claude’s deliberations. That separation lets an engineer watch two scores from the same edit. If training performance keeps rising and the test score stays flat, the system has an early signal that Claude may be fitting details peculiar to the training cases.

A feedback loop shows an analyzer using training failures to propose one edit while the model and harness stay fixed. Train and held-out test scores determine whether to keep the edit.Open full source frame
Evaluating one proposed change per round

The split answers a limited question. It can show whether a change transfers to other examples drawn from the evaluation. Its usefulness for predicting production performance still depends on how the original examples were selected. A benchmark full of unusual requests can produce train and test sets that share the same bias. Martin therefore keeps the engineer involved in auditing what the hill climber changes. The question to ask about that new OCR tool is which real users need it, even if the benchmark rewards the addition.

11:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:47 · section reference included

Give Claude the API skill it can edit and the documentation it must match

Martin tested the method on the Claude API skill itself. The skill gives Claude Code guidance on Anthropic’s SDKs and APIs, so its answers need to be grounded in Anthropic’s documentation. He built evaluation questions from that documentation and tested the existing skill. It scored 66%. This gave him both a specific failure to investigate and a bounded place to make changes. The evaluated model was reading the skill whose text the optimizer would be allowed to revise.

He then gave Claude Code access to the skill text and the underlying documentation. For each round, Claude could inspect a failure, consult the source, identify where the skill misrepresented the documented behavior, and revise its guidance before trying again. The documentation served as a reference independent of the current wording of the skill. An edit could therefore be assessed in two ways, by checking its fidelity to the source and by measuring whether the model using the revised skill answered the questions better.

Martin reports a substantial improvement from 66% without giving a final score. He singles out the fit between the task and the editable material. The evaluation asked about API behavior, the documentation supplied that behavior, and the skill was the piece of text meant to convey it to Claude. That short path from an edit to an observable answer made repeated autonomous experiments practical. He has found prompts and skills particularly amenable to this treatment.

A line chart tracks pass rate across three phases of skill and grader edits. It rises from 66.1% at baseline to 87.9% at round 24, with small dips at rounds 9 and 13.Open full source frame
Pass rate rises from 66.1% to 87.9% over 24 hillclimb rounds
14:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:57 · section reference included

Search model, effort and prompt together under a quality floor

The same runner can optimize for a different objective. Tell the hill climber to reduce cost, and it searches for a configuration that costs less while at least matching baseline performance. It can vary three things. Model selection changes which Claude does the work. Effort changes how much reasoning the model applies. Editing the prompt changes the instructions under which it solves the cases. The evaluation score gives the search a quality floor while the measured cost distinguishes configurations that clear it.

On the synthetic support benchmark, Martin started with Opus 4.8 at high effort and a baseline prompt. The hill climber first moved to Opus 5 at low effort. Performance jumped and cost fell a little. It then tried another model at low effort. That round delivered a much larger cost reduction but gave back some of the previous round’s performance gain. Claude next edited the prompt and recovered performance. The final configuration cost 90% less than the starting configuration and achieved a higher score.

Dashed arrows connect four points. Moving from Opus 4.8 to Opus 5 raises accuracy and lowers cost. Switching to Sonnet 5 lowers both, then a prompt edit restores accuracy to near 100% at about 1¢ per ticket.Open full source frame
Cost per ticket and train-split decision accuracy across four configurations

The intermediate results explain why Martin allows all three controls into the search. Stopping at the first model switch would have captured a quality improvement but only a small saving. Judging the next candidate solely by its drop from the preceding high score would have obscured a much cheaper configuration whose prompt still had room for improvement. The reference point for the cost objective was the original quality floor. The runner could measure the effect of each successive choice against it.

Cost optimization also gives a saturated evaluation a job it can still do. Even if further quality gains are out of reach, the set can test whether a cheaper configuration maintains the score. Martin has repeatedly seen a stronger model at low effort solve a problem in fewer tokens than a weaker model working at high effort. A higher price per token can then translate into a lower price for the completed task. The hill climber tries that possibility and measures it on the workload. The 90% figure describes the complete support-benchmark run, including the later prompt edit.

16:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:29 · section reference included

An overnight agent needs a history that survives a sandbox failure

Martin shifts from the economics of one attempt to the duration of the work an agent can undertake. He describes METR charts with time on the horizontal axis and the time a task would take a human on the vertical axis. He puts Claude Opus 3 at roughly four minutes of human-equivalent work, a horizon he associates with autocomplete and chat. During 2025, he says, models reached tasks on the order of an hour or two, as Claude Code gained popularity. He sees agents moving toward tasks representing a day or two of human work, while qualifying that this may extend beyond what the charts can measure.

That prospect changes what a useful harness has to survive. Imagine starting an agent in the evening and having its sandbox die halfway through the night. If its session log lived only there, the same failure that interrupted execution also took away the history needed to recover. Claude Managed Agents separates the harness that directs the work, which Martin calls the brain, from the sandboxes where it writes code and performs actions, the hands. A third component, the session log, has a lifetime independent of both.

The log is an append-only, recoverable object. If the sandbox dies, the system can boot a replacement. If the harness dies, an orchestrator can bring up another harness. In either case the session history survives and provides a basis for recovery. Martin’s example concerns recovering from that record. The design places the evidence of what happened outside the two components most immediately involved in carrying out the work.

Credentials have their own boundary. Managed Agents holds secrets in vaults associated with the tools. Claude can reference an API key or another secret while its value stays in the vault, outside the model’s view. An agent can thus work through tools that require credentials throughout a long run. For Martin, the prospect of an agent operating autonomously for days makes the placement of those credentials as much a harness-design question as the placement of the session log.

18:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:53 · section reference included

A scheduled Slack update brings the managed components together

Martin and a colleague, CJ, put together an example that gathers regular context updates. The engineer specifies the sources the agent should consult. A scheduled deployment runs on Anthropic infrastructure, the agent reads from those sources, and it can write updates to Slack. The example combines scheduling, securely stored secrets and memory in a customizable automation. To start it, Martin directs engineers to the Managed Agents onboarding workflow in the Claude API skill in Claude Code. Supplying the accompanying tutorial to that workflow lets Claude spin up the example for them to explore.

A schedule wakes the agent to read sources, update its state and consult reader preferences. Guardrails surround the agent and its duplicate-post check. An approved destination delivers the brief to the reader, who can update preferences.Open full source frame
How a scheduled agent reads sources and delivers a brief

Martin connects this back to a working pattern he sees among colleagues. An engineer can spend the day defining what success looks like on an evaluation and give Claude the night to improve a skill against it. The cache matters across the repeated turns, the evaluation determines which changes count as progress, and the harness has to keep the job recoverable while its owner is away. As Claude takes on more of the iteration, decisions about the cases, grader and permitted edits account for more of the engineer’s work.

22:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:09 · section reference included

Let Claude flag suspect traces before a person selects evaluation cases

An audience question asks how much of evaluation design can be automated. Martin recalls feedback from Hamel on the build-eval skill. In a customer-support example, the starting material was a collection of traces and the first decision was which conversations contained failures worth promoting into an evaluation. Claude did well at scanning the offline logs and pointing out potential errors, much as an online evaluator might flag a problematic interaction. That helps a person find cases worth reading in a large collection.

Hamel wanted the skill to walk the user through the underlying data first. Martin agrees and distinguishes the proposed workflow from the behavior available at the time. The skill went straight into analyzing the data, building an evaluation and presenting it. Martin had a pull request open to add an intermediate viewer. Claude would annotate candidate failures or suggest what to look at, then show the person the raw traces before examples were promoted. The order matters because the person deciding what belongs in the suite can inspect the interaction that gave rise to the annotation.

A follow-up asks about hill climbing with human feedback. Martin’s requirement is a signal per example that Claude can use to distinguish good from bad. Human judgments can supply that signal once they have been incorporated into the evaluation. This divides the work at a different point: people can establish which outcomes are desirable, and Claude can use the resulting judgments while it iterates.

24:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:25 · section reference included

Choose an editable file whose effect the evaluation can see

Asked where to start hill climbing, Martin returns to the relationship between an edit and an answer. With a prompt or skill, Claude can change a small piece of text that the evaluated model reads directly. With an agent harness, a single round can produce a much larger code diff. Some changes may affect behavior the evaluation never exercises. People have successfully hill-climbed harnesses, he says, but it takes more work to interpret a score change when the edit could have acted through several different parts of the system.

The ease of editing text has a downside he has observed in his own experiments. Claude is adept at nudging a prompt into a shape that works unusually well on the chosen examples. Even a train/test split leaves room for that prompt to fit a quirk shared by the entire set. A small, legible diff makes it easier to inspect what Claude did. The engineer still has to judge whether the new wording expresses a requirement that will recur in production.

Another question concerns when a prompt or standard operating procedure should become a skill. Martin often completes a workflow with Claude and then asks it to turn the repeated procedure into a skill. For an optimization run, he usually starts one step further along, with an existing skill or a crude piece of boilerplate that Claude can change. His API skill was essentially a distillation of the docs, directing Claude to particular pages for particular questions. It contained enough small gotchas to produce the low initial score, but it gave the evaluation a baseline and the optimizer a specific file to improve.

26:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:37 · section reference included

For open-ended answers, test whether the graders agree

A question about open-ended work gets at what the hill climber should trust as a score. Martin prefers a deterministic check wherever the task admits one. For a piece of writing that resists such a check, he starts with a rubric he could apply himself. The practical test is to read a sample answer alongside the criterion and decide whether he can actually grade it. Handing Claude an arbitrary zero-to-five scale leaves the distinctions between scores unresolved. A request to rate the text from one to ten introduces still more opportunities for noisy judgments.

For one criterion, Martin favors a crisp binary decision. If several qualities matter, use several binary classifiers so each grader has a specific question to answer. Then test the rubric itself. Give Sonnet, Opus and Fable the same work to grade using the same criteria. Broad disagreement is a warning that the rubric leaves too much to interpretation. Separately, run replicates to see how tightly the grades cluster across repeated attempts. The first check probes agreement between graders. The second asks whether a grader can reproduce its own judgment reliably enough to guide the next edit.

30:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:04 · section reference included

A choice between two pages can be easier to grade

An audience member suggests pairwise comparison. Martin gives the example of two web pages. A reviewer may know immediately which page they prefer yet struggle to give either page a grade from zero to five. The pair supplies a point of comparison that the numerical scale leaves unspecified. Martin has found that people and model judges can both make the comparison more easily when the quality of an individual artifact is hard to grade.

There is a separate question about which artifacts to put in front of the judge. By the “distribution” of an eval, Martin means the set of cases sampled for it and how well those cases represent the work someone cares about in production. An engineer can have an excellent way to judge each case and still select a collection that gives a misleading picture of the application. One particularly tempting way to do that is to build the entire collection out of failures from a single model.

Suppose Sonnet 5 misses a group of examples. On the evidence of those runs, they all look hard. Promote them to the eval, switch to Opus 5, and the new model might solve every one. Martin describes model capabilities as jagged. The particular pocket where Sonnet struggled may be quite accessible to Opus. An eval selected this way, through what he calls adversarial sampling, acquires the shape of one model’s weaknesses. A large score increase on upgrade can then tell the owner surprisingly little about the rest of the workload.

Martin would still promote a production failure that exposes something important. The selection question is what makes the case worth keeping across model versions. A product owner might recognize a difficult request from experience, see in the production data that it matters, or have a reason from first principles to expect the application to handle it. That rationale survives the day a different model gets the answer right.

32:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:02 · section reference included

A grading scale has to work for the person and the model applying it

An audience question presses Martin on the apparent preference for binary grading. Can an eval put outputs into several buckets? Yes. A binary decision is a shortcut for simplifying a difficult judgment. His objection concerns the familiar zero-to-ten scale whose author can name the endpoints but gives the grader little basis for distinguishing the values in between. Giving an output a seven looks precise. If the grading rules would support a six equally well, that precision is of little use to an optimizer measuring its next change.

Martin gives three conditions under which he would be comfortable with several grades. A human reviewer should be able to distinguish two grades using the rules. Repeated grading should have low variance. The distinctions should also be reproducible across models. Together, those checks establish that the categories mean something to a person and that the machinery applying them can recover that meaning consistently. A well-thought-through multicategory classification meets his objection just as well as a binary one.

Cost matters because the judge may run every time a candidate configuration produces an answer. Martin often wants a small, inexpensive model to do that work. Depending on the cost constraints, using Fable for routine grading could be an expensive way to administer an unclear rubric. It is worth asking whether the cheaper judge can make the distinctions the eval designer has asked for.

33:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

33:44 · section reference included

A personal automation can still produce work that deserves an eval

Martin decides whether to build an eval by looking at the consequence of the task. Consider someone in finance who automates part of their own workflow. The tool has one maker and perhaps one user, yet its output may go to people outside the organization. If the quality of that deliverable matters, Martin would build an eval. The same reasoning applies to a personal agent entrusted with sensitive work. For a calendar helper someone built to manage their own meetings, he could go either way. He has built few evals for his own personal automations.

The investment in tooling can follow the scale of the job. Asked about platforms, Martin mentions that he worked at LangChain before Anthropic and declines to endorse a particular provider. At small scale, he would be comfortable building a viewer for the data at hand. A bespoke problem may call for a very particular view of the records, and he finds Claude capable of building that interface quickly and inexpensively. He also describes work on a simple data viewer for the skill. The immediate need is modest: give the person making quality decisions a way to inspect the underlying examples.

34:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

34:37 · section reference included

When the hill climber stalls, inspect what the remaining failures have in common

Hill climbing means making a change, running the evaluation, inspecting the results and using that evidence to choose another change. Martin has seen the process get stuck many times. A common starting condition is that someone has invoked the hill climber on an existing suite without first running the eval-building command or otherwise inspecting the cases carefully. The score rises until the easy-to-interpret examples pass. The remaining failures continue to promise gains on paper, so the optimizer keeps working on them.

Imagine the last 20 percent of the suite contains ambiguous instructions or faulty grader examples. A run can fail because the author left the intended answer open or because the grader penalizes a defensible answer. In either case, repeatedly editing the application prompt is a poor way to discover the cause. The hill-climbing skill has a fallback for several rounds without progress. It groups the remaining failures and looks for common reasons they keep failing, including problems in the evaluation itself.

Martin saw that fallback operate in the second stage of one of his own runs. After progress stalled, the skill clustered the cases, investigated why the answers were being marked wrong, found several ambiguous cases and rewrote them. That sequence matters when interpreting the next score. An edit to a skill can make the application answer an existing question better. Rewriting the question clarifies the task against which subsequent answers will be judged. Both can move the work forward, but they establish different things about the application.

Asked how the approach relates to DSPy, Martin describes the hill climber as an open-source skill bundled with Claude Code. Its core is a set of instructions for setting up the optimization loop. A developer could vendor those instructions into another environment. Martin thinks the guidance might be adaptable to DSPy and credits Omar and the DSPy team. He has not used DSPy himself and offers the connection as a possible implementation, without claiming to have tested it.

36:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

36:39 · section reference included

Search model and effort together, with the current score as a floor

Choosing an effort setting is sufficiently unintuitive that Martin admits to finding it confusing himself. With an eval, he can turn the choice into a search. Establish the application’s baseline performance, let the hill climber try different models at low effort, and compare both their scores and their cost. Work through candidate configurations until performance falls below the baseline. That last measurement is informative even when the configuration will never ship. It locates a point at which the savings come at an unacceptable quality cost.

One such observation was missing from the comparison Martin had shown. Haiku 4.5 at low effort produced a significant regression. He distinguishes that result from Haiku 5.5, which he describes as a better model. The observed regression belongs to the 4.5 configuration he actually tried. It also explains why advice to start at low effort still calls for a performance check. The ability of a capable model to solve a task economically says little about how a different model will fare at the same setting.

Without an eval, Martin’s starting suggestion is a strong model such as Opus 5.5 at low effort. Asked why low rather than medium, he points out that the engineer can turn it up. Starting at maximum effort incurs the largest reasoning spend before there is evidence that the task benefits from it. Starting low gives the engineer an opportunity to see what the model can already do. Martin presents this as a useful order of experiments, with less certainty about the eventual choice when the application lacks a quality measure.

The economic intuition depends on the number of tokens consumed per task. Effort tells Claude how hard to think. A model with a lower base price per token can spend many tokens reasoning at high effort. A more capable model may reach an equally good answer with fewer thinking tokens at low effort. The reduction in tokens can be large enough to outweigh the higher price per token. Martin starts to call the stronger configuration more effective, then corrects himself. The advantage he is describing is token efficiency. Whether it also produces a better answer is a separate result to measure.

38:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

38:45 · section reference included

Budget the search and let a person define a consequential failure

The hill-climbing skill asks for a budget during setup and tracks spending against it during the round. That budget covers the experiments themselves. The per-task cost of the configuration the search eventually selects is a separate measure. If the final 20 percent of a suite is impossible to solve, the optimizer can spend heavily chasing those points. Inspecting the stubborn cases helps the owner decide whether another round is worth buying.

For the division of labor between Claude and a human, Martin draws on Hamel’s public use of the build-eval skill. Take a customer-support application. Before Claude reads a trace, someone who knows the product needs to decide what matters about a support interaction and why. Customer satisfaction is one possible concern. Specifying it gives Claude a reason to flag particular conversations. Simply handing over a pile of traces leaves a product decision implicit in whatever the model happens to find interesting.

Once the owner has specified the goal, Claude can read raw traces and surface interactions in which the application appears to have fallen short. Martin compares this to a tracing system with an online evaluator that scores customer satisfaction as conversations arrive. In both cases, the first output is a collection of candidates for inspection. The owner reads those conversations, works out what happened, and identifies the cases that warrant an eval. Only then does the request to build the suite have both examples and an account of why they belong.

At Anthropic, Martin says, responsibility for evals generally follows ownership of the product. People in many roles build them. Claude can take on work such as constructing the eval runner, which reduces how much specialized implementation experience the owner needs. The part that remains with the owner is deciding what a satisfactory result looks like and recognizing a miss that matters. For a support product, someone able to explain the consequence of a poor conversation can supply that guidance even if someone else would traditionally have written the runner.

41:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

41:53 · section reference included

Turn repeated preferences between generated UIs into a configuration comparison

Asked for a concrete use of a model as judge, Martin recalls an evaluation of generative UIs. Two configurations, A and B, produce candidate interfaces. The judge sees the pair and selects which is better. Repeating that judgment across many trials supplies evidence about the configurations beyond the fate of a single page. Martin recalls the results being combined into something like an Elo score. He is more certain about the pairwise judging procedure than about the precise aggregation method.

This gives the opening web-page example an operational use. The individual judgment is within reach of a reviewer who can recognize a preference but would struggle to assign an absolute grade. The accumulated judgments address the engineer’s question about which configuration to use. Martin has also seen language models judge retrieval and writing tasks, and classify traces as good or bad. Those applications share a need for judgment, though the form of the decision can differ. Selecting one of two interfaces and deciding whether a single trace merits attention are different instructions to give a grader.

44:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

44:54 · section reference included

A split eval set and a measurable loss can drive the same experiment loop

Martin traces some of the motivation for hill climbing to autoresearch and to models’ improving ability to correct their own work. The enabling condition is feedback they can interpret. A model runs an experiment, measures an objective, inspects the result and adapts its next experiment. Over a long series of attempts, that measurement can guide work a human would otherwise have to steer one change at a time.

There are two formulations in the discussion. The Claude Code command is designed for an eval set with a split, where changes to an application are assessed against a collection of cases. In Martin’s account of autoresearch, the system builds or modifies an artifact, such as a network, then measures a quantity such as loss. He does not recall the exact problem in Karpathy’s example and uses loss to illustrate the kind of feedback available. In one formulation the optimizer can inspect cases it missed. In the other it can inspect the result of an experiment on the artifact. Each has an observable outcome on which to base another attempt.

An audience member asks whether to start with hill climbing and extend it to autoresearch. Martin treats that as a choice of problem formulation. Someone working in a repository such as nanoGPT could try giving Claude the skill and asking it to adapt the loop to the experiment there. He presents that as something to try. The common procedure is already quite small: run a change, measure the result, use it to decide what to change next. The work of adaptation lies in specifying what Claude can change and what result it should measure.

46:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

46:03 · section reference included

Cost, quality and speed complicate the choice of a winning change

A hill climber can measure more than one objective. Martin’s own example pairs cost with performance. Claude can be told about both and can take both measurements into account as it compares candidate changes. Speed is a third dimension he would consider. A configuration that answers correctly and cheaply may still take longer to return the answer, a distinction a cost-and-quality comparison alone would miss.

Martin has mostly exercised the cost and performance pair. As objectives accumulate, choosing the next change becomes harder because different measurements can favor different candidates. He suggests making changes serially, one thing at a time, as a way to simplify the experiment. He gives no combined score or weighting scheme for the three dimensions. In the effort search, the existing baseline already gives the comparison a concrete interpretation: a cheap configuration that falls below the quality floor has failed one of the owner’s conditions.

48:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

48:32 · section reference included

Audit inherited prompts at migration time and check effort on hard tasks

Returning to prompt auditing, Martin calls it the simplest intervention to try. The prompt-audit skill looks for inherited patterns that the target model can do without. He is confident about removing known unnecessary guidance when migrating to a frontier model in the 5.5 series. He is much less confident about assigning a historical cutoff. His testing suggests that some older prompting habits may have begun to lose their usefulness around Opus 4.5, but he would not use that observation as a precise version boundary.

The skill includes model-specific caveats. Moving an application to Haiku, for example, raises a reasonable question about whether it benefits from more explicit guidance than the model used in the previous experiment. Martin offers that as a possibility, acknowledges his uncertainty, and brings the answer back to the actual migration target. The audit’s job is to identify guidance known to be unnecessary for the model receiving the prompt. A rule inferred from the newest Opus version need not settle every case for another model family.

A follow-up about effort clarifies what the control does. Martin describes it as guidance on how much thinking Claude should devote to the problem. It does not impose a hard token cap. Even at very high effort, a simple question can receive a short answer because Claude still exercises judgment about how much verification the task warrants. He has no eval at hand to quantify that behavior and gives it as his working explanation of the setting.

A hard coding problem is where Martin would pay particular attention to the measurement. At extra-high or maximum effort, Claude may spend substantially more tokens checking a result than it spends at medium or low effort. The final outcome can be much the same. That pattern explains why observing a small cost difference on simple inputs is insufficient evidence for setting the whole application to maximum effort. The expensive difference may emerge precisely on the difficult inputs that invite another round of verification.

49:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

49:32 · section reference included

Prompt optimization can overfit even when the model behaves as intended

Martin agrees that hill climbing resembles training in an important respect. Give a model a reward or objective and the ability to modify a system, and it will search for modifications that improve the objective. Train and test splits are useful in both settings. Here the model is already deployed and the editable components are runtime inputs such as prompts or skills. An application owner can apply the experimental discipline of training to those components in their own domain.

Could Claude know it is being evaluated? Martin acknowledges evaluation awareness as a concern Anthropic has discussed in the context of training. In these application experiments, the more immediate problem he has observed is overfitting. Give Claude a prompt to edit and a distribution on which it receives a score, and it can discover wording that raises the score significantly. The wording may work only for the questions represented in the suite. This result requires no special story about Claude detecting a test. It follows directly from the objective and the examples available to the optimizer.

A train/test split puts a check on that behavior. The optimizer can use training cases to choose edits, while performance on held-out cases provides a separate reading of whether the edits carry beyond those examples. The audience asks for a minimum useful dataset and an appropriate split. Martin has not studied different sizes systematically and gives neither a minimum count nor a fixed ratio. The skill has heuristics, and he suggests working with Claude on a size appropriate to the signal the owner wants to measure.

With very few examples, three problems become especially acute. The sample may omit important production work. It can be easy to tailor a prompt to the handful of examples it contains. And its score can be noisy, so a change in the result is harder to attribute to the edit. Holding out some examples addresses one source of overfitting, but both portions still inherit the choices made when the owner selected the original sample.

52:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

52:15 · section reference included

An hours-long task changes the cost of every iteration

Martin usually chooses an editable component such as a prompt because it is quick to change. An audience member asks about tasks that take a very long time. In that regime, preparing the next candidate can be the shortest part of the experiment. The owner still has to wait for the agent to execute the task before a quality signal comes back. A failed run costs elapsed time as well as compute, and each additional round incurs the delay again.

For experiments running many hours or days, the harness itself must remain reliable across that interval. Martin points to Anthropic’s internal use of Managed Agents. Long-duration hill climbing is feasible, but it brings infrastructure failures and time to feedback into the design of the experiment. Improving the prompt editor alone does little to accelerate a loop whose next decision depends on the outcome of a day’s work.

An agent also usually calls for a multi-turn eval. Martin suggests giving the build-eval command access to real traces, specifying that the eval should be multi-turn, and explaining what the owner wants to examine. Claude can inspect the traces and help select interactions to promote. A grader can then examine the whole trajectory, including the sequence of actions the agent took, or inspect the final outcome. The first answers a question about how the agent worked. The second answers a question about what it ultimately accomplished.

55:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

55:41 · section reference included

Keep the session log queryable after context compaction

Asked which context-engineering technique has helped reliability, Martin points to an idea he associates with recursive language models, or RLMs. Keep the context object independent of the model and its harness. In the Managed Agents design he describes, the harness can use tools to query a session log that lives separately. The messages presently in Claude’s working context are therefore only one view of the history available to the agent.

Compaction illustrates the payoff. Partway through a long task, the agent summarizes its message history to keep working. At that moment it has to decide which details are important enough to carry forward, even though it does not yet know the questions it will encounter later. If that summary is the only surviving record, an omitted detail is gone. With an independent log, the harness can return to the earlier interaction and retrieve the detail when it becomes relevant. Recoverability comes from preserving the record and giving the harness an interface for interrogating it.

Reliable history still leaves the problem of judging the work recorded there. Agents produce nondeterministic outcomes, and many useful tasks involve open-ended writing or judgment with no easily verifiable answer. Martin returns to clear grading rules, low variance and checks that multiple models can reproduce a judgment. An optimizer can work for hours on a carefully preserved trace and still get nowhere if the question is impossible or the grader treats several reasonable interpretations inconsistently. Time spent on verifier design determines what that long run can teach the owner.

57:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

57:23 · section reference included

Refresh saturated evals with production cases worth keeping

As models improve, Martin expects scores on a well-structured existing eval generally to rise. If the trend is absent, he would re-examine what the suite measures. At the other extreme, capable models can saturate a suite quickly. Designing harder cases that genuinely challenge models such as Fable or Opus 5.5 is recurring work. An old suite can still check behavior the team already achieved, even as it loses the ability to distinguish further gains.

Martin closes with a workflow he has seen among tracing providers. Deployed agents produce traces. An online evaluator scores them, and poorly scoring interactions become candidates for an offline eval set. The team hill climbs on selected production failures. Once the application handles them successfully, the cases become regression tests. Meanwhile, the deployed agent continues producing fresh traces, so the owner can look for the next consequential failure and see whether the rate of failures observed in production is falling.

LangChain, Arize and Braintrust are among the providers Martin associates with tooling for that outer loop. He has not followed their latest product changes closely. The difficult part he leaves with the product owner is upstream of any platform choice. What does the online evaluator recognize as good work in production? Which flagged interactions deserve promotion? Those decisions determine whether the next round of optimization improves something users actually experience.

1:01:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:01:00 · section reference included

Transcript download
Full transcript

Loading transcript…