How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Laurie Voss· Head of Developer Relations, Arize AI1:56:13

Read the talk

How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude

Voss introduces the full trace, evaluate, test, improve and CI loop.Open full source frame
Voss introduces the full trace, evaluate, test, improve and CI loop.

Laurie Voss traces a financial analyst agent, finds failures in its reports, tests its judges, and measures what prompt and model changes actually improve.

From a talk by Laurie Voss

At a glance

Ideas worth remembering

  • Read traces before choosing evals: the Tesla consensus combined percentages from different search results into an impossible total.

  • Give judges the evidence needed for their question. Faithfulness to supplied research is different from independent factual correctness.

  • Compare judge labels with carefully produced human labels, and inspect disagreements before tightening the rubric.

  • The prompt experiment improved actionability while faithfulness remained poor; the model comparison improved consistency partly by repeatedly recommending hold.

  • Reuse established evals in CI and sample real traffic for sustained quality changes after release.

A good-looking report can still be wrong

An agent can produce a polished financial report and still answer the wrong question. In Laurie Voss’s example, a search for Tesla finds the inventor instead of the car company; a well-formatted report built on that result can pass checks for grammar and tone while failing its purpose. An early agent error can propagate through every later step.

Voss, Head of Developer Relations at Arize AI, uses a deliberately simple Claude financial analyst to work through the problem. The agent researches a ticker and focus area, then writes a report. The workshop asks how to detect failures, compare changes, and prevent regressions without reading every output by hand.

A trace is a detailed execution record; an eval is a test applied to that record. A trace contains nested spans for steps such as agent turns, model calls and tool calls. Their inputs, outputs, timing and token counts help explain how an answer came to be. Evals then assess properties of those steps or their results.

The three evaluation methods have different strengths:

  • Code checks test deterministic properties such as valid JSON, length or required fields quickly and cheaply.
  • LLM judges apply a rubric—a set of rules—to questions of meaning, such as whether an answer follows its sources. They cost time and money and can themselves be wrong.
  • Human review finds unexpected failures and checks judges, but expert attention is scarce and repetitive labeling causes fatigue.

A test can also be too rigid. Voss describes a flight-change benchmark in which an agent achieved the requested seat change by temporarily upgrading a passenger and then downgrading them. The benchmark expected failure under the economy-class rules. The example motivates checking the required outcome without prescribing incidental steps, while still testing paths where the steps themselves matter.

11:2411:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:47 · section reference included

Trace the research and writing turns

The workshop uses TypeScript, Node 22 or later, an Anthropic key and a local Phoenix instance at localhost:6006. The supplied project separates instrumentation, agent code, prompts, evals and test queries; its checkpoints let participants resume at a later exercise. Phoenix collects traces, stores eval annotations and supports datasets and experiments.

The agent’s first turn decides what to search for and whether another search is needed. The resulting session ID is passed back as resume for the writing turn, so the writer can use the research conversation. A parent span wraps both turns into one trace, making the whole request easier to inspect.

Registration points the instrumentation at Phoenix using OpenTelemetry and OpenInference. One TypeScript detail is easy to miss: the Claude Agent SDK is an ES module, whose exports are read-only. manuallyInstrument returns an instrumented copy; the project imports the SDK through its instrumentation file. Using the original copy would leave the agent running but produce no trace.

The first live run exposes an ambiguity. The prompt asks the agent to write a report without saying where to put it. The agent tries to write a file, cannot obtain permission in its unattended environment, and asks whether it should display the report instead. How does the trace connect the searches to that failure? The nested tool calls show the research queries and their results, followed by the attempted file write; the problem is the destination instruction, not necessarily the research.

One request, two agent turns

Ticker and focusThe user requests a financial report.
Research turnparent trace — The agent chooses searches and reads their results.
Writing turnparent trace — The resumed session gives the writer access to the research.
ReportThe final result is available for evaluation.
The session carries research context into the writing turn; the parent span groups both turns for inspection.
Read the diagram as text
  • Ticker and focus. The user requests a financial report.
  • Research turn. parent trace — The agent chooses searches and reads their results.
  • Writing turn. parent trace — The resumed session gives the writer access to the research.
  • Report. The final result is available for evaluation.
  • Ticker and focus → Research turn: initiates.
  • Research turn → Writing turn: session resumed with context.
  • Writing turn → Report: produces.
0:421:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:31 · section reference included

Read the traces before choosing metrics

One run is not enough. The batch runs 15 queries, five at a time; a seed project supplies 16 earlier runs if the batch fails. The cases include an Apple–Microsoft comparison, Coca-Cola, the invented ticker ZZZQ, and the casual question “Yo, is Tesla a buy right now?” The fake ticker tests whether the agent admits it cannot identify a company instead of inventing one.

Define success with the people who understand the product. For this exercise, Voss wants current data, the right company, claims supported by the research and an explicit buy, hold or sell recommendation. The ZZZQ run identifies the ticker as invalid. A Coca-Cola report calls the company a good investment but stops short of the explicit decision Voss wants.

The Tesla report has a more subtle failure. It reports 50% buy, 52.6% hold and 13.1% sell—115.7%, rounded in the workshop to 116%. Which values produced that impossible consensus? The trace shows that the agent combined figures from different search results and treated them as one dataset. Voss identifies this as a reasoning error: the individual search results were usable, but the synthesis was not.

Read a dozen or more traces end to end before automating the assessment. First write plain-language observations, then group those notes into failure categories: choosing categories in advance can hide failures nobody anticipated. A coding agent can help classify and annotate larger collections, but human sampling remains useful. Voss’s analysis found permission requests, hallucinations and reasoning errors; priority depends on severity as well as frequency.

Synthetic queries can seed a test set before real usage exists, especially if they vary phrasing and difficulty. Voss cautions that generated queries can be conservative compared with real users. Once traffic exists, add representative real requests and surprising failures to the dataset.

31:2631:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

31:26 · section reference included

Start with a deterministic ticker check

The first code eval asks whether the report mentions each requested ticker. The script retrieves root spans by selecting spans with no parent, extracts each run’s structured input and output, and checks the requested tickers against the report. It returns a score and label, plus the missing ticker when it fails, then attaches the result to the span as a Phoenix annotation.

The demonstrated run reports 15 passes and two failures. One Amazon query focused on AWS profitability; the agent wrote about AWS and omitted the expected Amazon reference. A mention check cannot establish that a report is correct, but it can cheaply expose a missing requirement. Structured input makes the check easier because the ticker and focus are already separate fields.

Code evals can also parse a response, check database state or compare a claim with a reference API. When the reference changes over time, the comparison needs appropriate reference conditions. Avoid turning every implementation choice into a required sequence: a good outcome can be reached through different searches or tools.

50:2650:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

50:26 · section reference included

Give the judge evidence it can use

An LLM judge combines a model, a rubric and the data to be graded. Phoenix supplies built-in evaluators for questions such as correctness and faithfulness. Voss uses a more capable model to judge the agent and suggests testing a judge from another provider to reduce same-model preference. Judge tracing is suppressed so its own calls do not clutter the application traces.

The correctness evaluator gives all 16 reports zero. Why? Voss attributes the result to a judge relying on older model knowledge while grading newer financial data. Seven explanations complain that fresh information cannot be verified; others identify potentially real inconsistencies. The uniform score cannot distinguish those cases.

Faithfulness asks a narrower question: does the report follow the material the agent found? Its inputs are the user request, the report, and research context. An early version supplied only the research notes, causing the judge to flag figures that appeared in the raw searches but not the notes. The revised helper supplies both. This checks support in the retrieved material; it does not independently establish that the material itself is true.

The faithfulness judge marks 14 of 16 reports unfaithful. Its Microsoft explanation identifies claims absent from the supplied context, while an NVIDIA explanation points to conflated data points. Those explanations are useful leads for inspection, even though the judge’s classifications still need checking.

56:3057:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

56:30 · section reference included

A tool judge inherits ambiguity from the prompt

Some failures require examining the trajectory. Tool-selection evaluation considers the request, the available tools and the tools chosen; invocation evaluation looks at arguments, while response-handling evaluation asks what the agent did with a result. The trace records calls, but the workshop supplies the list of available tools separately.

Thirteen of 16 writing turns tried to save a file. How did the judge interpret that choice? One explanation treats writing to a file as plausible and instead objects to a redundant second call; only one explanation notes that the user never requested a file. The prompt merely said to write a concise report, so the judge and the agent had the same ambiguity. A passing tool judgment cannot supply an unstated destination requirement.

1:05:501:06:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:05:50 · section reference included

Write a judge for the product’s definition of useful

No general built-in evaluator knows this product’s definition of an actionable stock report. Voss’s custom judge looks for a specific recommendation, concrete risks and forward-looking analysis. These criteria come from observed failures, such as reports that summarize information without interpreting it.

The rubric gives the judge a role and context, states observable criteria, separates the request and report from instructions, optionally supplies labeled examples, and constrains the result to actionable or not actionable. The classification evaluator maps those labels to scores of one and zero. XML tags distinguish examples and data; in Voss’s trial, adding tags around previously unmarked examples changed the classifications because the judge could distinguish the examples from the report.

Examples can help or overconstrain a judge. With examples, Voss’s judge agreed with his labels on 15 of 16 reports in each of three runs; without them it agreed on 16, 16 and 15. The disputed Microsoft report gave a buy recommendation but no risks. Voss accepted it; an example requiring risks led the judge to reject it. The disagreement exposes an unresolved choice about the standard as well as a possible inconsistency in the human labels.

In the original set, the actionability judge labels five of 16 reports actionable. What is missing from the others? Its explanations cite a JPMorgan report without a buy, hold or sell recommendation and similar vague conclusions for Rivian, NVIDIA and Meta. The split supplies examples to compare when changing the prompt.

Version and test the rubric like code. Keep separate evaluators for separate dimensions: a single “God Evaluator” that scores accuracy, tone, formatting and policy makes failures difficult to diagnose. Also distinguish release-blocking guardrails, such as invented prices, from improvement targets that need not block a build.

1:10:331:11:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:10:33 · section reference included

Test the judge against human labels

The judge is a classifier: for each report, it predicts a label. Voss annotates 16 reports with human actionability labels and builds a golden dataset. Each example contains a report and its expected label, linked back to its source span. Sorting by span ID and placing every fourth example in the test split leaves 12 for development and four held out.

A meta-evaluation runs the judge as the experiment task and a small code check compares its label with the human label. The demonstrated judge agrees on all 12 development examples. That is a useful early signal, not evidence of production reliability from such a small set. Human labels also need the same criteria: Voss later recognizes that he had accepted the Microsoft report despite his own risk criterion.

Making the rubric stricter can change what the judge measures. Voss adds a sentence requiring all four criteria. Agreement on development falls from 12 of 12 to 11 of 12 because of Microsoft; an Apple example in the test split also changes because the judge considers its risk discussion insufficient. Specificity helps only when it expresses the desired standard.

For failure detection, precision asks how often a report flagged as not actionable really fails; recall asks how many of the genuinely failing reports the judge catches. Voss favors recall when an unnecessary review is cheaper than a bad report reaching a user. The right balance depends on the cost of each error.

Judges can favor an option because of its position, length, confident wording or similarity to their own model’s output. Swap option order where relevant and inspect disagreements. Humans disagree too, so a golden label is a considered judgment, not an automatic guarantee. Add difficult production cases over time, and include cases where a behavior should happen and where it should not—for example, both necessary and unnecessary web searches.

1:24:471:25:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:24:47 · section reference included

Change the prompt and measure what moved

An experiment compares versions on the same inputs with the same evaluators. Phoenix can collect failing traces into a dataset for focused iteration. The experiment runs the agent on each example and gives each evaluator the fields it needs, including research context for faithfulness.

Each prompt change addresses a finding: request an explicit recommendation, support risks with data, use only numbers found in the research, and return the report in the response instead of saving a file. This gives the experiment concrete questions to answer rather than treating a longer prompt as an improvement by itself.

The results separate the gains from the remaining problem. Actionability rises from five of 16 original reports to 48 of 48 outputs across three improved runs; the five previously passing examples still pass. Faithfulness goes from two of 16 to only three of 48, despite the instruction to use researched numbers. The chart’s comparison matters because making recommendations explicit improved the measured behavior while unsupported claims remained.

Voss compares actionability and faithfulness across prompt experiments.Open full source frame
Voss compares actionability and faithfulness across prompt experiments.

Repeat runs because the agent is nondeterministic. Voss distinguishes pass@K, success at least once in K attempts, from pass^K, success on every attempt. A coding assistant with retries may benefit from the first measure; a financial or support agent may need the second because a single bad answer matters. Ten to 20 examples can indicate a direction in development, while a production decision needs a larger, representative set.

1:38:551:39:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:38:55 · section reference included

A more consistent model can be consistently conservative

Voss recommends improving data quality first, then prompts, then model choice, with parameter tuning later. A model change can affect many behaviors at once and may cost more, so the same experiment should measure its quality and economics.

The model comparison discusses faithfulness, consistency and cost.Open full source frame
The model comparison discusses faithfulness, consistency and cost.

In the workshop’s comparison with a newer Claude Haiku model, faithfulness improves by more than threefold but remains poor, while cost rises fourfold. Recommendation consistency also improves. Inspecting the outputs reveals a catch: 41 of 45 recommendations are “hold.” A consistency metric rewards repeating the same call, but does not establish that the call is useful or well founded.

A capability eval measures a behavior that still needs work. Once a behavior passes reliably, keep it as a regression eval so later changes cannot quietly remove it. Voss also suggests writing an eval before a feature—for example, requiring a support agent to verify identity before issuing a refund—so the team has a measurable definition of done.

1:44:331:45:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:44:33 · section reference included

Put the checks in CI, then monitor real traffic

The workshop’s agent.eval.ts uses Vitest to run a regression set that includes Coca-Cola, an Apple–Microsoft comparison, Rivian and ZZZQ. Each test runs the agent and reuses the ticker and actionability evaluators. Mentioning the ticker is a guardrail; actionability is gated on an average of at least 0.8, allowing for noise in individual LLM judgments.

In the illustrated failing run, all four reports mention their ticker, but the Rivian report is not actionable. Why does the build miss the target? Three of four actionability passes yield 0.75, below 0.8. The scorecard and judge explanations provide more direction than a red build alone. The same suite can run in GitHub Actions with Phoenix as a service container.

The failing scorecard is discussed: all ticker checks pass, but actionability averages 0.75.Open full source frame
The failing scorecard is discussed: all ticker checks pass, but actionability averages 0.75.

Running the whole agent on every commit can be slow and expensive, so choose the regression scope deliberately. After launch, Voss recommends evaluating a sample of real traffic and alerting on a sustained drop rather than each isolated failure. Product, support and sales colleagues can help define good answers even if they do not write the TypeScript checks.

The first practical step can be smaller than the complete suite: instrument an agent and read a dozen traces. Those traces expose decisions and tool failures that a polished final answer can conceal.

1:51:251:51:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:51:25 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…