Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Read the talk

Evals for High-Stakes AI Agents

Opening refund example illustrates why green system metrics can conceal an agent failure.Open full source frame
Opening refund example illustrates why green system metrics can conceal an agent failure.

Annabell Schäfer and Doneyli De Jesus use a banking assistant to show how traces, expert review, evaluators, and deployment gates can reveal failures that ordinary system metrics miss.

From a talk by Annabell Schäfer and Doneyli De Jesus

At a glance

Ideas worth remembering

  • Trace tool calls and intermediate steps: an agent can report a successful action that it never performed.

  • Use domain experts’ examples and comments to define concrete failure modes, then choose deterministic or model-based evaluators for each.

  • Run experiments against stable datasets and inspect how a CI gate relates to the actual release process.

  • Calibrate model judges against human labels and revisit the sample as production behavior changes.

A green status can hide a missing refund

A support agent tells a customer that it has issued a refund. The customer thanks it; latency and system status are green. Weeks later, the customer returns because the refund never arrived. The trace would reveal the problem: the agent never called the refund tool. Annabell Schäfer opens with this example to show why a successful request is not necessarily a successful task.

The IT-support trace illustrates the root trace and individual agent steps.Open full source frame
The IT-support trace illustrates the root trace and individual agent steps.

A trace records the steps behind an answer: model generations, tool calls, their inputs and outputs, and associated metadata. In Schäfer’s IT-support example, a root trace gives aggregate time and cost while individual steps show what the agent actually did. Monitoring then helps find interesting traces at scale, using signals such as latency, cost, compliance checks, or potentially frustrated users.

Those observations can become a dataset: representative inputs paired with a sample answer, expected behavior, or required content. Experiments run variations against that stable set; evaluators score the results. When comparing prompts, changing only the prompt makes the result easier to interpret than changing the model, tools, and context at the same time. Production observations can then supply cases for the next round.

The starting point depends on risk. A low-stakes internal assistant might begin with tracing and frustration monitoring, adding offline cases over time. For a customer-facing agent in a regulated setting, Schäfer describes three ways to establish ground truth before relying on automation:

  • Expert curation: Domain specialists define and validate representative cases.
  • Shadow runs: Humans perform the real process while the agent handles the same cases in parallel, allowing its proposed handling to be compared with the manual work.
  • Human-in-the-loop operation: People review outcomes, and the team gradually separates cases it is confident automating from those that should retain human review.
1:542:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Trace the banking assistant without storing card numbers

Doneyli De Jesus demonstrates a fictional banking assistant whose test interface can impersonate different customer personas. That matters because guardrails may need to hold across customers with different circumstances. A question about international wire fees retrieves an answer from a knowledge base; the demonstration also exposes the tools, data sources, and configuration behind the response.

What happens when a customer types a card number into the chat? In the demonstration, a masking function identifies defined categories of personally identifiable information and redacts them before the trace is persisted in Langfuse. The trace shows the redacted question and a tree of model and tool steps, including account listing. This lets the team inspect behavior while keeping the demonstrated card number out of the observability store.

The presenter opens the trace and points out the redacted card number.Open full source frame
The presenter opens the trace and points out the redacted card number.

Scores add another way to find relevant interactions. When a customer asks whether to put savings in Bitcoin, the demo assistant declines to give investment advice and points toward an advisor. The input guardrail identifies investment advice as the primary risk, and a categorical score makes those traces filterable. A separate contact-center example transcribes a call about blocking a card, evaluates the text for violations, and links the trace to the audio, transcript, and assistant response.

11:5613:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:40 · section reference included

Build test cases around behavior

Customers will find uses the designers did not anticipate. A dataset gives the team a repeatable set of cases against which to measure changes, but its expected output need not be a word-for-word answer. It can specify intent or behavior: for example, refuse an out-of-scope request, cite a source, or take an appropriate action.

The dataset view introduces questions and expected outputs as repeatable test cases.Open full source frame
The dataset view introduces questions and expected outputs as repeatable test cases.

Different datasets can test different risks. De Jesus shows one for red-teaming prompt injection and attempts to bypass guardrails, and another focused on unresolved disputes. An existing spreadsheet can be imported as CSV. Teams can also add production traces to a dataset after deciding that those examples represent the behavior they want to preserve.

Reading traces remains useful even with automated scoring. It shows whether the agent took the required steps, called the right tools, and produced the intended output. The problem is volume: a person cannot inspect every one of thousands of production traces. Sampling and filtering make expert attention more practical.

20:2320:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:23 · section reference included

Turn expert comments into failure modes

De Jesus suggests selecting a representative sample of traces, potentially filtered to a topic such as disputes or card information, and putting it in an annotation queue for a domain expert. A small set of scores—yes or no, for example—plus a comment makes review fit around the expert’s existing work. The annotation remains attached to the trace.

How does one comment become something the team can measure? In the example, an expert marks a missing citation and comments that the citation needs to be included. Across many reviews, an LLM can group raw comments into recurring failure categories; the team can then define a score for whether an answer cites the expert’s expected source. Other demonstrated dimensions include correctness, formal register, and matching the customer’s language.

The experiment view presents citation and other quality dimensions derived from expert input.Open full source frame
The experiment view presents citation and other quality dimensions derived from expert input.

This division of work lets the expert describe what is wrong without first inventing a taxonomy. The extracted categories still need to become specific evaluation criteria: a useful score names the behavior the application should exhibit, rather than merely assigning a general impression of quality.

25:5426:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:54 · section reference included

Mix model judgment with deterministic checks

Once the quality dimensions are defined, choose how to grade each one. An LLM-as-a-judge can assess behavior that requires interpretation; a code evaluator can perform a deterministic check such as an exact match or a regular-expression check. The latter returns the same result for the same input, although that consistency only establishes the condition the code actually checks.

The evaluator setup offers model-based and code-based evaluation.Open full source frame
The evaluator setup offers model-based and code-based evaluation.

De Jesus recommends a mix because model judges can themselves make mistakes. Scores can be calculated on incoming traces or in experiments; a client application can also calculate scores during an online interaction. An experiment anchors the comparison on a dataset, then runs an agent configuration—prompt, model, tools, or context—against its cases before the team decides whether to use the change.

30:2430:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:24 · section reference included

Run the dataset when a prompt changes

The demonstrated automation watches for changes to versioned prompts and launches a GitHub CI job. The job runs an experiment on the dataset, applies the evaluators, and reports pass or fail against defined minimums. Prompt versions can also be managed in a separate version-control system; the important role of the change is to trigger the evaluation.

In the live sequence, De Jesus selects version four after seeing regressions, applies the production tag, and saves it. That prompt change triggers the CI run. The tag is applied before the demonstrated evaluation completes, so the recording does not establish that this particular setup prevents the tagged prompt from being used while the job runs.

What does a gate result actually contain? An earlier completed run shows the prompt version and labels, a passing result, and evaluations for required content, source recall, language match, correctness, and no unsolicited upsell. Each score can have a minimum threshold. De Jesus leaves the result sitting; creating a pull request for review or automatically merging a passing change is described as a possible extension, not as the action shown.

The earlier CI result shows a passing gate and its evaluators and minimums.Open full source frame
The earlier CI result shows a passing gate and its evaluators and minimums.

The demonstrated prompt-change evaluation

Save prompt versionVersion four receives the production tag.
Launch CI jobThe prompt change triggers the configured GitHub automation.
Run experimentThe agent configuration is evaluated against dataset cases.
Report gate resultEvaluator scores are compared with minimum thresholds.
The production tag is saved before CI evaluates the resulting configuration.
Read the diagram as text
  • Save prompt version. Version four receives the production tag.
  • Launch CI job. The prompt change triggers the configured GitHub automation.
  • Run experiment. The agent configuration is evaluated against dataset cases.
  • Report gate result. Evaluator scores are compared with minimum thresholds.
  • Save prompt version → Launch CI job: Triggers.
  • Launch CI job → Run experiment: Runs.
  • Run experiment → Report gate result: Produces scores.
33:5334:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

33:53 · section reference included

Use dashboards to decide where to investigate

A stakeholder dashboard can connect agent activity to operational questions. In the fictional demo, De Jesus reports 1,400 interactions over seven days that did not go to a human agent, at a reported cost of less than $7. He presents the comparison with contact-center labor as directional business evidence; the figures alone do not establish the full cost or quality of those interactions.

Which failures deserve attention first? The dashboard ranks out-of-scope responses, informal register, and unresolved disputes as the three leading failure modes over the previous seven days. De Jesus then filters traces for dispute not resolved = 0: those cases include disputes handed to a human or left unresolved when the session ended. The ranking identifies where to look; the filtered traces supply the details needed to investigate.

The trace filter narrows investigation to unresolved disputes.Open full source frame
The trace filter narrows investigation to unresolved disputes.

Other views capture thumbs-up and thumbs-down feedback and security-risk categories. In the example, 18% of the displayed feedback is thumbs down, and 89 interactions over seven days are flagged for investment-advice risk. A flag selects cases for inspection; it does not by itself establish that the assistant actually gave prohibited advice.

39:2339:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

39:23 · section reference included

Give coding agents useful feedback and human direction

Schäfer cautions against asking a coding agent to “fix my agent and make no mistakes,” even when it can access traces through a CLI or MCP. Her proposed balance keeps people and domain experts responsible for defining the problem, curating datasets, and deciding what good behavior means. Coding agents can help instrument the application, identify candidate failure modes, run evaluations, and try more experiments.

The presenter describes the balance between human direction and coding-agent automation.Open full source frame
The presenter describes the balance between human direction and coding-agent automation.

A practical starting point is to collect ground truth from historical data, production traces, or cases reviewed with experts, then encode hard limits in the evaluation set. For the banking example, one such limit is that the assistant should not give investment advice. Schäfer also points to the Langfuse agent skill for guidance on instrumentation, datasets, and evaluators.

46:0046:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

45:52 · section reference included

Calibrate the judge and choose where data lives

In the Q&A, Schäfer addresses consistency between an LLM judge and human reviewers. Have people label a representative sample, run the judge on the same cases, and tune it toward agreement with those labels. Multiple raters can help establish the human reference; a practical starting point may be one reviewer. Revisit the sample over time as production usage changes.

For production monitoring, Schäfer suggests initially tolerating extra false positives so the team catches more potential failures, then tuning the judge down over time. This is a sensitivity tradeoff, not a guarantee that every failure will be caught.

On deployment, De Jesus says Langfuse uses ClickHouse for its data store and can run as a managed cloud service or self-hosted on Amazon infrastructure. Data can also be exported to S3 or consumed through the API, CLI, or MCP. These choices address where the observability system runs and how teams retrieve its data.

52:1852:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

51:14 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…