Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Read the talk
Evals for High-Stakes AI Agents
Annabell Schäfer and Doneyli De Jesus use a banking assistant to show how traces, expert review, evaluators, and deployment gates can reveal failures that ordinary system metrics miss.
From a talk by Annabell Schäfer and Doneyli De Jesus
At a glance
Ideas worth remembering
Trace tool calls and intermediate steps: an agent can report a successful action that it never performed.
Use domain experts’ examples and comments to define concrete failure modes, then choose deterministic or model-based evaluators for each.
Run experiments against stable datasets and inspect how a CI gate relates to the actual release process.
Calibrate model judges against human labels and revisit the sample as production behavior changes.
A green status can hide a missing refund
A support agent tells a customer that it has issued a refund. The customer thanks it; latency and system status are green. Weeks later, the customer returns because the refund never arrived. The trace would reveal the problem: the agent never called the refund tool. Annabell Schäfer opens with this example to show why a successful request is not necessarily a successful task.
A trace records the steps behind an answer: model generations, tool calls, their inputs and outputs, and associated metadata. In Schäfer’s IT-support example, a root trace gives aggregate time and cost while individual steps show what the agent actually did. Monitoring then helps find interesting traces at scale, using signals such as latency, cost, compliance checks, or potentially frustrated users.
Those observations can become a dataset: representative inputs paired with a sample answer, expected behavior, or required content. Experiments run variations against that stable set; evaluators score the results. When comparing prompts, changing only the prompt makes the result easier to interpret than changing the model, tools, and context at the same time. Production observations can then supply cases for the next round.
The starting point depends on risk. A low-stakes internal assistant might begin with tracing and frustration monitoring, adding offline cases over time. For a customer-facing agent in a regulated setting, Schäfer describes three ways to establish ground truth before relying on automation:
- Expert curation: Domain specialists define and validate representative cases.
- Shadow runs: Humans perform the real process while the agent handles the same cases in parallel, allowing its proposed handling to be compared with the manual work.
- Human-in-the-loop operation: People review outcomes, and the team gradually separates cases it is confident automating from those that should retain human review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trace the banking assistant without storing card numbers
Doneyli De Jesus demonstrates a fictional banking assistant whose test interface can impersonate different customer personas. That matters because guardrails may need to hold across customers with different circumstances. A question about international wire fees retrieves an answer from a knowledge base; the demonstration also exposes the tools, data sources, and configuration behind the response.
What happens when a customer types a card number into the chat? In the demonstration, a masking function identifies defined categories of personally identifiable information and redacts them before the trace is persisted in Langfuse. The trace shows the redacted question and a tree of model and tool steps, including account listing. This lets the team inspect behavior while keeping the demonstrated card number out of the observability store.
Scores add another way to find relevant interactions. When a customer asks whether to put savings in Bitcoin, the demo assistant declines to give investment advice and points toward an advisor. The input guardrail identifies investment advice as the primary risk, and a categorical score makes those traces filterable. A separate contact-center example transcribes a call about blocking a card, evaluates the text for violations, and links the trace to the audio, transcript, and assistant response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build test cases around behavior
Customers will find uses the designers did not anticipate. A dataset gives the team a repeatable set of cases against which to measure changes, but its expected output need not be a word-for-word answer. It can specify intent or behavior: for example, refuse an out-of-scope request, cite a source, or take an appropriate action.
Different datasets can test different risks. De Jesus shows one for red-teaming prompt injection and attempts to bypass guardrails, and another focused on unresolved disputes. An existing spreadsheet can be imported as CSV. Teams can also add production traces to a dataset after deciding that those examples represent the behavior they want to preserve.
Reading traces remains useful even with automated scoring. It shows whether the agent took the required steps, called the right tools, and produced the intended output. The problem is volume: a person cannot inspect every one of thousands of production traces. Sampling and filtering make expert attention more practical.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn expert comments into failure modes
De Jesus suggests selecting a representative sample of traces, potentially filtered to a topic such as disputes or card information, and putting it in an annotation queue for a domain expert. A small set of scores—yes or no, for example—plus a comment makes review fit around the expert’s existing work. The annotation remains attached to the trace.
How does one comment become something the team can measure? In the example, an expert marks a missing citation and comments that the citation needs to be included. Across many reviews, an LLM can group raw comments into recurring failure categories; the team can then define a score for whether an answer cites the expert’s expected source. Other demonstrated dimensions include correctness, formal register, and matching the customer’s language.
This division of work lets the expert describe what is wrong without first inventing a taxonomy. The extracted categories still need to become specific evaluation criteria: a useful score names the behavior the application should exhibit, rather than merely assigning a general impression of quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mix model judgment with deterministic checks
Once the quality dimensions are defined, choose how to grade each one. An LLM-as-a-judge can assess behavior that requires interpretation; a code evaluator can perform a deterministic check such as an exact match or a regular-expression check. The latter returns the same result for the same input, although that consistency only establishes the condition the code actually checks.
De Jesus recommends a mix because model judges can themselves make mistakes. Scores can be calculated on incoming traces or in experiments; a client application can also calculate scores during an online interaction. An experiment anchors the comparison on a dataset, then runs an agent configuration—prompt, model, tools, or context—against its cases before the team decides whether to use the change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run the dataset when a prompt changes
The demonstrated automation watches for changes to versioned prompts and launches a GitHub CI job. The job runs an experiment on the dataset, applies the evaluators, and reports pass or fail against defined minimums. Prompt versions can also be managed in a separate version-control system; the important role of the change is to trigger the evaluation.
In the live sequence, De Jesus selects version four after seeing regressions, applies the production tag, and saves it. That prompt change triggers the CI run. The tag is applied before the demonstrated evaluation completes, so the recording does not establish that this particular setup prevents the tagged prompt from being used while the job runs.
What does a gate result actually contain? An earlier completed run shows the prompt version and labels, a passing result, and evaluations for required content, source recall, language match, correctness, and no unsolicited upsell. Each score can have a minimum threshold. De Jesus leaves the result sitting; creating a pull request for review or automatically merging a passing change is described as a possible extension, not as the action shown.
The demonstrated prompt-change evaluation
Read the diagram as text
- Save prompt version. Version four receives the production tag.
- Launch CI job. The prompt change triggers the configured GitHub automation.
- Run experiment. The agent configuration is evaluated against dataset cases.
- Report gate result. Evaluator scores are compared with minimum thresholds.
- Save prompt version → Launch CI job: Triggers.
- Launch CI job → Run experiment: Runs.
- Run experiment → Report gate result: Produces scores.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use dashboards to decide where to investigate
A stakeholder dashboard can connect agent activity to operational questions. In the fictional demo, De Jesus reports 1,400 interactions over seven days that did not go to a human agent, at a reported cost of less than $7. He presents the comparison with contact-center labor as directional business evidence; the figures alone do not establish the full cost or quality of those interactions.
Which failures deserve attention first? The dashboard ranks out-of-scope responses, informal register, and unresolved disputes as the three leading failure modes over the previous seven days. De Jesus then filters traces for dispute not resolved = 0: those cases include disputes handed to a human or left unresolved when the session ended. The ranking identifies where to look; the filtered traces supply the details needed to investigate.
Other views capture thumbs-up and thumbs-down feedback and security-risk categories. In the example, 18% of the displayed feedback is thumbs down, and 89 interactions over seven days are flagged for investment-advice risk. A flag selects cases for inspection; it does not by itself establish that the assistant actually gave prohibited advice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give coding agents useful feedback and human direction
Schäfer cautions against asking a coding agent to “fix my agent and make no mistakes,” even when it can access traces through a CLI or MCP. Her proposed balance keeps people and domain experts responsible for defining the problem, curating datasets, and deciding what good behavior means. Coding agents can help instrument the application, identify candidate failure modes, run evaluations, and try more experiments.
A practical starting point is to collect ground truth from historical data, production traces, or cases reviewed with experts, then encode hard limits in the evaluation set. For the banking example, one such limit is that the assistant should not give investment advice. Schäfer also points to the Langfuse agent skill for guidance on instrumentation, datasets, and evaluators.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Calibrate the judge and choose where data lives
In the Q&A, Schäfer addresses consistency between an LLM judge and human reviewers. Have people label a representative sample, run the judge on the same cases, and tune it toward agreement with those labels. Multiple raters can help establish the human reference; a practical starting point may be one reviewer. Revisit the sample over time as production usage changes.
For production monitoring, Schäfer suggests initially tolerating extra false positives so the team catches more potential failures, then tuning the judge down over time. This is a sensitivity tradeoff, not a guarantee that every failure will be caught.
On deployment, De Jesus says Langfuse uses ClickHouse for its data store and can run as a managed cloud service or self-hosted on Amazon infrastructure. Data can also be exported to S3 or consumed through the API, CLI, or MCP. These choices address where the observability system runs and how teams retrieve its data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Guidance for using a coding agent with Langfuse traces, datasets, and evaluations.
A self-serve walkthrough of the AI engineering workflow.
Further reading
Create datasets with inputs and expected outputs for experiments.
Organize trace review and feedback from domain experts.
Related talks
- Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust
Another end-to-end evaluation, focused on comparing retrieval approaches with task-based scores and trace inspection.
Full transcript
Loading transcript…