AI Engineer World's Fair 2025
Taming Rogue AI Agents with Observability-Driven Evaluation
Loading slides…
Read the talk
Taming Rogue AI Agents with Observability-Driven Evaluation
A banking chatbot can retrieve the right balance and still deliver a poor interaction. Evaluating each step reveals where progress stops, why tools go unused, and which changes need human review.
From a talk by Jim Bennett
Before you start: Familiarity with LLM tool calls, retrieval-augmented generation, and basic software testing will help you follow the examples.
A plausible book that does not exist
A summer reading list recommends Atonement by Ian McEwan, then offers another appealing title: The Last Algorithm by Andy Weir. The second book does not exist. Jim Bennett opens with this Chicago Sun-Times example because the recommendation had already crossed the boundary from generated text into something readers were expected to trust. An outside contributor had used AI to produce the list, and the newspaper subsequently published an apology. The publisher’s response clarifies that the material appeared in a syndicated supplement licensed from King Features, without Sun-Times newsroom review.
The same problem appears in settings with different consequences. Bennett cites Butler Snow attorneys using fabricated case law in an Alabama prison case, then Air Canada’s chatbot giving incorrect refund advice. These examples connect unreliable generation to organizational accountability: someone publishes, submits, or acts on the answer. Detecting the problem before that happens is the engineering challenge.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What does it mean for an agent to work?
An arithmetic unit test has a straightforward contract: adding two and two should return four. A conversational agent rarely has one exact acceptable output. Keyword checks might help with a simple answer, but a complex application can call an LLM, retrieve data, make a decision, invoke another agent, and then call a tool. Each decision changes what happens next. Checking only the final wording leaves much of that behavior unexamined.
Success must be defined before it can be measured. In a conversation, reaching the requested outcome and providing a good interaction are different requirements. Bennett introduces an LLM judge as a way to evaluate these less rigid criteria: give another AI the evidence and ask it to assess whether the application worked. He describes AI evaluators as roughly comparable to humans at this task; that is a motivation for using judges, not a guarantee that a judge will correctly assess every agent workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The balance arrives, but only after three turns
An audience member immediately raises the trust problem: if the first AI is unreliable, does asking another AI to check it create an unreliable loop? Bennett defers the answer and turns to a recorded banking chatbot demonstration. Recording matters here because a nondeterministic application is not guaranteed to fail in the same way during a live presentation.
The conversation proceeds through three user turns:
- The user asks for their account balance. The assistant responds that it does not have access to the account information.
- The user specifies the checking account. Instead of retrieving the balance, the assistant asks for the account’s name.
- The user supplies the name,
checking account. The assistant can now call a tool and retrieve the balance.
The eventual result supports one interpretation: the chatbot worked. The unnecessary back-and-forth supports another: it failed to provide a satisfactory interaction. A final-answer check alone cannot express both judgments. Evaluations need to examine the steps that led to the result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give each component an evaluation contract
Evaluating the application requires collecting the information flowing through it and deciding what to check at each boundary. The relevant questions differ by component:
- Tool use: Did the agent successfully call the tool needed for the task?
- Retrieval: Did the retrieval-augmented generation system supply the right information?
- Answer quality: Does the response make sense, and does it invent information?
In a multi-agent banking application, an orchestrator might call another agent, which calls a tool. Each component needs an evaluation appropriate to its role. The useful question becomes where the failure occurred, rather than whether the entire application passed or failed.
To produce a score, an evaluator can receive the user input, the retrieved context, and the generated output under a well-defined judging prompt. One or several LLM calls can assess the chosen metric. Bennett recommends using a stronger evaluator than the economical model serving the application: the application runs frequently, while a more expensive judge can assess a sample of its traces.
Bennett’s hypothetical workload evaluates 10,000 of one million daily traces with an expensive model. Those figures illustrate a cost tradeoff, not a measured deployment or an established sampling requirement. He also describes Galileo’s specialized small evaluation model as an alternative to a general-purpose judge. In either case, evaluator selection and carefully defined prompts are part of the evaluation system.
These evaluations belong in the workflow from the beginning: while selecting models and engineering prompts, then throughout development and CI/CD. Once the application is deployed, the same concerns remain relevant because users introduce inputs and behaviors that development testing did not anticipate. An existing production application needs evaluation just as much as a new prototype does.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate completion from progress
The banking traces become more informative when viewed through two separate metrics. Action completion asks whether the flow accomplished the requested task. Action advancement asks whether it moved toward the goal. Completion describes the outcome; advancement makes useful intermediate behavior visible.
| Banking interaction | Completion | Advancement |
|---|---|---|
| Denies access to account information | No | No |
| Asks for the checking account’s name | No | Yes |
| Retrieves the named account’s balance | Yes | Yes |
The clarification question is the important middle case. It does not answer the user, but it obtains information the agent needs to proceed. Separating that behavior from a refusal lets the developer see which requests made progress and which stalled. The final retrieval then satisfies both criteria. These distinctions provide a basis for improving prompts without collapsing every incomplete response into the same failure category.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expand the score into a trace
An aggregate score tells you where to look, but it does not explain the cause. Bennett expands an individual trace into its sequence of operations: an LLM call, a decision to invoke a tool, retrieval of data, and another LLM call to process that data into the response. Each level can expose the metrics relevant to that operation. A red score becomes a starting point for investigation rather than a diagnosis.
Observability connects an evaluation result to the component that needs attention. Understanding the agent architecture lets the developer distinguish a retrieval problem from a prompt problem and assign the work to the appropriate team. Without that granularity, an overall failure gives little guidance about what to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn trace evidence into a proposed change
Traces contain substantial unstructured data, which gives an LLM another useful job: explaining patterns behind low metrics. In the generated insights Bennett shows, the diagnosis is that the assistant sometimes fails to use the Get Balance tool when asked about account balances. This makes the problem more specific than a low completion score: the application has a relevant tool, but the model does not consistently select it.
That diagnosis leads to a product decision. For a general balance request, Bennett expects the assistant to retrieve balances for all of the user’s accounts rather than demand an account name first. The suggested change is to add explicit instructions to the system message so the agent follows that behavior. This is a proposed fix; the demonstration does not show the revised prompt running or measure its effect.
The system leaves the proposal for a human to review. Automatically changing the prompt would introduce another source of potentially incorrect behavior, along with another obligation to evaluate the changes. The developer decides whether the suggested behavior is appropriate and whether to apply it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The evaluator also needs human feedback
This returns to the audience’s original objection. A stronger or specialized judge can still produce an incorrect score. Humans therefore need to inspect the evaluations, assess the reasons behind them, and supply corrections. Bennett describes this ongoing process as Continuous Learning with Human Feedback, or CLHF. The associated Galileo workflow refines evaluator prompts and examples from feedback; it should not be understood as a demonstration of continuous model-weight updates.
The point of that feedback is to make the evaluator fit the application’s actual criteria. A low score needs an explanation that a knowledgeable reviewer can assess. Evaluation quality is something to refine over time, rather than a property guaranteed when a metric is first enabled.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make production failures actionable
The operational work starts by adding evaluations and choosing measurements that match the application. Depending on the use case, that might mean toxicity in inputs and outputs, hallucination, comprehensibility, retrieval quality, or a custom domain requirement. Define these criteria alongside the prompts, agent responsibilities, and application structure so that the system exposes the evidence needed to assess them.
Keep measuring after deployment. Real users exercise paths that even extensive development testing misses, so evaluations cannot remain a development-only gate. Bennett closes by calling for real-time prevention and alerting when an agent misbehaves, including waking an operator when necessary. No specific blocking mechanism is demonstrated, but the operational requirement is clear: production failures need to reach someone who can respond.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Documentation for evaluating whether an interaction makes meaningful progress toward a user's goal.
Documentation for Galileo's metric assessing whether an agent completes the requested task.
Explains how expert feedback can refine evaluation prompts and examples for a particular domain.
The publisher explains how an externally produced supplement containing AI-generated material reached readers.
Further reading
Galileo's 2025 introduction to agent metrics, tool evaluation and tracing across workflow steps.
A banking assistant example with Python and TypeScript setup instructions and Galileo evaluation integration.
Original research measuring agreement between LLM judges and human preferences, including documented judge biases.
Full transcript
Loading transcript…