AI Engineer Europe 2026
How agent o11y differs from traditional o11y
Loading slides…
Read the talk
How agent o11y differs from traditional o11y
Agents need more than latency and error monitoring: their variable behavior, text-heavy traces, and domain-specific failures change both the storage system and the improvement workflow.
From a talk by Phil Hetzel
Before you start: Familiarity with HTTP requests, LLM prompts, and tool calls will help; traces and spans are defined in the article.
Why does a working prototype stall before production?
A generative AI prototype works well enough to demonstrate. What would make you confident enough to put it into production? Phil Hetzel encountered that gap repeatedly during twelve years in consulting and systems implementation, including leading Slalom’s Global Databricks practice: customers were prolific at producing proofs of concept, but much less successful at deploying them. He began using Braintrust to address that problem before joining the company, where he leads solutions engineering.
The confidence problem runs in two directions. Once an agent is live, you need to know whether it performs as well as expected. When you change it, you need evidence that the next version deserves your confidence. Braintrust’s framing of agent quality connects these two activities: monitoring production behavior and evaluating experimental changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A fast response can still be wrong
Conventional application code deliberately constrains control flow. An agent can choose different paths while attempting the same task—the flexibility that makes LLMs useful also makes their behavior harder to characterize. Observability must therefore help explain why the agent selected one path rather than another, not merely how long the selected path took.
Time to first token, total tokens, duration, and latency remain useful. But a response can arrive promptly and still fail the task. Quality measurements ask additional questions:
- Grounding: Does the answer follow from the context the application gathered?
- Tool selection: Did the agent use the tools expected for this request?
- Brand alignment: Does the response follow the standards in the system prompt?
Technical success does not establish behavioral quality. To evaluate these questions, the trace must retain the relevant context, tool interactions, instructions, and response content. That is much richer evidence than a duration or status code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Large traces, immediate inspection
Agent traces combine a semi-structured hierarchy with substantial unstructured text. Hetzel reports seeing customer agent traces larger than one gigabyte and individual spans as large as twenty megabytes. These are reported examples, not typical trace sizes; he does not specify their composition or measurement method. The systems challenge includes ingesting and processing those payloads, but also making them usable.
As usage grows, AI engineers and product managers still expect to inspect interactions in real time. In the trace example Hetzel shows, the model calls and tool calls form a collection of spans, with large amounts of text inside those spans. A trace viewer has to expose both the execution structure and the content needed to understand the result.
The same data must support two read patterns:
- Interactive inspection: A user interacts with the agent, and an engineer wants to see that interaction almost immediately.
- Programmatic improvement: A developer issues SQL queries through a CLI and uses observability or evaluation traces to improve the application automatically.
Serving a recent interaction quickly and querying a large collection of traces are different access requirements. Both have to work against the same bulky records.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Visibility, indexing, and text search
Braintrust built a database specifically for this workload. The architecture separates immediate visibility from indexed query performance: incoming data enters a write-ahead log, while indexing supports filtering and analytical queries. The related Brainstore architecture explains the visibility mechanism: queries can read committed log entries alongside indexed data before background indexing finishes. Immediate visibility here means avoiding a wait for that indexing step, not zero ingestion latency.
Text search is a central requirement. Hetzel says Braintrust forked Tantivy to provide text indexing. His example is straightforward: find every trace containing the word Amazon. Answering that request efficiently across large trace bodies requires an index over their text, rather than relying only on structured fields such as duration or error status.
For a simple relational representation with one text row per span, the search intent can be expressed in SQL:
sql
SELECT DISTINCT trace_id
FROM span_text
WHERE text_content LIKE '%Amazon%';
Here, span_text names an illustrative table, and DISTINCT returns each matching trace once even if several spans match. The query expresses a substring search; a full-text index supplies its own token-matching semantics and avoids treating every search as a scan through every text body. The important requirement is retrieval by content across the trace.
When an audience member asks whether Tantivy is like OpenSearch, Hetzel compares it more directly to Apache Lucene, but written in Rust. The log, indexes, and text-search capabilities then need a unified SQL or SQL-like query interface. That is the purpose of the architecture diagram: incoming traces become visible, acquire indexes, and remain accessible through a common query path with results streamed to the UI.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The people who can recognize a failure
Operational observability usually serves systems engineers and product engineers. Agent quality brings in another group: people close to the users or the problem domain. Natural-language prompts give those contributors a direct way to influence behavior. Hetzel describes clinicians, registered nurses, wealth advisors, and lawyers inspecting traces to improve agents, and attributes stronger agent teams to collaboration between technical and nontechnical contributors. The person best equipped to recognize a bad answer may not be the person who operates the infrastructure.
This also connects observability to evaluations. In Braintrust’s framing, evaluations use the same basic machinery but run in batches with inputs known ahead of time. Trace storage, quality assessment, and human review are useful in both settings. The depth of the platform follows from supporting both complex data and the different people who need to work with it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What are people doing with the agent?
Once traces contain the substance of user interactions, another question becomes possible: can the system explain how people are using the agent? Hetzel describes a capability rolled out in Braintrust’s SaaS offering roughly a month before the talk. At the level described in the recording, a lightweight LLM, embeddings, and clustering support topic modeling over incoming traces. The resulting analysis surfaces user intent, sentiment, and possible issues.
The purpose is to shorten the distance between a problem observed in production and a fix tested through experimentation. Instead of requiring someone to discover every recurring issue by reading individual traces, grouping interactions can help reveal where to investigate next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From expert judgment to automated scores
The audience’s first distinction is between functional quality and non-functional performance. Hetzel agrees that traditional observability can measure technical performance; agent tracing also captures prompts, duration, time to first token, and cache hits alongside quality measures. These measurements are complementary: one set describes the execution, while the other assesses whether the agent did a good job.
A question about human annotation on the iceberg slide makes the quality workflow concrete. A product manager or domain expert reviews an incoming trace, but the useful artifact is more than a grade:
- Inspect the interaction. Determine whether the agent performed well or poorly.
- Explain the judgment. Record why the behavior deserves that grade.
- Identify the failure mode. Use those explanations, potentially with an LLM, to help construct scalable scoring functions.
- Automate the recurring check. Turn the recognized failure mode into something future traces can be scored against.
Human annotation helps discover what the automated evaluator should look for. The justifications connect domain expertise to repeatable quality checks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring production failures into experiments
The next question joins two concerns: integrating with agent frameworks for offline optimization, and choosing a custom database instead of an OLAP system such as ClickHouse. Hetzel says Braintrust previously used ClickHouse and moved away from it. He describes the founder as an early SingleStore employee, with the background to undertake database development, but gives a specific workload reason for the decision: the text indexes they needed were not adequately available in ClickHouse at that time. This is a historical explanation, not a measured efficiency comparison or a claim about present-day ClickHouse capabilities.
For the optimization question, the distinction is when the inputs become available:
| Workload | Inputs | Execution |
|---|---|---|
| Evaluations | Known in advance | Batch |
| Production observability | Arrive from users | Real time |
Braintrust uses the same system for both. That shared infrastructure makes a production interaction usable as evidence in an offline experiment.
When the questioner presses for the practical integration step, Hetzel gives a direct workflow: capture the trace, add it to an offline dataset, and experiment on it. The handoff is the recorded interaction becoming a test case. His answer specifies that dataset workflow rather than a framework-specific integration procedure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Leave room for failures you have not defined
The final question asks whether observability results must be quantitative. User satisfaction might be a number, but could a result instead be prose or a flag? Hetzel separates two kinds of analysis. Online scoring addresses known unknowns: you know the dimension you want to measure, even though you do not yet know how this interaction performed. Open-ended analysis addresses unknown unknowns: it derives insights without requiring every issue to have a predefined score.
Scores measure the failures you have learned to recognize; open-ended inspection helps you discover the next ones. A quality workflow needs space for both, so its understanding of agent behavior can grow beyond the checks already written.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Rust full-text search library inspired by Apache Lucene, with examples and development documentation.
Further reading
- Brainstore architectureArticle
Explains Brainstore’s object storage, write-ahead log, Tantivy indexing, and specialized trace queries.
Introduces automated grouping of production traces by user tasks, sentiment, and agent issues.
Updates since the talk
- Inside the Topics pipelineArticle
Details how trace summaries become embeddings, clusters, named topics, and classifications.
Instructions for assembling versioned test cases and adding production traces to evaluation datasets.
Search production logs through filters, SQL, and the CLI for debugging and dataset creation.
Documents production availability of ClickHouse’s native inverted text indexes in March 2026.
Full transcript
Loading transcript…