AI Engineer Europe 2026
Why building eval platforms is hard
Loading slides…
Read the talk
Why building eval platforms is hard
A loop and a spreadsheet can get evaluation started. Sustaining agent quality requires experiments, production feedback, and a data layer built for demanding trace workloads.
From a talk by Phil Hetzel
Before you start: Familiarity with LLM agents, prompts, and basic database queries will help; no prior evaluation-platform experience is required.
What keeps a working prototype out of production?
Why can a team build a convincing generative AI proof of concept and still struggle to put it into production? Phil Hetzel encountered that gap repeatedly while helping clients implement systems. Before leading solutions engineering at Braintrust, he spent 12 years in consulting: four at KPMG and eight at Slalom, where he led the global Databricks business unit. His clients could produce compelling prototypes, but getting those systems into production remained difficult. He began using Braintrust to help close that gap, then joined the company roughly a year before this talk. His perspective comes from helping customers build evaluation and agent observability into their development process.
The missing ingredient is confidence that survives contact with real usage. Hetzel describes Braintrust as an agent quality platform with two related pillars: evaluation builds confidence while a team experiments before deployment; observability checks whether that confidence remains justified once real users interact with the agent. Both concern the same behavior, viewed at different points in the agent’s life.
LLM variability makes this difficult. The flexibility that lets a model reason across many problems also makes its behavior harder to anticipate. As customers increasingly expect agentic interactions, that uncertainty becomes a product concern: an agent can create brand and compliance risks, as well as cost, maintenance, and systems problems. Evaluation is how a team starts checking whether its agent behaves as intended before those consequences reach customers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A spreadsheet and a for loop
The smallest useful eval setup is approachable: run an agent on several inputs, put its outputs into a spreadsheet, and add handwritten notes or scores. It needs only three ingredients.
| Ingredient | What it provides |
|---|---|
| Agent execution | A way to invoke the system being tested |
| Input examples | The information needed to initiate each run |
| Results UI | Outputs, notes, and scores people can inspect |
An input example is not necessarily just a question. It contains whatever information is necessary to invoke that agent. Google Sheets can be a perfectly reasonable first results UI. The problem is that these visible pieces are only the tip of the platform iceberg.
The work underneath involves several kinds of expertise. Product and AI engineers shape behavior; systems engineers make the agent run; subject-matter experts judge whether its responses make sense in the domain. Model complexity, collaboration across those roles, and infrastructure requirements compound one another. An eval platform must support that collaboration rather than assume engineers can evaluate the agent in isolation.
Start with the loop anyway. Each time the agent changes, rerun the examples and record the new outputs. That provides a low-barrier way to see differences, but the workflow soon becomes documentation rather than experimentation. Comparing runs across a growing spreadsheet is cumbersome, aggregate analysis is limited, and human scoring—valuable as it is—becomes difficult to scale.
The spreadsheet also limits who participates. Nontechnical contributors may have the strongest domain expertise or the closest contact with users, yet a manually maintained results sheet may not draw them into the process. Recreating or appending the sheet after every evaluation adds friction precisely where the team needs fast iteration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A nicer UI is not yet an experiment
The next step often begins with an engineer announcing, “I can just vibe code Braintrust.” There is real value in that step. Keep the for loop, replace the spreadsheet with a friendlier interface, and persist results in a database—Hetzel gives Neon as an example. Now the records are easier to retain, and the UI can fit the people who need to review them. But if users can only inspect what already happened, the result is still primarily a reporting tool. Better presentation does not by itself create an iteration workflow.
Experimentation requires letting users change something and evaluate the consequences. A playground supplies a sandbox around an agent configuration, exposing selected parameters that a user can adjust. Technical users also need an SDK-driven path; the UI should not be the only way to run the experiment.
Hetzel’s example exposes the system instructions for an agent that runs outside the eval platform. A user changes those instructions, compares two system-prompt configurations, and runs evals across both agent runs. The resulting scores make technical and functional behavior visible. The important boundary is that the platform can control an experiment without owning the agent’s execution environment: it needs access to the configuration, a way to run the agent, and a way to compare the results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use production failures to choose the next experiment
A playground makes experiments possible, but it does not tell the team what to test. Build scoring functions around failure modes, then use production traces to discover which failure modes actually occur. Real users expose behavior that a team’s initial examples may never have anticipated. This is where evaluation and observability become parts of the same system.
Hetzel traces that connection to an early Braintrust customer. The company initially offered only offline evals, but the customer was piping production traffic into its database and evaluating it there. Hetzel recalls a customer running a massive evaluation every hour of every day against production traffic. That workaround exposed the need for a direct tracing and observability workflow, rather than forcing live usage into the shape of an offline experiment.
The resulting loop is continuous:
- Observe the agent in production to see its actual behavior and the effect of changes.
- Analyze those interactions to find failures worth addressing.
- Evaluate using real examples brought back into an offline environment.
- Improve the agent and repeat the cycle throughout its lifetime.
The dataset is no longer just a collection of imagined requests. It becomes a way to bring observed behavior back into development.
That feedback loop expands the platform’s scope substantially. It is now a tracing and logging platform as well as an offline eval platform. Real interactions provide higher-signal examples, making offline evaluation closer to rerunning production in a safe environment. The same scoring functions can also run against observability traffic as online evals, supporting alerts when behavior warrants attention.
The organizational consequence is easy to underestimate: building the platform creates a continuing maintenance responsibility. Someone must operate it and extend it as agent development changes. The initial custom UI may be a small project; keeping the full evaluation and observability loop working becomes an ongoing job.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The data layer behind the trace viewer
Agent traces put unusual pressure on storage. They contain large amounts of text, mix semi-structured and unstructured content, and can arrive rapidly once an agent has substantial production usage. Hetzel illustrates the mismatch with the prospect of putting a one-gigabyte trace into a Postgres row: treating a trace as one ordinary record can create serious performance problems. The example is a storage-design warning, not a measured database limit.
The platform must serve two different access patterns at once.
| Access pattern | User expectation | Data-layer requirement |
|---|---|---|
| Inspect a recent trace | See what just happened | Low-latency ingestion and visibility |
| Analyze many traces | Understand behavior in aggregate | Efficient analytical reads |
Braintrust’s former architecture addressed these with separate persistence layers, including an open-source data warehouse for aggregate analysis. It stitched the sources together using BTQL, a domain-specific query language. Hetzel describes that language as unpopular with both users and the team. A third aggregation layer ran DuckDB in the browser. The arrangement worked for a time, but coordinating the layers added complexity.
Text-heavy customer workloads exposed the arrangement’s limits. Hetzel uses Notion as an example: large volumes of unstructured data created demand for full-text search across traces. His criticism concerns the suitability of Braintrust’s former combination of technologies for that workload. DuckDB does have a full-text-search extension; that capability alone does not establish that the former browser-based arrangement could satisfy the required trace-search workload.
Agent quality becomes a systems problem, not just a UI problem. The challenge is not simply storing more records. Users expect capabilities such as full-text search across millions of traces. A polished interface cannot supply those functions if the underlying data platform cannot retrieve and analyze the necessary content.
A span is one part of a trace, and context can make an individual agent span large. Hetzel contrasts traditional spans of a couple of kilobytes with agent spans he has observed at 10–20 megabytes because of their context. These are his observed examples, not a typical-size estimate. Combine large, highly unstructured payloads with rapid ingestion, aggregate reads, and low-latency inspection, and familiar database requirements become a difficult joint workload. None of those requirements is individually new; satisfying them together is the distinctive problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The next user may be a coding agent
The data platform also determines whether evaluation can support automated improvement. Hetzel describes a workflow in which a coding agent retrieves aggregate evaluation data, brings it into its context, and changes the evaluated agent within the same coding session. That requires useful access to the underlying data, independently of the trace viewer. He presents SQL access as important to making this workflow practical.
For a concrete example, suppose evaluation results live in an eval_results table with agent_version, failure_mode, and passed columns. A coding agent investigating version candidate_b could retrieve failures by category:
sql
SELECT
failure_mode,
COUNT(*) AS evaluated_cases,
SUM(CASE WHEN passed = FALSE THEN 1 ELSE 0 END) AS failed_cases
FROM eval_results
WHERE agent_version = 'candidate_b'
GROUP BY failure_mode
ORDER BY failed_cases DESC;
That gives the coding session a compact set of failure categories to investigate before proposing a change. It illustrates the aggregate-data access Hetzel calls for; the table is a teaching schema. For Braintrust specifically, the current SQL reference recommends SQL for new queries while retaining legacy BTQL support. SQL access does not imply unrestricted PostgreSQL compatibility.
Hetzel reports growing interest in these headless eval workflows: some users want Codex or Claude Code to help improve their agent without visiting the platform UI at all. The backend must therefore serve agents as well as human analysts. He briefly shows a backend architecture slide, but passes over the implementation discussion for time; the workflow requirement is the developed point here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find unknown usage patterns and govern collection
Retrieving known failures is only part of the job. A mature platform should also help teams discover behavior they did not know to look for. Hetzel points to topic modeling as a way to understand how people use an agent without manually reading a large collection of traces. Surfacing those unknown usage patterns helps decide where engineering time should go. As that analysis becomes part of development, both humans and agents need to be able to consume it.
Operating the platform at scale brings additional requirements:
- Role-based access control: govern access to platform data and capabilities.
- Data masking: account for sensitive content within the collected data.
- Automatic tracing: use an AI proxy or gateway to capture LLM interactions centrally, making tracing part of the request path rather than an optional developer action.
The gateway provides a point of central governance. These are further platform responsibilities, beyond experiment execution and results presentation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep media review in the trace
The closing audience question makes the data problem concrete again. Dynamic prompts can include files and videos, not just strings assembled through interpolation. The questioner asks how Braintrust handles that complexity when teams otherwise end up building custom solutions around evaluation tools. Hetzel narrows the question to multimodal inputs, outputs, and traces, and the questioner confirms that framing.
His answer separates media storage from trace review: put the media in object storage, reference it from the trace, and render it inside the trace interface. A reviewer can play audio or video where they are already inspecting the agent’s behavior, without leaving the platform. The attachment documentation describes current media-preview support; storing an attachment should not be confused with indexing all of its contents for search.
Prompt ownership remains a separate decision. When the questioner asks whether prompt management must live in Braintrust, Hetzel answers that it can, but does not have to. The platform can make multimodal execution inspectable while leaving prompt management elsewhere—just as the earlier playground can evaluate an agent running outside the platform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- How Brainstore worksArticle
Braintrust's architectural account of trace ingestion, indexing, querying, and the limitations of its previous storage stack.
- Introducing BrainstoreArticle
The original Brainstore announcement, with workload-specific benchmarks and explanations of its object-storage architecture.
DuckDB's official extension for indexing text and searching document collections.
Updates since the talk
Examples for attaching media and large JSON objects to traces, including preview behavior and indexing limitations.
Current SQL and legacy BTQL syntax, with sandbox, CLI, and programmatic query access.
Full transcript
Loading transcript…