Phil Hetzel is Head of Solutions Engineering at Braintrust, where he helps companies turn generative AI prototypes into reliable production systems. His work centers on agent evaluation, production observability, and the infrastructure required to measure whether AI applications actually serve their users.
From enterprise data to agent quality
Hetzel spent 12 years in consulting and systems implementation: four at KPMG and eight at Slalom Consulting, where he led the global Databricks business unit. His 2019 writing on Delta Lake examined how source-to-target comparisons, historical records, and upserts can reveal lost or misrepresented data before it compromises downstream decisions.
As generative AI gained enterprise attention, he watched clients produce convincing demonstrations without successfully deploying them. He argued for grounding applications in company data and user needs in an essay on closing the GenAI value gap, began using Braintrust with customers, and joined the company in 2025.
How he thinks about dependable AI
Human judgment becomes evaluation infrastructure. Hetzel advises teams to start with expert review and record why an output succeeds or fails. Those explanations become labeled examples, explicit failure modes, and scoring criteria. His gymnastics analogy for evaluations organizes the practice around a task, representative inputs, and appropriate scorers.
Automated judges must be tested too.LLM-as-a-judge validation compares model-generated assessments against human-labeled examples; statistical measures help establish whether automated graders reflect expert judgment. Deterministic checks remain useful for concrete constraints such as token consumption and excessive tool calls.
Production failures should strengthen subsequent tests. His production-to-evaluation feedback loop turns real user interactions into offline test cases, helping teams improve prompts, agent configurations, and tool behavior. Full-trace evaluation also exposes a harder problem: replaying interactions safely when external systems have changed or agent actions could modify live data.
Agent observability requires specialized data systems. Conventional uptime and latency metrics cannot establish whether an agent chose appropriate tools, grounded its answers, or met organizational standards. Hetzel describes large, semi-structured execution traces that require rapid ingestion, SQL analysis, full-text indexing, and clustering. He identifies Brainstore as infrastructure supporting these workloads, including natural-language log analysis and coding-agent queries.
Agent ownership should be cross-functional.His case for shared responsibility assigns data scientists judge validation, statistical rigor, and appropriate fine-tuning; engineers build and operate applications; domain experts define successful outcomes, refine prompts, and annotate failures. Pretrained foundation models make dependable AI a product, infrastructure, and organizational challenge—not the exclusive territory of any single discipline.
Building agents requires more than model expertise: product engineering, domain judgment, and evaluation each address a different part of the gap between a convincing prototype and a useful product.
Agents need more than latency and error monitoring: their variable behavior, text-heavy traces, and domain-specific failures change both the storage system and the improvement workflow.
Why does a working prototype stall before production?
Agent evaluation grows from documented human judgments into production replay, trace-level checks and careful reconstruction of the systems an agent touches.
Why do promising prototypes stall before production?
A loop and a spreadsheet can get evaluation started. Sustaining agent quality requires experiments, production feedback, and a data layer built for demanding trace workloads.
