▶ Watch ↗AI Engineer World's Fair 202617:39
Conference affiliation: Growth Engineer · Clickhouse · 2026
Annabell Schäfer is a product marketing engineer at Langfuse, the open-source platform for tracing and evaluating AI applications, now part of ClickHouse. Her work connects production monitoring with the decisions behind agent improvement: which failures deserve attention, what correct behavior means for a particular application, and whether a proposed change improves performance on examples it was not optimized against.
Annabell studied technology management at the Center for Digital Technology and Management from 2021 to 2023. Her work at Langfuse combines application and documentation contributions with technical writing and developer education.
In May 2026, Annabell and Lotte Verheyden jointly introduced Langfuse Academy, a free resource covering tracing, monitoring, datasets, experiments, and evaluation. It connects activities that teams need to use together: traces make an agent’s behavior inspectable, monitoring helps identify failures, and datasets let developers test changes before deployment.
A support agent can complete its model and tool calls successfully while leaving its user frustrated or misinformed. Annabell’s practical writing explains how to detect those conversational failures and turn them into specific tests, rather than treating infrastructure health or a general helpfulness score as sufficient evidence of quality.
In a workshop with Lotte Verheyden, Annabell applies these ideas to a TypeScript support agent built with the OpenAI Agents SDK. She configures a model-based judge for user disagreement, a code check for all-caps messages, and a scope check comparing the request with the system prompt. Explicit mappings select the messages each evaluator examines, so each score answers a defined question. Together, the presenters connect those production signals to representative datasets, offline experiments, and regression checks.
Annabell’s research-paper classification experiment makes the distinction between proposing a change and accepting it concrete. She paired GPT-5.4 nano with an optimizer using Claude Code and Claude Opus 4.8. The optimizer clustered classification errors and proposed focused prompt revisions; changes were retained only when accuracy improved on a separate validation set. An early revision raised accuracy from 68% to 78%, later iterations reached 83%, and an untouched test set scored approximately 80.2%.
Even paper classification exposed ambiguity: authors exercise discretion when labeling their papers. Annabell’s argument for domain expertise therefore extends beyond supplying examples. Specialists help identify implicit decision rules, representative cases, emerging failure modes, and appropriate stopping conditions. Automated error analysis can suggest revisions, but the quality criteria and separate validation examples determine whether those revisions count as progress.
Her account of Slite’s evaluation workflow illustrates another way to make those criteria precise. Slite moved from a manually graded spreadsheet to a reusable test dataset, specifying what an answer must contain rather than requiring one fixed wording. Annabell explains the team’s implementation while preserving the distinction between documenting its engineering choices and making them herself.
▶ Watch ↗AI Engineer World's Fair 202617:39
▶ Watch ↗53:32
▶ Watch ↗AI Engineer World's Fair 202644:22
Loading more talks and highlights…