Annabell Schäfer

Annabell Schäfer is a product marketing engineer at Langfuse, the open-source AI engineering platform acquired by ClickHouse in 2026. She helps teams replace vague promises of self-improving agents with measurable standards, realistic evaluation datasets, and expert oversight.

Schäfer studied technology management at the Center for Digital Technology and Management from 2021 to 2023. At Langfuse, her work spans application and documentation contributions, technical writing, developer education, and production evaluation. In May 2026, she and Lotte Verheyden introduced Langfuse Academy, a free educational resource covering tracing, monitoring, datasets, experiments, and evaluation.

Her guide to monitoring AI support agents identifies user frustration and response failures that conventional infrastructure monitoring misses. A technical account of Langfuse’s documentation-chatbot evaluations translates those production signals into annotation queues, regression datasets, grounding checks, and predeployment experiments.

• Domain-specific evaluation: Generic helpfulness or correctness scores rarely provide enough direction for automated improvement. Schäfer favors concrete tests: whether an answer uses retrieved information, preserves a product name, calls the appropriate tool, or exhibits a known failure. Her public position on evaluation prioritizes measuring meaningful behaviors over measuring everything.

• Validation-gated agent improvement: In a research-paper classification experiment, Schäfer paired GPT-5.4 nano with Claude Code and Claude Opus 4.8. The optimizer clustered classification errors, proposed focused prompt revisions, and retained changes only when separate validation accuracy improved. An early update increased accuracy from 68% to 78%; later iterations reached 83%, while an untouched test set scored approximately 80.2%. Even this seemingly objective task exposed ambiguity: authors exercise discretion when labeling their papers.

• Production-derived evaluation datasets: Schäfer builds regression suites from representative conversations, observed failures, human annotations, and concise explanations. Deterministic checks handle clear requirements; model-based evaluators address contextual judgments. For frequently changing documentation, grounding checks can outperform comparisons against answers that have become outdated.

• Expert-defined quality criteria: Domain specialists help identify representative examples, implicit decision rules, emerging failure modes, and appropriate stopping conditions. Automation can investigate patterns and suggest improvements, but validation boundaries and expert review determine whether those changes genuinely generalize.

Schäfer has also documented Hugging Face’s model-discovery agent and Slite’s evaluation workflow. Her public GitHub projects include agent_process_mining_cookbook, granola-connect, mini_agent, and skills.