▶ Watch ↗19:19
Conference affiliation: Langfuse
De Jesus’s personal project captures prompts, responses and tool calls in Langfuse. A hook processes new messages after each assistant response and groups traces by session.
github.comDe Jesus’s companion account of why he built the observability template and how it works.
doneyli.substack.com · Source ↗A pipeline described by De Jesus that evaluates models and whole agents against financial datasets, producing pass-or-fail results and reviewable traces. Human approval remains separate.
github.com · Source ↗De Jesus explains the financial AI gate, its evaluators, human review and feedback loops for keeping evidence current.
langfuse.comAn official open-source collection introduced by De Jesus and Al Brown. Its initial 28 prioritized rules covered schema design, query optimization and ingestion.
github.com · Source ↗Co-authored by Al Brown and De Jesus.
clickhouse.comDe Jesus’s newsletter on practitioner frameworks and production AI implementation.
doneyli.substack.com · Source ↗Doneyli De Jesus leads AI Engineering Solutions at Langfuse, part of ClickHouse, helping organizations evaluate agents for production. He focuses on the gap between model capability and application performance: an agent can retrieve the right financial information yet calculate the wrong answer. His work connects those failures to specific changes in tools, prompts and evaluation criteria.
Don, his preferred short name, began in IT help desk and built a career spanning more than two decades in data and AI systems, including work at Deutsche Bank, Verizon, Cloudera, Elastic and Snowflake. At Snowflake, he became a principal and executive advisor and helped lead the company’s first AI go-to-market effort. He joined during the pandemic, when customer engagement and dealmaking had moved online. His farewell account also describes mentoring colleagues and working across teams.
At Langfuse, Don works with an open-source platform for tracing agent execution, managing prompts and evaluating application behavior. His projects address both sides of that work: giving developers a record of what an agent actually did and helping domain experts decide whether its behavior meets their requirements.
In his account of a banking project, Don describes how his team helped a bank evaluate an analyst application for interpreting earnings information using the firm’s internal data and analytical practices. One model scored 97% on grounding its responses in the bank’s sources but only 60% on numerical accuracy. The team gave the agent a Python calculator; Don reported that numerical accuracy rose to about 95%, clearing the bank’s 85% threshold. The improvement came from changing how the application performed calculations.
That example explains his case for approving complete applications. Public model benchmarks cannot establish whether a particular combination of prompts, retrieval, tools and internal data meets a bank’s requirements. Evaluation must also distinguish different kinds of success. Finding the correct source figures and calculating the correct result are separate checks; a strong score on one cannot compensate for failure on the other. A compliance violation can block deployment even when the application passes its other tests.
Don’s deployment-gate implementation connects representative test cases, execution traces, quality scores and human review. It combines deterministic checks, checks on the agent’s tool-use path and model-based judges. A correct answer can still fail if the agent skipped a required tool. Changes to prompts, models or tools trigger another evaluation, while reviewed failures can become new test cases. Don reported that the banking team could rerun evaluations in minutes after establishing this process, compared with weeks previously. The gate supplies evidence for approval; it does not replace the people responsible for the decision.
Domain experts play a specific role in Don’s approach: they help define acceptable behavior and set thresholds for each use case. He also tries to make their participation manageable. Rather than asking a risk specialist to design an evaluation taxonomy, he recommends giving them a selected set of agent interactions, simple scoring options and room for plain-language comments. An LLM can then help group those comments into recurring failure categories.
In a workshop with Annabell Schäfer, Don demonstrated this process with a fictional banking assistant. An expert could flag a missing citation, retain the comment with the execution trace and use that feedback to define what subsequent evaluations should check. He advocated combining deterministic evaluators with LLM judges, choosing the method to fit the failure being measured rather than delegating every assessment to another model.
The demonstration also addressed what observability should retain. A masking function redacted defined personal information, including a sample credit card number, before it was persisted in Langfuse. Developers could inspect the assistant’s execution without storing that number in plain text in the tracing system.
Don’s other projects make agent behavior easier to inspect and give coding assistants more specialized guidance:
▶ Watch ↗19:19
▶ Watch ↗53:32
Loading more talks and highlights…