← All speakers

Doneyli De Jesus

Conference affiliation: Langfuse

Doneyli De Jesus leads AI Engineering Solutions at Langfuse, part of ClickHouse, helping organizations evaluate agents for production. He focuses on the gap between model capability and application performance: an agent can retrieve the right financial information yet calculate the wrong answer. His work connects those failures to specific changes in tools, prompts and evaluation criteria.

On this page

From enterprise data to AI applications

Don, his preferred short name, began in IT help desk and built a career spanning more than two decades in data and AI systems, including work at Deutsche Bank, Verizon, Cloudera, Elastic and Snowflake. At Snowflake, he became a principal and executive advisor and helped lead the company’s first AI go-to-market effort. He joined during the pandemic, when customer engagement and dealmaking had moved online. His farewell account also describes mentoring colleagues and working across teams.

At Langfuse, Don works with an open-source platform for tracing agent execution, managing prompts and evaluating application behavior. His projects address both sides of that work: giving developers a record of what an agent actually did and helping domain experts decide whether its behavior meets their requirements.

Approve the application

In his account of a banking project, Don describes how his team helped a bank evaluate an analyst application for interpreting earnings information using the firm’s internal data and analytical practices. One model scored 97% on grounding its responses in the bank’s sources but only 60% on numerical accuracy. The team gave the agent a Python calculator; Don reported that numerical accuracy rose to about 95%, clearing the bank’s 85% threshold. The improvement came from changing how the application performed calculations.

That example explains his case for approving complete applications. Public model benchmarks cannot establish whether a particular combination of prompts, retrieval, tools and internal data meets a bank’s requirements. Evaluation must also distinguish different kinds of success. Finding the correct source figures and calculating the correct result are separate checks; a strong score on one cannot compensate for failure on the other. A compliance violation can block deployment even when the application passes its other tests.

Don’s deployment-gate implementation connects representative test cases, execution traces, quality scores and human review. It combines deterministic checks, checks on the agent’s tool-use path and model-based judges. A correct answer can still fail if the agent skipped a required tool. Changes to prompts, models or tools trigger another evaluation, while reviewed failures can become new test cases. Don reported that the banking team could rerun evaluations in minutes after establishing this process, compared with weeks previously. The gate supplies evidence for approval; it does not replace the people responsible for the decision.

Turning expert feedback into tests

Domain experts play a specific role in Don’s approach: they help define acceptable behavior and set thresholds for each use case. He also tries to make their participation manageable. Rather than asking a risk specialist to design an evaluation taxonomy, he recommends giving them a selected set of agent interactions, simple scoring options and room for plain-language comments. An LLM can then help group those comments into recurring failure categories.

In a workshop with Annabell Schäfer, Don demonstrated this process with a fictional banking assistant. An expert could flag a missing citation, retain the comment with the execution trace and use that feedback to define what subsequent evaluations should check. He advocated combining deterministic evaluators with LLM judges, choosing the method to fit the failure being measured rather than delegating every assessment to another model.

The demonstration also addressed what observability should retain. A masking function redacted defined personal information, including a sample credit card number, before it was persisted in Langfuse. Developers could inspect the assistant’s execution without storing that number in plain text in the tracing system.

Tools that expose actions and supply expertise

Don’s other projects make agent behavior easier to inspect and give coding assistants more specialized guidance:

  • Claude Code observability. His personal Claude Code and Langfuse template captures prompts, responses and tool calls. After each assistant response, a hook processes new conversation messages and sends traces to a cloud or self-hosted Langfuse instance, grouping turns by session. Developers can examine tool inputs and outputs alongside the conversation rather than judging the assistant solely by its final response.
  • ClickHouse guidance for coding agents. Don and Al Brown co-authored the introduction to ClickHouse Agent Skills, an open-source collection covering schema design, query performance and data ingestion. Its initial release packaged 28 prioritized rules. The guidance addresses decisions that a small successful test may not expose: ordering choices, joins and insert batching can all become problems at larger scale.

Read the topics behind these talks

2 conference talks

Loading more talks and highlights…

References