Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse

Loading slides…

Read the talk

Why a bank gave its AI analyst a calculator

Doneyli De Jesus of Langfuse describes how a top-five bank built a deployment gate that could distinguish an agent that found the right financial data from one that gave a banker the right number.

From a talk by Doneyli De Jesus

At a glance

Ideas worth remembering

  • Start with cases the institution already trusts. Pair each input with its expected output and retain the trace when a candidate configuration runs it.

  • Score retrieval, arithmetic and tool use separately. The bank's 97% groundedness score concealed 60% numerical accuracy, and the calculate call gave engineers a specific intervention to verify.

  • Have domain experts set thresholds by use case. Give a disqualifying compliance failure its own release gate and route the failed trace to someone who can assess it.

  • Use reviewed failures to improve the golden dataset, then rerun it when the prompt, model or tools change.

A benchmark for the investment banker waiting on an earnings call

When an earnings call ends, an investment banker wants to know what an analyst at the firm makes of it. Getting a summary of the transcript quickly would accomplish part of the job. An experienced analyst also brings knowledge of the firm and how it operates to the reading. The bank wanted its proposed 10-K analyst to bring that context to bear while the information was still fresh. How could the people responsible for the application tell whether it had done that work well enough to put the result in a banker's hands?

Doneyli De Jesus leads AI engineering solutions at Langfuse by ClickHouse, an open-source platform for agent observability and evaluation. His team helped a top-five bank work out how to make release decisions like this one. The public benchmarks that accompany a new model could tell the engineers about its technical capabilities, sometimes across many dimensions. But the bank had its own datasets, ways of building models and uses for their output. It considered much of that work proprietary. A performance gain on an open dataset gave the bank little basis for estimating the gain an investment banker would see.

The analyst posed two separate engineering problems. The team had to make the firm's knowledge available to a model that had little reason to know it out of the box. Then it had to establish whether the resulting analysis was correct. As De Jesus recalled the bank's rule, “We don't certify models here. We certify applications.” A 10-K analyst and an investment analysis application could use the same underlying model and still call for different evidence at release time.

0:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Three constraints on where the evaluation can run

The first requirement De Jesus encountered in finance was on-premises operation. After years of moving work to the cloud, he had seen banks reconsider data centers they already had. The sensitivity of the material going into models made it worthwhile for some to repurpose those facilities and stand up GPU infrastructure for local inference. Some banks even had research divisions where people with PhDs trained and fine-tuned their own models. An evaluation method had to be applicable to whatever model the institution could run there.

The second requirement was gated network access. The bank wanted control over where its data went, even when a model provider guaranteed zero data retention. Even the appearance that a provider might train on the data mattered. The third requirement was an air-gapped deployment that kept the system isolated from external networks. Within the bank’s infrastructure, De Jesus emphasized that only people who needed the data should see it. That mattered for evaluation, too, because an agent’s trace could contain the internal material it retrieved.

3:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:11 · section reference included

Make a configuration change answerable to a repeatable test

Once the platform and models were in place, the bank still had three things to prove. The application had to perform the intended task, satisfy the bank's compliance requirements and produce something people could use. The third condition had an organizational consequence. Bankers who kept getting unusable analyses would stop using the application, and the experience could sour them on other AI-enabled work. The team wanted a pass-or-fail decision it could revisit whenever someone edited the system prompt, changed the model or gave the agent another tool.

De Jesus started with a golden dataset. In software engineering terms, its entries were the application's test cases. In his experience, banks often already had a set of examples they trusted. What they lacked was a way to exercise those examples against each candidate configuration. Keeping the questions fixed lets engineers return to a case after a change and examine what happened to the answer.

Running the agent on those cases produces traces. Each trace records evidence of what the team evaluated and how the agent handled it. A dataset entry can establish what the bank expected. The corresponding trace can show, for example, which tool the agent actually called on the way to its answer. Evaluators use that evidence to score the run along the dimensions the bank selects. Only then does the team have something to compare with a release threshold.

Arrows connect three boxes. A dataset and prompt version lead to a traced run with item scores, then to a verdict with evidence. Each result row links to a trace.Open full source frame
The contract from dataset to verdict
4:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:55 · section reference included

The small model could find a figure and still get the math wrong

The team tried models of different sizes and capabilities. One small model earned 97% on groundedness. Here, groundedness measured how well it extracted information from internal sources and based its response on data the bank actually held. That was a passing result. If the team had investigated only whether the analyst could find and use the bank's material, it would have had little reason to change this part of the application.

The same model scored 60% on numerical accuracy. The difference showed up when a financial answer required arithmetic. An answer could draw on the appropriate source and still report the wrong figure, even for a simple calculation. De Jesus attributed that weakness to asking a next-token predictor to do math. For the team, the distinction narrowed the problem considerably. The evidence pointed to the work the agent did with the retrieved information, after it had found the material it needed.

Two scores appear side by side. Groundness is 97% in black, and Numerical Accuracy (Math) is 60% in red.Open full source frame
Model scores for groundness and numerical accuracy
7:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:10 · section reference included

Put a tool at the calculation step and a gate after the workflow

De Jesus followed a test case from the golden dataset into the agent. The agent planned the task, retrieved data from official sources and reached a calculation step. That last step was where the team intervened. Arithmetic with given inputs is deterministic, so the team gave the agent a calculator it could call. The agent still had to work out what the question required and retrieve the relevant information. The calculator took over the operation that had been yielding unreliable numbers.

The gate sat at the end of that workflow. Numerical accuracy was one condition the application had to clear. Other dimensions, including regulatory compliance, could have thresholds of their own. A change that improved arithmetic therefore had to go through evaluation of the whole application. Even an average above 90% could leave a candidate below a required threshold in one category. That candidate would be held back from production.

A golden dataset feeds a four-step agent. Calculate is outlined in blue. The final gate requires all four thresholds at once, with 100% compliance outlined in red.Open full source frame
Four thresholds for passing the agent evaluation

This arrangement also gave the bank a way to consider a future model swap. The new model could run inside the agent with the same task, sources and tools. The bank could then apply the same quality requirements to what came out. Approval of the previous configuration supplied a basis for comparison. The new configuration would earn its own result.

8:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:46 · section reference included

Connect the expected answer, the calculator call and the measured gain

The example dataset had three elements per case: an input, an expected output and accompanying metadata. Together these gave the team a reference for assessing the agent's response and behavior. De Jesus then showed a scored agent run and singled out the calculate tool call in its trace. The dataset supplied the expectation before the run. The call was evidence from execution that the agent had used the capability the engineers introduced.

A Langfuse trace shows the calculate step selected in the run tree. Blue outlines identify the tool call, its arithmetic input, and its returned value. The equation below rounds the result to 24.26.Open full source frame
Tracing the calculate tool’s inputs and result

The instruction was to use the small piece of Python code whenever a number needed calculating. With that tool available, numerical accuracy rose from 60% to roughly 95%. The distinction between the tool and the measured result matters here. Python can calculate deterministically once it receives the inputs. The roughly 95% figure described how often the application got the numerical answer right on the evaluation. The improvement included the agent's ability to get from a question to a calculation and return the result to the user.

10:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:16 · section reference included

Use code, path checks and an LLM judge for different questions

De Jesus described three ways to score a run. The first was deterministic scoring in code. For the same inputs, the scorer produces the same output each time. A numerical result with a known expected value is a natural fit. The Python calculator and a deterministic scorer have different jobs here. One helps produce the answer during execution. The other assesses the result, so engineers can tell whether adding the tool improved performance across cases.

The second was a path check. An answer can look complete even when the agent skipped a required step. Inspecting the trace lets an evaluator ask whether the agent took the appropriate route. In this example, the presence of a calculate call answers a different question from whether the final figure was correct. An agent might happen to supply the right number on a case where it did the arithmetic itself. A check of the answer alone would miss that it bypassed the tool.

The third was an LLM judge for more qualitative aspects of the response. De Jesus favored a blend as applications became more complex. Code can settle reproducible questions, the path exposes missing actions, and a judge can assess qualities of an answer that are harder to express in code. Together those results gave the bank evidence it could tie to an identified change in a system prompt, model or set of tools. In the calculator example, the small model's application went from failing numerical accuracy to clearing the 85% gate for that use case.

Three code panels show how scoring works. A prohibited phrase scores zero, a path check maps each question type to a required step, and the judge weights faithfulness at 0.7 and completeness at 0.3.Open full source frame
Scoring with prohibited phrases, path checks, and a judge
11:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:32 · section reference included

Let the risk team decide which score can block a release

Selecting an evaluator still leaves a decision about what score is good enough. For an AI-enabled risk application, De Jesus wanted the people responsible for risk involved in setting that bar. They could judge what they would be comfortable putting in front of colleagues or customers. The order mattered. First identify the dimensions on which this particular application should be judged. Then set a threshold for each. The analyst's 85% gate offered no general answer to what another use case should require.

De Jesus described full runs on two datasets, one with about 150 questions and another with about 1,500. He illustrated the release decision with a configuration that passed on groundedness, completeness and tool use but scored zero on regulatory compliance. Compliance in this example was binary, scored at zero or 100. A configuration could therefore retrieve the right material, carry out the expected steps and supply a complete answer, yet fail a condition the bank considered binding.

Four scorecards show passing scores for groundedness, completeness and tool use. The red-outlined compliance card shows 0% against a required 100%, causing the gate to fail.Open full source frame
One prohibited phrase fails the compliance gate

Averaging across dimensions could let high scores for groundedness, completeness and tool use outweigh the compliance failure. The bank could still face a penalty or reputational damage for releasing that configuration. With a separate compliance gate, a zero blocked promotion even when the other scores passed.

13:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:45 · section reference included

Route failed traces to experts and keep the cases worth testing again

De Jesus described an annotation queue with a specific selection rule. Whenever regulatory compliance failed, the team could send the corresponding trace to a domain expert. That gave the reviewer the question, the answer and the steps behind it. The expert could inspect what happened and give engineers feedback on a particular failure, without having to find it among hundreds of successful cases. Engineers could use that feedback to improve the agent.

Two events then fed different parts of the process. Promoting a prompt or changing the configuration triggered another run through the gates. Expert feedback, meanwhile, made a reviewed case a candidate for the golden dataset. An expert might identify a better expected result for the dimension that failed. If the team included that case, the next candidate would face a test informed by an actual failure and someone qualified to judge it.

Solid arrows connect the golden dataset, four agent steps, four thresholds and a gate that feeds a sign-off portal and local exit code. Dashed arrows show two paths tested offline: a reviewed trace becomes a golden example, and promoting a prompt triggers another run.Open full source frame
Re-running the application gate when something changes
15:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:46 · section reference included

What the bank could repeat in minutes

De Jesus reported that the bank reduced the time it took to evaluate AI applications from weeks to minutes. He singled out risk and compliance as areas where evaluation can sometimes take months. With cases, scorers and thresholds in place, the team could rerun a changed configuration and focus attention on the dimensions that failed. The reported time saving applied to evaluation. Experts still had to establish what good work looked like and review failures that warranted their judgment.

De Jesus closed by pointing practitioners to a detailed account of the work and a repository for reproducing it. What another team could reproduce was the process. Its own data would determine the benchmark, the application would determine which behaviors to measure, and the people accountable for its use would decide which results justified a release.

17:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:16 · section reference included

Transcript download
Full transcript

Loading transcript…