AI Engineer World's Fair 2024

LLM Evals That Work IRL

Aparna Dhinakaran· Co-founder & Chief Product Officer, Arize18:49

Loading slides…

Read the talk

LLM Evals That Work IRL

Evaluate the decisions inside an LLM application, turn explained failures into regression cases, and test whether judge scores and model benchmarks measure the behavior you need.

From a talk by Aparna Dhinakaran

Before you start: Familiarity with LLM prompts, function calling and retrieval-augmented generation will help you follow the examples.

Is the model good, or is the application working?

What does it mean to evaluate an LLM application in the real world? A model can rank well on a benchmark while the application built around it still fails its users. Aparna Dhinakaran starts with this distinction: model evaluations help you choose a model; task evaluations tell you whether your application does its job.

Model Evals vs Task Evals slide showing a leaderboard beside an AI observability interface, with a question beneath each.Open full source frame
Model evals ask which model to choose; task evals ask whether the LLM app is working.

Model comparisons use tools such as the Open LLM Leaderboard, MMLU and needle-in-a-haystack tests to assess capabilities. Application builders need another layer of evidence: whether their particular workflow produces the intended result. The approaches include LLM judges, user feedback and heuristic checks. An LLM-as-a-judge evaluation takes an application response, optionally includes the context behind it, and supplies an evaluation prompt that tells another model what to assess. The resulting judgment is only useful if that prompt asks about the behavior the application actually needs.

0:170:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Evaluate the route before the response

Consider a chat-to-purchase application. A customer asks about buying a new Kindle. Before answering, an intent router decides which workflow should handle the request, commonly through function calling. If it sends a product recommendation request to customer support, the application has already made a consequential mistake before generating its final response. Dhinakaran describes routers as one of the most common agentic workflows she sees in production.

The selected branch can contain more LLM calls, ordinary application calls, search and traditional machine learning models. Evaluation therefore needs to follow the structure of execution: assess the router’s choice, then assess the components within the branch. A single final-answer judgment may reveal that something failed without identifying which decision caused it.

Function calling introduces two separate correctness questions:

EvaluationQuestion
Function selectionDid the request reach the right function?
Parameter extractionDid that function receive the right arguments?

A product-details request should select the product-details function. But selecting that function is insufficient if the router extracts the wrong product identifier. Correct routing and correct arguments are separate requirements, and the evaluation should preserve that distinction.

Function call evaluation slide with two definitions, three example rows, separate function and parameter evaluation columns, and precision, recall and F1 results.Open full source frame
Function selection and parameter extraction receive separate evaluations.
2:152:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:15 · section reference included

A promotions request takes the wrong branch

The demonstration uses Phoenix, Arize’s open source observability and evaluation product. A trace exposes the user’s request, the application response and the calls made along the way. The customer asks whether there are current promotions for a Samsung phone. Among the recorded requests, Dhinakaran opens one flagged for an incorrect function call.

The evaluator’s explanation makes the failure concrete: the request concerns promotions, but the generated call performs a product search. A function dedicated to promotions and discounts exists in the application, yet the router does not select it. The problem is not merely an imperfect answer. The first decision sends the rest of execution down a branch that does not specifically address the customer’s intent.

5:506:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:50 · section reference included

Turn a trace failure into an experiment

Traces, evaluations and explanations answer different debugging questions. The trace shows what happened. The evaluation identifies a failed requirement. The explanation connects that failure to a possible repair. For the promotions request, that connection points back to routing rather than requiring the team to inspect every downstream response.

The improvement loop is straightforward:

  1. Add the failed request to an evaluation dataset.
  2. Change the relevant prompt or function description.
  3. Run an experiment against the accumulated cases.
  4. Compare results to see whether the change improves the targeted behavior.

Dhinakaran shows experiments involving a modified prompt and identifies function descriptions as another possible source of error. Keeping evaluations attached to individual components helps the team decide where to make the next change.

For a regression case with an explicit expected call, a small deterministic check can retain the distinction between function choice and arguments. This Python example represents the same Samsung promotions failure with illustrative function names and a simplified product argument:

python

case = {
    "request": "Are there current promotions for this Samsung phone?",
    "expected": {
        "name": "get_promotions",
        "arguments": {"product": "Samsung phone"},
    },
}
observed = {
    "name": "search_products",
    "arguments": {"product": "Samsung phone"},
}


def evaluate_call(expected, actual):
    function_correct = actual["name"] == expected["name"]
    arguments_match = actual["arguments"] == expected["arguments"]
    return {
        "function_correct": function_correct,
        "arguments_match": arguments_match,
        "call_correct": function_correct and arguments_match,
    }


result = evaluate_call(case["expected"], observed)

Matching argument values cannot rescue the wrong function choice. The saved expectation also gives a subsequent prompt experiment a concrete target; it does not imply that the application has already been repaired.

7:467:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:46 · section reference included

Carry evaluation through the application lifecycle

A chatbot with multiple exchanges needs evaluations at more than one scope:

  • Session: assess the conversation across turns.
  • Trace: assess a particular request’s execution.
  • Span: assess an individual operation within that execution.

These levels let a team move from an observed failure to the component responsible for it. The process begins by benchmarking evaluators during development, then refining evaluation templates alongside the application’s components. In production, evaluations can run as monitoring jobs. The templates develop with the system rather than appearing only after it is finished.

Three-column lifecycle slide listing golden datasets and eval templates, evaluating experiments, and production evaluations and performance monitoring.Open full source frame
LLM evaluations span benchmarking, development and production.

Dhinakaran’s strongest recommendation from deployment experience is to run evaluations with explanations. A label such as incorrect identifies a problem but gives little direction about what to fix. An explanation that identifies a mismatch between promotions intent and product-search routing gives the team a specific place to investigate. Its practical value is the bridge from a judgment to a repair.

9:259:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:25 · section reference included

A numerical score may hide the difference you need

Evaluators can return numerical scores, categorical labels, multiple outputs or multiclass judgments. A number looks convenient for comparison, but its apparent precision does not establish that it tracks severity. Dhinakaran recommends against relying exclusively on numerical judge outputs, drawing on Arize’s numeric-evaluation research.

The experiment corrupts documents with spelling errors and asks an LLM judge to assess how bad those errors are. In the spoken example, documents with approximately 80% and 11% spelling corruption both received a judge score of 10. A substantial difference in the input therefore disappears in the reported score.

Across the tested foundational models, Dhinakaran reports scores concentrated near the extremes rather than spread smoothly across the scale. She singles out Mistral as comparatively better. As corruption increased, the score did not reliably increase with it. The lesson is specific: if you need a severity measure, test whether the judge distinguishes severity. These results support caution about numerical outputs, not a claim that every numerical evaluation fails.

11:2211:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:22 · section reference included

Retrieval depends on context length and placement

The final part of the talk turns from application debugging to model evaluation. A needle-in-a-haystack test hides a fact inside a larger context and asks the model to retrieve it. Varying both context length and the fact’s insertion depth tests two different pressures. The illustrated placements put the fact near the first 5% of the context or around 90% of the way through it.

For retrieval-augmented generation, this becomes a practical context-assembly question: does placing important information lower in the supplied document change the answer? The displayed comparison uses Claude 2.1 and GPT-4. These are historical model results, not a ranking of current systems; newer Opus results are mentioned but not shown.

Read the heatmaps with context size on the horizontal axis and insertion depth on the vertical axis. Green marks successful retrieval and red marks failure. Dhinakaran reports stronger retrieval from GPT-4 in this comparison, while highlighting difficulty retrieving facts placed earlier in longer contexts. The useful engineering conclusion is to test placement under your own context lengths and prompts. The displayed failure pattern does not establish a universally best position for every RAG application.

Two side-by-side red and green retrieval heatmaps labeled Claude 2.1 and GPT-4, with outlined regions highlighting different failure patterns.Open full source frame
Needle-in-a-haystack results for Claude 2.1 and GPT-4.
13:4113:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:41 · section reference included

Finding the fact is only the first task

Retrieval success does not establish that a model can correctly use what it retrieved. The next experiments add generation tasks: rounding numbers from financial documents, mapping dates and concatenating strings. Each requires the model to find the relevant information and then transform it.

The model ordering changes. Dhinakaran describes GPT-4 as performing almost four times worse despite its stronger retrieval. The companion generation study narrows that comparison: Claude 2.1 achieved nearly four times GPT-4’s initial success rate on date mapping. That is a result for a particular transformation task and initial prompt, not an aggregate advantage across all generation tasks.

The intervention changes the instruction: ask the model to explain itself before answering. Dhinakaran attributes Claude’s initial advantage to its more verbose responses, interpreting those intermediate explanations as helping it work through the transformation. That is an interpretation of the observed behavior, rather than proof of an internal mechanism. When the team requests an explanation before the answer from GPT-4, its generation performance improves substantially. The prompt is part of the evaluated configuration: a retrieval benchmark alone cannot tell you whether a model will perform the subsequent transformation, or how that performance will change when you alter the instruction.

16:0816:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:08 · section reference included

Resources

From the talk

  • Tools and installation instructions for tracing LLM applications, evaluating outputs, building datasets, and comparing experiments. The repository reflects current software.

Transcript download
Full transcript

Loading transcript…