AI Engineer World's Fair 2024
LLM Evals That Work IRL
Loading slides…
Read the talk
LLM Evals That Work IRL
Evaluate the decisions inside an LLM application, turn explained failures into regression cases, and test whether judge scores and model benchmarks measure the behavior you need.
From a talk by Aparna Dhinakaran
Before you start: Familiarity with LLM prompts, function calling and retrieval-augmented generation will help you follow the examples.
Is the model good, or is the application working?
What does it mean to evaluate an LLM application in the real world? A model can rank well on a benchmark while the application built around it still fails its users. Aparna Dhinakaran starts with this distinction: model evaluations help you choose a model; task evaluations tell you whether your application does its job.
Model comparisons use tools such as the Open LLM Leaderboard, MMLU and needle-in-a-haystack tests to assess capabilities. Application builders need another layer of evidence: whether their particular workflow produces the intended result. The approaches include LLM judges, user feedback and heuristic checks. An LLM-as-a-judge evaluation takes an application response, optionally includes the context behind it, and supplies an evaluation prompt that tells another model what to assess. The resulting judgment is only useful if that prompt asks about the behavior the application actually needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the route before the response
Consider a chat-to-purchase application. A customer asks about buying a new Kindle. Before answering, an intent router decides which workflow should handle the request, commonly through function calling. If it sends a product recommendation request to customer support, the application has already made a consequential mistake before generating its final response. Dhinakaran describes routers as one of the most common agentic workflows she sees in production.
The selected branch can contain more LLM calls, ordinary application calls, search and traditional machine learning models. Evaluation therefore needs to follow the structure of execution: assess the router’s choice, then assess the components within the branch. A single final-answer judgment may reveal that something failed without identifying which decision caused it.
Function calling introduces two separate correctness questions:
| Evaluation | Question |
|---|---|
| Function selection | Did the request reach the right function? |
| Parameter extraction | Did that function receive the right arguments? |
A product-details request should select the product-details function. But selecting that function is insufficient if the router extracts the wrong product identifier. Correct routing and correct arguments are separate requirements, and the evaluation should preserve that distinction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A promotions request takes the wrong branch
The demonstration uses Phoenix, Arize’s open source observability and evaluation product. A trace exposes the user’s request, the application response and the calls made along the way. The customer asks whether there are current promotions for a Samsung phone. Among the recorded requests, Dhinakaran opens one flagged for an incorrect function call.
The evaluator’s explanation makes the failure concrete: the request concerns promotions, but the generated call performs a product search. A function dedicated to promotions and discounts exists in the application, yet the router does not select it. The problem is not merely an imperfect answer. The first decision sends the rest of execution down a branch that does not specifically address the customer’s intent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a trace failure into an experiment
Traces, evaluations and explanations answer different debugging questions. The trace shows what happened. The evaluation identifies a failed requirement. The explanation connects that failure to a possible repair. For the promotions request, that connection points back to routing rather than requiring the team to inspect every downstream response.
The improvement loop is straightforward:
- Add the failed request to an evaluation dataset.
- Change the relevant prompt or function description.
- Run an experiment against the accumulated cases.
- Compare results to see whether the change improves the targeted behavior.
Dhinakaran shows experiments involving a modified prompt and identifies function descriptions as another possible source of error. Keeping evaluations attached to individual components helps the team decide where to make the next change.
For a regression case with an explicit expected call, a small deterministic check can retain the distinction between function choice and arguments. This Python example represents the same Samsung promotions failure with illustrative function names and a simplified product argument:
python
case = {
"request": "Are there current promotions for this Samsung phone?",
"expected": {
"name": "get_promotions",
"arguments": {"product": "Samsung phone"},
},
}
observed = {
"name": "search_products",
"arguments": {"product": "Samsung phone"},
}
def evaluate_call(expected, actual):
function_correct = actual["name"] == expected["name"]
arguments_match = actual["arguments"] == expected["arguments"]
return {
"function_correct": function_correct,
"arguments_match": arguments_match,
"call_correct": function_correct and arguments_match,
}
result = evaluate_call(case["expected"], observed)
Matching argument values cannot rescue the wrong function choice. The saved expectation also gives a subsequent prompt experiment a concrete target; it does not imply that the application has already been repaired.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry evaluation through the application lifecycle
A chatbot with multiple exchanges needs evaluations at more than one scope:
- Session: assess the conversation across turns.
- Trace: assess a particular request’s execution.
- Span: assess an individual operation within that execution.
These levels let a team move from an observed failure to the component responsible for it. The process begins by benchmarking evaluators during development, then refining evaluation templates alongside the application’s components. In production, evaluations can run as monitoring jobs. The templates develop with the system rather than appearing only after it is finished.
Dhinakaran’s strongest recommendation from deployment experience is to run evaluations with explanations. A label such as incorrect identifies a problem but gives little direction about what to fix. An explanation that identifies a mismatch between promotions intent and product-search routing gives the team a specific place to investigate. Its practical value is the bridge from a judgment to a repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A numerical score may hide the difference you need
Evaluators can return numerical scores, categorical labels, multiple outputs or multiclass judgments. A number looks convenient for comparison, but its apparent precision does not establish that it tracks severity. Dhinakaran recommends against relying exclusively on numerical judge outputs, drawing on Arize’s numeric-evaluation research.
The experiment corrupts documents with spelling errors and asks an LLM judge to assess how bad those errors are. In the spoken example, documents with approximately 80% and 11% spelling corruption both received a judge score of 10. A substantial difference in the input therefore disappears in the reported score.
Across the tested foundational models, Dhinakaran reports scores concentrated near the extremes rather than spread smoothly across the scale. She singles out Mistral as comparatively better. As corruption increased, the score did not reliably increase with it. The lesson is specific: if you need a severity measure, test whether the judge distinguishes severity. These results support caution about numerical outputs, not a claim that every numerical evaluation fails.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieval depends on context length and placement
The final part of the talk turns from application debugging to model evaluation. A needle-in-a-haystack test hides a fact inside a larger context and asks the model to retrieve it. Varying both context length and the fact’s insertion depth tests two different pressures. The illustrated placements put the fact near the first 5% of the context or around 90% of the way through it.
For retrieval-augmented generation, this becomes a practical context-assembly question: does placing important information lower in the supplied document change the answer? The displayed comparison uses Claude 2.1 and GPT-4. These are historical model results, not a ranking of current systems; newer Opus results are mentioned but not shown.
Read the heatmaps with context size on the horizontal axis and insertion depth on the vertical axis. Green marks successful retrieval and red marks failure. Dhinakaran reports stronger retrieval from GPT-4 in this comparison, while highlighting difficulty retrieving facts placed earlier in longer contexts. The useful engineering conclusion is to test placement under your own context lengths and prompts. The displayed failure pattern does not establish a universally best position for every RAG application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finding the fact is only the first task
Retrieval success does not establish that a model can correctly use what it retrieved. The next experiments add generation tasks: rounding numbers from financial documents, mapping dates and concatenating strings. Each requires the model to find the relevant information and then transform it.
The model ordering changes. Dhinakaran describes GPT-4 as performing almost four times worse despite its stronger retrieval. The companion generation study narrows that comparison: Claude 2.1 achieved nearly four times GPT-4’s initial success rate on date mapping. That is a result for a particular transformation task and initial prompt, not an aggregate advantage across all generation tasks.
The intervention changes the instruction: ask the model to explain itself before answering. Dhinakaran attributes Claude’s initial advantage to its more verbose responses, interpreting those intermediate explanations as helping it work through the transformation. That is an interpretation of the observed behavior, rather than proof of an internal mechanism. When the team requests an explanation before the answer from GPT-4, its generation performance improves substantially. The prompt is part of the evaluated configuration: a retrieval benchmark alone cannot tell you whether a model will perform the subsequent transformation, or how that performance will change when you alter the instruction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Tools and installation instructions for tracing LLM applications, evaluating outputs, building datasets, and comparing experiments. The repository reflects current software.
Further reading
Arize's 2024 spelling and emotional-content experiments examine score calibration, prompt sensitivity, and variation across repeated judgments.
Explains context-length and insertion-depth testing, random-number retrieval, and the effects of changing prompts.
Experiments with date mapping, string concatenation, money formatting, and arithmetic show how transformation tasks respond to prompt changes.
A historical account of how a relevant-sentence response prefix changed Claude 2.1's results on a long-context retrieval test.
Full transcript
Loading transcript…