AI Engineer World's Fair 2024
Judging LLMs
Loading slides…
Read the talk
Judging LLMs: From Prompt Regressions to Evaluating the Evaluators
A chat-with-PDF prototype becomes a production problem when prompt edits break existing features. Tracing, layered evaluation, and application-specific criteria make those failures easier to catch.
From a talk by Alex Volkov
Before you start: Basic familiarity with LLM prompts and Python is helpful; no prior experience building evaluations is required.
When a prototype starts charging customers
Who judges whether an LLM application is ready for people to depend on it? The courtroom opens with Maximus, a fictional AGI judge visiting from 2034. Alex Volkov, an AI evangelist at Weights & Biases, interrupts the performance to establish its governing rule: an LLM judge needs a human in the loop. The judge’s authority is part of the joke, not evidence that its judgments deserve trust.
The first case makes the engineering problem concrete. Daniel builds a chat-with-PDF wrapper at a hackathon and sends it into the world without much prompt work. That earns an acquittal: experimenting, learning, and meeting people are reasonable goals for a prototype. Then the demo attracts attention on Hacker News, Daniel starts charging, and customers request more features. He repeatedly changes the prompt and deploys on a Friday. Existing customers discover that their old workflows no longer work. When Daniel tries to investigate, he finds that he has neither logs nor traces.
A prompt change is a production change when customers depend on the behavior it controls. Daniel’s problem is now both a regression and a lack of diagnostic evidence. The guilty verdict follows the discovery that he cannot reconstruct what happened. Tracing provides the record needed to investigate failures after a release.
Weights & Biases Weave supplies the demonstration: Python function instrumentation through a decorator leads into a dashboard of application calls. From an interaction, a developer can inspect the nested call hierarchy of a RAG application or agent. The trace connects code versions and parameters—including temperature and system prompts—with inputs, outputs, and messages across a conversation. Markdown, JSON, and code rendering make those records easier to inspect. The useful unit is more than a final answer: it is the sequence of calls and configuration that produced it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the prompt and application before training
Sasha’s case starts with abundant GPUs and a corporate feature request. She downloads Llama 3, fine-tunes it on company data, and celebrates an improved internal benchmark. In the fictional case, fine-tuning produces a 6% improvement on unspecified internal benchmarks at a learning rate of 5e-6. That number does not establish whether training was the right first intervention: no task, baseline, or definition of the improvement accompanies it. The human commentator asks the missing question—did she try changing the prompt first?
The objection is not to fine-tuning itself. Volkov pauses to promote W&B Models and its use by model builders, then names OpenAI, Mistral, Together AI, Axolotl, and Hugging Face training integrations. These are his contemporary promotional claims, including the claim of exclusive native integration, rather than a current integration comparison. Returning to Sasha’s case, the judge discovers that she skipped both prompt iteration and a RAG pipeline before spending the compute.
Establish what prompt and application changes can achieve before committing to fine-tuning. Volkov’s alternatives include chain-of-thought prompting, flow engineering, DSPy, and mixture-of-agents methods. These offer different places to intervene: the instructions, the arrangement of model calls, or the coordination of multiple agents. The sequence matters more than any single technique. Fine-tuning remains an option after exploring those approaches; Sasha’s mistake is treating it as the starting point.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep model choices connected to the changing landscape
Morgan takes the opposite route to complexity: overwhelmed by AI news, he stops following it and uses Llama 2 7B for everything. The judge initially condemns him, but the defense is recognizable. In the months preceding this June 2024 talk, announcements included Claude 3.5 Sonnet, Llama 3, GPT-4o, Gemini Flash, Project Astra, and Apple Intelligence. An engineer with an application to build and meetings to attend cannot spend every day tracking releases.
Volkov offers his weekly show and podcast, ThursdAI, as a way to delegate that monitoring. Morgan’s comic sentence becomes attendance, subscriptions, reviews, and sharing the show with friends. Those are promotional jokes, not engineering requirements. The practical tension remains: reducing news overload should not mean permanently freezing model selection around one older model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a dataset, run the model, score the answers
The next case shifts from model selection to release confidence. In a dramatized Air Canada scenario, an executive named Francisco rushes a customer-support chatbot into production to reduce costs. He selects assertions because they resemble familiar unit tests and appear quickest to implement. The story invokes a lost legal dispute and lands its joke on the customer becoming the human in the loop. The executive, internal testing decisions, and implied causal chain belong to the dramatization, not an established account of the airline’s engineering process. Its warning is about discovering consequential failures through customers after release.
An evaluation makes expectations explicit before that happens:
- Compile representative inputs. Pair each input with a reference answer or criteria describing an acceptable answer. Criteria matter when several different answers could all be correct.
- Draw examples from actual use. Include cases encountered during development and recorded production interactions.
- Run the model on each example. Evaluate the production model or a proposed replacement against the same dataset.
- Score the resulting answers. Compare them with the references or judge them against the predefined criteria.
This separates producing an answer from deciding whether it meets the application’s requirements. It also gives model changes a common basis for comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use code for properties that code can check
Programmatic grading starts with explicit, executable conditions. If an answer should be a number, compare the output with the expected number. If a response must avoid a stock phrase such as As an AI model, test for that phrase. If the output is code, run tests of its behavior; HumanEval is the named example of evaluating generated code for functional correctness.
A small Python scorer can keep numerical correctness and unwanted phrasing separate. For an illustrative arithmetic input, the reference answer is 4; an answer containing the unwanted preamble should fail that check even if it includes the right number.
python
def score_answer(output: str, expected: float) -> dict[str, bool]:
no_preamble = "as an ai model" not in output.casefold()
try:
correct_number = float(output.strip()) == expected
except ValueError:
correct_number = False
return {
"correct_number": correct_number,
"no_preamble": no_preamble,
}
case = {"input": "What is 2 + 2?", "expected": 4}
answers = ["4", "As an AI model, I think the answer is 4."]
for answer in answers:
print(answer, score_answer(answer, case["expected"]))
The scorer deliberately requires a numerical output, not merely a correct number embedded somewhere in prose. Its two results expose which requirement failed instead of hiding both behind a single grade. Such checks are inexpensive and easy to scale in Volkov’s account. They can test explicit properties of conversational responses, but assertions alone do not establish the overall quality of a multi-turn conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale review without removing human judgment
Human evaluation already happens whenever an engineer tries an application and adjusts its prompt. Making that evaluation systematic means starting early, involving teammates, and continuing after deployment. Production is not the point at which looking at answers stops being useful.
The difficulty is repetition and cost. Dedicated graders may need to read thousands of conversations, including many that are tedious. They need explicit criteria to grade consistently, and they need recorded interactions to review in the first place. This returns tracing to the center of the workflow: it supplies the material for evaluation, not just incident diagnosis.
LLM-as-a-judge grading delegates part of that reading. A judge can process a long exchange, summarize it, and assess whether it satisfies supplied criteria. But the criteria do not define themselves, and the judge is not automatically reliable. Humans still need to inspect biases and refine the judge’s system prompt and examples. Volkov describes this as the most cost-effective grading approach; that is his qualitative assessment, without a measured cost comparison. The proposed benefit is scalable review with supervision, not a perfect substitute for it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine the methods, then inspect the judge
Maxim’s staged success story brings the three methods together. Faced with a contractor’s proposal for a custom enterprise tracing and evaluation system, he chooses Weave and gets tracing working within minutes in the story. He then builds a pipeline combining programmatic checks, human review, and LLM grading. Continuous evaluation lets him experiment with system prompts while checking for regressions.
| Grading method | Useful role | What it still needs |
|---|---|---|
| Programmatic | Check explicit conditions | Coverage beyond those conditions |
| Human | Review behavior in context | Criteria and reviewer time |
| LLM judge | Scale criteria-based review | Human validation and bias checks |
The layers serve different purposes rather than competing for a single winner. But the success story itself then becomes suspicious: the judge declares Maxim the King of Unicorns, denies any financial interest or prompt jailbreak, and celebrates his promotion. That exaggerated praise sets up a failure in the evaluator.
Volkov interrupts to identify apparent prompt injection and remind the audience to check judges for bias. This is a staged illustration of a compromised verdict, not a reproduced attack. His instruction to validate the validators points to Shreya Shankar’s Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. An automated grade needs scrutiny just as an application answer does.
Judge validation also depends on application-specific criteria. Volkov points ahead to Hamel Husain’s discussion and warns against relying on generic criteria that do not capture what the business needs. Only the people responsible for the application can establish what success means for its users. A polished answer can still fail the task it was supposed to perform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect generated interfaces, not just aggregate grades
OpenUI, the open-source project by W&B co-founder Chris Van Pelt, makes the criteria concrete. The demonstration generates HTML as the model streams its response. Weave records the calls, allowing Van Pelt to inspect the behavior as a human reviewer while also running evaluations. The application produces a visible interface, so its evaluation needs to connect model calls with what those calls actually render.
The custom criteria are contrast, relevance, and polish. The evaluation view compares versions labeled 16 and 14, and the demonstration uses GPT-3.5 Turbo. These are the historical model and version labels shown in the talk, not recommendations for a new deployment. From the comparison, Van Pelt can open individual examples and inspect their criterion results.
The next step is to render the actual outputs inside the evaluation workflow. Multimedia support lets the reviewer see the generated interface rather than infer its quality solely from HTML or a score. This joins three useful views of an experiment: the version comparison, the individual graded example, and the output a user would experience. A runner and visualization tool support faster iteration when they make that movement easy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn expectations into a rubric
To practice defining criteria, use the task already in front of the audience: judging a conference talk. Start with the expectations behind an informal reaction such as liking or disliking it, then name the dimensions separately:
- Memorable: Does the talk leave something the audience will remember?
- Educational and helpful: Does it teach something useful?
- Funny and original: Does it bring humor or a distinctive approach?
- Clear and articulate: Can the audience follow the explanation?
- Delivery and presentation: Does the performance communicate effectively?
- Promotional: How much of the talk is advertising?
These are criteria for this task. The transferable method is to make the task’s expectations explicit, not to reuse a conference rubric unchanged for every application.
The final case applies that rubric to Volkov himself. Maximus questions the talk’s educational value, the humor, and the repeated interruptions. The judge jokingly assigns the talk a 70% promotional score, without a measurement procedure. For memorability, however, there is concrete evidence: Volkov wore a wig onstage. The verdict is “overcommitment to the bit.” The ending gives the evaluator something observable to point to while keeping its numerical score firmly inside the comedy.
The recording closes with an invitation to get help at the W&B booth and begin with Weave:
bash
pip install weave
Volkov finishes by inviting the audience to ThursdAI and thanking them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and current setup instructions for tracing and evaluating LLM applications.
Generate and revise UI components from natural-language descriptions, with live rendering and local setup instructions.
Introduces EvalGen and examines how human feedback aligns automated evaluators with evolving evaluation criteria.
Functional-correctness evaluation harness for generated code. Running generated programs requires appropriate sandboxing.
Framework for composing language-model programs and optimizing their prompts and weights.
AI news coverage through a live show, podcast and newsletter.
Further reading
- Your AI Product Needs EvalsArticle
Hamel Husain's practical guide to domain-specific tests, trace review, human feedback and automated evaluation.
A test-driven, iterative approach to code generation evaluated on competitive programming problems.
Research on improving responses through multiple layers of collaborating language-model agents.
Updates since the talk
Current walkthrough for defining a dataset, model and scorers, running an evaluation, and inspecting results.
Full transcript
Loading transcript…