AI Engineer World's Fair 2025
Building Metrics That Actually Work — David Karam, Pi Labs
Loading slides…
Read the talk
Building Metrics That Actually Work
Turn application requirements into inspectable scoring signals, calibrate them against user feedback, and reuse the resulting scorer for model comparisons, prompt changes, and response selection.
From a talk by David Karam
Before you start: Basic familiarity with LLM prompts, JSON, and Python is helpful; following the original hands-on exercises also requires a Google account and familiarity with Colab.
What counts as a correct answer?
A tax-research answer can be correct in several ways. A business question translated into SQL creates a similar problem: how do you define success, then improve the system without repeatedly writing unit tests, reviewing results, and starting again? That is the opening problem raised by an attendee in David Karam’s workshop. The difficulty begins before choosing an evaluation tool: the team must decide what it wants the tool to measure.
For agents, evaluation also supplies feedback from the world. An experiment needs an observable result that distinguishes success from failure; an agent needs the same thing. Yet each application brings its own subjective requirements and probabilistic behavior, making evaluation work difficult to reuse. One attendee estimates that testing consumed 30% of feature-development time in their earlier QA work, versus 80% for evaluation in their AI work. Those figures describe that attendee’s experience, but the underlying frustration is shared: building the evaluator can become most of the job.
Novel applications introduce another gap: there may be no real-world dataset to use as a baseline. Here, test examples and metrics are separate design problems. Synthetic data supplies cases to exercise; a metric checks what happened on those cases. Generating convincing examples does not establish that the checker is accurate. Another attendee has already tried AI-based judging and found its judgments unreliable enough to require manual review again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evaluation part of development
The presenters bring experience from more than a decade at Google, including work on Search, where stochastic behavior made quality measurement central to development. The recurring process was to establish a benchmark, improve until it stopped exposing useful differences, then move to a harder benchmark. Metrics were calibrated against human judgments and user data along the way. The workshop applies that methodology to language-model applications; it does not promise a single evaluation recipe that finishes the job.
Karam recalls roughly 300 Search metrics, challenging teams that think only in terms of a handful of generic scores. The point is the breadth of quality requirements an application can accumulate. Some teams first need to define metrics; others already have them and need to connect those scores to thumbs-up/down feedback. That connection carries evaluation out of the test suite and into the online product’s feedback loop.
The workshop’s shared Google Doc collects the exercise steps, code, spreadsheet, and slide deck; the presenters say it will remain available afterward. The accompanying Slack channel is workshop-metrics, intended for questions during and after the exercises.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with cheap signals, then add coverage
Live testing—trying the application, inspecting its responses, and changing prompts—can take a simple product surprisingly far. Multistep agents create more opportunities for failure, and eventually a developer needs checks that can run repeatedly without inspecting every trace. The available methods serve different purposes:
| Method | Useful contribution | Main constraint |
|---|---|---|
| Live testing | Fast discovery and prompt iteration | Requires direct inspection |
| Human evaluation | Expert judgment | Rater time and cost |
| Code checks | Verifiable properties | Only covers what can be specified |
| Natural-language judging | Semantic requirements | Consistency of judgments |
The presenters argue that generative decoder models, built to produce varied responses, can be awkward tools for consistent scoring. Their proposed next step is to combine multiple simpler signals into a scoring system.
Begin with five to ten easily derived signals that you have reason to associate with good results. Test the application, inspect failures, revise the application or its metrics, and measure again. The initial set does not need to anticipate every failure; it needs to make the next development decision more informed.
Karam treats these methods as a toolbox. Start with live testing and trace monitoring, then add sophistication as the system grows and manual inspection becomes impractical. Each additional technique has a cost and a return. A scoring system emerges by layering useful checks, rather than by replacing every earlier method at once.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put domain knowledge in the scorer
A trusted evaluator makes domain knowledge reusable. Once it can distinguish better and worse outputs, it can guide meta-prompts or optimizers such as DSPy, and filter synthetic data for fine-tuning or reinforcement learning. The scoring definition becomes an input to improvement, rather than merely a final acceptance test.
The same scorer can operate at inference time. Increase temperature, generate four or five candidate responses, score each one, and return the best. The presenter calls this online reinforcement learning; the procedure described is more specifically best-of-N sampling, which selects among responses without updating model parameters. It can change which answer reaches the user while leaving the prompt and model unchanged.
Generic helpfulness, harmfulness, and hallucination checks do not capture every product goal. A trip plan can satisfy basic guardrails and still be bland. If the application is supposed to make someone excited about a destination, that requirement needs a place in the evaluation. This is where application-specific judgment becomes valuable: the scorer must express what makes this particular product good.
The exercise therefore starts with a small scoring system and tests it against good, bad, and synthetic examples. Participants revise it in a copilot, then move it into a Python Colab notebook. A spreadsheet provides another interface for changing criteria and inspecting results without editing the whole workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decompose subjective quality into inspectable signals
Search offers a useful analogy: score documents, then rank them to decide which result to show. That score combines signals such as popularity, title quality, content quality, feasibility, spam, and clickbait. Each signal answers a narrower question than whether the whole document is good. A scoring system for generated content can use the same decomposition.
At the bottom of the system, a signal may be an objective property checked with deterministic Python. Higher up, signals combine into a subjective judgment about the output. Keeping those layers visible makes the score inspectable: a developer can examine a failed criterion instead of trying to interpret a single opaque number from a complicated prompt.
Coverage matters as much as aggregation. Search cares about relevance, site quality, popularity, spam, and adult-content intent because each can affect the result a user should receive. Asking only whether an answer is helpful delegates the definition of helpfulness to the judge. Decomposition gives the application developer more control over that definition. Karam attributes lower variance and finer-grained analysis to measuring narrower, more objective properties. As failures reveal missing requirements, add signals that expose those failures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a scorer for meeting summaries
The demonstration uses a meeting summarizer. Its input is a transcript of several people talking; its output is structured JSON containing a summary, action items, key insights, and a title. This is small enough to inspect directly, but it already mixes structural requirements with semantic ones.
The copilot can begin from three kinds of material:
- System prompt: Derive scoring dimensions from the application’s instructions.
- Explicit criteria: Start with requirements the developer has already identified.
- Input/output examples: Infer dimensions from concrete cases.
A reasoning model decomposes broad requirements and displays the resulting scoring artifacts beside the chat. Karam notes that the same iterative decomposition can also be attempted in a preferred chat interface. Asked whether the process can work backward from examples, the presenter shows an Example entry point for collections of cases; a few examples can also be pasted directly into the prompt. The resulting criteria are a starting point for revision.
Each dimension is a signal. One asks whether the output includes insights from the meeting. Others can contain editable Python. Natural-language questions go to specialized scoring models, while the code expresses properties that can be checked directly. The importance labels critical, major, and minor control how much dimensions matter. The presenter explains that the combination function can eventually be learned from examples; this exercise gives participants direct control over the weights.
After developing the scorer with synthetic examples, participants apply it to a separately labeled set of meeting summaries carrying real thumbs-up/down feedback. Agreement is not guaranteed: it depends on the criteria they built. Karam frames the distinction as calibrated versus uncalibrated metrics. Correlation analysis and confusion matrices help answer the operational question: when this scorer gives a high score, does that correspond to an output users consider good? Decomposition makes disagreements easier to investigate, but it does not remove the need to check them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Break examples deliberately
The first probe is to generate a synthetic example from the system prompt and score it dimension by dimension. Next, the presenter asks for a bad example. The copilot uses its accumulated context to create an output missing information, then generates another with broken JSON. These targeted defects let a developer check whether the expected dimension notices the failure.
The same interface can edit the checker: change Python, remove a dimension, revise a question, or add a requirement. The concrete addition in the demonstration is a title containing fewer than twenty words. The underlying structural checks are straightforward to express in Python:
python
import json
def structural_checks(output: str) -> dict[str, bool]:
try:
document = json.loads(output)
except json.JSONDecodeError:
return {"valid_json": False, "title_under_20_words": False}
title = document.get("title") if isinstance(document, dict) else None
return {
"valid_json": True,
"title_under_20_words": (
isinstance(title, str) and len(title.split()) < 20
),
}
output = '{"title": "Launch planning", "action_items": []}'
checks = structural_checks(output)
For this example, JSON validity and title length are separate signals. Neither establishes whether the summary preserved the meeting’s meaning; that requires other dimensions.
An audience question extends the idea to changing product scope: what happens when a trip planner also starts making bookings? Supply the changed requirements and new examples, then request dimensions that test the added behavior. The human-guided copilot is useful for this kind of inspection, while Colab can send larger datasets through longer-running jobs. The presenter describes alternating between those modes—periodically pasting in user examples to inspect the scorer, while clients also run substantial batch processes. The demo finishes this revision with the new title-length dimension in place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare scores with user feedback in Sheets
The next step exports the criteria into a copy of the prefilled workshop spreadsheet. The presenter briefly mentions an Unsloth reinforcement-learning integration as another destination for Pi Scorer, but the hands-on path here is Sheets: copy the criteria from the copilot and replace the sheet’s defaults.
The presenter describes the spreadsheet exercise as scoring about 120 examples with user thumbs-up/down labels. Its confusion matrix separates agreement on positive feedback, agreement on negative feedback, and the two directions of disagreement. Participants can revise dimensions and observe whether alignment improves.
The walkthrough makes the procedure concrete:
- Replace the default criteria with your own labels, questions, weights, and Python checks.
- Inspect the data sheet, which contains input/output pairs and recorded user feedback across mixed difficulty levels.
- Select the input/output ranges and invoke
Score selected rangesunderExtensions. - Inspect the resulting confusion matrix, then change criteria or deliberately damage an output to see how its score responds.
For example, corrupting JSON should affect the structural dimension. If a damaged response still looks acceptable in the aggregate, the dimension or its importance needs investigation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose where evaluation runs
An attendee asks how teams evaluate at scale: every example, or a subset? The presenter points to the Python Colab, SDK guidance, and batch workflows. Pi’s scorers are described as designed for low-latency online use, although the spoken latency estimate is too imprecise to establish a usable performance target.
For offline sets, mix easy, medium, and hard examples so the benchmark covers routine behavior as well as difficult cases. More extensive synthetic-data generation is left to follow-up documentation. For live applications, sample production logs or run scoring directly online. The presenter uses Google’s online spam checks and decision systems as examples, and identifies the cost of conventional LLM judging as an obstacle to applying it on every request. The placement of the scorer therefore depends on both the feedback needed and the cost of obtaining it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A scorer built to return numbers
Asked how the system differs from LLM-as-a-judge, the presenter starts with repeatability: Pi is described as returning the same score for identical inputs and remaining stable under small variations. The architectural explanation is bidirectional attention with a regression head, rather than autoregressive generation of score tokens. This also avoids the pattern of generating a score and then generating a post-hoc justification for it.
The presenter says the models were trained on billions of tokens dedicated to scoring varied content, including code. The intended interface is simple: supply a question and the data, receive an inspectable score. The scoring model is expected to understand the criterion without a separate cycle of judge-prompt tuning. These are the presenter’s descriptions of the system’s design and training.
Individual scores then feed an aggregation model described as an extension of a generalized additive model. Calibration uses thumbs-up/down examples to learn how signals should influence the result. Crucially, a signal’s effect can be asymmetric: spam can make the entire response unacceptable, while the absence of spam need not earn positive credit. This is richer than treating every successful check as another point in a sum.
The presenter attributes the scorer’s speed to denser embeddings and fewer parameters enabled by bidirectional attention. At the time of the workshop, the stated language support was English and a handful of other languages; broader multilingual support was forthcoming, and multimodal support remained on the roadmap.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the same specification into Python
The final walkthrough moves into a prepared Colab that participants can finish independently. After the installation cells, the central object is the scoring specification. Copy it from the copilot’s code view into the notebook: it contains natural-language criteria with Python checks where needed. This is the definition to edit as requirements change, and the notebook can either be run cell by cell as supplied or modified.
The notebook loads the public Hugging Face dataset described in the workshop as containing the same thumbs-up/down examples used in Sheets. It runs the scorer, collects scores and predicted ratings, and constructs a corresponding confusion matrix. The displayed evaluation output puts the feedback, transcript, extracted output, score, and predicted rating together, making the Python workflow a continuation of the spreadsheet exercise.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare models and catch prompt regressions
With the scorer established, the notebook uses it to compare generators. The presenter reports that the model labeled 2.5 scores slightly higher than 1.5 on this meeting-summary task. The model family and exact variants are not identified in the narration. The presenter attributes the small gap to the task’s limited reasoning demands, and suggests that a smaller model such as Claude Haiku would expose a larger difference. The model-comparison demonstration uses about ten examples and five models, generating responses and scoring them with the participant’s criteria. Its result is a task-specific comparison under that scorer.
Prompt comparisons reuse the same structure: hold the scoring specification and test set fixed, change the system prompt, and inspect whether scores decline. On ten examples, the deliberately bad prompt receives lower scores than the good prompt in the demonstration. The contrast is intentionally pronounced to show how the workflow detects regressions before a prompt change is adopted.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the score to select the response
The last exercise holds one meeting transcript fixed and varies the number of generated responses. The single-response baseline uses temperature 0.7. With more samples, the system produces several distinct summaries, ranks them using the scoring system built during the workshop, and returns the highest-scoring candidate.
The selection step is small enough to keep separate from both generation and scoring:
python
from collections.abc import Callable, Sequence
def select_best_summary(
transcript: str,
candidates: Sequence[str],
score: Callable[[str, str], float],
) -> tuple[str, float]:
if not candidates:
raise ValueError("At least one candidate is required")
scored = [
(candidate, score(transcript, candidate))
for candidate in candidates
]
return max(scored, key=lambda item: item[1])
All candidates receive the same transcript and scoring function. The operation chooses an existing response; it does not rewrite the summary or train the generator.
The presenter reports rising scores as sample count increases in this single-transcript exercise, interpreting the change as better response quality. The demonstrated measurement is the scorer’s own output, rather than a separate user-quality measurement. That makes calibration consequential: once a score determines which answer the user receives, its definition of quality becomes part of the application’s behavior. The prepared Colab leaves participants with this complete generate-score-select workflow to run and explore after the workshop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Framework for building and optimizing language-model programs against task-specific metrics.
Further reading
Configuration guide for evaluating input/output pairs with Pi scoring criteria. Requires a separate Pi API key.
Full transcript
Loading transcript…