How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Loading slides…
Read the talk
How Netflix evaluates AI when there’s no single right answer
Netflix staff research scientist Cameron Wolfe explains how to turn expert judgment into a repeatable evaluation, measure how much a model’s score can fluctuate, and test whether an offline gain carries through to users.
From a talk by Cameron Wolfe
At a glance
Ideas worth remembering
Collect multiple human ratings on the same output. Investigate the disagreements, revise the scoring examples and use the reviewers’ eventual agreement as a benchmark for the judge.
Choose evaluation tasks by profiling what competing models can solve. Size the sample around the difference worth detecting and the variation observed across attempts.
Resample agent rollouts and judge ratings separately, and report uncertainty alongside each benchmark score.
Advance promising offline results through human review and a limited A/B test. Use failures found after rollout to decide whether the benchmark needs new tasks, clearer criteria or better calibration.
One evaluation trial runs an agent and grades its solution
Give a creative-writing agent the same assignment twice and it can produce two different pieces. Suppose a new version earns a higher score on that assignment. How much credit should go to the new model, and how much to the particular piece it happened to write? Cameron Wolfe, a staff research scientist at Netflix and author of the Deep (Learning) Focus newsletter, starts with this problem of noise. It arises even on tasks where everyone agrees on the right answer.
An evaluation suite contains a set of tasks. For each task, the agent produces a rollout, the course of one attempt, that eventually yields a proposed solution. A grader takes that solution and returns a result, which in the simplest case is pass or fail. The benchmark score summarizes what happened across those trials. Repeat the exercise with the same agent and a different set of attempts can produce a different pass rate.
Cost limits how much of that variation a team can average away by testing more tasks. Wolfe puts the cost of running some recent benchmarks at roughly $50,000 to $100,000. A team may consequently have to draw its conclusion from a relatively small set. A handful of tasks on which the agent happened to succeed or fail can then have appreciable influence on the reported difference between models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Creative writing adds a second source of variation
Many benchmarks concentrate on verifiable tasks. String matching can check whether a response matches an expected answer. Symbolic verification can check a mathematical solution, including whether two expressions are equivalent. Test cases let a grader execute checks against a proposed solution. In each case, a function implements the decision. An agent might arrive at a different answer on its next attempt, but the grader can apply the same check to either one.
Wolfe places real assignments on a spectrum of verifiability. At the creative-writing end, many quite different pieces could satisfy a prompt, and experienced readers can disagree about which ones succeed. An LLM judge can read a piece and assess its quality, either using general guidelines or checking the requirements of that particular assignment. Now hold the piece fixed and run the evaluation again. The judge itself can return a different assessment. Both the choice of completion and the grading of that completion contribute to the observed score.
Wolfe organizes the work into three parts. Human evaluation establishes what quality means for the task. Offline evaluation gives the team an affordable way to apply that definition across many experiments. Production monitoring supplies evidence from people actually using the system. The order of construction matters. Before a team can decide whether an automatic grader is doing a good job, it needs examples of judgments it has reason to trust.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Write criteria that two reviewers can apply to the same output
Human evaluation is Wolfe’s North Star for defining, measuring and improving quality. He starts with a small group of subject-matter experts and asks them to identify a few things they care about in the finished solution. For each criterion, the team has to specify what the reviewer should look for and how to record the judgment. A binary rating fits a decision about whether an output meets a requirement. A Likert scale lets a reviewer assign an ordered rating when the degree to which the output meets the criterion matters.
The instructions need enough detail to support the decision a reviewer faces with a piece of work in front of them. Wolfe calls for specific examples of how to rate each criterion. On a scale, a reviewer needs a basis for choosing one rating over another. On a binary check, they need to know what qualifies as meeting the requirement. Examples give the experts a way to make those distinctions concrete before the guidelines go into wider use.
Calibration begins by collecting multiple human ratings for the same data. Holding the output fixed makes a disagreement worth investigating. Ask the reviewers how they reached their decisions. Two readers may agree on what the piece contains yet interpret a criterion differently. Their explanations tell the team where to revise a definition or add a scoring example. Rate common examples again and measure agreement. The result of this iteration is both a clearer guide and an observed level of consistency among the people applying it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Size the sample for the improvement you need to detect
Once reviewers can apply the guidelines consistently, the next decision is how much work to send them. Wolfe frames this around a comparison between two models. What difference in performance would matter, and how many examples would the team need to distinguish a difference of that size from ordinary fluctuations? Power analysis gives the team a way to estimate that volume before committing to a round of human evaluation.
To size the evaluation, the team specifies the smallest performance gap it wants to detect and estimates the noise in its scores. Sampling several completions for the same task helps estimate how much results fluctuate between attempts. The team also chooses a significance level, which limits the risk of reporting a difference when none exists, and a target power, the chance of detecting a gap of the chosen size when it exists. With the same noise, significance level and target power, detecting a smaller improvement requires a larger sample.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Profile model attempts to find tasks that distinguish candidates
Which tasks enter the sample is a separate decision. Wolfe looks for diversity and realism together. The evaluation should cover different kinds of work and reflect what the agent will be asked to do in production. A team could collect many examples of one familiar assignment and estimate performance on it quite precisely while learning little about the other assignments its users bring.
Wolfe also looks for tasks that are difficult and discriminative. To assess that property, profile the candidates by sampling rollouts and inspecting which kinds of problems each can solve. A very hard problem that defeats every candidate yields the same result for all of them. A problem that some candidates solve more reliably than others can reveal a capability difference. Difficulty alone gives the benchmark designer little basis for telling those cases apart.
The examples also need to be accurate. Wolfe describes improving the collection over time through a combination of model-based and human filtering. These sampling decisions carry over to automated benchmarks. Replacing human review with a faster grader changes how much experimentation a team can afford. The choice of tasks still determines which capabilities its comparison can reveal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose between reusable guidelines and a task-specific rubric
Having people evaluate every new agent quickly creates a bottleneck. An offline evaluation keeps the task and the candidate’s solution but hands the quality decision to an automatic proxy. One approach gives an LLM judge high-level guidelines similar to the ones the human reviewers use. For every assignment in the collection, the judge reads the proposed solution and assesses how well it adheres to the same criteria. The work that went into defining quality carries across assignments.
A rubric changes what the judge is asked to do. Wolfe illustrates this with an assignment from what he calls the Dr. Seuss paper: culturally adapt a book into a Netflix film. Many adaptations could satisfy the brief. The rubric lists requirements specific to that assignment, and the LLM judge checks each one independently against the proposed adaptation.
With the general guidelines, an assessment tells the team how the work fared against a shared definition of quality. With the rubric, the team can inspect the judgment on each requirement of this adaptation. That specificity takes work for each new assignment and gives the judge a narrower decision to make on each check. In both approaches, the resulting metric depends on how well the LLM can carry out the judgment it has been given.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check the judge against human ratings and viewer behavior
Wolfe calls the next step meta-evaluation. Take examples from the calibrated human process whose ratings the team trusts and use them as golden data. Run the automatic evaluation on those same examples and compare its decisions with the human assessments. Binary accuracy can express how often a yes-or-no check matches the human decision. For scored outputs, a correlation measure can show whether pieces that humans rate more highly also tend to earn higher scores from the judge.
A Netflix project Wolfe worked on tested a judge against a different kind of evidence. His team established a causal link between the quality the LLM judge assessed and streaming behavior. When the judge rated something poorly, a viewer who began watching the show was more likely to abandon it. Agreement with an expert rating would establish that the judge could reproduce that judgment. Abandonment gave the team a specific consequence for viewers against which to assess the significance of a poor rating.
How much judge–human agreement is good enough depends on the assignment. Calibrated reviewers may still disagree relatively often on a highly subjective task. Wolfe uses the agreement those reviewers achieve with one another as a baseline and aims for a similar level between the judge and the humans. Once the grader meets that standard, the team can build a benchmark around it, applying the same sampling, profiling and filtering decisions it used for human review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeat both the generation and the grading
A judge that agrees with humans across a golden set can still vary from one run to the next. Wolfe recommends resampling at both places where variation enters the benchmark. Generate several rollouts for a task to see how much the agent’s work varies. Then have the judge assess a fixed solution several times to see how much the grading varies. If every new completion receives only one new judgment, a change in score could have come from either step.
Use power analysis to size the benchmark and report error bars alongside the scores. Repeated samples can reduce the variance of the estimated result. Confidence intervals make the remaining uncertainty visible to the person comparing candidates. That distinction matters when deciding whether to spend human-review time on a new version. A slightly higher average on one run gives the team less reason to advance a candidate when fluctuations of that size are common across repetitions.
At this point the team can run many ideas through the offline benchmark and return the promising candidates to its calibrated human reviewers. This is where the investment in automation pays off. The team gets to explore broadly while reserving close human assessment for changes with evidence of an improvement. A candidate that survives both comparisons is ready for a test of how the change affects users.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use an A/B test as a limited deployment
Production adds evidence unavailable from inspection of a finished answer. Users can give explicit feedback, such as a thumbs-up, a thumbs-down or a preference between responses in a chatbot. They can also leave implicit signals through their behavior, such as the length of a session or the decision to abandon something they started. The team needs a reason to connect its chosen measure to quality in this product. Session length by itself records elapsed use. Its interpretation depends on what users are trying to accomplish.
An A/B test gives the team a comparison against the system already in place. Expose a portion of users to the candidate and measure the chosen outcomes relative to the existing experience. The statistical question from benchmark design returns here in operational form. Given the size of the difference that matters and the variability of the metric, how long does the experiment need to run before the team can draw a meaningful conclusion?
Wolfe treats the experiment as a soft deployment. The agent gets real use while the decision about a full rollout is still open. An offline gain earned the candidate a human review, and favorable human judgments earned it this production comparison. The A/B result can now inform whether to extend the change to everyone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn production failures into cases for the next evaluation
After rollout, the team can monitor performance trends and inspect individual failures or new uses. Those observations can lead to different repairs. If a production task was missing from the benchmark, add an example so future candidates face it offline. If the existing criteria overlooked what made an answer poor, revise the quality guidelines. If the criterion already covers the failure but people or the judge apply it inconsistently, revisit calibration.
This gives each stage a role in the next model decision. Offline runs narrow a large field of candidates, human reviewers examine the strongest ones, and a limited deployment tests the result against the current user experience. A case discovered after deployment can travel all the way back to the experts who first defined quality. The next comparison can then measure a failure that every stage of the previous evaluation allowed through.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…