AI Engineer World's Fair 2026

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Loading slides…

Read the talk

Stop Evaluating Models Like It's the 50s

A benchmark score hides which questions a model gets right. Item response theory turns those response patterns into estimates of ability, item quality, and unexpected behavior.

From a talk by Alejandro Vidal

Before you start: Familiarity with benchmark accuracy and basic probability is enough; item response theory and its parameters are introduced as they are used.

What disappears inside an accuracy score

Take a benchmark leaderboard and split each model’s accuracy bar into the individual questions behind it. The total tells you how many answers were correct. It does not tell you whether those answers came from easy questions, difficult questions, or questions whose answer keys are wrong. Alejandro Vidal, founder of MindMakers, begins with this transformation using real benchmark data from Epoch AI. His background in psychology and computer science suggests another way to evaluate the resulting patterns: borrow the measurement tools used in psychometrics.

In psychometrics, a question is an item. Summing correct responses gives every item the same contribution to the score, despite differences in difficulty and usefulness. Vidal describes this approach as classical test theory and asks what becomes possible when the individual responses remain available.

The bars become a response matrix: one row per model, one column per item, and a correct or incorrect response in each cell. Now each item can be treated as its own variable. The matrix begins to expose structure that a leaderboard conceals, including the concentration of failures where weaker models meet difficult questions.

Slide titled “A benchmark is one number. It shouldn't be.” Nine model rows contain individual green and red responses, with text describing an answer vector rather than a sum.Open full source frame
Keeping each item separate reveals an answer matrix instead of a single benchmark score.
0:020:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:02 · section reference included

Put difficulty and ability on the same scale

At the other end of the matrix, strong models tend to answer easy questions correctly. To describe this pattern, assign each item a difficulty parameter, b, and a response curve mapping model ability to the probability of a correct answer. In the introductory curve, b is the ability level where the probability crosses 50%. Moving the curve right makes the item harder.

The model’s ability is θ, or theta—what Vidal calls intelligence. Black dots on the response matrix mark those estimates. Questions to the right of a model’s dot are harder than its estimated ability, and their responses are mostly incorrect. This is a probabilistic pattern, not a boundary beyond which success is impossible.

The highlighted row in the demonstration is labeled GPT-5.5 Pro. The frame shows θ = 1.20, b = −1.23, and a predicted success probability of 99%; Vidal rounds the difficulty to approximately −1.2 in his narration. The useful relationship is that the model sits well above the item’s difficulty. Reproducing the displayed probability also requires the curve’s slope and scaling convention; the two location values alone do not determine it.

Response matrix with black model-ability dots and a highlighted GPT-5.5 Pro row above a green curve. Dashed guides mark b = -1.23, theta = 1.20, and P = 99%.Open full source frame
Item difficulty and model ability meet on a shared response-probability curve.

Calibration supplies a reference population as well as an estimate. Vidal describes normal distributions for ability and difficulty and uses b = 0 to explain average difficulty. Normality is a modeling assumption, however, rather than a property guaranteed by item response theory. On a scale centered at zero, the precise interpretation is that an item with b = 0 gives a model with θ = 0 a 50% success probability. Models above and below that point have different probabilities; half the models do not each have exactly a 50% chance. This shared scale makes the reference behind an ability estimate explicit and supports uncertainty estimates.

2:102:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:10 · section reference included

Discrimination determines how much an answer tells you

This framework is item response theory, or IRT. Difficulty is only the first item parameter. The second, a, is discrimination: how sharply success probability changes with ability.

DiscriminationResponse curveMeasurement value
Large positive aSteeply increasingSeparates nearby ability levels
Near-zero aNearly flatCarries little ability signal
Negative aDecreasingStronger models succeed less often

Negative discrimination is especially interesting because it runs against the intended direction of the test. It can point toward a problem with the item rather than a weakness in the stronger models.

A standard mathematical expression for this mechanism is the two-parameter logistic model:

Pi(θ)=11+exp⁡[−ai(θ−bi)]P_i(\theta) = \frac{1}{1 + \exp[-a_i(\theta-b_i)]}

Here, Pᵢ(θ) is the probability of answering item i correctly. At θ = bᵢ, the exponent is zero and the probability is one half. The discrimination parameter controls the slope around that point. This formula makes the roles of the parameters explicit without assuming that it reproduces every numerical setting in Vidal’s interface.

Once the item curves are calibrated, a new model’s answers provide evidence about its ability. Vidal displays each item curve above a relative likelihood over θ. A correct answer contributes Pᵢ(θ); an incorrect answer contributes 1 − Pᵢ(θ). Under the usual conditional-independence assumption, these contributions multiply:

L(θ∣y)=∏iPi(θ)yi[1−Pi(θ)]1−yiL(\theta \mid \mathbf{y}) = \prod_i P_i(\theta)^{y_i} [1-P_i(\theta)]^{1-y_i}

The response yᵢ is one for a correct answer and zero for an incorrect answer. As the demonstration adds responses, their curves combine into a likelihood that identifies plausible ability values and supports a likelihood interval. More informative responses can narrow that interval; the result is more than a point on a leaderboard.

4:465:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:46 · section reference included

Two more correct answers can conceal a larger difference

Vidal next compares Claude Opus 4.1 and Gemini 3 Pro on the same set of questions:

ModelCorrect answers in Vidal’s example
Claude Opus 4.1245 of 337
Gemini 3 Pro247 of 337

The totals look very close. Vidal reports that the fitted IRT ability estimates are separated by almost one standard deviation, favoring Gemini 3 Pro. He interprets this as a substantial ability difference and notes that Gemini is the later model. The comparison belongs to this calibration and evaluation setup, rather than establishing an unconditional ranking between the models.

Which questions a model answers correctly matters, not just how many. To explain the gap, Vidal contrasts answering harder questions with accumulating correct answers on easier ones. In principle, a model could even answer fewer questions correctly while receiving a higher fitted ability estimate. That hypothetical reversal is distinct from the displayed totals, where Gemini has two more correct answers. Item parameters and likelihood intervals help make the comparison interpretable before the talk moves into practical applications.

6:396:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:39 · section reference included

Find questions that punish better answers

The first application is benchmark maintenance. With response data already available, Vidal proposes using a shared skill to inspect the item parameters, retain useful questions, and find candidates for repair. Plotting difficulty against discrimination separates informative positive-a items from nearly flat items and those whose relationship with ability runs backward.

The inspection procedure is straightforward:

  1. Flag items whose estimated discrimination is significantly below zero.
  2. Inspect the question and its gold answer, optionally asking another LLM to help.
  3. Decide whether the label needs correction, the wording needs improvement, or the item should be removed.

The statistical flag prioritizes review; it does not itself establish the correct answer. In Vidal’s first example, he does not know the answer personally, and ChatGPT apparently disputes the benchmark’s gold label.

The second example is more revealing. A question asks for the total number of passengers, but its gold answer is 583—the total people killed, including crew. An answer can therefore match a familiar number while failing to answer the question actually asked. A model that distinguishes passengers from crew may be marked wrong precisely because it read more carefully. The repair is to align the question and label; the passage does not supply a verified replacement passenger count. Other flagged items may have correct labels but poor construction, requiring revision or removal instead.

8:198:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:19 · section reference included

Spend evaluation tokens on informative items

Organizations often maintain their own benchmarks to choose models, particularly open-source models. Running every item against every candidate costs time and tokens. Calibrated items make it possible to ask how small an evaluation can become while still serving its purpose. Vidal uses a deliberately simplified selection rule: start with the item having the highest discrimination, then add items in descending order of a.

For the demonstrated benchmark, Vidal reports reaching a target correlation of 0.99 with the original ranking using about 97 of 484 items—almost a fivefold reduction in question count. Random selection performs substantially worse in his comparison. This is a ranking-preservation result for that example; the talk does not specify the correlation definition or a held-out evaluation protocol.

Why can so many questions be omitted? Vidal points to overlapping response curves and weak discrimination. Similar curves can provide similar information about the ability range being measured, while nearly flat curves contribute little ability signal. The objective is not to maximize the number of questions asked, but to obtain enough useful evidence. Rather than automatically deleting the rest of the benchmark, keep the item bank and administer subsets on different runs.

The benefit depends on the benchmark. In Vidal’s analysis of GPQA, selected and randomly chosen subsets perform similarly. He attributes that result to highly discriminative items with little overlap. This counterexample matters: item selection offers less advantage when the bank already contains useful questions throughout.

10:2810:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:28 · section reference included

Look for answers the fitted model did not expect

An item curve supplies an expected probability for every model–question pair. Compare that prediction with the observed response and you obtain a residual. These differences make it possible to investigate unusual behavior, including possible leakage, benchmark overfitting, or other contamination.

In the response matrix, Gemini 3 Pro answers an item incorrectly despite a fitted success probability of 86%. A simple raw residual subtracts the predicted probability from the binary observation:

ri=yi−Pi(θ)r_i = y_i - P_i(\theta)

The calculation for that response is:

python

predicted_probability = 0.86
observed_correct = 0
residual = observed_correct - predicted_probability
print(f"{residual:.2f}")  # -0.86

The negative sign indicates performance below the prediction. It identifies an answer worth investigating, without explaining why the model failed.

DeepSeek-R1 supplies the opposite case: a correct answer where success was not expected. That positive surprise does not prove overfitting either. The completed matrix highlights both kinds of anomaly, making it easier to distinguish unusual successes from unusual failures.

Model response matrix with two circled cells labeled as a strong model answering an easy item wrong and a weak model answering a hard item right. A DeepSeek-R1 tooltip shows a response curve and the observed correct answer.Open full source frame
Residual analysis highlights both an unexpected wrong answer and an unexpected right answer.

A single binary response can be noisy. Vidal suggests asking the same question multiple times and averaging the outcomes. Comparing an empirical success rate with the fitted probability may resolve an apparent outlier that was merely one unexpected sample.

13:4013:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:40 · section reference included

Use response consistency to investigate inference failures

The same analysis can operate across an entire model row. Vidal identifies o4-mini as the least consistent model in the displayed example: its mixture of correct and incorrect answers fits the difficulty ordering less neatly than expected. That pattern suggests a question for operations as well as model research: is the inference system behaving correctly?

An inference platform running the model incorrectly, or a faulty quantization, could produce unusual response patterns. These are possible causes to investigate, not diagnoses established by the plot. Some noise is inevitable; a useful response model should make success and failure approximately predictable, not turn the matrix into a perfect staircase. Residuals provide a way to inspect deviations from that expectation.

15:2015:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:20 · section reference included

Give each organization a different exposure pattern

Suppose a benchmark is expensive to build and valuable precisely because its questions remain private. Sharing the whole bank risks exposing it to future training. Vidal introduces a selective-exposure scheme under adaptive testing: evaluate models through a common anchor while giving each participating organization a distinct set of additional questions.

The proposed procedure preserves a common reference and records who has seen which difficult items:

  1. Randomly select a representative anchor set and use it with every participating model and organization.
  2. Assign each organization its own exclusive fingerprint set.
  3. Choose extremely difficult items for those fingerprint sets.
  4. When organizations release new models months later, run evaluations again and inspect residuals on the previously exposed questions.

The signal of interest is unusually strong performance on the particular fingerprint set an organization received, beyond what the new model’s estimated ability would predict.

Vidal demonstrates the comparison with synthetic data. One organization has an unusually large average residual on a specific fingerprint set relative to the other organization. This provides a targeted contamination signal rather than merely noticing that a newer model improved overall. The scheme is not bulletproof: it creates evidence for investigation, not proof that an organization trained on the benchmark.

16:3316:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:33 · section reference included

Compare groups at the same estimated ability

Psychometrics also asks whether an item behaves differently for different groups. Vidal transfers that question to model evaluation by splitting the dataset into open-weight and closed-weight models. Other grouping variables, or more than two groups, could be used when they answer a useful research question.

For each item, fit a separate response curve for each group. Compare the curves at matched ability values: if the item behaves similarly across groups, the curves should overlap and the probability gap should be close to zero. A gap indicates that group membership carries information about success beyond the shared ability estimate. In the demonstration, some items favor closed-weight models and others favor open-weight models.

Vidal withholds the actual questions to avoid leaking them. He observes common features among some questions favoring open-weight models, but cannot identify the cause. Such patterns could guide research into training differences; the curve comparison itself does not reveal how a model was trained.

18:3118:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:31 · section reference included

Use shared residual patterns to explore model relationships

The final application treats a model’s residual vector as a behavioral fingerprint. Instead of asking only how capable two models are, ask whether they make similar unexpected successes and failures after accounting for ability and item difficulty. Vidal constructs a correlation matrix and a simple projection to explore those relationships.

The display groups some models from the same lab or model lineage. It also shows patterns among DeepSeek distillations and Qwen models. The intuition is that shared history may leave shared error patterns, even when the models’ overall abilities differ.

Vidal reports a residual correlation of 0.38 for an example pair sharing a base model. He then considers several related comparisons:

  • Distillation and base model: Similar residuals could help investigate a suspected distillation relationship, including unauthorized distillation.
  • Different effort settings: Runs of the same model at different effort levels may retain a recognizable response pattern.
  • Successive versions: Relationships among Llama or Gemini versions may remain visible as the models evolve.

The pair behind the reported correlation is not named in the spoken example. Vidal frames this application as an area for research into model relationships, rather than an established everyday test for model ancestry.

20:0220:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:02 · section reference included

Beyond one ability number

A single ability axis is a starting point. Vidal closes by proposing multidimensional and hierarchical models that could distinguish skill levels across task types. A model might then have a profile of abilities rather than one scalar standing in for every kind of performance.

He also proposes combining benchmarks through IRT to improve their estimates and recommends a paper he calls Meta-Benchmark. The underlying opportunity is to relate evidence across tests instead of treating every benchmark score as an isolated measurement.

Other extensions would broaden what is measured: latency and token usage could contribute additional signals, while psychometric models could be used to study alignment and interpretability. Vidal also suggests that mechanistic interpretability could help explain the behavioral structure psychometrics uncovers. He is working on these directions, invites collaboration and requests for benchmark help, and closes by offering materials, skills, and benchmarks for experimentation. The practical starting point remains the response matrix: retain the individual answers so that evaluation can measure more than their sum.

22:0622:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:06 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…