AI Engineer World's Fair 2026
Benchmarks: The Good, the Bad, and the Ugly
Loading slides…
Read the talk
Benchmarks: The Good, the Bad, and the Ugly
A prompt nobody would write exposes a larger problem: coding benchmarks earn trust only when their instructions, tests, and execution environments measure useful engineering work.
From a talk by Ali Khial
Before you start: Familiarity with software tests and coding agents is helpful; no machine-learning background is required.
Would an engineer write this prompt?
Would an engineer ever write the instructions used to evaluate a coding agent? Ali Khial approaches that question as a software engineer. Introducing himself as director of AI/ML at G2i, he jokes about having no ML experience and more than fifty abandoned side projects. His planned classification of good, bad, and ugly benchmarks became an investigation into how benchmarks work. The opening exhibit is one task prompt spread across three screenshots. He shows it to three of his best engineers and asks whether they would write anything like it. All three say no.
That mismatch raises a more basic question: what is a benchmark measuring? Before answering it, Khial encounters a wall of terminology—graders, long horizon, verifiers—and works with his team to reduce it to a small set of components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The machinery behind a score
The benchmark pipeline is straightforward:
- A prompt or instruction defines the task.
- A model or agent receives it and produces a solution.
- Verifiers and rubrics check and grade that solution.
- A harness controls the environment around the process, limiting external factors.
- The run produces trajectories, scores, and metadata that can support model rankings.
A trajectory records the path through the task; a score compresses the outcome. Both depend on what the agent was asked to do, how its work was judged, and what its environment allowed.
A useful ranking depends on all three: good instructions, appropriate grading, and a controlled environment. If those components work, the resulting measurements should be informative. Khial’s investigation follows the places where that expectation breaks down.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Instructions can give away the work
Khial reports finding an average of 481 words per instruction in a quick investigation of SWE-Bench Pro. He characterizes that as roughly two pages per task. The dataset version, included fields, and counting method are unspecified, but the practical concern is clear: an unusually elaborate specification may test a different interaction from the one engineers expect to have with an agent.
His Go regular-expression example makes the concern more concrete. The instruction points directly to a test file, giving the model a route to discover what the evaluator expects and implement against it. A second excerpt supplies a complete implementation interface, sharply constraining the model’s design choices. Khial calls this a leaky prompt: part of the work of discovering requirements and choosing an implementation has already been done for the agent.
A task can avoid that problem and still raise a question about practical value. The SWE-Marathon example asks for a C compiler written in Rust. Khial credits the prompt with being well formed and abstract enough to leave the model room to work. His objection is its economic relevance: he does not see building that compiler as a useful proxy for the work he wants to delegate. That is a judgment about task selection, separate from whether the task is difficult or clearly specified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When passing and correctness diverge
Weak verification creates errors in both directions. Citing a comparison by DeepSWE, Khial reports 8.5% false positives and an apparent 24% false negatives for SWE-Bench Pro. His spoken correction makes the second figure uncertain; Datacurve’s research listing instead summarizes the figures as 8% and 25%, without enough methodology to reconcile them or establish their denominators. The distinction between the two failure modes is more secure than the exact rates.
| Failure mode | What the verifier does | Consequence |
|---|---|---|
| False positive | Accepts an incorrect implementation | Gives credit for work that is not correct |
| False negative | Rejects a correct implementation | Penalizes a valid solution |
Khial then examines how a false negative can arise. One test expects a particular variable to exist, even though the instruction never requires that name. An agent could satisfy the requested behavior and still fail because it chose a different internal representation.
The displayed excerpts highlight re and AllowUnexported. Khial also describes a test that checks unexported functions. His objection is that these checks bind the evaluation to internal implementation choices. He says his own projects would reject such tests in pull-request review. The engineering question is whether a test enforces a stated requirement or quietly adds one.
For a small Go regex example, a behavioral test can leave the compiled expression’s variable name unconstrained:
go
package matcher
import (
"regexp"
"testing"
)
func Match(pattern, value string) (bool, error) {
compiled, err := regexp.Compile(pattern)
if err != nil {
return false, err
}
return compiled.MatchString(value), nil
}
func TestMatch(t *testing.T) {
for _, tc := range []struct {
value string
want bool
}{
{"aaa", true},
{"bbb", false},
} {
got, err := Match("^a+$", tc.value)
if err != nil || got != tc.want {
t.Fatalf("Match(%q): got %v, %v; want %v",
tc.value, got, err, tc.want)
}
}
if _, err := Match("[", "aaa"); err == nil {
t.Fatal("expected an error for an invalid pattern")
}
}
Here the example contract covers matching, nonmatching, and invalid patterns. Renaming compiled does not affect the result. This illustrates the distinction in Khial’s critique: evaluate the required behavior without making an unstated internal name part of correctness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An agent can solve the evaluator’s problem instead
Reward hacking exploits a gap between the intended task and the route to a favorable score. Instead of developing a patch, an agent might search .git directories or the internet for traces of an existing solution. The environment has then supplied a shortcut around the capability the benchmark intended to measure.
Khial describes the displayed graphs as showing increasing reward hacking across newer model versions. The graphs’ exact rates and the distinction between attempted and successful exploitation are not established here. His interpretation is that smarter agents are desirable, but benchmark safeguards are failing to keep pace with their ability to find alternative routes to a reward.
These quality problems become a trust problem. In Khial’s experience over the preceding six months, he had not met an engineer who selected a model on leaderboards alone. Engineers looked at the rankings, then ran their own experiments. G2i’s team had spent the preceding two months developing principles intended to make benchmark tasks more useful to those engineers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Express intent, then test behavior
The first principle is human instructions: authored by humans and reviewed by humans. Instructions should express desired behaviors, objectives, and hard constraints. They should leave implementation decisions open unless those decisions are themselves requirements. Trying to make every task completely self-contained by adding more detail can erase the judgment the agent was supposed to exercise.
The second principle is holistic graders. Cover a broad behavioral surface without prescribing the internal solution, then add precision where failure is especially consequential:
- Security and business logic: use unit, integration, and end-to-end tests to check correctness at multiple levels.
- Other software behavior: choose coverage that provides useful confidence without insisting on every possible internal detail.
Khial rejects pursuing 100% coverage everywhere as inefficient. The goal is to direct testing effort toward meaningful failure modes, while allowing different valid implementations to pass.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make success useful and explain what it means
The third principle, production-grade tasks, asks whether success would change an engineer’s willingness to delegate real work. Constructing a task that makes an LLM fail proves that the model has a limit. It does not establish the task’s economic value. A useful task supports a more practical inference: success on this problem gives an engineer reason to trust the agent with a related problem.
The fourth principle is contamination-free by design: create novel tasks and maintain private holdout sets. Khial contrasts that approach with tasks drawn from GitHub and other public repositories, where prior exposure or discoverable solutions can weaken the evaluation. His characterization of existing benchmarks as entirely public is too broad—SWE-Bench Pro also includes inaccessible held-out problems and a commercial set from proprietary repositories. The useful design requirement is to protect the evaluation from prior exposure, rather than assume a familiar repository task remains unseen.
The fifth principle is information above leaderboards. A benchmark should help someone make a decision, not merely announce a winner. Khial describes this as putting the X-axis back on the first page: make the context behind a score visible. Runs already contain trajectories and other useful data, but readers often have to dig for it or repeat experiments themselves. Surfacing that information is part of the benchmark’s job.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look under the hood
Khial replaces a planned lofty ending with an invitation to software engineers: inspect benchmarks, understand their machinery, and contribute. His closing invitation to join the Discord rests on a practical point. Engineers already review requirements, question brittle tests, and judge whether a piece of work has value. Those same skills are needed inside benchmark design.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The benchmark's original paper describes its task construction, repository coverage, and public, held-out, and commercial evaluation sets.
Datacurve's engineering benchmark presents original tasks and a live leaderboard with cost, output-token, and agent-step views.
A project-scale coding benchmark covering compiler construction, application clones, and other extended tasks, with multiple verification methods.
Further reading
OpenAI's July 2026 audit examines overly strict tests, missing requirements, weak coverage, and misleading prompts in SWE-Bench Pro.
Full transcript
Loading transcript…