AI Engineer Europe 2026
Evals Are Broken, Use Them Anyway
Loading slides…
Read the talk
Evals Are Broken, Use Them Anyway
Similar benchmark scores can hide very different agents. Useful evaluation connects realistic tasks, isolated runs and failure traces to improvements that survive actual product use.
From a talk by Ara Khan
Before you start: Familiarity with coding agents, command-line tools and automated tests will help you follow the evaluation workflow.
When the same score does not mean the same model
If GPT-5.4 and Gemini 3.1 Pro Preview sit next to each other on a benchmark dashboard with similar scores, should you expect them to behave the same way? Ara Khan opens with that mismatch: a scalar score makes models look interchangeable even when using them reveals substantial differences. The dashboard is evidence about a particular evaluation, not a complete description of either model.
Khan connects that mistake to a tweet he saw that morning criticizing Meta for maximizing benchmark results. His objection is to treating a lab's headline performance as sufficient evidence of real-world quality. But abandoning measurement creates the opposite problem. Asked why they prefer Opus, someone might answer, “I like talking to her.” That preference says something about the interaction; it cannot establish everything a builder needs to know about reliability or task completion.
Evaluations are useful without being definitive. There are three increasingly involved ways to use them: interpret other people's evaluations, improve your own agent against an evaluation, and build an evaluation for a specific use case. This discussion develops the first two; constructing a new benchmark is a further undertaking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use external scores to decide what to investigate
Start by treating model-lab numbers as approximations. Khan's deliberately blunt advice is not to believe them at face value: sometimes they describe something useful, and sometimes they do not. He also references a tweet describing researchers and engineers who routinely dismiss evaluation numbers. The productive response is to supplement the score with actual trials and informed preferences, rather than turn either numbers or intuition into the sole authority.
Next, separate staying current from adopting immediately. Khan uses an Epoch aggregate index to illustrate how rapidly frontier leadership changes, and describes his own preferences changing despite testing models professionally. His practical recommendation is to wait a couple of weeks after a release, let early users expose its problems, and then try a switch if the model still holds up. A builder does not need to assume the workload of someone whose job is testing every new release.
Finally, look for evaluations that are recent, precise and relevant. Khan cites OpenAI's analysis of why SWE-bench Verified no longer measures frontier coding capabilities. The distinction matters: SWE-bench uses resolved GitHub issues and patches, and OpenAI's critique concerns flawed tests and contamination. Khan's subsequent Fibonacci and matrix-multiplication examples express his broader objection to elementary exercises standing in for software engineering; they are not an accurate description of that benchmark's task format. Freshness alone is insufficient, too: a new evaluation still needs credible tasks that represent the work you care about.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From intuition to tasks users actually perform
Improving an agent introduces a harder question: what should the evaluation represent? This is both an engineering problem and a problem of definition. You can build reliable test infrastructure and still choose the wrong tasks. Coding has an effectively unbounded space of requests, environments and failure modes, so a finite suite must approximate the work rather than exhaust it.
Khan describes encountering this problem at Cline, an open-source coding-agent company. The available standardized evaluations did not seem to measure users' day-to-day programming work. Cline initially relied on informal judgment; Khan says conversations with the Codex team and other teams reflected a similar stance. Eventually, dismissing the available measurements stopped being enough. Cline needed some way to measure progress and decided to construct tasks itself.
The raw material came from users who had opted in to share their Cline coding usage. Khan says the team offered users money, gathered examples of real work, and then spent substantial manual effort parsing and cleaning them into a usable dataset. The essential transformation was from messy usage records to problems another coding agent could attempt. Real-world provenance supplied relevance; careful task preparation was still necessary to make those examples evaluable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Grade the repair, not just the answer
A bounded factual question offers a simple evaluation shape: submit a prompt, receive an answer and grade it. Khan uses a question about how many toes a cat has, jokingly guessing eleven while admitting he does not know. The point is the limited answer space, not the guessed number. That kind of single-turn task is much easier to grade than an agent's extended interaction with a working environment.
Consider the request to make a broken MCP server work. An agent such as Claude Code might read repository files, search documentation, install dependencies, configure the environment, run Python scripts and finally run tests. Its response is only one artifact of that process. The evaluator must establish whether the resulting system works—and whether fixing it broke something else.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give every task a complete, isolated environment
During this work, Cline began using Terminal-Bench, which Khan credits to researchers at Stanford. Its tasks cover problems such as race conditions, database failures and infrastructure troubleshooting. The CLI interface allows different coding agents to attempt the same work. Khan describes some tasks taking a couple of minutes and others taking 30–40 minutes. Long runs expose extended behavior: an agent may make progress, circle back repeatedly or lose its way.
Khan describes an 89-task suite, matching Terminal-Bench 2.0's task count. To run a race-condition repair, the evaluator needs more than the instruction to fix it:
- Prepare an isolated environment, such as a virtual machine.
- Include the repository and the complete setup needed to reproduce the problem.
- Install the agent being evaluated—Cline, Claude Code, Codex or another CLI agent—inside that environment.
This makes the environment part of the evaluation. Without the required repository and setup, the run may measure a provisioning failure instead of the agent's ability to repair the race condition.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run the isolated tasks concurrently
Harbor, which Khan attributes to Laude Institute, addresses the infrastructure work around these runs. Instead of setting up and executing every task sequentially, define a standardized configuration for each task's Linux environment, CPU and memory, then run the isolated environments concurrently.
The execution backend can vary:
- Daytona: a cloud infrastructure option Khan names.
- Local Docker: an option if the machine can support the aggregate compute demand.
- Modal: the infrastructure Khan says Cline uses.
With enough capacity to run all tasks concurrently, the slowest task becomes the main limit on suite completion time. That is the fully parallel arrangement described here, rather than a guarantee independent of provisioning, scheduling or concurrency limits.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn failed runs into a repair queue
Once a baseline run produces a score, inspect the failures. Khan proposes a hypothetical run with fifty failures out of eighty-nine tasks. The next step is not to stare at the aggregate: have another agent examine the failed runs' traces, including every LLM call, and group them by cause. One task might fail because the agent skipped tests; another because its read-file tool was broken. Grouping failures makes recurring defects visible and identifies changes that could help multiple tasks.
A small Python example captures the bookkeeping after trace review. These two teaching records use Khan's failure categories; status keeps diagnosis separate from an implemented repair:
python
from collections import defaultdict
reviews = [
{"task_id": "task-a", "cause": "skipped_tests", "status": "proposed"},
{"task_id": "task-b", "cause": "broken_read_file", "status": "proposed"},
]
repair_queue = defaultdict(list)
for review in reviews:
repair_queue[review["cause"]].append(review["task_id"])
for cause, task_ids in sorted(
repair_queue.items(), key=lambda item: (-len(item[1]), item[0])
):
print(f"{cause}: {len(task_ids)} affected task(s): {', '.join(task_ids)}")
The grouping does not discover causes by itself; that work comes from examining traces. Its purpose is to turn those diagnoses into a queue of concrete interventions.
Interpreting the next score requires separating three things:
| What is being tested | What can distort the result |
|---|---|
| Model | Strong capability can mask a weak harness. |
| Agent harness | Tools and prompts may fit one model better. |
| Task quality | Success may measure the wrong work. |
Khan illustrates model–harness fit with Anthropic models used through Cursor, Droid and Claude Code: the same model can appear more capable when its surrounding agent makes better use of it. Task quality is equally consequential. A perfect score on unsuitable problems is still evidence about unsuitable problems. A useful result requires alignment between the model, harness and task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources and reasoning both change the outcome
Khan reports an initial score of approximately 43% in this Terminal-Bench discussion. Cline then changed container CPU and memory allocations, raised timeouts and adjusted thinking behavior. He does not give a final score or enough run configuration to isolate the contribution of each change. Resource settings therefore belong among the conditions of the result, not outside the evaluation as incidental infrastructure.
More thinking did not always help. Khan describes cases where asking the model to think more reduced response quality and produced repetitive loops, including repeating “I am a model” for roughly 2,000 tokens. Reasoning behavior is another variable to test, not a knob that can always be turned upward. Cline maintains an internal benchmark across model versions, including open-source models, to keep examining these differences.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the agent without fitting it to the test
Benchmark improvement falls into three zones:
- Repair obvious flaws. Fix bugs that crash the harness and resolve rate-limit failures. These defects prevent the system from meaningfully attempting the task.
- Tune model-specific behavior. Prompting techniques that work for Anthropic models may not work for the Codex or Gemini model families. Adjust instructions and try longer or shorter prompts to find where the harness is failing to elicit a model's capabilities.
- Stop before overfitting. Optimizing specifically to extract the highest benchmark score can become cheating, especially when the real objective has shifted to publishing an impressive number.
The second zone is where much of the useful engineering happens. It explains why a model that other people find excellent may perform poorly inside your agent: the surrounding prompts and tools may need different choices.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the score and the product experience together
Find a benchmark relevant to the problem, establish a score, and improve the harness against it. That repeated process is hill climbing. But the numerical result must remain connected to using the product: does the agent actually feel good to work with, and does it perform the work well? A good score is not a reason to stop asking those questions. Start somewhere, improve honestly, and keep both forms of evidence in view.
This also changes how a team judges unfamiliar models. Khan says Cline initially worked better with Anthropic models than with Gemini and some other model families. Hill climbing helped the team improve that support. The benefit was not merely a higher score: people who preferred those other models could now use Cline more effectively. Evaluation became a way to expand the product's usefulness rather than reinforce an existing model preference.
Khan closes by inviting people to contact him through the Twitter account shown in the recording, whether they want to work on these problems or discuss evaluation questions outside coding agents. The infrastructure examples here concern programming, but the underlying question extends further: how can a bounded set of tests help improve the open-ended work your users actually need done?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
OpenAI's February 2026 analysis of flawed tests and benchmark contamination in SWE-bench Verified.
Further reading
The original paper describing Terminal-Bench 2.0's 89 realistic terminal tasks, task environments, reference solutions, and verification tests.
Examples for selecting agents, filtering tasks, and running Terminal-Bench evaluations concurrently in cloud environments.
Explains resource requests and limits for Modal Functions and Sandboxes, including CPU allocation and memory-limit behavior.
Updates since the talk
Current documentation for task resource declarations, CPU and memory enforcement policies, provider support, and run-time overrides.
Full transcript
Loading transcript…