AI Engineer Europe 2026
Task Fidelity Scaling Laws — Kobie Crawford, Snorkel AI
Loading slides…
Read the talk
Task Fidelity Scaling Laws
A difficult agent task is useful only if it is solvable, correctly specified and reliably evaluated. Kobie Crawford examines how those properties affect reinforcement learning.
From a talk by Kobie Crawford
Before you start: Basic familiarity with agent tool use, automated tests and reinforcement learning will help; no knowledge of Snorkel’s platform is required.
Does task quality actually matter?
If data quality determines what a model can learn, what counts as good data when the model is learning to act? At Snorkel, the training material increasingly includes entire tasks and the environments in which agents attempt them. Kobie Crawford, a developer advocate there, describes a company that produces datasets for foundation-model improvement. Its origins run from Stanford research and a CEO’s PhD work through an open-source library to customer datasets. Since the company’s start in 2019, the through line has been data quality; agent training turns that concern into task quality.
The research question is whether that quality has a measurable effect on training outcomes. Rather than assume that better tasks produce better models, the team sets out to test the connection empirically.
The setting is Terminal-Bench-style work: a task definition inside a containerized environment, where an agent must take actions to accomplish an objective. Architecture and harness choices also affect performance. The question here is whether task quality remains consequential underneath those choices—and how to isolate its contribution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four conditions for an accepted task
Frameworks such as Harbor and OpenEnv provide environments for agent evaluation, benchmarking and reinforcement learning. Containerization supports reproducibility, isolates runs and allows rollouts to happen in parallel. But a reproducible container is only part of a useful task: the task logic and the environment must also pass scrutiny.
| Criterion | What must hold |
|---|---|
| Achievable | The agent can accomplish the task. |
| Non-trivial | Success requires substantive work. |
| Functionally correct | The task logic behaves as intended. |
| Reliable environment | The execution environment works dependably. |
Snorkel’s research harness tests all four criteria. A task that passes every check enters the accepted bucket; otherwise it enters the rejected bucket. Those buckets supply the operational distinction between higher- and lower-quality tasks for the comparisons that follow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Do accepted tasks require more work?
Before using the buckets in training, the team checks whether acceptance correlates with the execution behavior it wants. Crawford describes runs with Sonnet 4.5 and Codex, mentioning GPT-5.2, GPT-5.1 and an imprecise “4.0” designation in the Codex mix. This is a historical comparison, not a single fixed Codex model configuration.
Crawford reports that accepted tasks averaged twice as many tool calls as rejected tasks in these evaluation runs. Accepted tasks also had lower pass rates and required more output tokens. He interprets the combination as evidence of more steps, greater engagement with external tools and more reasoning effort. These are behavioral indicators of difficulty in this comparison, rather than a definition of difficulty based on tool count alone.
A lower pass rate is not enough to establish a better training task. It can mean the agent encountered a real capability limit, but it can also mean the task failed in a way that provides no meaningful learning signal. The next analysis therefore asks what caused the failures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate capability failures from broken tasks
The useful distinction is between a model that cannot reach a required logical conclusion and an environment that prevents any model from completing the task. The first can reveal a capability worth training. The second makes failure inevitable in that execution setting, regardless of the agent’s ability.
The team categorizes failures and compares the percentage falling into each category for accepted and rejected tasks. Crawford singles out the logic-error and incomplete-task bars, noting reversals in which bucket has greater representation. The comparison concerns the composition of failures, not simply the overall proportion of tasks that fail.
His interpretation is that accepted tasks produce cleaner failures: the agent fails because the required work is genuinely difficult. Those attempts can become useful samples for improving the model. An execution defect may also produce a failed run, but it does not necessarily expose a skill the model can learn to perform better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hold training resources constant and change task quality
With behavioral and failure analyses supporting the quality distinction, the team moves to reinforcement learning. It trains the same model twice, holding the compute budget and number of tasks constant while changing which task bucket supplies the training material. This tests whether the accepted tasks do more than look better under inspection: do they produce more improvement?
| Training tasks | Reported improvement over the base model |
|---|---|
| Rejected, lower quality | About 1% |
| Accepted, higher quality | About 6% |
These are Crawford’s reported results under the matched model, compute-budget and task-count conditions. The talk does not specify the trained model, evaluation set or performance metric, or whether the percentages mean relative changes or percentage-point gains. Crawford separately calls the difference a “five x uplift”; the rounded values alone do not establish that multiplier.
For Snorkel, the result reinforces an emphasis on human expertise in dataset and RL-environment generation. Crawford presents expert participation as an important contributor to quality, with the platform supporting that work at scale; the comparison itself does not isolate the contribution of experts from other aspects of task construction.
He closes the presentation by pointing to Snorkel’s research work and its curated benchmarks, including the Agentic Coding Benchmark for Terminal-Bench-style tasks. The emphasis carries from training into evaluation: task construction needs enough rigor that differences in model performance are interpretable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Can a model learn around bad tasks?
The first audience question proposes a third training condition: alongside accepted-only and rejected-only datasets, train on all the tasks. Perhaps a model can learn despite the rejected examples when good examples remain available. Crawford does not have a direct result for that experiment.
Instead, he describes work with the Terminal-Bench team, comparisons of Terminal-Bench 1 and 2, and internal analysis of SWE-bench variants. When tracking model improvement and benchmark saturation, tasks that never complete can become a source of noise. Some remain unsolved because they cannot be completed as constructed, rather than because the models still lack the necessary ability.
That is an evaluation problem distinct from the proposed training experiment. Impossible tasks can mask progress and make it harder to tell whether models are improving in their presence. It does not answer whether a mixed training dataset would let the model overcome, ignore or suffer from the rejected tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hidden requirements create artificial difficulty
A second question asks how much difficulty comes from the input: more prescriptive instructions may make a task easier, while less prescriptive instructions may make it harder. Crawford identifies underspecification as a common reason for rejection. The task omits a desired, testable outcome, but the backend tests still require it. The agent then appears to fail a difficult task when the real problem is a mismatch between the request and the verifier.
For a concrete example, consider a request to implement unique_names(names) that returns each name once. A verifier might require first-seen order without the request ever mentioning order:
python
def unique_names(names):
return sorted(set(names))
names = ["Mina", "Ari", "Mina"]
assert unique_names(names) == ["Mina", "Ari"]
The implementation returns each name once, but the assertion rejects its alphabetical ordering. If preserving first-seen order is the intended requirement, the task should say so: “Return each name once, preserving its first occurrence in the input.” That proposed specification repair makes the existing assertion interpretable; it does not change the implementation or make the test pass.
A related defect is an implicit testing dependency that appears neither in the task specification nor in the model’s context. Without knowing that the dependency is required, the agent lacks information needed to approach the task correctly. Both cases separate missing information from genuine reasoning difficulty.
The audience then challenges the assumption that incomplete work is failure at all: real problem solving often involves iteration, not one-shot completion. Crawford agrees with that broader goal. Benchmarks deliberately choose outcomes that can be verified on the backend, while aiming to teach skills that also help with unverifiable results and subsequent steps in an iterative process. That transfer is an intended benefit, conditional on constructing the tasks well, rather than a demonstrated result of this experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When correctness has several acceptable forms
The next question extends the problem to longer horizons and tasks without straightforward verification. Crawford says Snorkel has projects across those areas, though he does not describe a specific long-horizon method. Coding and math offer relatively direct correctness checks; more subjective domains require different evaluation machinery.
Through its Open Benchmarks Grants program, Snorkel partners with people developing benchmarks and evaluations in less verifiable areas. Crawford mentions an unnamed organization exploring emotional, human-centric tasks. In that setting, correctness need not collapse to one acceptable answer: multiple outcomes can be valid while receiving different scores along a spectrum.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make human and model judgments comparable
Multiple acceptable outcomes introduce another question: how should a system handle annotators who disagree, especially across multi-step tasks? Crawford acknowledges that longer horizons, more steps and more dimensions make assessment harder. Snorkel combines human annotators with LLM judges, both to scale expert contributions and to assess agreement.
The shared instrument is a rubric with detailed criteria. It covers high-level qualitative judgments as well as more quantitative comparisons, giving people and LLM judges the same dimensions to assess. Experts supply ground-truth information that informs what the LLM judges should look for.
Quality assessment then compares agreement between humans and between humans and LLM judges. Crawford describes an aim of high agreement without giving numerical thresholds. Explicit pass/fail tests remain the easier case, but the guiding principle extends beyond them: the assessment process itself must be checked, so that its judgments can support meaningful conclusions about the agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Crawford's introduction to the benchmark: 100 coding tasks, four difficulty tiers, expert validation, and evaluation through Harbor.
Crawford explains Terminal-Bench 2.0, improvements to task validation, Snorkel's contributions, and the Harbor framework.
Framework for evaluating agents and running environment-based optimization experiments.
Framework for creating and using isolated execution environments for agentic reinforcement learning.
Research hub covering expert data, evaluation, reinforcement learning, and public benchmarks.
Snorkel's program supporting open benchmarks and evaluation research.
Further reading
Original research describing evaluation of agents on realistic command-line tasks.
Foundational research on creating training data through weak supervision.
Full transcript
Loading transcript…