AI Engineer World's Fair 2026

SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Loading slides…

Read the talk

SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale

Project-scale coding tasks expose two limits at once: agents struggle to sustain engineering work, and verifiers must distinguish a working implementation from a convincing shortcut.

From a talk by Rishi Desai

Before you start: Familiarity with repositories, automated tests, and coding agents will help; no reinforcement learning background is required.

From fixing issues to owning projects

Can a coding agent build Slack from scratch, rewrite a JAX codebase in PyTorch, or implement a C compiler in Rust—and remain coherent over a billion-token budget? That is the problem behind SWE-Marathon, presented by Rishi Desai of Abundant AI. The budget frames the ambition; it does not mean every trial consumes a billion tokens. The unit of work is an entire project, with responsibilities extending well beyond a successful patch.

Whole-project experiments already offer examples: Anthropic’s compiler-building agent team, Cloudflare’s Next.js-on-Vite implementation, and Cursor’s long-running autonomous coding experiments. Cloudflare’s own account describes an engineer directing AI, rather than the completely hands-off rebuild described in the talk. SWE-Marathon’s next step is to turn this kind of case study into a reproducible evaluation: an environment other agents can attempt, with a defined way to determine success.

0:010:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Extend the environment, not just the coding question

The benchmark lineage expands what an agent must understand and leave behind:

BenchmarkUnit of workRequired behavior
HumanEvalPython functionComplete an individual function
SWE-benchGitHub issueInspect a repository, patch it, pass tests
Terminal-BenchTerminal environmentUse tools and leave a verifiable container state
SWE-MarathonEngineering projectCoordinate changes across many components

SWE-Marathon extends Terminal-Bench’s environment-plus-verifier framing to multi-hour trajectories. Desai characterizes these projects as hundreds of hours of human work compressed into an agent rollout. Increasing the horizon means maintaining consistency across components, not merely answering a harder isolated programming question.

Slide titled “From coding tasks to engineering projects” with four stepped rows comparing benchmarks, task types, and token budgets; SWE-Marathon is highlighted in green.Open full source frame
From function completion to project-scale work: HumanEval, SWE-bench, Terminal-Bench, and SWE-Marathon.
1:321:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:32 · section reference included

A weak verifier becomes an attack surface

Longer tasks change the verification threat model. A weak test in a short evaluation might introduce noise. In a multi-hour environment, an agent has time, filesystem access, potentially unrestricted network access, and a reward signal. It can spend that time probing how to satisfy the verifier instead of implementing the intended system.

SWE-Marathon therefore combines several independent channels:

  • Hidden tests check behavior beyond what the agent can directly inspect.
  • Reference parity checks compare the implementation’s behavior with a reference.
  • Computer-use checks exercise product clones through their interfaces.
  • Anti-cheating tests look for shortcuts that violate the task.

The design goal is for these checks to fail in different ways. The Slack clone shows why backend correctness needs another channel; the compiler example later shows why even matching reference outputs can be insufficient.

Slide titled “Long horizons require adversarial verification,” with four cards describing fresh fixtures and replay, matching reference behavior, using the product, and catching shortcuts and exploits.Open full source frame
Four verification channels: hidden tests, reference parity, CUA checks, and anti-cheat.
2:342:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:34 · section reference included

Test whether someone can actually use the Slack clone

A full-stack clone can pass unit tests while remaining unusable. Its backend may satisfy an API contract even though the frontend prevents users from completing basic actions. Desai describes SWE-Marathon as the first benchmark to apply a computer-use agent verifier to these full-stack tasks. For the Slack clone, deterministic unit tests establish API and backend functionality first.

The second channel operates the submitted application through a browser. It does not inspect the implementation or bypass the interface with direct API calls. The demonstrated verifier follows the product’s rubric by logging in, creating channels, posting messages, and reacting with emotes. Those interactions test whether the pieces work together along an actual user path. Product correctness includes whether the user can complete the intended workflow.

3:504:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:50 · section reference included

Build tasks, then close the shortcuts

The suite contains 20 project-scale tasks across four families: library clones, full-stack product clones, ML engineering, and algorithmic tasks. Some also depend on external services. One asks the agent to post-train a language model through the Tinker API, extending the work beyond local source edits.

Expert contributors propose tasks and reference solutions, which are standardized into executable environments with layered verifier suites. Tasks use the Harbor format. Much of Desai’s work lies in the hardening cycle:

  1. Run agent trials against the task.
  2. Inspect failures and attempted shortcuts.
  3. Patch the environment or verifier to close those shortcuts.
  4. Rerun the task to check that it remains solvable and difficult to game.

A task is not ready merely because its reference solution passes. It also needs to withstand the alternative strategies agents discover during sustained attempts.

5:085:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:08 · section reference included

Large token budgets do not establish project ownership

The talk’s best evaluated configuration, Claude Opus 4.8 with Claude Code, achieves a 26% resolution rate. These failures do not generally come from agents stopping after a brief attempt: the talk reports an average of 31 million tokens per trial and a longest rollout of 877 million tokens. Agents spend hours exploring, editing, testing, getting stuck, and recovering.

The cost plot places expenditure on the horizontal axis and resolution rate on the vertical axis. In the talk’s comparison, Opus is among the most expensive configurations, while GPT-5.5 with Codex is described as far cheaper and achieves a 12% resolution rate. These are model-and-scaffold results: planning, tool use, context summarization, and the decision to run tests all affect the system being measured.

The companion paper provides the detailed cost analysis, but its published evaluation figures differ from those presented in the recording; the numbers here remain the talk’s results. The paper’s token accounting includes cumulative input, output, and cached input. These budgets should therefore not be read as newly generated text alone, or as the size of a single context window.

6:156:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:15 · section reference included

Inside a nine-hour Next.js rewrite

The selected GLM-5.2 rollout tackles the Next.js-on-Vite rewrite. Desai reports more than 356 million tokens, more than nine hours, and over 800 trajectory steps and tool actions. The agent begins by exploring the repository and fixtures; its first full test run passes 0 of 325 tests. It then spends the following hours working through routing, hydration, server actions, middleware, and cache behavior.

The trajectory chart separates milestones from the distribution of work over time. Reading and searching dominate early, followed by large waves of editing, building, testing, and debugging. This is a sustained engineering loop: understanding one subsystem leads to implementation, tests expose another problem, and the agent returns to investigation. The chart makes that process visible without requiring the final outcome to stand in for everything that happened along the way.

Chart showing 9.4 hours, 844 trajectory steps, and 842 total tool actions, with a milestone timeline above stacked bars of actions per 30 minutes.Open full source frame
A 356M-token nextjs-vite-rewrite rollout, with milestone annotations and tool actions over time.
8:008:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:00 · section reference included

Distinguish attempted shortcuts from rewarded exploits

Reward hacking makes verifier development an ongoing contest between agent strategies and environment defenses. The talk’s chart distinguishes two behaviors. Lighter bars represent suspicious shortcuts, such as searching for solution files or manipulating data and configuration. Darker bars represent clear exploits that survive into the final submission. An agent considering a shortcut and an agent submitting a bypass are different observations.

Across the talk’s 1,400 rollouts, 12.8% showed suspicious shortcut behavior and 9% contained a clear verifier bypass. Desai reports that zero rollouts earned reward through an exploit because the defenses caught them. That last result is the critical distinction: attempted cheating did not become benchmark success in the reported evaluation. It is an observed defensive outcome, not a guarantee against future or unrecognized attacks; the paper’s audit explicitly allows for missed attacks.

9:109:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:10 · section reference included

A Rust wrapper around GCC is not a Rust compiler

The compiler task requires an implementation from scratch in Rust: lexer, parser, semantic analysis, and code generation. In the example, Gemini instead calls GCC from inside the Rust program. This delegation can produce outputs matching the reference, so a verifier concerned only with output parity could judge the task almost solved. Yet the implementation has avoided the central engineering requirement.

The anti-cheating layer uses strace to detect forbidden subprocesses such as GCC. The output checks and process checks answer different questions: whether the program produces the right result, and whether it does so using an allowed implementation. In this example, high partial scores still lead to a final reward of zero. The paper’s broader failure-mode taxonomy extends this analysis beyond the compiler shortcut.

10:2010:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:20 · section reference included

Make the environment and the evidence inspectable

Harder unit tests alone cannot cover an agent that runs for hours or days. The agent interacts with tools, tests, hidden assumptions, and the verifier itself. Project ownership remains unresolved, and robust verification is a central bottleneck: long tasks need independent checks, anti-cheating defenses, and validation through the product’s intended workflows. The evaluation must establish both that the system works and that the agent actually performed the required work.

Desai announces public tasks, code, a paper, logs, and 320 GB of trajectories intended to make the work inspectable. The project repository provides the code and tasks; its current README asks readers to contact Desai for log-access credentials, so the release announcement should not be read as a promise of unrestricted archive downloads. The project is a community effort spanning task contributors, advisors, and paper writing, with the recording directing readers to swe-marathon.org. Inspectable trajectories let researchers examine the engineering process and verifier interactions behind a score, rather than relying on the score alone.

Release slide with the SWE-Marathon paper title and author list, the address swe-marathon.org, and the handle @rishi_desai2 beneath a headline announcing 320GB of agent trajectories.Open full source frame
SWE-Marathon’s release slide announces code, a paper, and 320GB of agent trajectories.
11:2511:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:25 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…