AI Engineer World's Fair 2026

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Loading slides…

Read the talk

Everything Is a Rollout

Harbor connects agent evaluation, production work, and training through one execution loop: give an agent a task, preserve its trajectory, and grade the resulting state.

From a talk by Alex Shaw and Ryan Marten

Before you start: Familiarity with coding agents, command-line tools, and basic model evaluation will help; no reinforcement-learning background is required.

When reading the code was enough

A coworker sends you a 400-line pull request. You get coffee, sit down, admire the small refactors and clear method names, and leave a comment about an API that needs to be more extensible. This is the 2018 code-review scene Alex Shaw borrows from StaySaaSY to introduce Harbor, the agent evaluation and reinforcement-learning environment framework he works on at Laude Institute. The surrounding nostalgia includes Avengers: Infinity War, the arrival of GPT-1, Musical.ly becoming TikTok, and the novelty of cordless AirPods. The familiar engineering ritual is the important part: reading the implementation gives the reviewer a working explanation of what it will do.

Slide showing a StaySaaSY post about coffee, elegant micro-refactors, method names, and API extensibility during code review.Open full source frame
StaySaaSY’s description of reviewing a 400-line pull request in 2018.

That workflow already feels distant to Shaw, even though he places the change only six to eighteen months earlier, depending on when someone adopted coding agents. His deliberately sharp characterization of the earlier era is that software engineering meant knowing what the code would do before running it. François Chollet supplies the next step: generated code should be treated as a black-box artifact whose behavior and generalization must be established through empirical evaluation. Once confidence depends on observing behavior, agent development needs tools for experiments, not just tools for inspecting implementations.

0:160:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

A phone number reveals the trade-off

Consider a program that extracts phone numbers from text using a regular expression. Its accepted format is explicit in the implementation. Shaw says he can predict the regex program’s behavior with 100% confidence across one million executions. A small illustration makes the boundary visible:

python

import re

texts = [
    "Call 415-555-0132.",
    "Call (415) 555 0132.",
]
pattern = re.compile(r"\b\d{3}-\d{3}-\d{4}\b")

for text in texts:
    print(pattern.findall(text))

The first input matches; the second does not. Predictability does not mean broad coverage—it means the rule’s limitations are also predictable.

Shaw then replaces the regex with a model call asking GPT-5.5 to extract the phone number. He expects this version to handle unusual formatting that the regex would miss. But he cannot promise identical, known output over the same hypothetical million executions. This is a simple task that he expects to work almost every time; the uncertainty becomes more consequential as tasks grow more complex.

The change is broader than generated code. Shaw edits Chollet’s statement twice: agentic coding becomes agents in general, and generated code becomes agent performance. The behavior of the whole agent is what must be evaluated empirically. Inspecting its prompt, tools, or source code does not establish how reliably it will accomplish a task.

François Chollet post with red edits replacing “Agentic coding” with “agents are” and “Generated code” with “agent performance.”Open full source frame
An annotated François Chollet post generalizes the claim to agents and agent performance.
3:303:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:30 · section reference included

The machine-learning loop, translated

Treating agent development as machine learning exposes both the tools it needs and the mistakes it can repeat. Shaw maps familiar ML concepts onto the parts an agent builder can change:

Machine learningAgent development
Training dataEnvironments
Test and validation setsEvals, also built from environments
Model weightsSkills, prompts, tools, model choice or updates
Loss functionEnvironment rewards and feedback
OptimizerText optimization or a coding-agent loop
Gradient descent stepA pull request into the repository
OverfittingReward hacking or ordinary overfitting

An optimizer such as GEPA, or simply a coding agent running in a loop, can propose changes to the agent’s behavior. The repository change plays the role of an update step; evaluation determines whether that update helped. The analogy also preserves the warning: optimizing against a reward can produce reward hacking, and repeated improvement on familiar tasks can still overfit.

Traditional machine learning accumulated libraries, platforms, and companies around these operations. Harbor and other emerging frameworks are beginning to provide corresponding infrastructure for agents. Shaw frames the opportunity expansively: he describes Databricks as worth hundreds of billions of dollars and suggests that agent users and builders may already outnumber historical ML practitioners by orders of magnitude. These are his market estimates; the concrete engineering requirement is empirical evaluation. How do you actually measure an agent doing work?

5:225:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:22 · section reference included

From an environment to an evaluation result

An environment needs three things:

  • Instruction: What the agent is supposed to accomplish.
  • Sandbox: A virtual computer in which it can attempt the work.
  • Verifier: A way to decide whether the requested outcome was achieved, using programmatic tests, a rubric, or another agent inspecting the result.

A time limit or another stopping condition bounds the attempt. Harbor packages the specification in a directory so that it can move between people and systems. Shaw uses environment for this whole bundle; Harbor’s documentation calls the bundle a task and distinguishes it from the execution environment. The directory format makes the instruction, starting conditions, and grading procedure portable together.

A rollout turns that specification into an observed attempt:

  1. Start the task’s sandbox and give the agent access to it.
  2. Run the agent, either outside the sandbox while executing commands into it, or inside the sandbox itself.
  3. Continue until a stopping condition, retaining the agent’s trajectory.
  4. Give the resulting sandbox state to the verifier for grading.
  5. Stop the sandbox and retain the verifier’s reward or set of rewards.
  6. Aggregate rewards across the dataset’s rollouts to produce the agent’s evaluation result.

The trajectory records the attempt; the reward grades its outcome. Keeping both matters because a score tells you whether an attempt succeeded, while the trajectory gives you material for understanding and improving the behavior that produced it.

This lifecycle is a minimal version of the execution machinery. Harbor also supports multistep rollouts, verification in a separate sandbox, artifact collection, and simulated users. Those variations change what happens inside the loop without removing its central sequence: attempt the task, preserve evidence, and assess the result.

Harbor’s open-source framework combines three roles: an environment format, a parallel rollout runner, and a registry of training and evaluation environment sets. Its intended execution contract is any agent, any model, any sandbox, and any task. At the time of the talk, Shaw estimates that the registry contains 300–400 evaluation sets and says two or three Harbor-based benchmarks appeared that day. The purpose of a shared format is to make these tasks interoperable, increasing the speed at which people can exchange and use evaluation or training data.

8:028:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:02 · section reference included

Own the eval before choosing the model

Shaw’s proposed audience is every company that uses computers: if a workflow happens on a computer, investigate whether an agent can automate part or all of it. He cites Satya Nadella’s appearance on the Applied Compute podcast as expressing the right order of operations: identify the outcome that matters, establish how to grade it, and then welcome all models.

Build the eval first, then optimize against it. Owning the grading procedure lets a company compare models on its own work rather than relying on a brand, a public benchmark, or another organization’s priorities. It also makes the cost–performance trade-off explicit: the most capable option is not automatically the best choice at every price, and the cheapest option is useful only if it achieves the required outcome.

Shaw identifies four targets from the evaluations he sees people building:

  • Agents building your product. Internal repositories supply tasks that reflect the work your engineers actually do. Ramp SWE-Bench is his example of an internal-codebase benchmark that supports informed agent and model selection instead of maximizing token spend.
  • Agents using your product. Evaluate whether agents can accomplish tasks through the product, then improve the product’s usability for them. Headless access becomes part of the product surface; Shaw points to HubSpot and Stripe as examples of developer-oriented products.
  • Agents powering product features. Measure the behavior of agent capabilities delivered to customers.
  • Agents automating internal processes. Grade the work agents perform inside the business.

A company can need several of these evaluations at once. They answer different questions: whether an agent can change the software, operate it, deliver one of its features, or carry out the surrounding business work.

11:1711:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:17 · section reference included

Running evaluations in parallel

A person—or an agent—can start an evaluation through harbor run. Shaw’s recorded example uses Terminal-Bench 2.1 with Codex and GPT-5.5, configured for 64 parallel rollouts. He then demonstrates a launch command intended to move orchestration to Harbor’s servers, describing that hosted capability as rolling out soon. This is the talk’s interface and configuration; current dataset syntax differs, so the current evaluation guide is the appropriate reference for a new run.

Hosted execution enables the workflow Shaw wants: submit a large job, leave it running, and inspect the results later. His example of waking up to ten thousand completed rollouts is an illustration of that workflow, not measured overnight throughput. The demonstrated evaluation does not finish within the video. That wait is itself a reason to parallelize: long individual attempts make sequential execution a slow way to test an agent change.

The tasks themselves can also become a product. Experts specify work they want automated and sell those tasks to labs training models on the corresponding capabilities. Shaw describes the market around Harbor data and other data types together as probably worth multiple billions of dollars. The mechanism is straightforward: a portable task gives a buyer a repeatable situation in which to train or measure a desired skill.

14:1514:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:15 · section reference included

Production work uses the same machinery

Running an agent with a model in a sandbox on a task does not require evaluation or training to be the end goal. The output can be useful production work. Shaw calls this agentic MapReduce: distribute inputs across many sandboxed agents, run them in parallel, and aggregate their results, potentially with another agent.

The inputs might be trajectories to inspect for reward hacking, receipts to process for reimbursements, Obsidian notes to search semantically, or pull requests to examine while answering a question. In each case, the map stage handles separate pieces of work and the reduce stage combines their findings. Shaw describes this as an emergent use of Harbor rather than its original purpose.

15:4816:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:48 · section reference included

Turn session corrections into the next evaluation

Harbor Exec makes that pattern concrete. Shaw says the feature was built for this usage and would go away soon, without naming a replacement. In the recorded harbor exec demonstration, the inputs are his Codex sessions from the June twenties. The map instruction asks an agent to find occasions when he corrected the coding agent and write an analysis.json containing the mistake, reason, and correction.

The reduce instruction asks for recurring mistakes and failure categories to be summarized in a concise feedback.md. The two stages deliberately use different agents or models. Cursor CLI handles the map work because Shaw values its speed and cost there; he names Fable 5 for reduction because he cares about the quality of the combined analysis. Modal supplies the cloud sandboxes.

Shaw limits the demonstration to 32 input sessions; that is an input limit, not a stated concurrency setting. He suggests that a larger run could process a thousand or ten thousand sessions. The demo is orchestrated locally on his computer while the agents execute in cloud sandboxes. After the parallel Cursor rollouts, the reduce step produces the recurring-mistakes report.

That report closes a development loop. Recurring mistakes can become the next set of Harbor tasks, giving Shaw a targeted way to evaluate a different agent or improve a skill. Historical user corrections are therefore more than a record of frustration: they identify behaviors worth turning into repeatable tests.

Harbor report titled “Feedback: Recurring Mistakes & Failure Categories,” with three numbered categories and a cross-cutting pattern section.Open full source frame
The demo’s feedback report lists recurring mistakes and failure categories.
16:5016:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:50 · section reference included

Rollout outputs become optimization inputs

The same preserved outputs support several forms of improvement:

MethodRollout material used
Supervised fine-tuningTrajectories
Reinforcement learningToken trajectories and rewards
Text-based or evolutionary optimizationTrajectory text and evaluation feedback

Shaw points to Tinker’s Harbor integration for reinforcement learning. GEPA and related evolutionary methods instead use textual evidence to search for improvements to a harness or skill. The execution loop stays recognizable; what changes is the process that consumes its outputs and proposes the next update.

18:3618:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:36 · section reference included

A shared format across different kinds of work

The closing examples show how widely the task-and-rollout structure can apply. Shaw introduces Frontier Suite as an ultra-long-horizon software-engineering benchmark, then Handshake’s BankerToolBench for investment banking. He cites swyx’s statement that Cognition migrated all its evaluations to Harbor, followed by RuneBench, which measures agents playing RuneScape.

Shaw also names Scale’s Atlas Suite and reports that Poolside uses Harbor for all its model-training evaluations. The Scale Labs announcement shown in the recording describes refreshed SWE-Atlas Codebase QnA leaderboard results, use of Harbor for all models, and a dataset available in Harbor format. These examples put the same execution infrastructure beneath both benchmark publication and repeated model-development evaluations.

Scale Labs post describing refreshed SWE-Atlas Codebase QnA leaderboard results, use of Harbor for all models, and a dataset in Harbor format.Open full source frame
Scale Labs announces its SWE-Atlas transition to Harbor.

The survey then moves to systems that improve agents: Kevin Gu’s AutoAgent takes a Harbor eval as the input to a self-optimizing agent, and AfterQuery used Harbor to post-train a model. Shaw names Cognition’s Frontier Code benchmark, LangChain’s integration of Deep Agents and its sandboxes, and Snorkel’s Senior SWE-bench, which he describes as released that day to measure work under ambiguity with behavioral feedback.

Harbor is also growing its team. Shaw closes by inviting interested contributors to contact him through Twitter DMs and directing viewers to the documentation link on his profile. The invitation follows the range of work just shown: building tasks, running agents, understanding failures, and feeding the resulting evidence into the next improvement.

19:0919:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:09 · section reference included

Resources

From the talk

  • Definitions of tasks, datasets, agents, container environments, trials, and jobs.

Updates since the talk

Transcript download
Full transcript

Loading transcript…