AI Engineer World's Fair 2025
CI in the Era of AI: From Unit Tests to Stochastic Evals
Loading slides…
Read the talk
CI in the Era of AI: From Unit Tests to Stochastic Evals
Zed’s streaming editor shows how broad agent evaluations can expose failures that become focused model tests and, ultimately, deterministic regression tests.
From a talk by Nathan Sobo
Before you start: Familiarity with automated tests, LLM tool calls, and basic Rust syntax will help with the code examples.
How do you test an editor that edits for you?
An agent that searches a repository and edits its files adds a new reliability problem to a code editor: the editor must work correctly even when the model produces unexpected output. Nathan Sobo introduces that problem with a massively sped-up recording of agentic editing in Zed. Built from scratch in Rust rather than forked from VS Code, Zed organizes its rendering around the GPU, much like a video game. Sobo describes roughly 1,200 lines of shader code and a rendering target of 120 frames per second. Making generated edits dependable requires the same empirical attention as making the editor itself responsive.
Zed already has an extensive testing culture. Sobo estimates tens of thousands of tests; his joke that the editor would otherwise crash every eight seconds conveys how much the product depends on them. One collaborative editing test starts a server and two clients, passing in a simulated scheduler to control concurrent activity. The shown test runs 50 randomized iterations.
The scheduler makes failures reproducible. If a particular interleaving of network packets breaks the system, the team can replay that interleaving, freeze it, and investigate it under full control. Sobo’s reference to a failure on a hypothetical 375,000th iteration illustrates why replay matters; it is not the run count of the displayed test. Until the AI features arrived, he reports that this control let Zed keep even concurrency tests deterministic and avoid flaky CI tests.
An LLM changes that arrangement. Even if the team controlled sampling directly, changing an input token could change the generated output. A reproducible software harness does not make model behavior stable across changed inputs. Zed therefore needed to embrace stochastic behavior. Its first major agent evaluation resembled SWE-bench: a broad task that asks an agent to make a repository change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A broad failure becomes a search test
A data-driven evaluation starts with inputs and examines outputs. A conventional software test usually expresses a more specific contract through pass/fail assertions. Zed brought those approaches together: its evaluation program compiles a headless copy of the editor, checks out a repository, and runs the agent. The conversation itself can be a function, allowing test code to drive the interaction and make granular assertions about what the agent did.
That granularity matters when a large task fails. There are many possible causes, and an overall failure gives little guidance about which one to repair. In the demonstrated task, the agent must add a window argument to a tool’s run trait method in a specified file. The original grep tool returned too little useful context around the match. The problem lay partly in what the model could see.
The repair became a deterministic search test: set up the project, search for fn run, and check the returned context. Zed uses Tree-sitter to expand matches to syntactic boundaries. The displayed expected result includes the full function signature through its return type, giving the agent a coherent unit of code to inspect. Sobo reports a substantial improvement in agent behavior from this change, without supplying a numerical score. A stochastic evaluation had exposed an ordinary algorithmic defect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make edits stream, then test the protocol
Editing initially happened through tool calls. In the implementation Sobo describes, their key/value content arrived together rather than streaming usefully into the editor; this is a limitation of that implementation, not a claim about every tool-calling API. Zed changed the interaction to a two-step protocol:
- Make a small tool call describing the edits.
- Request old-text/new-text blocks that can stream back incrementally.
At this stage, Zed reused the same model for the second request because the context was already in its cache. The tradeoff was a new parsing responsibility: the editor now had to interpret streamed text and tolerate irregular model output.
The team added an evaluation function to its ordinary test suite. In the shown evaluation, Sobo requests 200 runs and says the eventual build gate requires all 200 to pass. That was a threshold the team reached over time, not its starting point. The test loads a conversation with the model and uses an LLM judge to assess the result; other cases use programmatic assertions. Repeated evaluation establishes a measurable acceptance condition, while failures identify smaller problems to investigate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three algorithms beneath a reliable streamed edit
Much of the work underneath the model is conventional software engineering. The input may vary, but the implementation can still have precise, testable behavior:
- Incremental parsing. The demonstrated parser test runs 100 iterations, randomly splitting the input at arbitrary chunk boundaries. A packet boundary must not change how the completed text is interpreted.
- Approximate matching. A dynamic programming algorithm finds an approximate match when model-generated old text differs slightly from the buffer. Sobo credits this tolerance with preventing many tool-call failures. The matching algorithm itself can be tested deterministically.
- Streaming diffs. As new text arrives, the editor compares it with the old text. If an old passage is absent from the current output, the editor must distinguish a deletion from text that has not arrived yet. That decision is another deterministic part of the system.
These tests isolate the machinery that turns model output into edits. Live-model evaluations remain useful, but they need not carry the entire burden of validating parsing, matching, and diff behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompt improvements leave cases the code must handle
Insertions at the beginning or end of a document revealed a weakness in the old-text/new-text protocol. The model sometimes supplied an empty old-text tag. Without old text, the matching algorithm has no useful anchor for locating the change. A prompt instruction improved the behavior, but Sobo reports that empty old-text tags still appeared roughly 1–2% of the time after the prompt change. The team therefore also simulated that output in tests to exercise robust handling of the case.
Another failure crossed XML tag names: an old-text block would close with a new-text tag. Sobo reports that mismatched tags initially caused failures about 40% of the time; adding “Always close all tags properly” brought success to about 95%, leaving roughly 5% to handle. These figures concern the described tag-handling case, with no model version, sample size, or sampling settings specified. The remaining malformed output became input to a deterministic robustness test. Prompting reduced the frequency of a failure; parser behavior determined what happened when it still occurred.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover an edit whose indentation is wrong
Indentation exposed a different kind of near miss. The evaluation places an inner function inside an outer function and asks the model to replace todo!() with return 42. A minimal Rust version of that starting shape is:
rust
fn outer() -> i32 {
fn inner() -> i32 {
todo!()
}
inner()
}
The model could identify the right function and the right replacement while flattening its indentation. Its old text might start with fn inner at the left margin, even though the buffer contains leading spaces. Rejecting the edit solely for that difference would discard otherwise useful output.
Zed handles this by detecting an indentation delta between the matched buffer text and the model’s text, then normalizing the edit. For the Rust example, a flattened replacement such as:
rust
fn inner() -> i32 {
return 42;
}
needs to retain its relative indentation while moving into the outer function’s indentation level. The resulting buffer should be:
rust
fn outer() -> i32 {
fn inner() -> i32 {
return 42;
}
inner()
}
The important operation is to reconcile the model’s indentation with the existing buffer, rather than treating every whitespace difference as a failed match. Sobo then drives that behavior into a deterministic fixture using lorem ipsum dolor sit with an unusual indentation pattern. An outdented replacement changes ipsum dolor sit to ipsum dolor sit amet; indentation normalization lets the test handle it correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Escaping stops at a prompt repair
Rust raw strings introduced another failure mode. Their syntax can preserve quotation marks inside the string, but a simple request to tweak the content sometimes prompted the model to transform the escaping as well. Sobo particularly observed Gemini producing HTML escape codes, adding backslashes, and double-escaping newlines. Those transformations could change text that the edit was supposed to preserve.
For this case, the team used a prompt fix. At the time of the talk, it had not added further algorithmic detection or repair for the escaping behavior. That marks the actual boundary of the implementation: indentation had a normalization algorithm backed by deterministic tests, while the escaping problem remained addressed through model instructions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move each understood failure into a smaller test
Rigorous testing still supplies the foundation. What changes is the need to run model interactions repeatedly and define a pass/fail threshold. The search and streaming-edit examples show that many failures arise in familiar software concerns: the context a tool returns, the boundaries a parser accepts, or the way text is matched and transformed.
The development process moves through three levels:
| Test level | What it exercises | What a failure reveals |
|---|---|---|
| Broad agent eval | A complete repository task | A broken user experience |
| Focused stochastic test | One behavior with a live model | A recurring model-output problem |
| Deterministic regression test | A specific algorithm or captured case | A reproducible implementation defect |
The narrower test does not replace the broader one. It makes an understood failure cheaper to reproduce and more precise to diagnose. At the time of the talk, Zed ran these evaluations in its existing test suite and infrastructure, without a special external evaluation framework. The empirical method and much of the engineering skill remained the same; model variability became another input the system had to accommodate.
Sobo closes by inviting contributions to the Zed source repository, describing the editor as open source under the GPL. His practical outcome is personal: with the Claude 4 models, he reports finally being able to write Rust agentically with high efficiency. The parser fixes, matching behavior, focused evaluations, and ordinary regression tests are the work that made that experience possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Zed's source code, build instructions, contribution guidance, and component licensing.
Documentation for generating parsers and maintaining syntax trees incrementally as source code changes.
The original benchmark paper evaluates models on editing repositories to resolve real GitHub issues.
Further reading
- Zed's agentic editing launchArticle
The May 2025 launch announcement introduces the Agent Panel and demonstrates agentic editing in Zed.
- Maintaining 120 FPS in GPUIArticle
A technical account of Metal rendering, buffering, and macOS display synchronization problems behind smooth scrolling.
Full transcript
Loading transcript…