# CI in the Era of AI: From Unit Tests to Stochastic Evals

Nathan Sobo

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/WXy8Yy9xGss-ci-in-era-ai-from-unit-tests) · [Video](https://www.youtube.com/watch?v=WXy8Yy9xGss)

## CI in the Era of AI: From Unit Tests to Stochastic Evals

Zed’s streaming editor shows how broad agent evaluations can expose failures that become focused model tests and, ultimately, deterministic regression tests.

Familiarity with automated tests, LLM tool calls, and basic Rust syntax will help with the code examples.

### How do you test an editor that edits for you?

[0:15](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=15s)

An agent that searches a repository and edits its files adds a new reliability problem to a code editor: the editor must work correctly even when the model produces unexpected output. Nathan Sobo introduces that problem with a massively sped-up recording of agentic editing in [Zed](https://zed.dev/blog/fastest-ai-code-editor). Built from scratch in Rust rather than forked from VS Code, Zed organizes its rendering around the GPU, much like a video game. Sobo describes roughly 1,200 lines of shader code and a rendering target of 120 frames per second. Making generated edits dependable requires the same empirical attention as making the editor itself responsive.

Zed already has an extensive testing culture. Sobo estimates tens of thousands of tests; his joke that the editor would otherwise crash every eight seconds conveys how much the product depends on them. One collaborative editing test starts a server and two clients, passing in a simulated scheduler to control concurrent activity. The shown test runs 50 randomized iterations.

![Rust test code in Zed showing a server, two clients, a shared project, buffer setup, and concurrent edits.](https://ai.engineer/api/talk-essay-media/5fdf658a9e597601cb515a6388dd58b73490343a009f5bf716c4c1d12b23f33c)

A collaborative editing test creates two clients, shares a project, and performs concurrent edits.

The scheduler makes failures reproducible. If a particular interleaving of network packets breaks the system, the team can replay that interleaving, freeze it, and investigate it under full control. Sobo’s reference to a failure on a hypothetical 375,000th iteration illustrates why replay matters; it is not the run count of the displayed test. Until the AI features arrived, he reports that this control let Zed keep even concurrency tests deterministic and avoid flaky CI tests.

An LLM changes that arrangement. Even if the team controlled sampling directly, changing an input token could change the generated output. **A reproducible software harness does not make model behavior stable across changed inputs.** Zed therefore needed to embrace stochastic behavior. Its first major agent evaluation resembled [SWE-bench](https://arxiv.org/abs/2310.06770): a broad task that asks an agent to make a repository change.

### A broad failure becomes a search test

[2:57](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=177s)

A data-driven evaluation starts with inputs and examines outputs. A conventional software test usually expresses a more specific contract through pass/fail assertions. Zed brought those approaches together: its evaluation program compiles a headless copy of the editor, checks out a repository, and runs the agent. The conversation itself can be a function, allowing test code to drive the interaction and make granular assertions about what the agent did.

That granularity matters when a large task fails. There are many possible causes, and an overall failure gives little guidance about which one to repair. In the demonstrated task, the agent must add a `window` argument to a tool’s `run` trait method in a specified file. The original grep tool returned too little useful context around the match. The problem lay partly in what the model could see.

The repair became a deterministic search test: set up the project, search for `fn run`, and check the returned context. Zed uses [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) to expand matches to syntactic boundaries. The displayed expected result includes the full function signature through its return type, giving the agent a coherent unit of code to inspect. Sobo reports a substantial improvement in agent behavior from this change, without supplying a numerical score. A stochastic evaluation had exposed an ordinary algorithmic defect.

![Zed editor displaying expected grep output with the run function signature highlighted, including its window argument and return type.](https://ai.engineer/api/talk-essay-media/2f8291bab3bab1265431da4d597e57ce0dd0f2f63cb68153baca7fdf826d3904)

The grep test’s expected output includes the full function signature.

### Make edits stream, then test the protocol

[5:04](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=304s)

Editing initially happened through tool calls. In the implementation Sobo describes, their key/value content arrived together rather than streaming usefully into the editor; this is a limitation of that implementation, not a claim about every tool-calling API. Zed changed the interaction to a two-step protocol:

1. Make a small tool call describing the edits.
2. Request old-text/new-text blocks that can stream back incrementally.

At this stage, Zed reused the same model for the second request because the context was already in its cache. The tradeoff was a new parsing responsibility: the editor now had to interpret streamed text and tolerate irregular model output.

![Streaming Edits notes listing tool-calling limitations, XML chunks split across packets, and imperfect model behavior.](https://ai.engineer/api/talk-essay-media/aba0e8836e2949359da4c1c30d9fbc0544c5cfd7f2eecba4e65a86f053aa6938)

Streaming edits face tool-calling, incremental parsing, and model-output challenges.

The team added an evaluation function to its ordinary test suite. In the shown evaluation, Sobo requests 200 runs and says the eventual build gate requires all 200 to pass. That was a threshold the team reached over time, not its starting point. The test loads a conversation with the model and uses an LLM judge to assess the result; other cases use programmatic assertions. Repeated evaluation establishes a measurable acceptance condition, while failures identify smaller problems to investigate.

### Three algorithms beneath a reliable streamed edit

[6:48](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=408s)

Much of the work underneath the model is conventional software engineering. The input may vary, but the implementation can still have precise, testable behavior:

- **Incremental parsing.** The demonstrated parser test runs 100 iterations, randomly splitting the input at arbitrary chunk boundaries. A packet boundary must not change how the completed text is interpreted.
- **Approximate matching.** A dynamic programming algorithm finds an approximate match when model-generated old text differs slightly from the buffer. Sobo credits this tolerance with preventing many tool-call failures. The matching algorithm itself can be tested deterministically.
- **Streaming diffs.** As new text arrives, the editor compares it with the old text. If an old passage is absent from the current output, the editor must distinguish a deletion from text that has not arrived yet. That decision is another deterministic part of the system.

These tests isolate the machinery that turns model output into edits. Live-model evaluations remain useful, but they need not carry the entire burden of validating parsing, matching, and diff behavior.

### Prompt improvements leave cases the code must handle

[8:06](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=486s)

Insertions at the beginning or end of a document revealed a weakness in the old-text/new-text protocol. The model sometimes supplied an empty old-text tag. Without old text, the matching algorithm has no useful anchor for locating the change. A prompt instruction improved the behavior, but Sobo reports that empty old-text tags still appeared roughly 1–2% of the time after the prompt change. The team therefore also simulated that output in tests to exercise robust handling of the case.

Another failure crossed XML tag names: an old-text block would close with a new-text tag. Sobo reports that mismatched tags initially caused failures about 40% of the time; adding “Always close all tags properly” brought success to about 95%, leaving roughly 5% to handle. These figures concern the described tag-handling case, with no model version, sample size, or sampling settings specified. The remaining malformed output became input to a deterministic robustness test. **Prompting reduced the frequency of a failure; parser behavior determined what happened when it still occurred.**

### Recover an edit whose indentation is wrong

[9:40](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=580s)

Indentation exposed a different kind of near miss. The evaluation places an inner function inside an outer function and asks the model to replace `todo!()` with `return 42`. A minimal Rust version of that starting shape is:

```rust
fn outer() -> i32 {
    fn inner() -> i32 {
        todo!()
    }

    inner()
}
```

The model could identify the right function and the right replacement while flattening its indentation. Its old text might start with `fn inner` at the left margin, even though the buffer contains leading spaces. Rejecting the edit solely for that difference would discard otherwise useful output.

Zed handles this by detecting an **indentation delta** between the matched buffer text and the model’s text, then normalizing the edit. For the Rust example, a flattened replacement such as:

```rust
fn inner() -> i32 {
    return 42;
}
```

needs to retain its relative indentation while moving into the outer function’s indentation level. The resulting buffer should be:

```rust
fn outer() -> i32 {
    fn inner() -> i32 {
        return 42;
    }

    inner()
}
```

The important operation is to reconcile the model’s indentation with the existing buffer, rather than treating every whitespace difference as a failed match. Sobo then drives that behavior into a deterministic fixture using `lorem ipsum dolor sit` with an unusual indentation pattern. An outdented replacement changes `ipsum dolor sit` to `ipsum dolor sit amet`; indentation normalization lets the test handle it correctly.

### Escaping stops at a prompt repair

[11:07](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=667s)

Rust raw strings introduced another failure mode. Their syntax can preserve quotation marks inside the string, but a simple request to tweak the content sometimes prompted the model to transform the escaping as well. Sobo particularly observed **Gemini** producing HTML escape codes, adding backslashes, and double-escaping newlines. Those transformations could change text that the edit was supposed to preserve.

For this case, the team used a prompt fix. At the time of the talk, it had not added further algorithmic detection or repair for the escaping behavior. That marks the actual boundary of the implementation: indentation had a normalization algorithm backed by deterministic tests, while the escaping problem remained addressed through model instructions.

### Move each understood failure into a smaller test

[12:15](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=735s)

Rigorous testing still supplies the foundation. What changes is the need to run model interactions repeatedly and define a pass/fail threshold. The search and streaming-edit examples show that many failures arise in familiar software concerns: the context a tool returns, the boundaries a parser accepts, or the way text is matched and transformed.

The development process moves through three levels:

| Test level | What it exercises | What a failure reveals |
| --- | --- | --- |
| Broad agent eval | A complete repository task | A broken user experience |
| Focused stochastic test | One behavior with a live model | A recurring model-output problem |
| Deterministic regression test | A specific algorithm or captured case | A reproducible implementation defect |

The narrower test does not replace the broader one. It makes an understood failure cheaper to reproduce and more precise to diagnose. At the time of the talk, Zed ran these evaluations in its existing test suite and infrastructure, without a special external evaluation framework. The empirical method and much of the engineering skill remained the same; model variability became another input the system had to accommodate.

Sobo closes by inviting contributions to the [Zed source repository](https://github.com/zed-industries/zed), describing the editor as open source under the GPL. His practical outcome is personal: with the **Claude 4** models, he reports finally being able to write Rust agentically with high efficiency. The parser fixes, matching behavior, focused evaluations, and ordinary regression tests are the work that made that experience possible.

## Resources

- [Zed source repository](https://github.com/zed-industries/zed) — Zed's source code, build instructions, contribution guidance, and component licensing.
- [Zed's agentic editing launch](https://zed.dev/blog/fastest-ai-code-editor) — The May 2025 launch announcement introduces the Agent Panel and demonstrates agentic editing in Zed.
- [Tree-sitter documentation](https://tree-sitter.github.io/tree-sitter/) — Documentation for generating parsers and maintaining syntax trees incrementally as source code changes.
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770) — The original benchmark paper evaluates models on editing repositories to resolve real GitHub issues.
- [Maintaining 120 FPS in GPUI](https://zed.dev/blog/120fps) — A technical account of Metal rendering, buffering, and macOS display synchronization problems behind smooth scrolling.

## Slides

### Programmatic evaluation example

The example pins a Zed repository revision. The eval runner builds a headless copy of Zed, checks out a repo, runs the agent, and supports programmatic assertions about its actions.

[3:08](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=188s) [Rust code defining an evaluation example and its metadata in Zed.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d3452f25c75ebac2151819d60da903b11fb6bdeaeff999fbb2ff555065a64e4.webp)

use std::path::Path;

use agent_settings::AgentProfileId;
use anyhow::Result;
use async_trait::async_trait;

use crate::example::{Example, ExampleContext, ExampleMetadata, JudgeAssertion, LanguageServer};

pub struct AddArgToTraitMethod;

#[async_trait(?Send)]
impl Example for AddArgToTraitMethod {
    fn meta(&self) -> ExampleMetadata {
        ExampleMetadata {
            name: "add_arg_to_trait_method".to_string(),
            url: "https://github.com/zed-industries/zed.git".to_string(),
            revision: "f69aeb6311dde3c0b8979c293d019d66498d54f2".to_string(),
            language_server: Some(LanguageServer {
                file_extension: "rs".to_string(),
                allow_preexisting_diagnostics: false,
            }),
            max_assertions: None,
            profile_id: AgentProfileId::default(),
            [illegible]

Interface text:
Zed Nightly
zed  ai-worlds-fair
00_intro.md
01_deterministic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_ba[illegible]
worlds_fair_talk/slides/03_programmatic_eval_example.rs
Nathan Sobo, 21 hours ago
AIE
Microsoft
smol ai

### Programmatic evaluation assertions

The eval checks each tool file for a window argument, then checks for the underscore prefix. Granular assertions help isolate failures in a larger evaluation.

[3:52](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=232s) [Selected Rust code checks added window arguments and asserts the results.](https://assets.ai.engineer/sites/aiecode2025/sha256/55c589c1ba182701384b55993ba645dc3f1475d744acfb8142422d36f2a90da7.webp)

let edits = cx.edits();

for tool_name in add_ignored_window_paths {
    let path_str = format!("crates/assistant_tools/src/{}.rs", tool_name);
    let edits = edits.get(Path::new(&path_str));

    let ignored = edits.map_or(false, |edits| {
        edits.has_added_line("        _window: Option<gpui::AnyWindowHandle>,\n")
    });
    let uningored = edits.map_or(false, |edits| {
        edits.has_added_line("        window: Option<gpui::AnyWindowHandle>,\n")
    });

    cx.assert(ignored || uningored, format!("Argument: {}", tool_name))
        .ok();

    cx.assert(ignored, format!("`_` prefix: {}", tool_name))
        .ok();
}

[illegible] Adds unignored argument to `batch_tool`
[illegible]

Interface text:
Zed Nightly
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs  08_ba[illegible]
worlds_fair_talk/slides/03_programmatic_eval_example.rs > impl Example for AddArgToTraitMethod > async fn conversation
Nathan Sobo, 21 hours ago
84:23 (12 lines, 475 characters)  Rust
AIE  Microsoft  smol ai

### Original grep output

The original grep output cuts off the Tool::run signature after three arguments, leaving the model incomplete context for adding a window argument.

[4:02](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=242s) [Zed displays a grep match with a partial Rust function signature.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed092b99cca703bd40bd4c90efdd6c68198c3ac7ba3f147716a31ebb1c1eedf3.webp)

Found 1 matches:

## Matches in crates/assistant_tool/src/assistant_tool.rs

### Lines 238-241

```rs
    }

    /// Runs the tool with the provided input.
    fn run(
        self: Arc<Self>,
        input: serde_json::Value,
        request: Arc<LanguageModelRequest>,
```

Interface text:
Zed Nightly
zed  ai-worlds-fair
00_intro.md
01_deterministic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_ba[illegible]
worlds_fair_talk/slides/04_grep_tool_output_old.md
Nathan Sobo, 21 hours ago
1:1  Markdown
AIE
Microsoft
smol ai

### Programmatic evaluation prompt

The eval asks the model to add a window argument across a trait and its callers. The speaker uses the task to show how incomplete grep output can expose a failure and motivate a deterministic test.

[4:14](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=254s) [Zed displays an evaluation prompt to add a window argument to a trait method and update its implementations and call sites.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7a46c4cf38e13c0b08c34b15f1a85f97e3f408a4264259d5140bc4fab5784cd.webp)

max_assertions: None,
profile_id: AgentProfileId::default(),
existing_thread_json: None,
max_turns: None,
    }
}

async fn conversation(&self, cx: &mut ExampleContext) -> Result<()> {
    const FILENAME: &str = "assistant_tool.rs";
    cx.push_user_message(format!(
        r#"
        Add a `window: Option<gpui::AnyWindowHandle>` argument to the `Tool::run` trait method in {FILENAME},
        and update all the implementations of the trait and call sites accordingly.
        "#
    ));

    let _ = cx.run_to_end().await?;

    // Adds ignored argument to all but `batch_tool`

    let add_ignored_window_paths = &[
        "code_action_tool",
        "code_symbols_tool",
        "contents_tool",

Interface text:
Zed Nightly
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs
worlds_fair_talk/slides/03_programmatic_eval_example.rs > impl Example for AddArgToTraitMethod > async fn conversation
33:95  Rust
AIE  Microsoft  smol ai

### Grep test setup

The deterministic test searches Rust files for fn run and checks the expected match. Tree-sitter expands the match to syntactic boundaries, giving the result a trait, function and line range.

[4:32](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=272s) [Zed displays a Rust test that searches for a function and checks the expected grep result.](https://assets.ai.engineer/sites/aiecode2025/sha256/dab8b2ebc4c13b2651d5c5d1adf4e550f79ec5d6b64b2f067207d6da6e48e67d.webp)

#[gpui::test]
async fn test_grep_function_args_and_body(cx: &mut TestAppContext) {
    let project = setup_syntax_test(cx).await;

    // Test: Line with a function argument
    let input = serde_json::to_value(GrepToolInput {
        regex: "fn run(".to_string(),
        include_pattern: Some("**/*.rs".to_string()),
        offset: 0,
        case_sensitive: false,
    })
    .unwrap();

    let result = run_grep_tool(input, project.clone(), cx).await;
    let expected = r#"
        Found 1 matches:

        ## Matches in crates/assistant_tool/src/assistant_tool.rs

        ### trait AssistantTool › fn run › L238-241

        [illegible]rs
        [illegible] Runs the tool with the provided input.

Interface text:
AIE
Zed Nightly
zed ai-worlds-fair
00_intro.md
01_deterministic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_ba[illegible]
worlds_fair_talk/slides/05_grep_tool_output_new.rs › async fn test_grep_function_args_and_body
Nathan Sobo, 21 hours ago
7:26 (8 characters)
Rust

### Testing grep output

A deterministic test searches for fn run and checks that Tree-sitter expands the match to syntactic boundaries, including the function’s full argument list.

[4:45](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=285s) [Zed displays the expected grep result with the Rust function's full argument list.](https://assets.ai.engineer/sites/aiecode2025/sha256/a69779c263ed5516a4d7ac67af7e8ac80b16de82151223a4b23c76483d8c967d.webp)

let result = run_grep_tool(input, project.clone(), cx).await;
let expected = r#"
    Found 1 matches:

    ## Matches in crates/assistant_tool/src/assistant_tool.rs

    ### trait AssistantTool › fn run › L238-241

    ```rs
    /// Runs the tool with the provided input.
    fn run(
        self: Arc<Self>,
        input: serde_json::Value,
        request: Arc<LanguageModelRequest>,
        project: Entity<Project>,
        action_log: Entity<ActionLog>,
        model: Arc<dyn LanguageModel>,
        window: Option<AnyWindowHandle>,
        cx: &mut App,
    ) -> ToolResult;
    [illegible]
[illegible]dent();

Interface text:
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs  08_ba[illegible]
worlds_fair_talk/slides/05_grep_tool_output_new.rs › async fn test_grep_function_args_and_body
Nathan Sobo, 21 hours ago
35:11 (13 lines, 416 characters)  Rust
AIE  Microsoft  smol ai

[5:00](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=300s) [Rust test sets up a search and shows the beginning of its expected result.](https://assets.ai.engineer/sites/aiecode2025/sha256/16ace851e217c28a25c2546a92fe74c7a76553acfab349470d335003b8128da8.webp)

#[gpui::test]
async fn test_grep_function_args_and_body(cx: &mut TestAppContext) {
    let project = setup_syntax_test(cx).await;

    // Test: Line with a function argument
    let input = serde_json::to_value(GrepToolInput {
        regex: "fn run(".to_string(),
        include_pattern: Some("**/*.rs".to_string()),
        offset: 0,
        case_sensitive: false,
    })
    .unwrap();

    let result = run_grep_tool(input, project.clone(), cx).await;
    let expected = r#"
        Found 1 matches:

        ## Matches in crates/assistant_tool/src/assistant_tool.rs

        ### trait AssistantTool › fn run › L238-241

        [illegible]rs
        [illegible] Runs the tool with the provided input.

Interface text:
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs  08_ba[illegible]
worlds_fair_talk/slides/05_grep_tool_output_new.rs › async fn test_grep_function_args_and_body
Nathan Sobo, 21 hours ago
16:25  Rust
AIE  Microsoft  smol ai

### Streaming edits

Zed describes an edit in a tool call, then asks the model to stream old and new text. The stream needs incremental parsing and must handle malformed tags and inconsistent whitespace.

[5:04](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=304s) [Slide lists challenges with tool-call streaming, incremental parsing, and imperfect model behavior.](https://assets.ai.engineer/sites/aiecode2025/sha256/254b0c95734b27e82bf6e1e1049078b2173979434e0849b7c6cd5e0e38d11d23.webp)

# Streaming Edits

Show the model edits as they happen, token by token.

## Challenges

1. Tool calling doesn't stream
   - JSON values must be complete before they are streamed
   - We can't use tool calling alone if we want to see streaming text
   - We ask it to stream `<old_text>` and `<new_text>` blocks

2. Parsing Complexity: XML tags arrive in random chunks
   - `</old_te` + `xt>` (split across network packets)
   - Must buffer and parse incrementally

3. Imperfect Model Behavior: Models don't follow instructions perfectly
   - Wrong closing tags: `<old_text>...</new_text>`
   - Inconsistent indentation and whitespace
   - Escaping

Interface text:
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs
worlds_fair_talk/slides/06_streaming_edits.md › # Streaming Edits
Nathan Sobo, Yesterday
1:1  Markdown
AIE

### Stochastic unit test example

The test runs an edit evaluation 200 times and requires every run to pass. An LLM judges the resulting diff to check whether the documentation comments were translated to Italian.

[5:46](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=346s) [Rust test configures repeated evaluation of an edit that translates documentation comments.](https://assets.ai.engineer/sites/aiecode2025/sha256/151a9c3779b0a09ff45b708dba6957e5bdd063157ff6bda07828be1897f9767d.webp)

// When AI enters the equation, we need a new approach
// We test AI features by sampling their behavior:

#[test]
fn eval_translate_doc_comments() {
    let input_file_path = "root/canvas.rs";
    let input_file_content = include_str!("evals/fixtures/translate_doc_comments/before.rs");
    let edit_description = "Translate all doc comments to Italian";
    eval(
        200,
        1.,
        EvalInput::from_conversation(
            vec![
                message(
                    User,
                    [text(formatdoc! {"
                        Read the {input_file_path} file and edit it (without overwriting it),
                        translating all the doc comments to italian.
                    "})],
                ),
                message(
                    Assistant,
                    [tool_use(

Interface text:
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs  08_ba
worlds_fair_talk/slides/07_stochastic_unit_tests.rs
Nathan Sobo, Yesterday
4:1 (7 characters)  Rust
AIE  Microsoft  smol ai

[6:21](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=381s) [Code shows an edit tool call followed by an assertion judging whether documentation comments were translated to Italian.](https://assets.ai.engineer/sites/aiecode2025/sha256/9cb2371e900f236eaf88799ca6687251332c20f1d962455dcc013d461567f638.webp)

[tool_result("tool_1", "read_file", input_file_content)],
                ),
                message(
                    Assistant,
                    [tool_use(
                        "tool_2",
                        "edit_file",
                        EditFileToolInput {
                            display_description: edit_description.into(),
                            path: input_file_path.into(),
                            mode: EditFileMode::Edit,
                        },
                    )],
                ),
            ],
            Some(input_file_content.into()),
            EvalAssertion::judge_diff("Doc comments were translated to Italian"),
        ),
    );
}

Interface text:
zed  ai-worlds-fair
00_intro.md  01_deterministic_testing_at_zed.rs  02_traditional_eval.toml  03_programmatic_eval_example.rs  04_grep_tool_output_old.md  05_grep_tool_output_new.rs  06_streaming_edits.md  07_stochastic_unit_tests.rs  08_ba
worlds_fair_talk/slides/07_stochastic_unit_tests.rs > fn eval_translate_doc_comments
Nathan Sobo, Yesterday
51:81 (62 characters)  Rust
AIE  Microsoft  smol ai

### Stochastic test setup

The test samples an assistant asked to translate Rust documentation comments into Italian. The speaker uses an LLM as judge to check the result.

[6:41](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=401s) [Code sets up a sampled evaluation asking an assistant to translate documentation comments to Italian.](https://assets.ai.engineer/sites/aiecode2025/sha256/c172a57b0a8cfd46cd8026f828d1413fb69225d9f302a44d72805fd37b4dc69c.webp)

// When AI enters the equation, we need a new approach
// We test AI features by sampling their behavior:

#[test]
fn eval_translate_doc_comments() {
    let input_file_path = "root/canvas.rs";
    let input_file_content = include_str!("evals/fixtures/translate_doc_comments/before.rs");
    let edit_description = "Translate all doc comments to Italian";
    eval(
        200,
        1.,
        EvalInput::from_conversation(
            vec![
                message(
                    User,
                    [text(formatdoc! {"
                        Read the {input_file_path} file and edit it (without overwriting it),
                        translating all the doc comments to italian.
                    "})],
                ),
                message(
                    Assistant,
                    [tool_use(
                        [illegible]

Interface text:
Zed Nightly
zed  ai-worlds-fair
[illegible]stic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
worlds_fair_talk/slides/07_stochastic_unit_tests.rs > fn eval_translate_doc_comments
Nathan Sobo, Yesterday
10:13  Rust
AIE
aws

### Testing random parser chunks

The test runs 100 iterations, splitting the input at random boundaries and feeding each chunk to the edit parser to check that it handles arbitrary chunking.

[6:46](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=406s) [A test generates random chunk boundaries and feeds the chunks to an edit parser.](https://assets.ai.engineer/sites/aiecode2025/sha256/52a27dac171390397ac54e3528a9d57999b1ae4684584dd518387534e1d76d28.webp)

// Some streaming edit problems can be tested the old-fashioned way!

// 1. Parser must handle chunks split ANYWHERE
#[gpui::test(iterations = 100)]
fn test_parser_random_chunks(mut rng: StdRng) {
    let input = "<old_text>hello world</old_text><new_text>goodbye</new_text>";

    // Generate random chunk boundaries
    let mut chunks = vec![];
    let mut pos = 0;
    while pos < input.len() {
        let chunk_size = rng.gen_range(1..=10);
        let end = (pos + chunk_size).min(input.len());
        chunks.push(&input[pos..end]);
        pos = end;
    }

    // Parser MUST handle any chunking
    let mut parser = EditParser::new();
    let events: Vec<_> = chunks.iter().flat_map(|chunk| parser.push(chunk)).collect();

    [illegible]t_eq!(
        events,
        [illegible]

AIE
aws

Interface text:
Zed Nightly
zed  ai-worlds-fair
[illegible]tic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
worlds_fair_talk/slides/08_basic_streaming_edits_unit_tests.rs › fn test_parser_random_chunks
Nathan Sobo, 2 days ago
Update Zed
6:56  Rust

### Checking parser events

Across 100 iterations, the test splits input at random boundaries and checks that the parser emits the expected old-text and new-text events.

[7:07](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=427s) [The parser test asserts that randomly chunked input produces old-text and new-text events.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb9235fd97f0d501c2b1a62c051ee23710c5d6dc11832e8fe7a6e759c6280173.webp)

// Generate random chunk boundaries
let mut chunks = vec![];
let mut pos = 0;
while pos < input.len() {
    let chunk_size = rng.gen_range(1..=10);
    let end = (pos + chunk_size).min(input.len());
    chunks.push(&input[pos..end]);
    pos = end;
}

// Parser MUST handle any chunking
let mut parser = EditParser::new();
let events: Vec<_> = chunks.iter().flat_map(|chunk| parser.push(chunk)).collect();

assert_eq!(
    events,
    vec![Event::OldText("hello world"), Event::NewText("goodbye")]
);
}

// 2. Fuzzy matcher algorithm (without LLM input)
[illegible]uzzy_match_algorithm() {
[illegible]uffer = "fn calculate_price() {\n    // TODO\n}";

Interface text:
AIE
Zed Nightly
zed  ai-worlds-fair
[illegible]tic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
worlds_fair_talk/slides/08_basic_streaming_edits_unit_tests.rs > fn test_parser_random_chunks
Update Zed
6:56  Rust

### Fuzzy matching test

The fuzzy matcher test finds code despite extra whitespace. The speaker says approximate matching reduced tool-calling failures when model output was slightly wrong.

[7:18](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=438s) [A code editor shows a test that finds a match despite extra whitespace.](https://assets.ai.engineer/sites/aiecode2025/sha256/a5402db92872e23aa83657ff3fbdc89dc8c74d9a6f7fddcd197dae0d7ffe58e3.webp)

    let mut parser = EditParser::new();
    let events: Vec<_> = chunks.iter().flat_map(|chunk| parser.push(chunk)).collect();

    assert_eq!(
        events,
        vec![Event::OldText("hello world"), Event::NewText("goodbye")]
    );
}

// 2. Fuzzy matcher algorithm (without LLM input)
#[test]
fn test_fuzzy_match_algorithm() {
    let buffer = "fn calculate_price() {\n    // TODO\n}";
    let query = "fn  calculate_price()  {"; // Extra spaces

    let matcher = FuzzyMatcher::new(buffer);
    let range = matcher.find(query);

    assert_eq!(range, Some(0..24)); // Found despite whitespace
}

[illegible]eaming diff computes edits incrementally
[illegible]treaming_diff() {

Interface text:
AIE
zed ai-worlds-fair
[illegible]ic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
worlds_fair_talk/slides/08_basic_streaming_edits_unit_tests.rs › fn test_fuzzy_match_algorithm
Nathan Sobo, 2 days ago
Update Zed
31:22 Rust
Microsoft
smol ai

### Streaming diff test

The test feeds new text to a streaming diff in chunks and checks the first keep operation. The diff must distinguish deleted old text from text the model has yet to emit.

[7:37](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=457s) [A code editor shows a streaming-diff test that processes new text in chunks.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9b2bbe2d16b10395393c9f30405ed8c616a1937a5488d6977f8ac0d0d53a8aa.webp)

let range = matcher.find(query);

assert_eq!(range, Some(0..24)); // Found despite whitespace
}

// 3. Streaming diff computes edits incrementally
#[test]
fn test_streaming_diff() {
    let old_text = "fn calculate() {\n    todo!()\n}";
    let mut diff = StreamingDiff::new(old_text);

    // Simulate new text arriving in chunks
    let ops1 = diff.push_new("fn calc");
    assert_eq!(
        ops1,
        vec![
            CharOp::Keep(7),  // "fn calc"
        ]
    );

    let ops2 = diff.push_new("ulate_total(");
    [illegible]t_eq!(
        [illegible]ps2,
        [illegible]

Interface text:
Zed Nightly
zed  ai-worlds-fair
[illegible]tic_testing_at_zed.rs
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
worlds_fair_talk/slides/08_basic_streaming_edits_unit_tests.rs › fn test_streaming_diff
Nathan Sobo, Yesterday
Update Zed
48:16  Rust
AIE
Microsoft
smol ai

### Streaming diff operations

The test checks insert, keep and delete operations as new text arrives. The diff must distinguish text that was deleted from text that has not streamed yet.

[8:01](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=481s) [A code editor shows later streaming-diff assertions with insert, keep and delete operations.](https://assets.ai.engineer/sites/aiecode2025/sha256/21a0b0801181fc8241e5d7e926af23e9a5683c390fd555fce4375229980e9629.webp)

[illegible]
]
);

let ops2 = diff.push_new("ulate_total(");
assert_eq!(
    ops2,
    vec![
        CharOp::Insert("_total"), // Insert: "_total"
        CharOp::Keep(5),          // "ulate"
        CharOp::Delete(2),        // Remove: "()"
        CharOp::Keep(1),          // "("
    ]
);

let ops3 = diff.push_new("items: &[Item]) {\n    items.iter().sum()\n}");
assert_eq!(
    ops3,
    vec![
        CharOp::Insert("items: &[Item]"),
        CharOp::Keep(4),    // ") {\n"
        CharOp::Delete(10), // Remove "    todo!()"
        CharOp::Insert("    items.iter().sum()"),
        CharOp::Keep(2), // "\n}"

Interface text:
Zed Nightly
zed  ai-worlds-fair
02_traditional_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
worlds_fair_talk/slides/08_basic_streaming_edits_unit_tests.rs › fn test_streaming_diff
Nathan Sobo, Yesterday
70:28 (17 lines, 462 characters)
Rust
AIE
Microsoft
smol ai

### Handling empty old text

The model sometimes emitted empty old text when inserting at document edges. A prompt improved the eval, but the problem still occurred 1–2% of the time, so a deterministic test covers the edge case.

[8:09](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=489s) [Code comments describe a prompt change and a deterministic test for empty old text.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad576ab4724171b5c6fd16771540ddcd9d407d7f02986eb2de4eccba64d010a6.webp)

// As we run the unit eval, we discover problems
// Some of which can be solved alorithmically and tested deterministically

// This prompt change helped:
//
// - `<old_text>` cannot be empty

// But the model still wasn't perfect: So we then wrote a deterministic test to
// gracefully handle the edge case:
#[gpui::test(iterations = 100)]
async fn test_empty_old_text(cx: &mut TestAppContext, mut rng: StdRng) {
    let agent = init_test(cx).await;
    let buffer = cx.new(|cx| {
        Buffer::local(
            indoc! {"
                abc
                def
                ghi
            "},
            cx,
        )
[illegible]apply, _events) = agent.edit(

Interface text:
AIE
Zed Nightly
zed  ai-worlds-fair
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
worlds_fair_talk/slides/09_empty_old_text_problem.rs
3:1 (1 character)
Rust

### Empty old text revisited

A prompt rule improved the eval, but empty old-text tags still appeared 1–2% of the time. The team added a deterministic test to handle the edge case.

[8:16](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=496s) [The editor highlights the instruction that old text cannot be empty.](https://assets.ai.engineer/sites/aiecode2025/sha256/c0fc78e6ae0ae09b88c2b8388c7ad366159b304b7330930e5a4f95196ad0e448.webp)

// As we run the unit eval, we discover problems
// Some of which can be solved alorithmically and tested deterministically

// This prompt change helped:
//
// - `<old_text>` cannot be empty

// But the model still wasn't perfect: So we then wrote a deterministic test to
// gracefully handle the edge case:
#[gpui::test(iterations = 100)]
async fn test_empty_old_text(cx: &mut TestAppContext, mut rng: StdRng) {
    let agent = init_test(cx).await;
    let buffer = cx.new(|cx| {
        Buffer::local(
            indoc! {"
                abc
                def
                ghi
            "},
            cx,
        )
[illegible]apply, _events) = agent.edit(

Interface text:
zed  ai-worlds-fair
[illegible]nal_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
worlds_fair_talk/slides/09_empty_old_text_problem.rs
Nathan Sobo, Yesterday
6:17 (10 characters)  Rust
AIE
Microsoft
smol ai

### Simulating an empty old-text tag

The test simulates an empty old-text tag followed by a second edit. The speaker says the model still produced empty tags 1–2% of the time after a prompt change, so the test checks that the case can be handled.

[8:53](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=533s) [A test simulates model output containing an empty old-text tag followed by another edit.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca9d9c95802af766a8a44a51666bc1dd2a5f1a1011390767da9754d8eb44b199.webp)

        buffer.clone(),
        String::new(),
        &LanguageModelRequest::default(),
        &mut cx.to_async(),
    );
    cx.run_until_parked();

    simulate_llm_output(
        &agent,
        indoc! {"
            <old_text></old_text>
            <new_text>jkl</new_text>
            <old_text>def</old_text>
            <new_text>DEF</new_text>
        "},
        &mut rng,
        cx,
    );
    apply.await.unwrap();

    pretty_assertions::assert_eq!(
        [illegible]uffer.read_with(cx, |buffer, _| buffer.snapshot().text()),
        [illegible]ndoc! {"

Interface text:
Zed Nightly
zed  ai-worlds-fair
[illegible]nal_eval.toml
03_programmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
worlds_fair_talk/slides/09_empty_old_text_problem.rs › async fn test_empty_old_text
Nathan Sobo, Yesterday
42:26 (12 lines, 273 characters)
Rust
AIE
aws

### Handling mismatched XML tags

A prompt raised the pass rate from 60% to 95%, but mismatched tags persisted. The parser test uses a reduced Sonnet 3.7 output that closes an <old_text> block with </new_text>.

[8:57](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=537s) [Code comments report pass rates before and after a prompt change and introduce a forgiving parser test.](https://assets.ai.engineer/sites/aiecode2025/sha256/515714ee1324d052ce8d683630b8e5f3942ef24058016aac2ad8cad67426b330.webp)

// Another tricky case with XML: Tag mismatches

// Initial pass rate: 60%

// PROMPT FIX:
// Added: "Always close all tags properly"

// After prompt fix: 95% pass rate
// That last 5% wouldn't budge, so we made the parser forgiving:

#[gpui::test(iterations = 1000)]
fn test_mismatched_tags(mut rng: StdRng) {
    let mut parser = EditParser::new();
    assert_eq!(
        parse_random_chunks(
            // Reduced from an actual Sonnet 3.7 output
            indoc! {"
                <old_text>
                a
                b
                c
                </new_text>
                <new_text>

Interface text:
Zed Nightly
zed  ai-worlds-fair
[illegible]mmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
10_mismatched_tags.rs
worlds_fair_talk/slides/10_mismatched_tags.rs
Nathan Sobo, Yesterday
AIE
Microsoft
smol ai

[9:14](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=554s) [Code editor shows an XML tag mismatch test and notes on improving the parser.](https://assets.ai.engineer/sites/aiecode2025/sha256/3dffcaf017d786a7964a067133e241e95ad43058c261db1073923cf9bf483774.webp)

// Another tricky case with XML: Tag mismatches

// Initial pass rate: 60%

// PROMPT FIX:
// Added: "Always close all tags properly"

// After prompt fix: 95% pass rate
// That last 5% wouldn't budge, so we made the parser forgiving:

#[gpui::test(iterations = 1000)]
fn test_mismatched_tags(mut rng: StdRng) {
    let mut parser = EditParser::new();
    assert_eq!(
        parse_random_chunks(
            // Reduced from an actual Sonnet 3.7 output
            indoc! {"
                <old_text>
                a
                b
                c
                </new_text>
                <new_text>

Interface text:
Zed Nightly
zed  ai-worlds-fair
[illegible]mmatic_eval_example.rs
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
10_mismatched_tags.rs
worlds_fair_talk/slides/10_mismatched_tags.rs
Nathan Sobo, Yesterday
AIE
aws

### Evaluating indented code edits

The eval asks an agent to replace a todo inside a nested function. The speaker says models sometimes flattened the indentation, breaking the edit.

[9:49](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=589s) [Code editor shows a nested function and an evaluation asking an agent to replace a todo with return 42.](https://assets.ai.engineer/sites/aiecode2025/sha256/98f39d900a81a3f77fd3fe8260566eccf393b008217041172bd11aea976a5ea1.webp)

// EVAL 3: Models stripped indentation, breaking everything

#[test]
fn eval_indented_code_editing() {
    eval(100, 0.95, "Edit nested function", || async {
        let buffer = Buffer::local(
            indoc! {"
                fn outer() {
                    fn inner() {
                        todo!()
                    }
                }"
            }
        );

        // Ask to modify the inner function
        agent.edit(
            buffer.clone(),
            "Replace todo with return 42",
            &conversation,
        ).0.await;

        // Models sent:

Interface text:
Zed Nightly
zed  ai-worlds-fair
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
10_mismatched_tags.rs
11_indentation.rs
worlds_fair_talk/slides/11_indentation.rs › fn eval_indented_code_editing
Nathan Sobo, Yesterday
Update Zed
5:5  Rust
AIE
Microsoft
smol ai

### The nested function's edit target

The eval asks an agent to replace a todo inside a nested function with return 42. The speaker says models sometimes flattened the indentation in their edits.

[9:58](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=598s) [Code editor highlights the todo inside a nested function.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9df34577df35319097f79a91e6f48535e2f5194e026cefbb66ea07da8306b1f.webp)

// EVAL 3: Models stripped indentation, breaking everything

#[test]
fn eval_indented_code_editing() {
    eval(100, 0.95, "Edit nested function", || async {
        let buffer = Buffer::local(
            indoc! {"
                fn outer() {
                    fn inner() {
                        todo!()
                    }
                }"
            }
        );

        // Ask to modify the inner function
        agent.edit(
            buffer.clone(),
            "Replace todo with return 42",
            &conversation,
        ).0.await;

        // Models sent:

Interface text:
Zed Nightly
zed  ai-worlds-fair
04_grep_tool_output_old.md
05_grep_tool_output_new.rs
06_streaming_edits.md
07_stochastic_unit_tests.rs
08_basic_streaming_edits_unit_tests.rs
09_empty_old_text_problem.rs
10_mismatched_tags.rs
11_indentation.rs
worlds_fair_talk/slides/11_indentation.rs › fn eval_indented_code_editing
Nathan Sobo, Yesterday
Update Zed
10:32 (7 characters)
Rust
AIE
aws

## Full transcript

[0:00](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=0s) [upbeat music] Hey, everybody.

[0:15](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=15s) Thanks for coming to my talk. Uh, I'm Nathan Sobo. I'm the co-founder of Zed. We are an AI-enabled code editor, and what sets us apart is we are not a fork of VS Code.

[0:27](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=27s) We are implemented from scratch. [laughs] [clapping] Uh, implemented from scratch in Rust. We literally engineered the entire system like a video game around about 1,200 lines of shader program that run on the GPU, and the rest of the system is organized to deliver frames at 120 frames per second really quickly.

[0:47](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=47s) And recently, I recorded a video because time is short, and I-- this is obviously sped up massively, but we launched agentic editing in Zed, and I wanted to talk about the approach that we took to test this, um, and be empirical ab-- and deliver a reliable product that does this effectively.

[1:04](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=64s) So here's what we're gonna be talking about. The rest of the talk is gonna be looking at code and just talking about our experience. Um, so

[1:13](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=73s) testing, evaluation, what's the difference? Um, first, I just wanna start by saying, like, without an empirical approach to the software that we've been building since twenty-eighteen, twenty twenty-one, depends on how long you measure it, Zed would crash every eight seconds.

[1:29](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=89s) Like, uh, we have probably tens of thousands of tests at this point. And here's an example of the extreme to which we take empirical software development. We literally are starting a server, creating two clients, and we have a simulated scheduler that gets passed in here, and we run fifty different iterations of this with every random interleaving of

[1:48](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=108s) anything that could possibly be concurrent. Um, so that if the three hundred and seventy-fifth thousandth, and this is only fifty, but, uh, iteration of this particular interleaving of network packets, um, goes wrong, we can replay that over and over again, freeze it, and have full control.

[2:06](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=126s) So we're very hardcore about testing and being empirical at Zed. But up until very recently, until shipping some of these AI-related features, we've been able to be fully deterministic.

[2:17](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=137s) Even with sort of non-deterministic things, we've been able to really lock things down and never have flaky tests on CI. But as soon as an LLM enters the picture, that's all out the window, right?

[2:28](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=148s) Because we could like, you know, assuming that [chuckles] we're using, you know, uh, frontier models, but s- even if we were to use our own model and to, like, control the sampling off the logits, you change one token in the input, and you're gonna get a completely different output.

[2:43](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=163s) So it's just a fundamentally different problem where we have to embrace stochastic behavior. And so as we built this system, our first eval that we really hammered on was something that if you've seen something like SWE-bench probably looks pretty familiar.

[2:57](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=177s) It's very data-driven, and it seems like in the machine learning world, like this is what an eval is, input, output. But in the programmatic software world, an eval is more like a test that passes or fails.

[3:11](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=191s) Like, we come from this very different perspective of automated testing. And so right away, you know, we had this traditional data-driven eval, and then backing that, you know, in the same program we als-- uh, that runs all these evals basically compiles a headless copy of Zed, checks out a repo, runs the agent, tries to make it do

[3:30](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=210s) things. Um, we right away got into, like, making the e- eval more programmatic. So you can see here the conversation is literally a function. And then our ability to sort of come in here and write code that performs assertions about what the agent did, even, you know, getting, uh, quite a bit more granular.

[3:49](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=229s) 'Cause the problem is when that big eval fails, it's like, what do we do? [chuckles] There's a million ways that that thing could go wrong. And so when we wrote this eval, we were able to drive out, uh, one really simple failure mode, which is when we run the grep tool, this is what our original, like, dumb implementation

[4:06](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=246s) of the grep tool looked like. So you can see in this case, we're saying we wanna add, uh, a window argument to this tool run trait method in this particular file name, right?

[4:17](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=257s) But if this is what the model sees, then we're in trouble, right? And so what we ended up, you know, driving from this stochastic test here is a more deterministic test, right?

[4:31](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=271s) Where I'm gonna go ahead and set up the project, uh, perform this search for fn run, and then, uh, we use Tree-sitter, which is, um, Max's parsing, uh, parser generator framework, to actually expand out the match to the syntactic boundaries.

[4:48](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=288s) Um, and so that was able to, you know, drive a sub-- a substantial improvement in the behavior of the agent. So right there, like, that was an interesting discovery of just being empirical and driving out what was ultimately an algorithmic problem with a stochastic test.

[5:04](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=304s) But then we g- moved on to the editing, and when we first implemented editing, it was just done with tool calls. But the problem with tool calls, if anybody's worked with them, is they don't really stream very well.

[5:14](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=314s) Like, you get key, value, key, value, and it all comes back at once. So we, what we ended up doing is deciding instead we perform a small tool call that just describes the edits, then we loop back around for the moment to just the same model because it has everything already loaded in its cache and ask it

[5:33](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=333s) to emit these old text, new text blocks. Um, but there's some challenges there. Like, now we have to parse that data is coming back and all kinds of weird stuff that the model's gonna be doing.

[5:45](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=345s) So I'll just dive in and show you, um, some of the examples. So here's like--

[5:51](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=351s) uh, there are a bunch of different, uh, tests. Like, these are part of our main regular old test suite, and we just added this eval function, and we say, "I wanna run this eval two hundred times."

[6:02](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=362s) And in this particular case-

[6:04](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=364s) Didn't start this way, [chuckles] but eventually we got this way. We want 100% of these 200 examples to pass, or this test should, like, literally fail to build. So we kind of set a watermark that way, and then here you can see I'm just, like, loading in a conversation with the LLM, um, and then ultimately, uh, in this

[6:24](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=384s) case, I'm using, uh, LLM as judge. But there are some cases you saw earlier where we do things more programmatically. This case, we're just, like, verifying this thing worked correctly.

[6:34](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=394s) Um, and so we go from a test like this, a stochastic test, into the particular problems, um, that drive this. So I mean... But it's funny, a lot of the things that went wrong here were really just, like, basic things, right?

[6:48](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=408s) Like, and so again, these are things that we can-- they're non-deterministic, but the non-determinism is b- non-determinism is sort of bounded. So in this case, we're gonna run 100 iterations of this very simple parsing test, where we're randomly chunking up the input into arbitrary boundaries and just making sure that the parser for this data, um, you know,

[7:12](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=432s) can handle arbitrary chunking o- of the text. Similarly here, fuzzy matching algorithm, that's a critical piece of, you know, if the model's fuzzy or, uh, generates something slightly wrong, being able to do this dynamic programming that gets us an, a, a, an approximate match really saved a lot of the tool calling failures for us.

[7:34](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=454s) But again, something that could be deterministically tested. And then finally, there's this idea of a streaming diff that as the model is emitting new text, we need to be care- comparing the new text it's emitting with the old text and then making a dynamic decision i- of if I don't see some text, is that in-- that was

[7:53](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=473s) present in the old text, is that because it was deleted or it just hasn't been streamed out yet? So that's another, you know, completely, uh, deterministic thing that still was critical to making this system work correctly.

[8:06](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=486s) So yet another, like, deterministic test. Um, but then we get into some of the fun stuff with, uh, uh, the behavior of the model itself. So one thing that we noticed right away was, um, the model when it was trying to insert at the top or the end of the document in some cases would just have an

[8:27](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=507s) empty old text tag, which like if you're doing old text, new text matching is not very useful. And so we found that we were able to just add this, you know, simple thing to the prompt and improve the eval.

[8:39](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=519s) But again, [chuckles] it would still do it, you know, one or two percent of the time. And so just being able to test that, uh, you know, we can handle that case robustly, um, you know, ended up being really helpful.

[8:52](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=532s) And this is just like, again, I'm in this case simulating the output of the LLM. Um,

[8:59](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=539s) uh, another case that we encountered a lot was that it would mismatch the XML tags, and so we would have old text, and then it would end like this with new text.

[9:10](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=550s) Um, and so again, we were able to get a certain distance with the prompt. We initially were sort of 40% of the time we were getting this mismatch and blowing up.

[9:20](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=560s) We were able to move that by saying, "Always close all tags properly," to like 95%, but we still have that last like five percent. What do we do with that?

[9:30](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=570s) And so again, it's just like driving it into a deterministic test where we have old text, new text, you know, and we need to just assert that even though we got...

[9:40](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=580s) You know, we just need to be robust and accept crazy stuff from the LLM, like, uh, in its output. Um, another interesting case was indentation. So here's an eval that, uh, that we wrote, which was this idea of having an outer function, and then inside here there's an inner function, and we have this todo macro here.

[10:02](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=602s) And we wanna say, "Replace this todo with return 42," and we would see the model doing things like this, right? Where the indentation level was completely flattened out, right?

[10:13](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=613s) It says, "Replace fn inner," but fn inner has like all this leading indentation on the front of it. Um, but otherwise, like it's perfectly fine. And so in this case, what we did is this strategy of like detecting this indent delta.

[10:29](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=629s) So we figure out basically sort of re-normalize the indent, if that makes sense. So if the text in the buffer is indented by a certain amount but we otherwise match, then we also detect the text in the, that the LLM emitted and compute this delta, and then again, driving it back to a deterministic test of, um, here

[10:51](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=651s) we build, uh, a buffer with lorem ipsum dolor sit with this very interesting indentation pattern. And then you can see here, we wanna replace ipsum dolor sit with ipsum dolor sit amet where it's out dented, right?

[11:07](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=667s) And you can see that when we actually do our, uh, indentation normalization, we're able to handle that correctly. Um, another tricky thing that we ran into was this weird escaping behavior where in these certain cases where we have weird constructs, like this is a Rust, um, sort of a raw string basically that can include, uh, things like

[11:30](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=690s) quotes inside of it due to this interesting escaping syntax. And doing something as simple as saying, just like, "Tweak this," we would notice this is the behavior we want.

[11:40](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=700s) We would notice, especially Gemini doing all kinds of crazy stuff like doing HTML escape codes, um, doing this sort of backslash escaping. Yeah, e- with new lines and stuff, we would see like, you know, uh, crazy escaping that it would do, [chuckles] uh, double escaping the new lines.

[11:59](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=719s) Um, and so in this case, like we-- that was just a pure prompt fix and didn't really... We didn't try yet, at least, to kind of detect- any of this escaping any further, although obviously, like, that's an opportunity, yeah, to keep going even further.

[12:15](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=735s) Um, so yeah, I mean from, from my perspective, like, the lessons we learned from this entire experience is that, like,

[12:24](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=744s) rigorous testing is fundamental to building reliable software, period. Uh, we have this new fancy parlance of, like, an eval, which comes, I think, out of the machine learning field of kind of input/output and having lots of examples of that.

[12:38](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=758s) But I think a lot of the techniques of just traditional, good old-fashioned software engineering are still really applicable. But we have to embrace this more statistical approach, where we're running it 100 times, 200 times, and asserting a threshold of pass versus fail.

[12:54](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=774s) Um, but I think, I mean, this is a real-world example of just the stuff that we saw trying to implement streaming edits or an agent that can go search and edit.

[13:04](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=784s) A lot of those problems are not, I don't know, they're not advanced [laughs] machine learning problems or anything. It's just stupid things that the model will try to do that we need to account for.

[13:15](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=795s) Um, and so this, I, I think this motion of sort of starting with the zoomed-out eval, then zooming into a sort of stochastic unit test that's still random and interacting with the model but is more focused in on a particular aspect of the experience, and then finally, like driving that even further to an actual good old-fashioned test,

[13:35](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=815s) that's been the process that we've discovered. But we so far haven't needed to use, like, special external tools, um, or eval frameworks or anything of like that. It's just in our test suite so far, and kind of using the same underlying infrastructure that's used to do any other kind of software testing.

[13:54](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=834s) And honestly, using just a lot of those same skills. Um, so for us, it was like just be empirical, just like we've always been, but it's this new X factor of the LLM doing all these crazy things.

[14:09](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=849s) Um, yeah, so that's, that's a little story of how we ... And this is all open source, so you can check it out. Uh, Zed is under the GPL license.

[14:18](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=858s) I'd love help improving it all, actually. Um, maybe some of you are like, "What are you doing? You could do better in the following five ways." I'd love, uh, contributions, if that's interesting to folks.

[14:28](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=868s) And, um, yeah, it's working. I'm, uh, with the Claude 4 models, I'm finally able to write Rust agentically really efficiently, and I'm just loving it. So, um, that's how we got there.

[14:42](https://www.youtube.com/watch?v=WXy8Yy9xGss&t=882s) I appreciate your attention. [audience applauds] [upbeat music]
