AI Engineer Europe 2026

How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed

Ben Kunkle10:50

Loading slides…

Read the talk

How We Built Zeta2: Training an Edit Prediction Model in Production

Ben Kunkle follows Zed’s training pipeline from production snapshots and repaired teacher predictions to cheaper data filtering and live editor experiments.

From a talk by Ben Kunkle

An edit prediction on every keystroke

An edit predictor looks at the code around your cursor and predicts the next change you will make. That is the task behind Zeta 2, the model whose training process Ben Kunkle, edit predictions lead at Zed, walks through here. Its immediate task is to propose an edit within a supplied region of code.

The editor can supply more than the current text. Recent edits reveal what you have been doing; the cursor identifies where you are working; nearby type and variable definitions help explain the code. Diagnostics and errors provide additional clues. But the prediction runs on every keystroke, so it must return quickly enough to remain useful while you type. That combination of rich context, a narrow task and frequent invocation motivates a small, specialized, fine-tuned model.

0:150:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Turn production snapshots into checked teaching targets

Training starts with opt-in production snapshots. These preserve the context available when a prediction was requested, including the related definitions collected by the editor. Zed turns those snapshots into training examples through distillation: give a frontier model the captured input and ask it to produce an edit prediction. The frontier model acts as the teacher; Zeta 2 learns to produce the resulting target.

A capable teacher does not automatically produce consistent training data. Across many requests, its answers vary, and some fail the requirements of the edit-prediction task. The team refines the teacher prompt to obtain more useful predictions, then checks the generated output instead of assuming that every answer is suitable for training.

The checking and repair sequence addresses concrete failures:

  1. Run static heuristics to detect predictions that undo what the developer just typed.
  2. Check whether a prediction violates the editable-region boundary.
  3. Send a failing prediction to another frontier model with the specific failure described, asking for a corrected prediction.

This repair step turns a flawed teacher answer into a usable candidate target. After repair, the teacher’s edit becomes the expected output for the student.

1:021:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:02 · section reference included

Cache the teaching work, vary the student input

The teacher and repair work is reusable across experiments. Zed caches those results before applying experiment-specific formatting. One experiment might include diagnostics; another might omit them or change how much edit history the student receives. The desired output format can also vary. These choices change how an example is presented for training without requiring the teacher to generate its answer again. Each experiment then trains its student and runs a final set of offline evaluations.

Change the input, keep the teaching signal

A small constructed record shows two experiments reusing the same checked-and-repaired teacher edit.

One cached training target
REUSED
total = price * quantity

Produced by the teacher; checked and repaired before the experiments split.

Without diagnostics

Student input

{
  "code": "total = price * quant"
}

BASE INPUT · code around the cursor

Target same cached teacher edit

With diagnostics

Student input

{
  "code": "total = price * quant",
  "diagnostics": [
    "quant is not defined"
  ]
}

ADDED INPUT · diagnostic message

Target same cached teacher edit

Formatting a new student input does not require another teacher prediction.

The pipeline uses JSONL: each line contains one large JSON object representing an example. Successive stages add fields or rearrange existing ones, so the record can carry information through the whole process. The files become large, but the representation stays straightforward. Kunkle reports training sets of roughly 100,000 examples at the upper end, with smaller experiments using 10,000–50,000 examples.

2:182:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:36 · section reference included

Use later code as evidence of intent

The developer eventually writes the code they want. That observation motivates an experiment with what Kunkle calls settled data: wait until editing stops in the region supplied to the model, then snapshot the resulting code. Because Zed is the editor, it can observe both the original prediction context and the later state of that same region.

The later state is not automatically a clean answer to the original prediction task. The developer may change direction, or an agent may rewrite the region entirely. A prediction that was reasonable when requested can look wrong against code produced after that change of intent. Training directly on every eventual result would mix useful examples with outcomes that were not predictable from the original input.

One proposed filter generates ten teacher predictions from the same original context, then compares them with the settled region using a measure such as Levenshtein distance. If a candidate is close, that provides evidence that the later code was reachable from the earlier context. But sampling makes the cost multiply: generating ten teacher predictions for each of 100,000 examples requires one million frontier-model requests. Kunkle describes that approach as prohibitively expensive.

3:534:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:53 · section reference included

Find useful near misses with the student

The student trained on the original teacher predictions offers a cheaper way to perform this filtering. Kunkle reports that its edit-prediction quality is approaching the teacher’s. For this filter, he describes sampling a student checkpoint about 50 times per example at much lower cost than repeatedly calling the frontier teacher. The comparison remains the same: look for a generated prediction close to the settled region.

The distances help distinguish different kinds of training material:

  • Far from the settled code: the mismatch suggests noise or a change of intent.
  • Very close: the continuation may already be obvious to the model. Kunkle’s example is an addition function where the second operand is easy to anticipate.
  • Almost right: the intermediate cases can expose something useful to learn, such as an unfamiliar function introduced after the student’s training-data cutoff.

The interesting examples are therefore not simply those the model already reproduces most easily. Near misses can identify gaps between what the student knows and the code developers now want to write.

Even after filtering, Zed generally does not use the raw settled region as the training target, because it remains noisy. Instead, the team can train on the generated prediction closest to that region. The later code helps select a target without being treated as an unquestioned answer.

5:295:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:29 · section reference included

Evaluate more than one reasonable next edit

Offline evaluation uses a held-out test set, keeping the examples used to assess a model separate from those used to train it. That separation lets the team compare experiments without testing them on their own training material.

Zed’s deltaChrF compares character n-grams of different lengths. The original chrF paper provides background on character n-gram scoring, rather than a definition of Zed’s custom metric. Alongside similarity, the team tracks reversal ratio: predictions that undo exactly what the developer just typed. This measures a specific unwanted behavior that a general similarity score may not adequately expose.

Kunkle also points to production kept rate as another signal. For offline evaluation, there is a further complication: a context often permits several reasonable next edits. The team generally evaluates each held-out context against three distinct frontier-model predictions. A student answer close to one of those references can be useful even if it differs from the others.

7:037:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:03 · section reference included

Deploy experiments and measure what survives

Offline scores do not necessarily correlate with what developers want in their editor. Zed therefore evaluates models through deployed experiments as well. The experiment page shows two live variants: one receives 15% of production traffic, and the other receives the remainder.

After deployment, the team can adjust an experiment’s traffic allocation, including from 15% to 20%, or promote it to the live model. A separate dashboard tracks acceptance rate and latency. Kunkle identifies Zeta 2 among the experiments managed through this process.

The team is also exploring signals that look beyond immediate acceptance:

  • Kept rate compares the text after a prediction with the later settled region to assess how many characters were retained.
  • Diagnostic-error counts compare the number of errors before and after a prediction.

These ask different questions. A suggestion can be accepted, subsequently edited, and affect diagnostics in ways that acceptance alone does not capture. Retention and error changes give the team additional ways to judge an edit’s usefulness.

7:548:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:54 · section reference included

The stopping rule is ten seconds of inactivity

The closing audience question returns to a practical detail: how does the editor decide that a region has settled? A Git commit might indicate that the developer is satisfied with a block of code. Does Zed use that kind of signal?

The implementation Kunkle describes does not inspect Git commits. Zed snapshots the region after ten seconds without edits to that area. This is a rough collection heuristic, not a declaration that the developer has finished or approved the code. If they keep editing the same location without a ten-second pause, the region is not captured until such a pause occurs.

9:379:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:37 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…