AI Engineer Europe 2026

Fast Models Need Slow Developers

Loading slides…

Read the talk

Fast Models Need Slow Developers

Faster code generation makes smaller changes, frequent verification, and real-time steering more practical—but only if developers stop optimizing for unattended output.

From a talk by Sarah Chieng

Before you start: Familiarity with AI coding assistants, Git diffs, and automated tests will help; inference concepts are explained as they arise.

When waiting becomes a development habit

Write a massive prompt, try to get the whole feature in one shot, accept a huge commit, or keep ten agents thinking at once. These are familiar ways to work around slow code generation: while the model runs, give it more work or find something else to do. The problem appears when generation speeds up but those habits remain.

Sarah Chieng introduces Codex Spark with a reported generation rate of 1,200 tokens per second, compared with roughly 40–60 for the Sonnet and Opus families. She frames that as approximately a 20× difference; these are her historical comparisons, not a controlled benchmark with specified workloads. If a workflow was producing bad code slowly, faster inference lets it produce bad code much faster. Generation speed does not repair the feedback loop around generation.

Sarah Chieng stands beside the complete title slide, with cream and orange lettering on black.Open full source frame
Fast Models Need Slow Developers.

Introducing herself as Cerebras’s head of developer experience, Chieng describes the excitement of showing developers inference without the familiar wait. That experience creates a new practical question: how should people work when code arrives faster than they can understand it? Her historical overview puts popular model families around 50–150 tokens per second over the preceding two years, even as intelligence and context windows grew. Spark represents the change in that otherwise relatively stable dimension.

0:170:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Move memory closer to computation

The speedup begins below the developer tools. Hardware, model architecture, and inference software are being optimized together. At the hardware layer, a central constraint is the memory wall: computation needs model weights and KV-cache data, and moving those values can dominate the time spent generating a token. Chieng attributes 50–80% of inference latency to hardware and memory movement, without specifying a workload or measurement configuration.

Her simplified hardware comparison contrasts NVIDIA GPUs accessing off-chip high-bandwidth memory, or HBM, with designs that place more memory near compute. This is a bandwidth argument, not a complete inventory of GPU memory. Cerebras and Groq are examples of companies pursuing proximity between storage and computation; on the Cerebras wafer, SRAM is distributed across the chip so cores can directly access the values they need. The complete wafer diagram makes that spatial relationship visible.

Slide titled The Memory-Wall shows a WSE-3 diagram labeled 900,000 cores and states that each core has direct access to memory.Open full source frame
The memory wall: bringing memory close to each core.
3:153:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:15 · section reference included

Give prefill and decode different hardware

Disaggregated inference takes specialization a step further by separating two phases of a request. Chieng connects this development to commercial activity, including NVIDIA and Groq and the announced Cerebras–AWS collaboration. Her description of NVIDIA buying Groq needs a correction: Groq’s announcement describes non-exclusive technology licensing and staff transfers while Groq remained independent; it does not substantiate the acquisition price she mentions. The AWS example was also an announced system under development, with customer availability planned for later, rather than an already demonstrated deployment at scale.

PhaseWorkMain pressure in this explanationHardware fit
PrefillProcess input tokens and populate the KV cacheComputeCompute-optimized hardware
DecodeGenerate successive output tokensMemory bandwidthMemory-optimized hardware

Prefill can process the supplied sequence in parallel. Decode is sequential because each new token depends on what came before it. Running both on one device asks the same hardware to serve two different resource profiles. Separating them allows each phase to run where its dominant cost is better supported, while the KV cache carries the prompt’s computed state into generation.

4:324:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:32 · section reference included

Compute selectively and reuse prior work

Moving up the stack, model design determines how well a workload fits its hardware. Layer dimensions, model size, and memory requirements all matter. A mixture-of-experts model activates only a subset of its experts for each token, rather than running the entire expert set. This separates total model capacity from the computation performed for an individual token: a large collection of parameters need not all participate in every step.

REAP—Router-Weighted Expert Activation Pruning—goes beyond selective activation to remove experts from the stored model. Chieng introduces it through the intuition of discarding experts that a particular use case does not activate. More precisely, the method ranks expert importance using router gate values and activation norms; pruning is not restricted to experts that never activate. Routing reduces the work selected for a token, while pruning reduces the expert set available at all.

At the inference-software layer, Chieng names Together, Baseten, Modal, and Fireworks and highlights KV-cache reuse. Caching preserves previously computed keys and values so generation does not have to recompute those representations for the entire preceding sequence. New tokens still perform attention over relevant cached information; caching avoids repeated work rather than eliminating attention.

5:365:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:36 · section reference included

Optimize for understood work, not agent count

At the top of this increasingly fast stack sits the developer. Chieng’s examples of online coding culture include six Claude Code terminals, a swarm of more than 500 agents, and eight agents spread across five screens. Those displays make concurrency look like progress. Her concern is the code that nobody verifies: increasing its production rate also increases the potential rate of technical-debt accumulation. Spark is the immediate example, but the workflow problem applies to a broader future of faster models.

Model selection therefore needs a third dimension alongside intelligence and cost: speed. Different parts of a task need different strengths.

  • Planning and long-horizon work: Use a larger model such as GPT-5.4 or GPT-5.3 to reason through the task and produce the plan.
  • Bounded execution: Give Codex Spark agents the individual steps, with a clear scope and an order of work.

The point is to assign a fast executor something specific to accomplish, rather than treating every agent as an independent owner of the whole project.

A successful difficult session can also become a reusable skill. First let the larger model work through the unfamiliar task. Then capture the successful trajectory as a verifiable, repeatable workflow. A smaller, faster agent can subsequently repeat that established procedure in the background. The reusable asset is the working procedure and its checks, not merely the final output of the original session.

7:057:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:05 · section reference included

Spend the saved waiting time on validation

Some activities become more attractive when asking the model for another pass no longer creates a long interruption. Chieng calls validation “basically free” in this regime. That describes reduced generation waiting, not zero execution time or cost: tests, browser automation, and human review still take time. The useful change is that checks can become part of each development step instead of a postponed final phase.

The proposed checks cover different failure modes:

  • Test suites: Check behavior against expected results.
  • Linting and pre-commit hooks: Catch recurring mechanical problems before they enter a commit.
  • Diff reviews: Examine what the agent actually changed.
  • Browser-based QA: Inspect behavior in the interface where users encounter it.

Run the relevant checks as the work develops, not only when the entire feature is finished or the code is about to be pushed. Smaller changes make each check easier to interpret and each correction easier to contain.

9:5710:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:57 · section reference included

Use variety to bring human taste into the result

Consider a request for a midnight-blue navigation bar with four icons. A single generated result may be acceptable without being the one you would choose. Chieng proposes asking for 15 variants in the time an older model would take to produce one; this is an illustrative timing comparison, not a supplied benchmark. Her larger example uses five agents producing 15 variants each, yielding 75 candidates for selection.

This is a different use of output volume from accepting a large pile of unreviewed implementation. The candidates are alternatives to compare, not changes that all need to enter the product. The same approach can help with research directions, architecture options, and graphic design. In the accompanying slide, five dark sidebar designs make the selection task concrete: the developer can inspect the differences rather than accept the first plausible answer.

Tip 4 slide shows five dark navigation sidebar designs under With Codex Spark and 1 min → 5 Outputs, alongside example prompt and response text.Open full source frame
Explore cherry picking: five sidebar alternatives with Codex Spark.

Chieng describes selection as a way to supply taste that the model does not reliably provide itself. Otherwise, the developer may spend substantial time creating a reference, finding examples, or writing a prompt detailed enough to nearly constitute the work. Generating alternatives shifts some of that effort from specification to judgment: decide what works after there is something concrete to evaluate.

10:4710:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:47 · section reference included

Stay present, constrain changes, and clean up

The old interaction pattern was to start a session, get food or scroll, and come back when the model had finished. Fast inference makes a more immediate exchange possible. Ask the agent to gather context across the repository and explain how the system works, then use that explanation to make implementation decisions. The model should help the developer decide; the developer should remain responsible for the decision.

That requires staying close enough to the work to understand it before someone asks for an explanation. Chieng suggests actively steering two or three sessions instead of leaving ten agents unattended. Useful boundaries include prohibiting file deletion, imposing a maximum diff size, and restricting permitted operations. During implementation, narrow the instruction further: change only the relevant area, postpone type changes, or stop and redo an approach as soon as it looks wrong. Fast responses make these interventions part of the normal conversation.

Apply the same short cycle to refactoring. After each completed checklist task, ask for a cleanup pass: remove unused imports, eliminate unnecessary lines, and make function structure consistent. This uses the same reduced-waiting opportunity as validation. Cleanup becomes part of completing a small task, rather than an increasingly large job saved for the moment before committing.

12:0712:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:07 · section reference included

Bound the task and persist what the next session needs

Faster generation also makes context management more urgent. Chieng’s illustration assumes context accumulates 20× faster: a window that previously filled in ten minutes would then fill in thirty seconds. This is conditional arithmetic, not a measured compaction interval. The relevant question is how quickly the workflow accumulates context, not just how quickly the model emits tokens.

Break large tasks into smaller, bounded goals before the session becomes crowded. Chieng advises avoiding the 80–100% context-occupancy region because compaction may lose information; that range is her working heuristic, not a universal trigger threshold. A bounded task should be understandable without carrying the entire history of the project in the conversation.

The next step is to put durable memory somewhere a fresh session can read it. Chieng proposes four files:

FileDurable responsibility
AGENTS.mdProject guidance and agent responsibilities
plan.mdOverall plan and ordered checklist
progress.mdCompleted work, remaining work, and where to resume
verify.mdChecks required before advancing

In her setup, AGENTS.md includes agent and subagent definitions. Its general role is persistent project instructions, rather than a required subagent-definition format; the other three filenames are workflow conventions. A new session reads the recorded progress to identify its next task, then uses the verification instructions before moving forward.

For the midnight-blue navbar example, a useful initial progress.md could make the distinction between planned work and verified work explicit:

markdown

# Progress

## Goal
Create a midnight-blue navigation bar with four icons.

## Completed
- None yet.

## Next task
Generate candidate designs for human selection.

## Remaining
- Record the selected design.
- Implement only the selected design.
- Review the diff and clean up the changed code.
- Run the checks in verify.md and record their results.

## Verification
Pending. No implementation has been accepted.

This record gives a fresh executor a concrete starting point without pretending that a proposed design has already been implemented or checked. In the planner–executor arrangement, GPT-5.3 or GPT-5.4 Codex creates the plan and GPT-5.3-Codex-Spark executes its checklist one item at a time. The files carry the continuity between those sessions.

14:3014:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:30 · section reference included

Make careful development easier to sustain

The closing slide brings the workflow back to concrete Codex controls: /permissions, /experimental, /skills, /review, and /rename, alongside /new, /resume, and /fork for conversation management. These are the controls shown in the recording, not a guarantee of availability in every current client. They connect the earlier ideas to permissions, reusable procedures, review, and deliberate session boundaries.

Terminal-style slide lists /permissions, /experimental, /skills, /review, /rename, /new, /resume, and /fork with brief descriptions.Open full source frame
Codex controls for permissions, skills, review, and conversation management.

The developer-experience payoff is the ability to do quality work without turning every question or correction into a long wait. Faster feedback makes it more practical to stay with the code, examine a small change, request a revision, and verify the result while the problem is still fresh. The model can move quickly while the developer takes the time needed to understand what should happen next.

16:5417:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:54 · section reference included

Resources

From the talk

  • OpenAI's launch account of the fast coding model, including its initial capabilities, context window and performance framing.

  • The partnership announcement explains splitting prefill onto Trainium and decode onto Cerebras, connected through KV-cache transfer.

  • REAP the ExpertsPaper6:16

    Research on compressing mixture-of-experts models by pruning experts using router weights and activation magnitudes.

  • Official code and experiment instructions for REAP expert pruning and comparison methods.

  • Stop Shipping AI SlopArticle

    Sarah Chieng and Sherif Cherfa's practical guide to fast coding workflows, bounded changes, verification and persistent project state.

  • An explanation of how autoregressive transformers reuse cached token information during generation.

Updates since the talk

Transcript download
Full transcript

Loading transcript…