AI Engineer World's Fair 2024

Code Generation and Maintenance at Scale

Morgante Pell· CEO, Grit18:54

Loading slides…

Read the talk

Code Generation and Maintenance at Scale

Maintaining large codebases requires more than generating plausible edits: agents need precise search, compiler feedback, warm environments, and checkpoints that keep failed attempts from compounding.

From a talk by Morgante Pell

Before you start: Familiarity with pull requests, TypeScript type checking, and language-model context windows will help you follow the examples.

Start with the application already running

How do you move an existing, revenue-producing business application onto Kubernetes? That was the recurring problem behind Morgante Pell’s work at Google Cloud. In his recollection, 90% of customer requests concerned existing applications; his illustrative customer had a business application generating $100 million in revenue. Kubernetes, Kustomize, and Terraform templates served that installed base: customers needed to run software they already depended on.

That experience explains Grit’s focus on maintenance. A prompt that produces a new application makes an appealing demonstration, but everyday engineering involves changing large systems whose behavior already matters—software that, for example, keeps flights running on time. The unit of work is a change to an existing system, with all its dependencies and operational constraints.

0:390:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:39 · section reference included

One expert directs many agents

AI developer tools address different constraints:

ApproachWhat it expands
IDE assistanceA developer’s speed while writing code
Agents that lower the floorThe tasks less technical users can attempt
Agents that raise the ceilingThe scope an experienced engineer can manage

Autocomplete already fits an established development workflow. Generating an application for a less technical user extends access, but specifications and edge cases still require engineering judgment. Grit concentrates on the third category: principal engineers often know what should change, but cannot personally carry out every change across an organization.

AI Developer Tools slide with three purple columns comparing IDE developer assistants and two approaches to AI agents.Open full source frame
Three categories of AI developer tools: IDE assistants, lowering the floor, and raising the ceiling.

Pell describes the target as the top 1% of engineers, using agents to extend their reach. Adoption can therefore remain concentrated: he reports that roughly 95% of engineers at customer organizations do not touch Grit. His illustrative hundred-engineer team has one deeply involved operator generating hundreds of PRs. The other engineers can receive the resulting work without adopting another daily tool.

1:451:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:45 · section reference included

From individual lines to coordinated changes

An IDE is a scalpel: precise, local, and well suited to editing individual lines. Changing hundreds of repositories or opening a thousand files calls for a different tool. Pell’s bulldozer analogy describes working at a higher level of abstraction—directing a large change rather than manually moving every piece of code.

Slide titled “Tools need to change” showing a scalpel on the left and a bulldozer on the right.Open full source frame
Tools need to change: a scalpel versus a bulldozer.

The demand grows as generation produces more software to maintain. Pell reports that some customers were producing 20–30% more code, accompanied by more PRs and CI activity. His scenario of AI adoption growing from 5% to 50% is a projection: wider use would increase the volume of code that needs management after it reaches production.

3:033:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:03 · section reference included

An OpenTelemetry migration becomes an orchestration task

The concrete customer example is an organization with thousands of repositories and developers that wanted to migrate its existing logging instrumentation to OpenTelemetry. OpenTelemetry itself supports logs; this was a change to the customer’s instrumentation approach, not the elimination of logging. Traditionally, such a project requires educating hundreds of teams, explaining how to instrument their code, and getting each team to replace the relevant library usage.

The bottleneck is organizational as much as technical. A program manager tracks repositories in a spreadsheet while teams negotiate roadmap time. Pell describes conventional efforts of this kind as consuming tens of thousands of developer hours. That cost can make a useful migration stay postponed for years; the same coordination problem helps explain why old platforms and large COBOL systems persist.

With agents, one engineer directs the change while compute performs the repeated edits and repairs. For this customer migration, Pell reports under 100 developer hours plus thousands of compute hours. He says the project, postponed for years, was completed in one week through roughly 1,000 PRs, including repair iterations, merging, and migration. These are figures from his customer account, not a general productivity guarantee. The change in workflow is the consequential part: engineering effort moves from coordinating every implementation to directing and reviewing the operation.

3:554:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:55 · section reference included

Plan, execute, and review

The workflow has three stages:

  1. Plan from a codebase index. Semantic indexing uses embeddings to represent file intent. Static analysis supplies code structure, imports, and the dependency graph. Both inform where and how to make the change.
  2. Delegate the edits. Language models assign file modifications to subagents. Those agents use GritQL, Grit’s code query and transformation language, alongside model-generated diffs to locate and modify relevant code.
  3. Review through PRs. The principal engineer directing the migration reviews the changes. Repository owners can respond with comments that inform subsequent work without ever opening Grit’s interface.

The PR remains the point where a centrally directed migration meets the developers responsible for each repository.

5:335:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:33 · section reference included

Plausible code needs compiler feedback

Finding the right location does not make an edit correct. Pell demonstrates the distinction with Claude 3.5 Sonnet, asking it to convert linter JSON into diagnostics for Grit’s VS Code extension. He says the prompt contained about 100,000 tokens of extension context, including everything required for the task. The model produced a plausible conversion to Language Server Protocol diagnostics; neither the model’s output nor a human reading made the defect obvious.

Type checking and linting exist precisely because plausible code can still be invalid. But the first retry supplied no such evidence: Pell told Claude the code had broken in production and asked why. The response proposed an irrelevant fix. Reconsidering the same code without a concrete failure signal did not identify the underlying problem.

The TypeScript compiler, tsc, exposed incompatible Grit and LSP position and range types. Feeding that diagnostic back to the model led it to use the available range-conversion helper and correct the mismatch. The helper had already been in context; the compiler error made its relevance explicit. More context and better feedback solve different problems. Agents need access to the tools that make human developers effective, including compilers and linters, rather than relying on visual plausibility alone.

8:388:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:38 · section reference included

Keep validation warm

A useful repair loop generates code, builds or type-checks it, and feeds the resulting diagnostics into another model call. Getting that loop to work reliably is a major part of building an agent. Its latency, however, can be dominated by ordinary developer infrastructure.

Pell’s customer example takes ten minutes for a from-scratch type-checking build, compared with roughly 30 seconds for model generation. That is not even the production build. Repeating the cold validation after every edit makes each repair expensive in elapsed time, and a sequence of attempts compounds the delay.

Slide titled “… but they are slow” showing Prompt, Build, and Fix stages labeled 30s, 10 minutes, and 30s.Open full source frame
A ten-minute build dominates the prompt–build–fix workflow.

An IDE avoids rebuilding everything after every keystroke. Its language server prepares an in-memory index, keeps it available, and incrementally reconciles affected code as files change. Pell illustrates a warm loop with 30 seconds of generation, one second of TS Server recomputation, and another 30 seconds for repair. These timings describe his example, not a general language-server benchmark.

For an agent that already owns a TypeScript language service, the validation step can request diagnostics from that retained service:

typescript

import ts from "typescript";

export function diagnosticsForEdit(
  service: ts.LanguageService,
  fileName: string,
): ts.Diagnostic[] {
  return [
    ...service.getSyntacticDiagnostics(fileName),
    ...service.getSemanticDiagnostics(fileName),
  ];
}

The service’s host must expose updated file versions and snapshots after an edit; recreating the service for every call would lose the intended reuse. Pell recommends the same kind of persistent tooling that supports an IDE. The distinction is retained state versus repeated cold work: the TypeScript compiler also supports incremental compilation, so a CLI is not inherently incapable of caching.

11:0711:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:07 · section reference included

Return to a known good state

In the ideal loop, successful repairs eventually produce a commit and a migration PR. In practice, an agent can get stuck on an error, cycle through unsuccessful techniques, and then repeat them. Its context accumulates failed approaches and misleading explanations. Pell reports that Grit’s chance of producing a successful PR fell dramatically after more than ten consecutive prompts. His response is to save a known good state and return to it when a path stalls, rather than indefinitely extending the same failing chain.

Restoring source files alone would leave another cost: rebuilding the prepared environment. Grit also preserves the in-memory graph used by TS Server. Firecracker snapshots make it possible to capture the microVM’s memory and execution state. Pell describes forking that prepared state into ten isolated environments so different edits can begin with the expensive preparation already done. The checkpoint is therefore more than a Git revision; it includes the warm machinery needed to evaluate the next attempt.

13:1313:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:13 · section reference included

Explore broadly, retain a short successful chain

Once prepared environments can be forked, the system begins to resemble distributed infrastructure more than a single assistant running on a laptop. Pell describes flows with six to ten agents working concurrently from a known good state. Each reports back with a candidate result. Evaluation combines model-based assessments with concrete signals such as remaining compiler errors and passing unit tests.

The next step is to choose which state becomes the new checkpoint. Pell uses a database voting analogy: favorable, similar candidate states inform a quorum-like choice, and the selected state becomes the starting point for another round of forks. This describes the selection strategy, not a specified formal consensus protocol.

Pell’s example spends 30–40 generations on a PR while retaining only four generations in its final selected chain. Total exploration can be large without forcing the successful path to carry every failed attempt. Checkpoint selection separates the amount of work performed from the length of the history that continues forward.

14:4314:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:43 · section reference included

Whole-file output makes every attempt more expensive

Parallel exploration has an inference bill. Regenerating large files across repeated attempts emits the same unchanged code again and again. Whole-file generation is still the simplest initial implementation, and Pell recommends starting there. But the demonstration also shows its characteristic failure: Sonnet substitutes a comment saying the beginning of a function remains unchanged instead of reproducing the required code.

Output tokens also cost more than input tokens. Pell cites a historical $5/$15 input/output price pair for GPT-4o, and $3/$15 for Claude 3.5 Sonnet. Sonnet’s launch announcement specifies those rates per million tokens. These are talk-era price examples, not current API quotations.

A large context window does not imply an equally large response budget. Pell contrasts models accepting 1.5–2 million context tokens with outputs of roughly 4,000 tokens, without identifying the models in that example. The broader constraint is that autoregressive output remains costly and bounded. For a small modification to a large file, emitting just the edit avoids spending the response on unchanged content.

15:5216:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:52 · section reference included

Use an edit format that preserves irrelevant code

Localized edits need a format the model can produce and the tool can apply reliably. Pell compares the alternatives this way:

FormatMain difficulty
Unified diffCorrect line numbers and diff structure
Search-and-replace through JSON function callsEscaping code inside strings
GritQL loose snippet replacementMatching a before snippet to the intended source

For unified diffs, he highlights unreliable line-number arithmetic and argues that diffs are less aligned with the full-file code models commonly encounter in training. JSON function calls avoid that particular format, but quotes, newlines, and other code characters require escaping, adding output overhead.

Grit’s loose replacement instead asks for a tutorial-like transformation: replace this snippet with that snippet. The model emits the before and after code, and the tool loosely matches the before snippet to the existing source before applying the replacement. Enough structure must remain to locate the intended code, but irrelevant contents can be omitted. In Pell’s example, that means not reproducing the existing body of makeMatch when it is unnecessary to describe the change. The edit format makes preservation the tool’s responsibility instead of asking the model to regenerate everything it should leave alone.

16:5717:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:57 · section reference included

An interface for editing at multiple scales

The interface shown at the end still resembles CI: a conventional workflow view sitting above thousands of executing agents. Yet the engineer’s task has expanded beyond inspecting one run. It now includes understanding and directing changes across an entire codebase.

Pell’s proposed direction is closer to SimCity: zoom out to understand the whole system, zoom in to inspect a particular area, and edit at different levels of detail. This is an interface aspiration rather than a demonstrated shipped capability. It follows directly from the work the agents make possible: when one engineer can direct changes across many repositories, the interface must help that engineer see and control the system at that scale.

Slide titled “The next unsolved problem is UI” with an illustrated city labeled with classes and interfaces.Open full source frame
The next unsolved problem is UI: a city-like view of software structure.
18:0218:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:02 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…