AI Engineer World's Fair 2024
Code Generation and Maintenance at Scale
Loading slides…
Read the talk
Code Generation and Maintenance at Scale
Maintaining large codebases requires more than generating plausible edits: agents need precise search, compiler feedback, warm environments, and checkpoints that keep failed attempts from compounding.
From a talk by Morgante Pell
Before you start: Familiarity with pull requests, TypeScript type checking, and language-model context windows will help you follow the examples.
Start with the application already running
How do you move an existing, revenue-producing business application onto Kubernetes? That was the recurring problem behind Morgante Pell’s work at Google Cloud. In his recollection, 90% of customer requests concerned existing applications; his illustrative customer had a business application generating $100 million in revenue. Kubernetes, Kustomize, and Terraform templates served that installed base: customers needed to run software they already depended on.
That experience explains Grit’s focus on maintenance. A prompt that produces a new application makes an appealing demonstration, but everyday engineering involves changing large systems whose behavior already matters—software that, for example, keeps flights running on time. The unit of work is a change to an existing system, with all its dependencies and operational constraints.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One expert directs many agents
AI developer tools address different constraints:
| Approach | What it expands |
|---|---|
| IDE assistance | A developer’s speed while writing code |
| Agents that lower the floor | The tasks less technical users can attempt |
| Agents that raise the ceiling | The scope an experienced engineer can manage |
Autocomplete already fits an established development workflow. Generating an application for a less technical user extends access, but specifications and edge cases still require engineering judgment. Grit concentrates on the third category: principal engineers often know what should change, but cannot personally carry out every change across an organization.
Pell describes the target as the top 1% of engineers, using agents to extend their reach. Adoption can therefore remain concentrated: he reports that roughly 95% of engineers at customer organizations do not touch Grit. His illustrative hundred-engineer team has one deeply involved operator generating hundreds of PRs. The other engineers can receive the resulting work without adopting another daily tool.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From individual lines to coordinated changes
An IDE is a scalpel: precise, local, and well suited to editing individual lines. Changing hundreds of repositories or opening a thousand files calls for a different tool. Pell’s bulldozer analogy describes working at a higher level of abstraction—directing a large change rather than manually moving every piece of code.
The demand grows as generation produces more software to maintain. Pell reports that some customers were producing 20–30% more code, accompanied by more PRs and CI activity. His scenario of AI adoption growing from 5% to 50% is a projection: wider use would increase the volume of code that needs management after it reaches production.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An OpenTelemetry migration becomes an orchestration task
The concrete customer example is an organization with thousands of repositories and developers that wanted to migrate its existing logging instrumentation to OpenTelemetry. OpenTelemetry itself supports logs; this was a change to the customer’s instrumentation approach, not the elimination of logging. Traditionally, such a project requires educating hundreds of teams, explaining how to instrument their code, and getting each team to replace the relevant library usage.
The bottleneck is organizational as much as technical. A program manager tracks repositories in a spreadsheet while teams negotiate roadmap time. Pell describes conventional efforts of this kind as consuming tens of thousands of developer hours. That cost can make a useful migration stay postponed for years; the same coordination problem helps explain why old platforms and large COBOL systems persist.
With agents, one engineer directs the change while compute performs the repeated edits and repairs. For this customer migration, Pell reports under 100 developer hours plus thousands of compute hours. He says the project, postponed for years, was completed in one week through roughly 1,000 PRs, including repair iterations, merging, and migration. These are figures from his customer account, not a general productivity guarantee. The change in workflow is the consequential part: engineering effort moves from coordinating every implementation to directing and reviewing the operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Plan, execute, and review
The workflow has three stages:
- Plan from a codebase index. Semantic indexing uses embeddings to represent file intent. Static analysis supplies code structure, imports, and the dependency graph. Both inform where and how to make the change.
- Delegate the edits. Language models assign file modifications to subagents. Those agents use GritQL, Grit’s code query and transformation language, alongside model-generated diffs to locate and modify relevant code.
- Review through PRs. The principal engineer directing the migration reviews the changes. Repository owners can respond with comments that inform subsequent work without ever opening Grit’s interface.
The PR remains the point where a centrally directed migration meets the developers responsible for each repository.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find every relevant call, not just similar text
For the migration, the search task is to find the error logs that should change. Chunking the repository and retrieving embedding matches does not sufficiently define that set. Log-like language may appear in a user-facing UI alert, which is not the telemetry call being sought. A fixed top-k result count also conflicts with the task: the engineer may not know whether the migration has ten targets or ten thousand until the search runs. Migration search needs to enumerate a constrained set, not merely return a few plausible passages.
The demonstrated GritQL query combines three kinds of evidence in sequence:
- Syntax: match a logger function call and capture its arguments.
- Meaning: use
liketo compare those arguments with an example error message such asan error occurred. Pell describes this as an embedding comparison with a cosine-similarity threshold. - Provenance: follow the import graph to require that the logger comes from
log4j.
The semantic threshold has not disappeared. It now operates inside structural and dependency constraints, so an error-like message is insufficient on its own to qualify a call for replacement. The import condition further restricts the migration to the logging implementation the engineer intends to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Plausible code needs compiler feedback
Finding the right location does not make an edit correct. Pell demonstrates the distinction with Claude 3.5 Sonnet, asking it to convert linter JSON into diagnostics for Grit’s VS Code extension. He says the prompt contained about 100,000 tokens of extension context, including everything required for the task. The model produced a plausible conversion to Language Server Protocol diagnostics; neither the model’s output nor a human reading made the defect obvious.
Type checking and linting exist precisely because plausible code can still be invalid. But the first retry supplied no such evidence: Pell told Claude the code had broken in production and asked why. The response proposed an irrelevant fix. Reconsidering the same code without a concrete failure signal did not identify the underlying problem.
The TypeScript compiler, tsc, exposed incompatible Grit and LSP position and range types. Feeding that diagnostic back to the model led it to use the available range-conversion helper and correct the mismatch. The helper had already been in context; the compiler error made its relevance explicit. More context and better feedback solve different problems. Agents need access to the tools that make human developers effective, including compilers and linters, rather than relying on visual plausibility alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep validation warm
A useful repair loop generates code, builds or type-checks it, and feeds the resulting diagnostics into another model call. Getting that loop to work reliably is a major part of building an agent. Its latency, however, can be dominated by ordinary developer infrastructure.
Pell’s customer example takes ten minutes for a from-scratch type-checking build, compared with roughly 30 seconds for model generation. That is not even the production build. Repeating the cold validation after every edit makes each repair expensive in elapsed time, and a sequence of attempts compounds the delay.
An IDE avoids rebuilding everything after every keystroke. Its language server prepares an in-memory index, keeps it available, and incrementally reconciles affected code as files change. Pell illustrates a warm loop with 30 seconds of generation, one second of TS Server recomputation, and another 30 seconds for repair. These timings describe his example, not a general language-server benchmark.
For an agent that already owns a TypeScript language service, the validation step can request diagnostics from that retained service:
typescript
import ts from "typescript";
export function diagnosticsForEdit(
service: ts.LanguageService,
fileName: string,
): ts.Diagnostic[] {
return [
...service.getSyntacticDiagnostics(fileName),
...service.getSemanticDiagnostics(fileName),
];
}
The service’s host must expose updated file versions and snapshots after an edit; recreating the service for every call would lose the intended reuse. Pell recommends the same kind of persistent tooling that supports an IDE. The distinction is retained state versus repeated cold work: the TypeScript compiler also supports incremental compilation, so a CLI is not inherently incapable of caching.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return to a known good state
In the ideal loop, successful repairs eventually produce a commit and a migration PR. In practice, an agent can get stuck on an error, cycle through unsuccessful techniques, and then repeat them. Its context accumulates failed approaches and misleading explanations. Pell reports that Grit’s chance of producing a successful PR fell dramatically after more than ten consecutive prompts. His response is to save a known good state and return to it when a path stalls, rather than indefinitely extending the same failing chain.
Restoring source files alone would leave another cost: rebuilding the prepared environment. Grit also preserves the in-memory graph used by TS Server. Firecracker snapshots make it possible to capture the microVM’s memory and execution state. Pell describes forking that prepared state into ten isolated environments so different edits can begin with the expensive preparation already done. The checkpoint is therefore more than a Git revision; it includes the warm machinery needed to evaluate the next attempt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Explore broadly, retain a short successful chain
Once prepared environments can be forked, the system begins to resemble distributed infrastructure more than a single assistant running on a laptop. Pell describes flows with six to ten agents working concurrently from a known good state. Each reports back with a candidate result. Evaluation combines model-based assessments with concrete signals such as remaining compiler errors and passing unit tests.
The next step is to choose which state becomes the new checkpoint. Pell uses a database voting analogy: favorable, similar candidate states inform a quorum-like choice, and the selected state becomes the starting point for another round of forks. This describes the selection strategy, not a specified formal consensus protocol.
Pell’s example spends 30–40 generations on a PR while retaining only four generations in its final selected chain. Total exploration can be large without forcing the successful path to carry every failed attempt. Checkpoint selection separates the amount of work performed from the length of the history that continues forward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Whole-file output makes every attempt more expensive
Parallel exploration has an inference bill. Regenerating large files across repeated attempts emits the same unchanged code again and again. Whole-file generation is still the simplest initial implementation, and Pell recommends starting there. But the demonstration also shows its characteristic failure: Sonnet substitutes a comment saying the beginning of a function remains unchanged instead of reproducing the required code.
Output tokens also cost more than input tokens. Pell cites a historical $5/$15 input/output price pair for GPT-4o, and $3/$15 for Claude 3.5 Sonnet. Sonnet’s launch announcement specifies those rates per million tokens. These are talk-era price examples, not current API quotations.
A large context window does not imply an equally large response budget. Pell contrasts models accepting 1.5–2 million context tokens with outputs of roughly 4,000 tokens, without identifying the models in that example. The broader constraint is that autoregressive output remains costly and bounded. For a small modification to a large file, emitting just the edit avoids spending the response on unchanged content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use an edit format that preserves irrelevant code
Localized edits need a format the model can produce and the tool can apply reliably. Pell compares the alternatives this way:
| Format | Main difficulty |
|---|---|
| Unified diff | Correct line numbers and diff structure |
| Search-and-replace through JSON function calls | Escaping code inside strings |
| GritQL loose snippet replacement | Matching a before snippet to the intended source |
For unified diffs, he highlights unreliable line-number arithmetic and argues that diffs are less aligned with the full-file code models commonly encounter in training. JSON function calls avoid that particular format, but quotes, newlines, and other code characters require escaping, adding output overhead.
Grit’s loose replacement instead asks for a tutorial-like transformation: replace this snippet with that snippet. The model emits the before and after code, and the tool loosely matches the before snippet to the existing source before applying the replacement. Enough structure must remain to locate the intended code, but irrelevant contents can be omitted. In Pell’s example, that means not reproducing the existing body of makeMatch when it is unnecessary to describe the change. The edit format makes preservation the tool’s responsibility instead of asking the model to regenerate everything it should leave alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An interface for editing at multiple scales
The interface shown at the end still resembles CI: a conventional workflow view sitting above thousands of executing agents. Yet the engineer’s task has expanded beyond inspecting one run. It now includes understanding and directing changes across an entire codebase.
Pell’s proposed direction is closer to SimCity: zoom out to understand the whole system, zoom in to inspect a particular area, and edit at different levels of detail. This is an interface aspiration rather than a demonstrated shipped capability. It follows directly from the work the agents make possible: when one engineer can direct changes across many repositories, the interface must help that engineer see and control the system at that scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and examples for searching and rewriting code using snippets, metavariables and structural conditions.
Explains saving and restoring microVM memory and state, including copy-on-write memory sharing.
Introduces OpenTelemetry's logging signal and integration with existing logging libraries.
The original model announcement documents its input/output pricing and 200K context window.
Further reading
- Grit's May 2024 releaseArticle
Morgante Pell describes GritQL open sourcing, build-log-driven repairs and migration tooling shortly before the talk.
Describes the language-service interfaces and text snapshots used for incremental parsing.
Explains how the compiler saves project-graph information to accelerate subsequent builds.
Full transcript
Loading transcript…