AI Engineer Code 2025

Fixing the PR Bottleneck — Matt Pocock, AIHero

Loading slides…

Read the talk

Fixing the PR Bottleneck

Agents can produce pull requests faster than people can review them. Matt Pocock explains how better tests, a separate standards reviewer, risk-aware PR descriptions and retrospectives can reduce the work that reaches a human.

From a talk by Matt Pocock

At a glance

Ideas worth remembering

  • Inspect what tests prove: constant assertions, source-text ordering checks and overly simple mocks can pass while leaving important behavior untested.

  • Deep modules give agents a small interface through which to test substantial behavior, reducing dependence on implementation details.

  • A separate review context can apply repository-specific standards and commit fixes before a human receives the PR.

  • Allocate review by reversibility and blast radius; a small diff can still trigger irreversible consequences.

  • Use retrospectives to turn recurring findings into better checks, standards, navigation and tool choices.

A software factory needs brakes

Pull requests were piling up before coding agents arrived. Agents increase the pressure by making more work possible without increasing the attention available to review it. In this AI Engineer Paris talk, Matt Pocock approaches the bottleneck through the processes around a PR: improve the artifact before it reaches a person, then make the remaining decisions easier.

The software factory adds another source of acceleration: work can start without a human requesting each change. A classifier might turn an incoming item into a fix, a reproduction or a notification. A PlanetScale report about slow database queries might trigger another workflow. These are examples of automated initiation, with deterministic code pushing work into the factory.

Permanent acceleration produces what Pocock calls a “slop cannon”: more poor PRs than anyone can reasonably inspect. There is a second cost beyond the review queue. Code is the environment the next agent works in; confusing code gives future changes a worse starting point. The brakes therefore protect both today's reviewer and tomorrow's implementation.

Three layers carry different jobs:

  • Automated checks: Linting, tests, type checking and code-quality metrics enforce repeatable conditions.
  • Automated review: An agent examines problems the checks missed, including the structure of the codebase.
  • Human review: A person inspects the resulting PR and makes the remaining judgments.

The intended speedup comes from reducing interventions at the final layer. More code reaching human review is only useful if less of it needs repair.

Deterministic checks are an attractive first investment because running them primarily costs CPU cycles, rather than model tokens or reviewer attention. When a check catches an agent's bug, fixing it does consume tokens—but those tokens buy a concrete correction. That makes frequent checks inexpensive relative to discovering the same problem during human review.

0:010:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Green CI can certify the wrong thing

A green CI run establishes that the checks passed. It does not establish that the change is ready to merge. Pocock calls automated and human review “lie detectors” because they must also question what the checks actually prove. Three examples show how a test can look reassuring while missing the behavior that matters.

Reasserting a constant. The implementation sets an X post character limit to 280; the test expects that constant to equal 280. Both contain the same answer. The assertion checks the stored value without exercising how the application enforces the limit. It also couples the test to an internal identifier: renaming the constant requires changing the test even if the user-visible behavior stays the same. This is the tautological-test problem—an assertion can preserve the shape of the implementation while contributing little independent evidence about its behavior.

Reading source order instead of UI order. Another test is meant to verify that a pitch detail page places its video section after its content plan. Rather than rendering the page, it reads the module as text, finds the content-plan and video references, and compares their positions. The observed result is the order of strings in a file. A source refactor can therefore break the test without changing the rendered page. To understand why this test is fragile, the useful distinction to inspect is the file-position comparison versus the intended screen-position check.

The test compares where “Content Plan” and “Videos” occur in the source file, then asserts that the Videos string comes later.Open full source frame
The test compares where “Content Plan” and “Videos” occur in the source file, then asserts that the Videos string comes later.

typescript

describe("pitch detail page section order", () => {
  it("renders Videos section after Content Plan", () => {
    const content = fs.readFileSync(PITCH_DETAIL_PATH, "utf-8");

    const contentPlanIndex = content.indexOf(">Content Plan<");
    const videosIndex = content.indexOf('title="Videos"');

    expect(contentPlanIndex).toBeGreaterThan(-1);
    expect(videosIndex).toBeGreaterThan(-1);

    expect(videosIndex).toBeGreaterThan(contentPlanIndex);
  });
});

Mocking away failure. The useAudioBoost function uses the browser's AudioContext API. Its test substitutes fake methods for that API, but real AudioContext behavior includes error modes under particular conditions. The stub cannot produce those failures, so the test cannot detect them. The specific browser conditions are not identified in the talk; the supported failure is narrower and still consequential: replacing a dependency with an always-successful imitation removes its error paths from the test.

These failures do not require an agent to cheat deliberately. An agent can follow an instruction to write tests and produce assertions that are easy to satisfy because they inspect implementation details or omit difficult behavior. The design question becomes: how can the repository make meaningful checks easier to write, and superficial checks harder to substitute?

4:504:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:50 · section reference included

Give tests a small interface with substantial behavior behind it

Codebase design can change the tests an agent naturally writes. Pocock uses John Ousterhout's idea of deep modules from A Philosophy of Software Design: a small interface hides a substantial implementation. The contrasting shallow module exposes many functions that individually do little. With more internal details available to callers, there are more opportunities for tests to depend on those details.

What makes a deep module useful for testing? The comparison below separates the interface a test should call from the implementation it should leave hidden. Tests written through the small interface can exercise substantial behavior while allowing the internals to change. That benefit depends on actually directing the agent to use the interface; merely creating one does not prevent tests from reaching inside the module.

Pocock's module-deepening skill proposes opportunities rather than automatically declaring a codebase fixed. It produces an HTML report with before-and-after designs, including a demonstrated proposal to reduce duplication and create a deeper, testable module. The next action is to implement the proposal. The useful question for that report is what moves behind the interface and what duplicated work disappears—not simply whether the resulting diagram looks tidier.

The report compares three repeated visibility checks with three components calling shared predicates, making the proposed reduction in duplication visible.Open full source frame
The report compares three repeated visibility checks with three components calling shared predicates, making the proposed reduction in duplication visible.

A companion codebase-design skill gives the team shared terms. Locality concerns how related code is located together and how changes travel through it. Leverage describes how much useful behavior a caller gets from a simple function call. The skill also names seams. Shared vocabulary gives reviewers and agents a consistent way to describe design changes, instead of relying on competing architectural labels.

What does the test have to know about the module?

Deep module ADeep — Small interface; large hidden implementation. Tests use the interface rather than reaching into implementation details.
Shallow module BShallow — Large interface; many callable functions that individually do little. More structure is exposed to callers and tests.
A deep module exposes a small interface over substantial behavior. A shallow module exposes more small operations, increasing the opportunity for tests to depend on internal structure.
Read the diagram as text
  • Deep module A. Deep — Small interface; large hidden implementation. Tests use the interface rather than reaching into implementation details.
  • Shallow module B. Shallow — Large interface; many callable functions that individually do little. More structure is exposed to callers and tests.
8:228:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:22 · section reference included

Move standards into a separate review context

Good design principles still need enforcement. Pocock's counterintuitive advice is to keep the detailed coding standards out of the implementer agent's context. That agent already has three substantial jobs: find the relevant code, make the change, and debug it by running checks. Adding a large standards burden asks the same context window to handle another kind of work.

The code-review skill instead receives the diff and reads a repository standards file. It runs in a subagent with its own context window and budget. The diff narrows the search, although the reviewer still explores surrounding code to understand the change. Its primary task is assessing standards, without carrying the original implementation and debugging workload.

This splits the work into two passes: make it work, then make it good. Pocock relates it to red-green-refactor, with implementation and refactoring handled in separate contexts. He reports that the split has worked well for him; the talk does not supply a measured comparison of review quality or context usage. The practical mechanism is specialization: fewer competing responsibilities leave the reviewer more room to apply detailed standards.

The file placement expresses that division. Put the standards in codingstandards.md, read by the review agent, rather than placing all of them in agents.md where they enter the implementer's general instructions. The advice is about where detailed standards receive attention: a separate review pass gives them a defined job instead of letting them compete with the task of getting the change working.

11:3012:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:30 · section reference included

A reviewer should reduce work, not generate comments

A generic automated reviewer faces an awkward tradeoff. Broad rules can produce false positives that do not fit the repository; narrow rules can become specific to one language or environment. A TypeScript-focused reviewer, for example, is less useful to a Rust team. Pocock's experience trying to build a generic bug and security reviewer leads him to recommend developing the team's own automated review and standards over time.

Existing team documentation is useful input. Rules that sit unread in documents can become criteria the reviewer applies to every change. Shared standards also let the team refine the reviewer around its actual code, rather than accepting irrelevant findings as the unavoidable cost of automation.

The reviewer's output matters as much as its criteria. A stream of verbose PR comments leaves the human to read each suggestion, decide whether it is valid, and arrange the fix. Pocock's preferred default is commits: the reviewer changes the problems it can resolve, so the human receives an improved artifact. Questions and uncertain findings can remain comments. This turns review capacity into corrections rather than another queue of decisions.

14:2214:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:22 · section reference included

Make merge danger and the change itself easy to see

Once checks and automated review have improved the code, the PR still needs to help a human understand it. Pocock introduces a PR skill that was in progress at the time of the recording. Its first principle is that reviews deserve different levels of attention. The distinction starts with whether merging is a one-way or two-way door.

A two-way door is a change whose consequences can easily be reversed. A one-way door has consequences that reverting code cannot undo. A small change that sends an email to 60,000 people is the memorable example: reverting the commit does not unsend the email. Expensive migrations and data loss also demand careful review. The size of the diff is therefore a poor substitute for understanding what execution will do.

Reversibility needs a companion question: what is the blast radius? A “merge danger” summary at the bottom of the PR records both the door classification and how far a failure could spread. A reversible change with localized effects can receive lighter attention. The summary makes the reason inspectable, rather than asking a reviewer to infer risk from a reassuring description.

The next question is what actually changes. Pseudocode and diagrams can make that answer faster to grasp. Pocock credits the Show Me skill from the HumanLayer skills repository, which uses images and diagrams to explain changes. His CLI example shows an added command and two added flags. A useful visual answers where those additions fit in the command interface; its exact command and flag names are not given in the transcript. Sequence diagrams serve a different question: what happens, and in what order?

The summary places `clip restore <id>` among the commands and marks `--archived` on `clip list` and `clip get` as the two new flags.Open full source frame
The summary places `clip restore <id>` among the commands and marks `--archived` on `clip list` and `clip get` as the two new flags.
16:2316:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:23 · section reference included

Turn recurring review comments into changes to the system

Human review also exposes weaknesses in the process that produced the code. Teams share skill files and steering instructions, so a recurring mistake can have a shared cause. Pocock's target is simple: never write the same review comment twice. Fixing one PR is valuable; changing the check or instruction that would catch the same problem next time makes the intervention compound.

The Retro skill supplies the mechanism. Its input can be one agent session, a PR together with its session, or a week's PRs and reviews. It examines that work and suggests automated checks and coding standards for subsequent changes. The session matters because it contains how the agent reached its result, giving the retrospective more to inspect than the final diff alone.

Retro examines several distinct sources of wasted effort:

  • Missing checks and standards: Propose automated checks or updates to codingstandards.md so later changes encounter the lesson earlier.
  • Navigation pointers: Examine how easily the agent found information, and whether a pointer in agents.md would improve its next search.
  • Tool economy: Inspect whether different tools could perform the session's work with fewer tokens.
  • Instruction bloat: Look for oversized steering files or skills that contribute to poor results, and opportunities to organize them better.

These are separate interventions. A missing test, an expensive tool choice and a hard-to-find document need different repairs.

The ending keeps human attention tied to consequences. Pocock's recommendation is that every one-way door receive review, while every two-way door need not. Stronger checks, a reviewer that fixes known problems, and a PR that explains risk make that allocation more practical. Retro then gives the team a way to use the attention it did spend to improve the next change.

19:4420:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:44 · section reference included

Transcript download
Full transcript

Loading transcript…