How to Build an AI Code Review Agent - Daksh Gupta, Greptile
Loading slides…
Read the talk
How to Build an AI Code Review Agent
Daksh Gupta walks through the gap between a sandbox that posts PR comments and a useful reviewer: targeted bug detection, historical evaluations, different models for writing and reviewing, codebase context, autonomous testing, and selective inference spending.
From a talk by Daksh Gupta
At a glance
Ideas worth remembering
Define the bug types worth catching and the nitpicks worth suppressing. Low-value comments can waste human attention and cause coding agents to add unnecessary defensive code.
Evaluate reviewer changes against historical incident PRs, and keep adding new failure modes. Fewer comments or cheaper inference do not establish that the reviewer still catches important bugs.
When you can identify which model wrote a change, try reviewing it with a different model family. Greptile reports about a 10% higher bug catch rate with this approach. The talk leaves the test population and baseline catch rate unspecified.
Codebase flows help identify contextual bugs such as authentication bypasses; documentation, tickets, logs, and telemetry help connect implementation to intended and observed behavior.
Autonomous UI testing needs both a running PR environment and documented flows. Smaller models can handle context selection and application operation while frontier models focus on consequential review decisions.
A webhook gets you comments. Validation takes more.
When engineers—and increasingly their colleagues outside engineering—can generate code quickly, review becomes the bottleneck. Daksh Gupta, co-founder of Greptile, builds agents that review and test pull requests with codebase context. His starting point is practical: some companies have requirements specialized enough to justify building their own reviewer, and the first working version is surprisingly small.
Put a coding agent in a sandbox. Connect GitHub or GitLab so that opening a pull request sends its contents to the agent through a webhook. Give the agent a prompt asking for a review, then post its feedback through the review API. Codex or Claude Code can supply the agent; Gupta recommends a model-independent harness, a choice that becomes important when the reviewer needs to use a different model from the code’s author. In his sketch, the PR receives feedback about five minutes later.
That proves the plumbing works. It also exposes several different failures:
- Missed bugs: The agent produces comments without necessarily finding the defects most likely to cause an outage or incident.
- Missing context: A change can depend on library behavior, architectural direction, or interactions across thousands of repositories. The PR contents alone do not explain those relationships.
- High inference costs: Gupta reports seeing internal implementations spend between $3 and $25 per pull request. Repeated exploration and file reading can quietly accumulate substantial token bills.
- Nitpicky comments: Technically correct but overly pedantic findings consume attention that should go to consequential problems.
- An unstable stream of findings: Running the reviewer again on the same PR can reveal another batch of issues, making the review feel as though it will never finish.
Nitpicks have a second cost beyond annoying humans. Review comments often become instructions for another coding agent, which attempts to resolve all of them without deciding whether each requested change is worthwhile. A pedantic suggestion can therefore turn into another layer of defensive code. The causal chain matters: noisy review does not merely hide useful feedback; it can actively steer implementation toward unnecessary complexity.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tell the reviewer which bugs matter—and what to leave alone
The cheapest improvement is to replace the generic review prompt with a more selective one. A request such as “Review this PR” and return comments in GitHub’s API format specifies the output shape, but says little about what deserves a comment. Gupta separates that missing guidance into two parallel instructions:
- Search for specific failure types. Examine previous outages and incidents, identify the kinds of mistakes the company repeatedly makes, and build a catalog. Ask the reviewer to look for those bug types rather than conducting an unrestricted search for anything questionable.
- Define and exclude nitpicks. Explain which technically defensible objections are not useful review findings. Gupta’s examples include demanding three levels of defensive checks or objecting to slightly untidy error handling without establishing a consequential failure.
The difficulty is knowing whether a quieter prompt is better. A small wording change can produce a large change in comment volume. Fewer comments might mean the agent stopped being pedantic, or it might mean the agent stopped catching real bugs. A reviewer can demonstrate the presence of a defect; a clean review cannot establish the absence of defects. Because catastrophic failures are rare, an apparently successful prompt change can remain in production for a long time before a serious miss becomes visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replay the pull requests that caused incidents
Historical incident PRs give prompt tuning a more useful test than comment counts. Start with the company’s incident records, collect the pull requests responsible for those failures, and run each candidate version of the reviewer against them. The question becomes concrete: how often does this version catch the bugs already known to have caused trouble? That comparison lets a team reduce noise while checking whether it has also lost detection ability.
The dataset needs to keep changing. Gupta estimates that about 80–90% of bugs repeat prior mistakes, which makes historical cases valuable, but coding agents also develop new failure modes. An evaluation set that covers yesterday’s mistakes can miss tomorrow’s. Broad coverage is difficult for an individual company because rare bug types require a very large number of PRs to encounter.
This is one of the advantages Gupta attributes to Greptile’s cross-company corpus: it can test changes against a much wider range of pull requests than most internal tools can. Internal incident data still gets a company meaningfully far, particularly on its own recurring problems. The limitation is coverage, not the usefulness of replaying known failures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use a different model to review the code
Model independence now earns its place in the architecture. Gupta’s research found different error patterns in Claude and GPT models, including a greater tendency toward authentication bypasses in the GPT models he studied. His proposed explanation is that a model can carry similar blind spots into generation and review: the reasoning that failed to notice a problem while writing code may fail again while checking it.
The correction is model inversion. If Claude Code wrote the change, route the review to a GPT model; use the opposite direction for GPT-written code. Gupta reports a 10% higher bug catch rate when flipping models. The talk does not specify the experimental conditions or whether that means a relative increase or percentage-point gain, so the result supports trying and evaluating cross-model review—not calculating a corresponding reduction in production incidents.
The routing problem is identifying what wrote the PR. Gupta describes fingerprints such as a codex/ branch-name prefix or a Claude co-author attribution in PR or commit metadata. He estimates these clues cover about 80% of cases. Manually named branches and other exceptions make the rest harder. An internal catalog of coding-agent usage or telemetry from development machines can supply additional evidence, but the model’s identity is not always recoverable from the code submission alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Authentication bugs require a map of the codebase
A reviewer can use a different model and still misunderstand the system it is reviewing. Greptile began as a codebase-context company, working on retrieval over large repositories. Gupta rejects the idea that grep alone solves that problem: imprecise search encourages the agent to read large quantities of files, then load their contents into model context. The expensive pattern is repeated search, file read, and cache write.
Greptile instead builds an internal knowledge base that catalogs the system’s parts. An authentication entry, for example, describes how authentication works, the flows involved, and the files that implement them. The entry helps the reviewer find the relevant code without reconstructing the entire subsystem through broad searches on every PR.
Authentication bypasses make the need for this context concrete. To decide whether a missing check is a bug, the reviewer needs to know whether execution has already entered an authenticated environment at that point in the codebase. A local code fragment does not necessarily answer that question. The flow and its surrounding files supply the relationship needed to distinguish a real bypass from a check that belongs elsewhere.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A living wiki connects implementation, behavior, and intent
Code can run correctly and still implement the wrong thing. Gupta uses the image of Slack becoming an IDE: someone sends a sentence or two asking an agent to fix something, but those sentences leave much of the intended behavior unstated. The agent may misinterpret the request or make an arbitrary choice where a human colleague would draw on shared knowledge.
Tickets and documentation supply some of that missing knowledge. Jira or Linear can carry the work’s intent; Confluence or Notion can explain the product and its conventions; the codebase index explains the implementation. Engineers often underestimate how much accumulated organizational knowledge helps them fill gaps in a short request. Giving the agent access to these sources makes more of that background available during review.
How do those sources help both review quality and search cost? Greptile places a living wiki between the context sources and the agent. The wiki combines knowledge of the codebase, application behavior from production telemetry and logs, and roadmap information from documentation and tickets. It supplies background for reasoning about a change and helps direct code searches toward relevant files.
The two outputs matter separately. Organizational context helps the reviewer assess whether the change fulfills its intended purpose. More precise retrieval reduces the amount of code it needs to read to make that assessment. Gupta describes the goal emphatically as never reading an unnecessary file; the supported mechanism is the wiki’s guidance about where to look, rather than a demonstrated guarantee of perfect retrieval.
One context layer, two uses
Read the diagram as text
- Codebase knowledge. sources — Subsystems, implementation flows, and the files involved.
- Production behavior. sources — Application knowledge from telemetry and logs.
- Documentation and tickets. sources — Product and roadmap information beyond the PR.
- Living wiki. context — Combines knowledge of the company, application, and codebase.
- Targeted code search. uses — Uses background knowledge to choose which files to read.
- Review agent. uses — Reasons with organizational context and retrieved code.
- Codebase knowledge → Living wiki: Supplies implementation knowledge.
- Production behavior → Living wiki: Supplies observed application behavior.
- Documentation and tickets → Living wiki: Supplies product and roadmap context.
- Living wiki → Review agent: Supplies background context.
- Living wiki → Targeted code search: Guides file selection.
- Targeted code search → Review agent: Supplies relevant file contents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Render the change and test the user flow
Reading React changes can leave the reviewer satisfied while the rendered interface is unusable on a phone. Gupta’s example is a button that technically appears in the application but is hidden on an iPhone or falls outside the initial visible area. Static review examines the implementation; running the application exposes the result the user actually encounters.
Greptile runs the customer’s stack in dependency-rich, configurable virtual machines and uses a computer-use agent to test it end to end. Two prerequisites make that possible:
- Documented application flows: The agent needs a map of behaviors to exercise, such as authentication and sign-up.
- A reproducible development environment: A sufficiently simple build script must start the stack with its required dependencies. Gupta describes helping customers prepare this environment as one of the trickier implementation tasks.
How do the environment and flow map fit together? The setup Gupta describes has a dev box running the PR branch with its dependencies, plus a flow map that tells the agent what paths to test. The running app supplies something the agent can operate; the map supplies the behaviors it should exercise. Vision models can then assess the rendered result, including the phone-layout problem that reading the changed files did not reveal.
The useful change is from a judgment about source code to an observation about behavior: a button can exist in the implementation yet remain inaccessible in the rendered interface. Gupta is optimistic enough to call end-to-end QA nearly solved by improved computer use, but that is his assessment, not a coverage result established here. Even in this approach, documented flows and a working environment remain necessary inputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend frontier inference on the decisions that need it
A context-aware agent that also runs the application can still be expensive. After describing autonomous testing, Gupta again puts a review in the range of several dollars to roughly $20. The next optimization is to stop assigning every subtask to a frontier model. He reports that frontier models account for about 20% of Greptile’s mix at the time of the talk, reserved for the most critical work, while smaller models handle much of the exploratory activity and file reading.
Where can a cheaper model take over without replacing the most consequential reasoning? Gupta’s illustrative split separates the jobs:
- Finding context: A small model locates useful information.
- Testing user flows: A lightweight computer-use model operates the application.
- Analyzing and submitting the review: A frontier model handles bug analysis, the review process, and submission.
The distinction is task-specific: finding a file and deciding whether a change contains a serious bug do not automatically need the same model or inference budget.
Greptile’s actual implementation divides the work into 20 subtasks, each with a specific set of models, and a small research team continually revisits those assignments. An internal implementation need not begin with that degree of specialization. Gupta suggests getting much of the benefit by replacing simple tasks with simple models first.
This brings the historical PR dataset back into the design. A smaller model can make a subtask faster and cheaper while quietly reducing the final reviewer’s ability to catch bugs. Without known failure cases to replay, the cost improvement is immediately visible and the quality regression may remain hidden until an outage. Model routing therefore needs evaluation of the resulting review quality, not just a lower token bill.
The final optimization targets what enters the frontier model’s context at all. A cheap model can inspect candidate context pieces and choose which ones to include. Gupta also proposes using a decision model to rerank those pieces before loading them into the frontier model’s cache. This is a proposed allocation mechanism: spend inexpensive inference selecting information so that the more expensive model reads less irrelevant material. In his framing, frontier-model cache is one of the system’s most expensive and scarce resources.
That leaves a concrete build-or-buy decision. A company with unusually specialized needs can build around its own incident history, application flows, and organizational context, but it also takes on environment setup, dataset maintenance, and model-selection work. Gupta closes by offering help to teams that choose to build. The small webhook prototype remains a useful beginning; the ongoing work is making its findings worth acting on at a cost the company can sustain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI
Extends the review-bottleneck discussion with false-positive filtering, multiple review passes, organizational quality standards, and evaluation of the reviewers themselves.
- Why AI Didn't Actually Make You Ship Faster — Gabriel Spencer-Harper, Meticulous
Provides a complementary approach to runtime verification: replaying browser workflows and comparing visual changes, with explicit mechanisms for repeatability and test selection.
- Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Develops the evaluation work behind safe prompt and model changes, including representative datasets, expert labels, experiments, and quality gates.
Full transcript
Loading transcript…