How to Build an AI Code Review Agent - Daksh Gupta, Greptile

Daksh Gupta· co-founder/CEO, Greptile18:27

Loading slides…

Read the talk

How to Build an AI Code Review Agent

Codebase context, running code, and task-specific model selection support reviewing and testing a pull request.Open full source frame
Codebase context, running code, and task-specific model selection support reviewing and testing a pull request.

Daksh Gupta walks through the gap between a sandbox that posts PR comments and a useful reviewer: targeted bug detection, historical evaluations, different models for writing and reviewing, codebase context, autonomous testing, and selective inference spending.

From a talk by Daksh Gupta

At a glance

Ideas worth remembering

  • Define the bug types worth catching and the nitpicks worth suppressing. Low-value comments can waste human attention and cause coding agents to add unnecessary defensive code.

  • Evaluate reviewer changes against historical incident PRs, and keep adding new failure modes. Fewer comments or cheaper inference do not establish that the reviewer still catches important bugs.

  • When you can identify which model wrote a change, try reviewing it with a different model family. Greptile reports about a 10% higher bug catch rate with this approach. The talk leaves the test population and baseline catch rate unspecified.

  • Codebase flows help identify contextual bugs such as authentication bypasses; documentation, tickets, logs, and telemetry help connect implementation to intended and observed behavior.

  • Autonomous UI testing needs both a running PR environment and documented flows. Smaller models can handle context selection and application operation while frontier models focus on consequential review decisions.

A webhook gets you comments. Validation takes more.

When engineers—and increasingly their colleagues outside engineering—can generate code quickly, review becomes the bottleneck. Daksh Gupta, co-founder of Greptile, builds agents that review and test pull requests with codebase context. His starting point is practical: some companies have requirements specialized enough to justify building their own reviewer, and the first working version is surprisingly small.

Five problems with a basic reviewer: missed bugs, missing context, expensive reviews, nitpicky comments, and new findings on every run.Open full source frame
Five problems with a basic reviewer: missed bugs, missing context, expensive reviews, nitpicky comments, and new findings on every run.

Put a coding agent in a sandbox. Connect GitHub or GitLab so that opening a pull request sends its contents to the agent through a webhook. Give the agent a prompt asking for a review, then post its feedback through the review API. Codex or Claude Code can supply the agent; Gupta recommends a model-independent harness, a choice that becomes important when the reviewer needs to use a different model from the code’s author. In his sketch, the PR receives feedback about five minutes later.

That proves the plumbing works. It also exposes several different failures:

  • Missed bugs: The agent produces comments without necessarily finding the defects most likely to cause an outage or incident.
  • Missing context: A change can depend on library behavior, architectural direction, or interactions across thousands of repositories. The PR contents alone do not explain those relationships.
  • High inference costs: Gupta reports seeing internal implementations spend between $3 and $25 per pull request. Repeated exploration and file reading can quietly accumulate substantial token bills.
  • Nitpicky comments: Technically correct but overly pedantic findings consume attention that should go to consequential problems.
  • An unstable stream of findings: Running the reviewer again on the same PR can reveal another batch of issues, making the review feel as though it will never finish.

Nitpicks have a second cost beyond annoying humans. Review comments often become instructions for another coding agent, which attempts to resolve all of them without deciding whether each requested change is worthwhile. A pedantic suggestion can therefore turn into another layer of defensive code. The causal chain matters: noisy review does not merely hide useful feedback; it can actively steer implementation toward unnecessary complexity.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Tell the reviewer which bugs matter—and what to leave alone

The cheapest improvement is to replace the generic review prompt with a more selective one. A request such as “Review this PR” and return comments in GitHub’s API format specifies the output shape, but says little about what deserves a comment. Gupta separates that missing guidance into two parallel instructions:

Target specific bug types and skip nitpicks.Open full source frame
Target specific bug types and skip nitpicks.
  • Search for specific failure types. Examine previous outages and incidents, identify the kinds of mistakes the company repeatedly makes, and build a catalog. Ask the reviewer to look for those bug types rather than conducting an unrestricted search for anything questionable.
  • Define and exclude nitpicks. Explain which technically defensible objections are not useful review findings. Gupta’s examples include demanding three levels of defensive checks or objecting to slightly untidy error handling without establishing a consequential failure.

The difficulty is knowing whether a quieter prompt is better. A small wording change can produce a large change in comment volume. Fewer comments might mean the agent stopped being pedantic, or it might mean the agent stopped catching real bugs. A reviewer can demonstrate the presence of a defect; a clean review cannot establish the absence of defects. Because catastrophic failures are rare, an apparently successful prompt change can remain in production for a long time before a serious miss becomes visible.

6:046:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:04 · section reference included

Replay the pull requests that caused incidents

Historical incident PRs give prompt tuning a more useful test than comment counts. Start with the company’s incident records, collect the pull requests responsible for those failures, and run each candidate version of the reviewer against them. The question becomes concrete: how often does this version catch the bugs already known to have caused trouble? That comparison lets a team reduce noise while checking whether it has also lost detection ability.

Historical pull requests with known bugs form a repeatable dataset for comparing review results and tuning prompts.Open full source frame
Historical pull requests with known bugs form a repeatable dataset for comparing review results and tuning prompts.

The dataset needs to keep changing. Gupta estimates that about 80–90% of bugs repeat prior mistakes, which makes historical cases valuable, but coding agents also develop new failure modes. An evaluation set that covers yesterday’s mistakes can miss tomorrow’s. Broad coverage is difficult for an individual company because rare bug types require a very large number of PRs to encounter.

This is one of the advantages Gupta attributes to Greptile’s cross-company corpus: it can test changes against a much wider range of pull requests than most internal tools can. Internal incident data still gets a company meaningfully far, particularly on its own recurring problems. The limitation is coverage, not the usefulness of replaying known failures.

7:348:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:34 · section reference included

Use a different model to review the code

Model independence now earns its place in the architecture. Gupta’s research found different error patterns in Claude and GPT models, including a greater tendency toward authentication bypasses in the GPT models he studied. His proposed explanation is that a model can carry similar blind spots into generation and review: the reasoning that failed to notice a problem while writing code may fail again while checking it.

The slide reports an approximately 10% higher bug catch rate in Greptile’s research; experimental details are not shown.Open full source frame
The slide reports an approximately 10% higher bug catch rate in Greptile’s research; experimental details are not shown.

The correction is model inversion. If Claude Code wrote the change, route the review to a GPT model; use the opposite direction for GPT-written code. Gupta reports a 10% higher bug catch rate when flipping models. The talk does not specify the experimental conditions or whether that means a relative increase or percentage-point gain, so the result supports trying and evaluating cross-model review—not calculating a corresponding reduction in production incidents.

The routing problem is identifying what wrote the PR. Gupta describes fingerprints such as a codex/ branch-name prefix or a Claude co-author attribution in PR or commit metadata. He estimates these clues cover about 80% of cases. Manually named branches and other exceptions make the rest harder. An internal catalog of coding-agent usage or telemetry from development machines can supply additional evidence, but the model’s identity is not always recoverable from the code submission alone.

9:049:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:04 · section reference included

Authentication bugs require a map of the codebase

A reviewer can use a different model and still misunderstand the system it is reviewing. Greptile began as a codebase-context company, working on retrieval over large repositories. Gupta rejects the idea that grep alone solves that problem: imprecise search encourages the agent to read large quantities of files, then load their contents into model context. The expensive pattern is repeated search, file read, and cache write.

Changed code reaches a service and an authentication check across repository boundaries.Open full source frame
Changed code reaches a service and an authentication check across repository boundaries.

Greptile instead builds an internal knowledge base that catalogs the system’s parts. An authentication entry, for example, describes how authentication works, the flows involved, and the files that implement them. The entry helps the reviewer find the relevant code without reconstructing the entire subsystem through broad searches on every PR.

Authentication bypasses make the need for this context concrete. To decide whether a missing check is a bug, the reviewer needs to know whether execution has already entered an authenticated environment at that point in the codebase. A local code fragment does not necessarily answer that question. The flow and its surrounding files supply the relationship needed to distinguish a real bypass from a check that belongs elsewhere.

1:1211:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:04 · section reference included

A living wiki connects implementation, behavior, and intent

Code can run correctly and still implement the wrong thing. Gupta uses the image of Slack becoming an IDE: someone sends a sentence or two asking an agent to fix something, but those sentences leave much of the intended behavior unstated. The agent may misinterpret the request or make an arbitrary choice where a human colleague would draw on shared knowledge.

Codebase crawls and telemetry from reverts and incidents feed a living wiki used by the sandboxed review agent.Open full source frame
Codebase crawls and telemetry from reverts and incidents feed a living wiki used by the sandboxed review agent.

Tickets and documentation supply some of that missing knowledge. Jira or Linear can carry the work’s intent; Confluence or Notion can explain the product and its conventions; the codebase index explains the implementation. Engineers often underestimate how much accumulated organizational knowledge helps them fill gaps in a short request. Giving the agent access to these sources makes more of that background available during review.

How do those sources help both review quality and search cost? Greptile places a living wiki between the context sources and the agent. The wiki combines knowledge of the codebase, application behavior from production telemetry and logs, and roadmap information from documentation and tickets. It supplies background for reasoning about a change and helps direct code searches toward relevant files.

The two outputs matter separately. Organizational context helps the reviewer assess whether the change fulfills its intended purpose. More precise retrieval reduces the amount of code it needs to read to make that assessment. Gupta describes the goal emphatically as never reading an unnecessary file; the supported mechanism is the wiki’s guidance about where to look, rather than a demonstrated guarantee of perfect retrieval.

One context layer, two uses

Codebase knowledgesources — Subsystems, implementation flows, and the files involved.
Production behaviorsources — Application knowledge from telemetry and logs.
Documentation and ticketssources — Product and roadmap information beyond the PR.
Living wikicontext — Combines knowledge of the company, application, and codebase.
Targeted code searchuses — Uses background knowledge to choose which files to read.
Review agentuses — Reasons with organizational context and retrieved code.
Independent source families feed the living wiki. Its information helps the reviewer understand the change and guides the retrieval of relevant code; the arrows show information flow, not a sequence of execution stages.
Read the diagram as text
  • Codebase knowledge. sources — Subsystems, implementation flows, and the files involved.
  • Production behavior. sources — Application knowledge from telemetry and logs.
  • Documentation and tickets. sources — Product and roadmap information beyond the PR.
  • Living wiki. context — Combines knowledge of the company, application, and codebase.
  • Targeted code search. uses — Uses background knowledge to choose which files to read.
  • Review agent. uses — Reasons with organizational context and retrieved code.
  • Codebase knowledge → Living wiki: Supplies implementation knowledge.
  • Production behavior → Living wiki: Supplies observed application behavior.
  • Documentation and tickets → Living wiki: Supplies product and roadmap context.
  • Living wiki → Review agent: Supplies background context.
  • Living wiki → Targeted code search: Guides file selection.
  • Targeted code search → Review agent: Supplies relevant file contents.
12:3413:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:34 · section reference included

Render the change and test the user flow

Reading React changes can leave the reviewer satisfied while the rendered interface is unusable on a phone. Gupta’s example is a button that technically appears in the application but is hidden on an iPhone or falls outside the initial visible area. Static review examines the implementation; running the application exposes the result the user actually encounters.

Greptile runs the customer’s stack in dependency-rich, configurable virtual machines and uses a computer-use agent to test it end to end. Two prerequisites make that possible:

  • Documented application flows: The agent needs a map of behaviors to exercise, such as authentication and sign-up.
  • A reproducible development environment: A sufficiently simple build script must start the stack with its required dependencies. Gupta describes helping customers prepare this environment as one of the trickier implementation tasks.

How do the environment and flow map fit together? The setup Gupta describes has a dev box running the PR branch with its dependencies, plus a flow map that tells the agent what paths to test. The running app supplies something the agent can operate; the map supplies the behaviors it should exercise. Vision models can then assess the rendered result, including the phone-layout problem that reading the changed files did not reveal.

Two prerequisites for runtime testing: a PR branch with its dependencies and running app, and a map from code to user flows.Open full source frame
Two prerequisites for runtime testing: a PR branch with its dependencies and running app, and a map from code to user flows.

The useful change is from a judgment about source code to an observation about behavior: a button can exist in the implementation yet remain inaccessible in the rendered interface. Gupta is optimistic enough to call end-to-end QA nearly solved by improved computer use, but that is his assessment, not a coverage result established here. Even in this approach, documented flows and a working environment remain necessary inputs.

14:2615:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:26 · section reference included

Spend frontier inference on the decisions that need it

A context-aware agent that also runs the application can still be expensive. After describing autonomous testing, Gupta again puts a review in the range of several dollars to roughly $20. The next optimization is to stop assigning every subtask to a frontier model. He reports that frontier models account for about 20% of Greptile’s mix at the time of the talk, reserved for the most critical work, while smaller models handle much of the exploratory activity and file reading.

Where can a cheaper model take over without replacing the most consequential reasoning? Gupta’s illustrative split separates the jobs:

  • Finding context: A small model locates useful information.
  • Testing user flows: A lightweight computer-use model operates the application.
  • Analyzing and submitting the review: A frontier model handles bug analysis, the review process, and submission.

The distinction is task-specific: finding a file and deciding whether a change contains a serious bug do not automatically need the same model or inference budget.

Small-model context retrieval, browser-agent user-flow tests, and frontier-model bug analysis sit between planning and submitting a review.Open full source frame
Small-model context retrieval, browser-agent user-flow tests, and frontier-model bug analysis sit between planning and submitting a review.

Greptile’s actual implementation divides the work into 20 subtasks, each with a specific set of models, and a small research team continually revisits those assignments. An internal implementation need not begin with that degree of specialization. Gupta suggests getting much of the benefit by replacing simple tasks with simple models first.

This brings the historical PR dataset back into the design. A smaller model can make a subtask faster and cheaper while quietly reducing the final reviewer’s ability to catch bugs. Without known failure cases to replay, the cost improvement is immediately visible and the quality regression may remain hidden until an outage. Model routing therefore needs evaluation of the resulting review quality, not just a lower token bill.

The final optimization targets what enters the frontier model’s context at all. A cheap model can inspect candidate context pieces and choose which ones to include. Gupta also proposes using a decision model to rerank those pieces before loading them into the frontier model’s cache. This is a proposed allocation mechanism: spend inexpensive inference selecting information so that the more expensive model reads less irrelevant material. In his framing, frontier-model cache is one of the system’s most expensive and scarce resources.

That leaves a concrete build-or-buy decision. A company with unusually specialized needs can build around its own incident history, application flows, and organizational context, but it also takes on environment setup, dataset maintenance, and model-selection work. Gupta closes by offering help to teams that choose to build. The small webhook prototype remains a useful beginning; the ongoing work is making its findings worth acting on at a cost the company can sustain.

15:4316:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:14 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…