Catch the Bugs Your Code Review Misses - Killian Carlsen-Phelan & Taylor Luttrell-Williams, Sonar
Loading slides…
Read the talk
Catch the Bugs Your Code Review Misses
Killian Carlsen and Taylor Luttrell use Gitar on a sample ticketing API to expose a cancellation bug outside the diff, check an all-in-pricing rule, and diagnose why a used ticket scans twice. They enable automatic fixes and merging through separate controls.
From a talk by Taylor Luttrell-Williams and Killian Carlsen-Phelan
At a glance
Ideas worth remembering
A green test suite can miss a new interaction with unchanged code. In Box Office, transferring a ticket worked, but canceling its original order afterward still voided the friend's ticket; Gitar located the correction in orders.py.
Project guidance and conditional rules serve different roles. The all-in-pricing requirement defines the expected behavior, while a rule triggered by pricing-file edits asks Gitar to inspect and report on affected customer-facing fields.
Automatic fixes and automatic merging are separate permissions. The third exercise enables both, with a required GitHub test check gating merge; its final merge is not confirmed before the walkthrough moves on.
Verification can compare a change with linked requirements, dependent repositories, and other in-flight PRs. Those checks address different ways a locally valid change can still be incomplete or incompatible.
External context supplies requirements to check; Knowledge retains feedback about Gitar's reviews and preferred fixes. The Q&A distinguishes these capabilities without resolving the exact ingestion path for the audience's requested integration.
More code leaves more verification work
AI coding agents can produce changes faster than a fixed-size engineering team can review them. The resulting queue leaves pull requests waiting even when implementation has become faster. In this Sonar workshop, Taylor Luttrell and Killian Carlsen introduce Gitar as a code change verification agent intended to take on both finding problems and fixing them before merge. The question is broader than whether code is secure or passes tests: does the change actually do what the project needs?
Gitar starts a review when a pull request opens. Its inputs include the diff, PR details and comments, CI logs, project-specific instructions, and repository rules. Findings and suggested fixes appear directly on the PR, where a developer can respond through comments or permit automated action. This makes the review part of the delivery workflow rather than a separate report someone must translate into edits.
That ability to act makes false positives more consequential. An incorrect finding costs review attention and tokens; an attempted fix can also introduce an unintended change. Taylor describes Gitar as tuned to reduce that noise, distinguish genuine CI breakages from flaky tests or infrastructure failures, and repeat fixes until CI passes. These are product claims in the introduction, rather than measured precision or reliability results from the workshop. The hands-on examples make a narrower set of behaviors concrete.
The session uses three prepared pull requests: an out-of-the-box review, a review guided by project business rules, and a CI failure followed by automated remediation. Everything runs in the browser, using a GitHub account and Gitar installed as a GitHub App; no local development environment is needed for the exercises.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare the fork and start three independent reviews
The workshop repository is sonarsamples/AIENewYorkWorkshopGitar. Fork it into the organization where the Gitar app is installed, and leave Copy the main branch only unchecked. The other branches contain the exercises; copying only main would leave the demonstration changes behind.
Enable the fork's workflows in GitHub Actions so the tests can run. Then check Gitar's Settings → Configurations and Repositories to confirm that the fork is visible. These are separate setup needs: the agent must be able to read the repository, and GitHub must be allowed to execute its workflows.
The sample application, Box Office, is a small Python ticketing API for a concert venue. Its ordering code in orders.py supports purchases, cancellations, and refunds. Its check-in logic handles scanning a ticket's QR code for admission. A test suite exercises the existing behavior and runs when a PR opens. Those two areas—what an order can do to a ticket, and what happens when a ticket is scanned—supply the important interactions in the exercises.
Open PRs for Stage 1, Stage 2, and Stage 3 against the fork, not the upstream repository. Killian catches that base-repository mistake during setup and corrects it. All three reviews start before the walkthrough proceeds through them: Gitar takes a few minutes to work, so running the reviews concurrently avoids waiting for each one to finish before starting the next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
PR 1: Transfer a ticket, then cancel its original order
The first PR adds ticket transfers without any custom Gitar configuration. A buyer should be able to purchase two tickets, send one to a friend, and attend the concert together. The new behavior includes basic validation: a missing ticket, an invalid email address, or an already-voided ticket must not be transferred. Tests cover those checks, and CI is green.
While the review runs, Killian distinguishes this contextual check from the static-analysis layer supplied by SonarQube. His example is SQL injection analysis that follows data through multiple files and functions. Here, the additional task is to compare the intended transfer feature with existing application behavior. The workshop presents the two approaches as complementary: tracing dangerous code paths and interpreting a feature's business consequences answer different questions.
Gitar posts the finding “Buyer can refund and void a ticket after giving it away.” The failure depends on the order of operations. A buyer transfers a valid ticket to a friend, then cancels the original order. The cancellation still voids the transferred ticket, leaving the friend without a usable ticket. The existing test almost reaches the problem, but checks only the opposite sequence:
| Sequence | Behavior covered in the demonstration |
|---|---|
| Void the ticket, then try to transfer it | Transfer is rejected by the existing validation and test. |
| Transfer the ticket, then cancel the original order | The friend's ticket is voided; this is the missed case. |
Checking one ordering does not establish that the reverse ordering is safe.
Where does the new feature need a correction? Gitar points to the original ordering code in orders.py, outside the transfer change itself. Its review connects the new operation to the older cancellation behavior. That is why inspecting only the added lines can miss the bug: the transfer code can look reasonable while an unchanged operation still has the power to invalidate its result.
A developer can select Apply fix or comment Gitar fix to request a correction. Gitar then adds a commit to the PR and checks the changed version again, rather than stopping at its first proposed patch. In this case, the intended correction adds validation to orders.py. Killian leaves that work running while moving to the second PR; its completed result appears later in the walkthrough.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
PR 2: Teach the review what a customer-facing price means
The second PR adds a $3.50 service fee per ticket. A $50 ticket therefore costs $53.50. The hypothetical company has an additional requirement: every price shown to a customer must be the all-in price. Correctly adding a fee at checkout is insufficient if another customer-facing price still leaves it out. As Killian puts it, surprise fees are a pet peeve; here, that preference becomes an explicit review requirement.
A reviewer cannot reliably infer a rule that exists only in a senior engineer's head or a company wiki. The repository configuration gives Gitar that missing input through two distinct mechanisms:
- Review guidance —
.gitar/review/: Markdown files describe business logic and project expectations that should steer reviews. - Conditional actions —
.gitar/rules/: A rule names criteria that a PR can meet and an action to take when they are met. In this exercise, changes to pricing files trigger a review of the edited customer-facing fields and whether their values include all fees.
The difference is operational. Review guidance supplies the meaning of a correct price; the conditional rule makes a pricing change trigger a particular inspection and report. The output should help a reviewer locate the affected customer-facing fields, rather than merely repeat that the company prefers transparent pricing.
Rules can also retrieve context beyond the repository. Killian proposes a rule that notices a Slack link in a PR description, reads the linked discussion, and checks the implementation against the feature the engineers discussed. The link is the trigger and a pointer to requirements; it is not itself the requirement. This Slack example is described as a possible configuration, rather than executed as another exercise.
What does the pricing rule make inspectable? Gitar posts a review listing the pricing fields and identifying whether they represent all-in prices, then flags a violation. The important distinction is the source of correctness: a fee calculation can be technically valid and still violate the company's customer-facing promise. The walkthrough leaves this PR unmerged after showing the finding; it does not demonstrate a pricing repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
PR 3: Repair a double-scan regression behind a merge gate
Before starting the third exercise, Killian returns to the first PR. Gitar has completed its fix, added the extra check in orders.py, and approved and verified the change. This completes the earlier example: the finding led to an edit in the existing cancellation code, not just a comment on the new transfer feature.
The third PR expands automation to include merging, but first sets up a GitHub gate. Enable Allow auto merge, create an active branch ruleset targeting the default branch, and select Require status checks to pass with the test check required. These settings permit automated merging while preventing the workshop PR from merging with a failing test. The repository README contains the setup instructions for following the exercise later.
The change refactors ticket scanning and updates tickets to point to their ticket-page URLs. This time, CI is already red. The failing test checks that a ticket cannot be scanned twice: once a ticket admits someone, reusing it must not admit another person. The refactor has missed a check that preserves this one-use behavior.
What does the failed check mean in application terms? Gitar's CI diagnosis says that scanning an already-used ticket a second time returns HTTP 200. That connects the test failure to a concrete wrong result: the second scan succeeds when it should be rejected. Unlike the first PR, where green tests missed a feature interaction, this PR has a test that detects the regression. Gitar uses the failing CI evidence to explain what needs repair.
The work being automated is familiar: inspect logs, diagnose the failure, edit the code, push a commit, wait for CI, and repeat if it fails again. Killian describes the waiting and repeated requests for review as a “time-consuming, money-burning process.” He enters two commands on the PR:
Gitar auto apply on: Allow Gitar to push subsequent fixes without a separate approval click for each one.Gitar auto merge on: Allow Gitar to merge once it detects no remaining issues and the pipeline passes.
The first grants permission to change the branch. The second grants permission to finish the delivery step.
How can automatic repair continue without allowing a failing PR to merge? A failing check sends work back to diagnosis and another permitted fix. Merging depends on the successful checks and clean review, with GitHub's required test check acting as an additional prerequisite. The automation therefore depends on the checks and project requirements supplied to it; the first PR has already shown why green CI alone cannot establish every intended behavior.
Gitar acknowledges the commands, and Killian explains that repository-wide approval and merge settings are also available in its dashboard. The commands scope automation to this exercise so the other PRs are not automatically merged. At the last explicit status check, Gitar is still working; the walkthrough moves on before confirming the third PR's final fix or merge. The demonstrated result is the diagnosis and enabled automation, with successful completion still pending at that point.
Repair permission and merge permission have different jobs
Read the diagram as text
- Diagnose CI failure. repair — Read the failure: a second scan of an already-used ticket returns HTTP 200.
- Auto apply enabled. permissions — Permit fix commits without approving each attempt.
- Push a fix. repair — Add a corrective commit to the PR branch.
- Verify the changed PR. repair — Check the updated change and rerun the pipeline.
- Auto merge enabled. permissions — Permit merging when no issues remain and the pipeline passes.
- GitHub required check. permissions — An active default-branch ruleset requires the test status check to pass.
- Merge. delivery — The intended endpoint; completion is not confirmed before the demo moves on.
- Diagnose CI failure → Push a fix: Failure diagnosis informs the correction.
- Auto apply enabled → Push a fix: Authorizes repeated fix commits.
- Push a fix → Verify the changed PR: Validate the new commit.
- Verify the changed PR → Diagnose CI failure: If CI fails again.
- Verify the changed PR → Merge: Only after the pipeline passes and no issues remain.
- Auto merge enabled → Merge: Authorizes automatic merging.
- GitHub required check → Merge: Blocks merging unless test passes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look beyond one diff, one ticket, and one repository
The closing feature tour extends the same contextual approach to work tracking. Functional validation compares a PR with the ticket it implements, asking whether the change is complete as well as clean. Taylor names Jira, Linear, and Slack as supported context sources in the recording, with connections configured through Settings → Integrations. A linked ticket or discussion gives the reviewer a statement of the requested behavior against which to compare the implementation. The README's Take It Further section offers additional exercises beyond the three-PR walkthrough.
Knowledge addresses feedback on the reviews themselves. Taylor describes it as learning conventions and remembering false positives from reviewer feedback, weighted by reviewer seniority. It also exposes the rules behind a review, where they came from, and why they apply. The weighting is described as a feature; the session does not explain its calculation or demonstrate a before-and-after change in review behavior.
Two other checks widen the set of changes considered together:
- Cross-repo analysis: A PR may merge cleanly in its own repository while breaking code in another repository. This check looks beyond the repository receiving the change.
- Cross-PR analysis: Two in-flight PRs may each pass independently but break the application when combined. This check compares changes that are still being developed, rather than treating each green PR as an isolated result.
The distinction is the relationship being inspected: dependencies across repositories versus interactions among concurrent changes.
The dashboard tour opens a finding to show its location, proposed fix, and explanation of why it matters. These closing capabilities are a brief tour rather than additional worked examples. They extend the first exercise's question—what existing behavior could this feature break?—to requirements outside the code and changes outside the current PR.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Q&A: External context and learned review preferences are different inputs
The final questions sharpen a distinction that the feature tour could otherwise blur. Gitar has MCP connectivity and supports configurable integrations for ingesting context. Architecture guidelines, for example, tell the agent what standards to compare code against. Knowledge serves a different purpose: if Gitar suggests a fix and a reviewer explains that the team normally solves that kind of problem another way, it can save the preference for future fixes.
Killian leaves the precise route for the audience's requested knowledge ingestion unresolved, saying he would need to follow up on whether it belongs in the Knowledge feature itself. The useful distinction remains: supplying project requirements changes what the review checks, while correcting the reviewer's preferred remedy changes how it proposes to fix a problem. Connecting a source and learning a convention should not be treated as the same operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
Full transcript
Loading transcript…