AI Engineer World's Fair 2025

How to Improve your Vibe Coding — Ian Butler

Ian Butler· CEO, Bismuth7:30

Loading slides…

Read the talk

How to Improve Your Vibe Coding

Reliable agent-assisted coding requires more than generating fixes: scope the bug search, preserve the code relationships that matter, and demand evidence that repairs work.

From a talk by Ian Butler

Before you start: Familiarity with coding agents, code diffs, and automated tests will help you apply the review workflow.

Who catches the bugs your coding agent misses?

If a coding agent introduces a bug, can you trust it to find and fix that bug later? Ian Butler, CEO of Bismuth, opens with this problem. Bismuth builds an end-to-end coding agent, which he compares to Codex. After several months evaluating agents’ ability to find and fix bugs, his team had released a benchmark that he describes as having launched the previous day. The initial finding is troubling: agents miss bugs while also producing substantial numbers of false reports.

Butler reports that Devin and Cursor each had a true positive rate below 10% for bug finding. Here, SM-100 uses true positive rate to mean valid reports divided by all reported bugs—not the fraction of all existing bugs detected—and specifies pass@1 evaluation. There is also a separate discovery problem: when the team plants bugs in larger codebases, agents struggle to navigate to the relevant code and locate the defect. A tool that generates code faster than it can reliably inspect it can leave an accumulating repair burden.

Butler reports that three of six agents had a true positive rate of 10% or less across more than 900 reports. In one task, an agent returned 70 issues, all of them false. That example makes the practical cost clear: somebody must still investigate the output, and a long list of plausible-sounding issues can consume attention without improving the code.

Slide titled “The Hard Truth” states that three of six agents had 10% or fewer true positives, alongside a photo of a person holding their head.Open full source frame
The hard truth: few true positives among more than 900 bug reports.

Butler reports a 97% false positive rate for Cursor over more than 100 repositories and more than 1,200 issues. These are the historical evaluation totals quoted in the talk; the exact agent-specific result and repository scope were not independently confirmed. The consequence he emphasizes is alert fatigue: developers stop investigating reports, trust erodes, and real defects become easier to ship.

0:160:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Three ways to improve the search

Butler turns those evaluation experiences into three practical adjustments for working beside an agent in an IDE:

  • Bug-focused rules: Use the agent’s rules file to supply scoped instructions about security issues and logical defects.
  • Context management: Preserve relationships between pieces of code. Butler observed agents losing logical links to material they had already read as they moved through a repository. This matters because significant bugs often involve deeply nested, multistep behavior.
  • Thinking models: Prefer models that spend additional effort reasoning through the problem. Butler reports stronger bug-finding performance from these models and recommends selecting them in tools such as Claude Code and Cursor.
1:592:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:59 · section reference included

Name the bug classes, then require validation

Start by giving the agent a concrete security lens. Butler recommends incorporating the OWASP Top Ten into rules files. This is web-application security awareness guidance, not an exhaustive taxonomy of software defects; the linked 2021 edition provides historical context, although Butler does not specify an edition. The mechanism is priming: supplying security criteria makes those criteria part of what the model considers while reading code. Butler says performance was substantially lower when security or bug-related information was absent.

Next, replace a generic request to find bugs with explicit classes of failure. Butler names authentication bypasses, prototype pollution, and SQL injection. These give the agent a more definite search target than an unrestricted repository review. Finally, require tests before accepting a repair: the agent must write them, get them passing, and establish that the reported defect has actually been fixed. A passing test suite is useful only if it exercises the behavior at issue.

“The Rule(s) of Cool” slide lists OWASP-focused rule files, explicit classes of issues, and verification with concrete test cases.Open full source frame
Bug-focused rules: specify security checks, name issue classes, and require fix validation.

A rules-file entry can express that workflow directly:

markdown

# Bug review

Use the OWASP Top Ten as security review guidance.
Focus this review on:
- Authentication bypasses
- Prototype pollution
- SQL injection

For each proposed fix:
1. Add a concrete test covering the reported defect.
2. Verify that the test exposes the defect before the fix.
3. Apply the fix and get the test passing.
4. Explain how the test verifies the repaired behavior.

Do not accept a fix until its validation tests pass.

Butler grounds this recommendation in work across roughly 100 repositories and thousands of issues from multiple agents. Structured rules replace vague requests with explicit criteria for both discovery and acceptance, which he reports produces higher-quality output.

3:103:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:10 · section reference included

Keep the code relationships in context

A bug search depends on retaining the connections between files. As agents reach their context limits, they may summarize or compact previously read material. Butler reports that their ability to detect and understand bugs drops substantially after this happens. The user therefore has an active role in managing what remains available to the agent.

His context-management procedure has three parts:

  1. Supply the diff. Show the changed code so the agent can follow cause and effect, rather than reconstructing the change from a final snapshot.
  2. Preserve key files. Keep the code needed to understand the behavior from being summarized away or removed from the context window.
  3. Request a component inventory. Have the agent work step by step through the classes, variables, and their uses across the codebase.

The inventory is especially useful because it makes the relationships explicit. Instead of relying on a fading recollection of earlier reads, the agent has an organized account of which components exist and where they are used. Butler reports that agents became more capable of finding bugs after performing this inventory.

4:264:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

Deeper reasoning still leaves uneven coverage

Butler’s preference for thinking models comes from both their results and the reasoning traces his team inspected. He describes models exploring several considerations in the code, identifying promising paths, and then investigating those paths more deeply. Across Bismuth’s benchmark, he reports that this process found deeper bugs than non-thinking models did.

That improvement still leaves a coverage problem. Across hundreds of repositories and thousands of issues, Butler observed that the headline number of bugs found could remain similar while the actual bugs identified changed from one run to the next. A stable bug count does not imply stable bug coverage. Two reviews can look equally productive in aggregate while finding different defects.

“Thinking Models Rock” slide has three columns titled Better Detection, Fewer False Alarms, and Substantial Difference, with text acknowledging LLM limitations.Open full source frame
Thinking models offer better detection and fewer false alarms, while retaining limitations.

Butler interprets this variability as evidence that agents do not yet inspect a file comprehensively, even when they use thinking models. His complaint that users should not need to run an agent a hundred times expresses the burden of repeated review, rather than a measured threshold for complete coverage. Consistent, comprehensive discovery remains unresolved. Thinking models are his preferred option because they were more thorough and performed better in the benchmark, but that advantage does not remove the variability.

5:285:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:28 · section reference included

From automated repairs to inspectable results

At the time of the talk, Butler presents Bismuth as a system that automatically creates pull requests, integrates with GitHub, GitLab, Jira, and Linear, scans for vulnerabilities, and provides code reviews. He also advertises on-premises deployment for organizations that need it. These are the product capabilities described in the recording, rather than a statement of current availability.

The closing QR code directs viewers to Bismuth’s site and its benchmark link. Butler names SM-100 and invites viewers to inspect the methodology, results, full dataset, and exploration of agents’ bug-finding and bug-fixing performance. The evaluation repository and dataset explorer provide routes into those materials. The ending returns the question to evidence: which bugs did an agent find, which reports were real, and which repairs actually worked?

Closing slide titled “May Your Vibes Be Immaculate!” shows a large purple QR code, bismuth.sh, and contact information.Open full source frame
A QR code points viewers toward the website and benchmark link.
6:446:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:44 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…