AI Engineer World's Fair 2025
How to Improve your Vibe Coding — Ian Butler
Loading slides…
Read the talk
How to Improve Your Vibe Coding
Reliable agent-assisted coding requires more than generating fixes: scope the bug search, preserve the code relationships that matter, and demand evidence that repairs work.
From a talk by Ian Butler
Before you start: Familiarity with coding agents, code diffs, and automated tests will help you apply the review workflow.
Who catches the bugs your coding agent misses?
If a coding agent introduces a bug, can you trust it to find and fix that bug later? Ian Butler, CEO of Bismuth, opens with this problem. Bismuth builds an end-to-end coding agent, which he compares to Codex. After several months evaluating agents’ ability to find and fix bugs, his team had released a benchmark that he describes as having launched the previous day. The initial finding is troubling: agents miss bugs while also producing substantial numbers of false reports.
Butler reports that Devin and Cursor each had a true positive rate below 10% for bug finding. Here, SM-100 uses true positive rate to mean valid reports divided by all reported bugs—not the fraction of all existing bugs detected—and specifies pass@1 evaluation. There is also a separate discovery problem: when the team plants bugs in larger codebases, agents struggle to navigate to the relevant code and locate the defect. A tool that generates code faster than it can reliably inspect it can leave an accumulating repair burden.
Butler reports that three of six agents had a true positive rate of 10% or less across more than 900 reports. In one task, an agent returned 70 issues, all of them false. That example makes the practical cost clear: somebody must still investigate the output, and a long list of plausible-sounding issues can consume attention without improving the code.
Butler reports a 97% false positive rate for Cursor over more than 100 repositories and more than 1,200 issues. These are the historical evaluation totals quoted in the talk; the exact agent-specific result and repository scope were not independently confirmed. The consequence he emphasizes is alert fatigue: developers stop investigating reports, trust erodes, and real defects become easier to ship.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ways to improve the search
Butler turns those evaluation experiences into three practical adjustments for working beside an agent in an IDE:
- Bug-focused rules: Use the agent’s rules file to supply scoped instructions about security issues and logical defects.
- Context management: Preserve relationships between pieces of code. Butler observed agents losing logical links to material they had already read as they moved through a repository. This matters because significant bugs often involve deeply nested, multistep behavior.
- Thinking models: Prefer models that spend additional effort reasoning through the problem. Butler reports stronger bug-finding performance from these models and recommends selecting them in tools such as Claude Code and Cursor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Name the bug classes, then require validation
Start by giving the agent a concrete security lens. Butler recommends incorporating the OWASP Top Ten into rules files. This is web-application security awareness guidance, not an exhaustive taxonomy of software defects; the linked 2021 edition provides historical context, although Butler does not specify an edition. The mechanism is priming: supplying security criteria makes those criteria part of what the model considers while reading code. Butler says performance was substantially lower when security or bug-related information was absent.
Next, replace a generic request to find bugs with explicit classes of failure. Butler names authentication bypasses, prototype pollution, and SQL injection. These give the agent a more definite search target than an unrestricted repository review. Finally, require tests before accepting a repair: the agent must write them, get them passing, and establish that the reported defect has actually been fixed. A passing test suite is useful only if it exercises the behavior at issue.
A rules-file entry can express that workflow directly:
markdown
# Bug review
Use the OWASP Top Ten as security review guidance.
Focus this review on:
- Authentication bypasses
- Prototype pollution
- SQL injection
For each proposed fix:
1. Add a concrete test covering the reported defect.
2. Verify that the test exposes the defect before the fix.
3. Apply the fix and get the test passing.
4. Explain how the test verifies the repaired behavior.
Do not accept a fix until its validation tests pass.
Butler grounds this recommendation in work across roughly 100 repositories and thousands of issues from multiple agents. Structured rules replace vague requests with explicit criteria for both discovery and acceptance, which he reports produces higher-quality output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the code relationships in context
A bug search depends on retaining the connections between files. As agents reach their context limits, they may summarize or compact previously read material. Butler reports that their ability to detect and understand bugs drops substantially after this happens. The user therefore has an active role in managing what remains available to the agent.
His context-management procedure has three parts:
- Supply the diff. Show the changed code so the agent can follow cause and effect, rather than reconstructing the change from a final snapshot.
- Preserve key files. Keep the code needed to understand the behavior from being summarized away or removed from the context window.
- Request a component inventory. Have the agent work step by step through the classes, variables, and their uses across the codebase.
The inventory is especially useful because it makes the relationships explicit. Instead of relying on a fading recollection of earlier reads, the agent has an organized account of which components exist and where they are used. Butler reports that agents became more capable of finding bugs after performing this inventory.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deeper reasoning still leaves uneven coverage
Butler’s preference for thinking models comes from both their results and the reasoning traces his team inspected. He describes models exploring several considerations in the code, identifying promising paths, and then investigating those paths more deeply. Across Bismuth’s benchmark, he reports that this process found deeper bugs than non-thinking models did.
That improvement still leaves a coverage problem. Across hundreds of repositories and thousands of issues, Butler observed that the headline number of bugs found could remain similar while the actual bugs identified changed from one run to the next. A stable bug count does not imply stable bug coverage. Two reviews can look equally productive in aggregate while finding different defects.
Butler interprets this variability as evidence that agents do not yet inspect a file comprehensively, even when they use thinking models. His complaint that users should not need to run an agent a hundred times expresses the burden of repeated review, rather than a measured threshold for complete coverage. Consistent, comprehensive discovery remains unresolved. Thinking models are his preferred option because they were more thorough and performed better in the benchmark, but that advantage does not remove the variability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From automated repairs to inspectable results
At the time of the talk, Butler presents Bismuth as a system that automatically creates pull requests, integrates with GitHub, GitLab, Jira, and Linear, scans for vulnerabilities, and provides code reviews. He also advertises on-premises deployment for organizations that need it. These are the product capabilities described in the recording, rather than a statement of current availability.
The closing QR code directs viewers to Bismuth’s site and its benchmark link. Butler names SM-100 and invites viewers to inspect the methodology, results, full dataset, and exploration of agents’ bug-finding and bug-fixing performance. The evaluation repository and dataset explorer provide routes into those materials. The ending returns the question to evidence: which bugs did an agent find, which reports were real, and which repairs actually worked?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains the benchmark's bug-discovery, report-validity, remediation, and pull-request-review metrics.
Bismuth's repository containing benchmark documentation, agent drivers, evaluation scripts, and raw result traces.
Security risk categories and guidance for web applications, including broken access control, injection, and authentication failures.
Further reading
Ian Butler and Nick Gregory's longer 2025 presentation on SM-100's dataset, evaluation design, and agent results.
Updates since the talk
- Bismuth API implementation archiveRepository
Public source archive of Bismuth's main API and agent implementation, marked read-only in October 2025.
Full transcript
Loading transcript…