AI Engineer World's Fair 2025
Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory
Loading slides…
Read the talk
Agents reported thousands of bugs. How many were real?
SM100 separates finding a known defect from producing believable bug reports, exposing the gap between coding ability and reliable software maintenance.
From a talk by Ian Butler and Nick Gregory
Before you start: Familiarity with pull requests, basic agent tool loops, and application state will help; no security-research background is required.
Can agents maintain the software they help write?
Can a software agent reliably find and fix bugs in an existing codebase? Generating a feature is only part of the job. Once software exists, someone must understand its behavior, identify failures, and repair them without breaking something else. That is the maintenance question Ian Butler and Nick Gregory bring to their benchmark work at Bismuth.
Butler introduces himself as Bismuth’s CEO, with a background in data engineering, machine learning, search, and Zillow’s A/B testing platform. He and Gregory previously built a technical-documentation search product in 2019, indexing hundreds of millions of documents for IDE use and applying AI summarization to internal knowledge bases. Gregory, Bismuth’s CTO, brings experience building endpoint-security tools at Google and researching software-exploitation detection—work whose techniques transfer to inspecting code with agents. Baseten supplied credits and compute for the benchmark’s DeepSeek R1 and Llama 4 Maverick runs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where coding evaluations leave gaps
Existing evaluations include HumanEval, SWE-bench, aider’s Polyglot benchmark, and LiveCodeBench. They cover different coding tasks; SWE-bench, in particular, evaluates resolving existing GitHub issues. The useful distinction here is therefore not simply new code versus old code. Repairing a described issue and discovering an unknown defect are different tasks.
The rest of the software development lifecycle introduces responsibilities that a coding score does not automatically measure:
- Planning: Understand business requirements, existing systems and designs, and potential solutions before choosing what to build.
- Development and testing: Implement behavior and check it—the area Gregory describes as relatively well covered by existing evaluations.
- Code review: Inspect a proposed change for problems. Gregory identifies this as a comparatively unbenchmarked area despite growing adoption of LLM review tools.
- Deployment: Write configuration, establish monitoring, and integrate with other systems.
- Maintenance: Fix bugs, upgrade dependencies, and perform migrations within an established codebase.
Bismuth concentrates on code review and maintenance: work that still involves writing code, but starts from different information and demands different judgments.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finding a bug means understanding the system
Feature development and debugging both require an understanding of architecture, connections between components, and data flow. Debugging can demand an even deeper understanding than writing the original feature: the investigator must establish not only what a function does, but which inputs reach it and how its outputs affect the rest of the system.
Butler reports that agents often inspect only part of that behavioral space. A run finds a subset of the bugs; thinking models help somewhat, but still explore a limited number of possibilities. This produces errors in both directions: agents miss defects a human would notice quickly and endorse supposed bugs a human would dismiss. Patching the bugs they do find is usually easier, but the discovered bugs are relatively simple. That success says less about repairing difficult defects than it first appears.
Existing bug-detection benchmarks from software security do not fully address this gap either. Gregory describes evaluations built for static analysis and program repair that emphasize familiar patterns: null-pointer dereferences, buffer overflows, or SQL injection. Some concentrate on a single language, such as Java. Their security focus also leaves out ordinary functional failures. A copy-paste mistake can break an application for its users without creating an authentication bypass. SM100 is designed to broaden that evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What counts as a bug?
Butler describes SM100 as 100 triaged, validated, and classified bugs from over 84 public repositories. All had already been remediated in their projects. The selection spans obvious defects requiring little domain knowledge through problems demanding substantial project expertise at a senior or staff engineering level. Python, TypeScript, and JavaScript were selected for their popularity and presumed model strength; Go provides a systems-language comparison.
The inclusion rule is an objective defect: an explicit security or logic problem with a concrete failure consequence, such as data loss or a crash. Context determines whether that consequence can actually occur. A function might lack a bounds check, yet every caller might guarantee that its input stays within bounds. Reporting the missing check alone does not establish a reachable system failure. Feature requests, optimization opportunities, style, formatting, and design preferences are excluded to avoid turning the evaluation into a debate about taste.
Each bug is annotated along several separate dimensions:
- Severity and context: How serious the defect is, where the relevant code is defined, and where it is called.
- Required knowledge: How much project-specific understanding a human would need to identify it.
- Discovery difficulty: How difficult it remains to find even for someone with that understanding.
- Consequence: Whether the failure causes data loss, a crash, a security problem, or another concrete outcome.
Separating knowledge from difficulty matters. An expert can understand a subsystem well and still need considerable investigation to locate a defect within it.
These classifications aim to measure repeatable capability rather than exceptional success. Butler invokes a reported o3 zero-day discovery and an experiment involving roughly 100 runs over the same context. The useful distinction is between demonstrating that a model can discover a vulnerability and establishing how reliably it does so in ordinary use; the run budget alone does not establish that discovery required every attempt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four outputs, with a bounded search
SM100 separates four questions that would be obscured by a single coding score:
| Evaluation | Question |
|---|---|
| Needle discovery | Can the agent find the target defect without being told what it is? |
| Report validity | How many of its reported bugs survive manual assessment? |
| Introduction-time review | Can it spot the defect in the PR or commit introducing it? |
| Remediation | Can it fix a discovered bug without breaking the rest of the codebase? |
The distinction between discovery and report validity is essential. An agent can find the target and still burden a developer with many incorrect reports. Review changes the starting conditions: the agent receives the introducing change, reducing the search problem and supplying immediate context. Remediation is assessed only after a bug has been discovered.
Searching an entire large repository for one target would be prohibitively slow, but pointing directly at the defective lines would make discovery too easy. The benchmark narrows the search through subsystems:
- Divide the repository into groups of related files, such as a frontend area or a particular API subsystem.
- Identify the subsystems containing files modified by the known corrective PR or commit.
- Supply the files from those subsystems to the agent without describing the target bug.
- Have the agent inspect the selected subsystem as a whole, rather than only the changed lines.
This withholds the defect description while keeping exploration tractable. It is still a scoped discovery task: the known fix influences which subsystem the agent receives, even though it does not explicitly reveal the defect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finding the target is not the same as producing useful reports
A basic agent is easy to assemble: put a model in a loop with shell access, code editing, a thinking tool, bug reporting, and a finish action. It can navigate files and report plausible problems. Butler says the basic loop found five or six bugs while producing a 97% false-positive rate. A useful detector therefore depends on more than tool access: model choice, prompting, information delivery, system design, and navigation strategy all contribute.
In the reported SM100 discovery evaluation, Bismuth found 10 of 100 target bugs; the next-leading system found seven. Even the leading result leaves most targets undiscovered, making this an evaluation with substantial room for improvement.
The presentation calls the share of valid reports the true positive rate. Here that means report precision—valid bug reports divided by all bug reports—not recall over all defects in the codebase. Butler reports the following historical results:
| Agent | Valid reports / all reports |
|---|---|
| Claude Code | 16% |
| Bismuth | 25% |
| Codex | 45% |
These systems also produced fewer irrelevant findings, which Butler attributes to tighter scoping. Devin, Cursor Agent, and Cosine reportedly produced roughly 900–1,200 findings, with Cursor reaching about 1,300, while only 3–10% of their reports were valid. Producing a larger list is not equivalent to finding more useful bugs.
PR review improves the starting position by supplying the introducing change. Butler gives Codex a corrected figure of 27%, then mentions Devin and Bismuth; their exact figures are redacted in the retained recording text. The denominator remains unresolved: the published protocol specifies 80 eligible PR-review problems, and the historical result table is unavailable. Codex’s reported figure should not be converted into a count or recalculated percentage. The comparison also excludes false-positive accounting; it asks only whether the review found the known target.
Bismuth is model-agnostic, although Butler says it typically uses Anthropic models because they have been easier to serve to customers through Vertex. He reports that Bismuth outperformed Claude Code in multiple categories while using Anthropic’s underlying models. The surrounding agent system can therefore materially affect the result even within the same model family.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What changes when the model runs in a basic loop?
The basic-loop comparison strips the system back to the simple tool loop described earlier. DeepSeek R1 and Llama 4 Maverick generated hundreds of findings but had very low report precision. Sonnet 4 and o3 did better, though their findings remained noisy:
| Model in basic loop | Target bugs found | Valid reports / all reports |
|---|---|---|
| DeepSeek R1 | 1 | 1% |
| Llama 4 Maverick | Not stated | 2% |
| Sonnet 4 | 6 | 3% |
| o3 | 2 | 6% |
The last column uses the speaker’s corrected wording: these are true-positive shares, not false-positive rates. Model choice changes both target discovery and the usefulness of the resulting report list.
Butler emphasizes that the best popular agent outside the newly launching Bismuth reached only 7% on target discovery. He treats this as a warning about maintaining complex software, describing maintenance as the back 90% of engineering work. That is his framing of the workload, rather than a measured allocation of engineering time. The quantitative results presented here establish difficulty discovering defects; they do not provide a separate ranking of complex-bug repair success.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Seventy reports, and a missed state transition
Report quality becomes a practical limit when a human must inspect the output. Butler reports that three of six agents had valid-report shares of 10% or less, and one agent produced 70 reports for a single issue. Even if the correct finding appears somewhere in that list, the developer must still separate it from dozens of distractions.
The concrete example is an ordinary form-state defect. Only Bismuth and Codex found the illustrated issue. After submission, isDirty was never set to false; Butler describes the form as failing to clear and continuing to indicate that it contained filled-out state. This is not a spectacular security failure, but it directly affects the user’s experience of whether submission completed correctly.
A compact TypeScript example isolates the missing transition. Here isDirty represents unsaved edits; a successful save should clear that flag, while a failed save should leave it unchanged:
typescript
type FormState = {
values: { message: string };
isDirty: boolean;
};
async function submitForm(
state: FormState,
save: (values: FormState["values"]) => Promise<void>,
): Promise<void> {
await save({ ...state.values });
state.isDirty = false;
}
For a form containing message: "Hello" with isDirty: true, successful submission changes the flag to false. Omitting the final assignment leaves it true. The example preserves the field value to isolate the dirty-state transition; clearing input values is a separate UI operation. This is the kind of small state inconsistency an agent must follow through the surrounding behavior to recognize.
The dirty flag after a successful submission
Constructed example: The message field, Hello value, TypeScript function, and successful-save operation are teaching details. Preserving the field value isolates the source-grounded dirty-state transition; this is an expected repaired outcome, not an executed test.
{"values":{"message":"Hello"},"isDirty":true}Operation: Apply the illustrated submitForm repair: await a successful save, then assign isDirty = false.
values.message
"Hello"
"Hello"
isDirty
true
false
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Broader search, deeper reasoning
Across the tested agents, Butler sees a common limitation: reasoning is narrow, including with thinking models, and often does not go deeply enough even along the paths it selects. The proposed improvement has two parts—consider more possible failure paths, then investigate selected paths more thoroughly.
Repeated runs expose the problem in another way. The total number of reported bugs stays roughly consistent, but the identities of those bugs change. That pattern suggests the agent is not building a comprehensive inventory of a file’s behavior on each run. Butler hypothesizes that context biases the model toward one interpretation or avenue of investigation at the expense of others. The team mentions possible remedies but does not specify an algorithm in this presentation.
Butler contrasts SM100’s weak results with SWE-bench scores he cites in the 60–80% range, without identifying the systems, benchmark variants, or evaluation dates behind that range. The distinction is the task: strong performance resolving a supplied issue does not establish reliable discovery of an unknown failure in deployed software. Maintenance demands targeted search, program comprehension, cross-file reasoning, and recognition of bug patterns within the actual system.
There are encouraging signs in the more focused findings from Codex and Bismuth, including occasional complex issues and the form-state defect. But these are early capabilities. A maintenance agent must connect a suspicious local pattern to a reachable failure, and then report it precisely enough that a developer can act on it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Making maintenance progress measurable
Gregory closes with a warning that widely used agents can introduce bugs, alongside cautious optimism about newer systems’ ability to reason through code and use context effectively. The presentation does not separately measure bug-introduction rates, but its discovery results make the maintenance challenge concrete: code generation alone does not supply dependable ongoing inspection.
SM100 gives teams a way to track whether different techniques improve that inspection over time. Better discovery, more accurate reports, and dependable repairs would benefit developers across the lifecycle, not just benchmark standings. Gregory points viewers to the benchmark’s numbers and details and names bismuth.sh as the company’s site. The next advance must make the reports more useful to the engineer who has to decide what to fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Definitions of bug discovery, report validity, remediation, and PR-review metrics, including the single-run evaluation protocol.
The benchmark for resolving real-world GitHub issues, useful context for distinguishing issue-guided repair from bug discovery.
Coding benchmark results with detailed model configurations, dated runs, and pass results.
Code and instructions for evaluating generated programs through functional correctness and pass@k.
Official project hub linking the coding benchmark’s paper, implementation, dataset, and leaderboard.
Full transcript
Loading transcript…