AI Engineer World's Fair 2026
Using LLMs to Secure Source Code
Loading slides…
Read the talk
Using LLMs to Secure Source Code
A security agent needs more than source code: it needs system context, a safe testbed, independent verification, and a path from confirmed vulnerabilities to reviewed patches.
From a talk by Eugene Yan
Before you start: Basic familiarity with HTTP APIs, Python, SQL, and containers will help you follow the order-service example; no security-engineering background is required.
What happens when finding vulnerabilities gets easier?
If a model can find vulnerabilities faster than engineers can fix them, what should a security team build next? Eugene Yan introduces this problem from his work as Anthropic technical staff, after several months helping security teams use Claude to find and repair vulnerabilities in code and systems. The lessons span dozens of organizations: improving discovery matters, but so does preparing the work that follows it.
Task horizons offer one way to understand the change in capability. Like the METR evaluations of knowledge work, the UK AI Security Institute’s cybersecurity evaluation expresses task difficulty in terms of how long a human would take. Its tasks include identifying and exploiting weaknesses, reverse engineering, and web exploitation. The matching AISI evaluation measures 80% success with a 2.5-million-token budget per task; most human durations are expert estimates. These are narrow benchmark tasks, rather than a direct measure of success against defended production systems.
The chart shows models completing progressively longer cybersecurity tasks. Yan also points to recent results above the previous regression line: a step in capability, though that alone does not establish a new sustained growth rate. The operational question is what that additional capability produces—and who handles the output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From more findings to more unfinished work
Firefox makes the change tangible. Yan reports roughly 20 security fixes per month in 2025, roughly 60 and 70 in February and March, and roughly 400 in April—about 20 times his rounded prior-year baseline. Mozilla’s account gives 423 security bugs fixed across April releases, including 271 Mythos Preview findings fixed in Firefox 150. Those are bug counts, not CVE counts; additional Mythos findings also shipped in other releases. The talk describes the April jump as sevenfold, but its rounded monthly figures do not establish that comparison precisely.
The stakes are familiar from shared infrastructure. With Log4Shell, an attacker-controlled string passed to a Java logging library could lead to code execution on the system logging it. Yan cites a Belgian Defense Ministry breach within days and an unnamed fintech incident affecting two million users. Heartbleed, in OpenSSL, supplies the earlier example: a flaw in a widely used dependency can spread risk across many otherwise unrelated systems.
Anthropic’s initial Glasswing update, dated May 22, 2026, shows the downstream workload. Yan rounds the snapshot to more than 1,000 open-source repositories, 23,000 candidate findings, 6,200 initially model-rated high or critical, 1,600 disclosures to maintainers, and about 100 upstream patches. The severity label is provisional: in the published assessed subset, 1,094 of 1,752 candidates retained high or critical severity. Candidates, verified severity, disclosures, and patches are different stages of work.
Discovery is becoming easier; verification, triage, and patching now constrain the useful output. A scanner’s queue is not a list of repaired systems. The harness must carry findings far enough that maintainers can act on them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the setup once, then run the loop
Mozilla’s earlier experience, before its Mythos Preview work, explains why the agentic harness matters. Initial experiments showed promise, but false positives made them impractical to scale. A model equipped to investigate findings can distinguish real bugs from speculation that does not reproduce. The model and its surrounding tools do the work together.
Teams tend to converge on six steps. The first two are investments in a particular codebase; the remaining four form the operating loop.
| Phase | Step | Purpose |
|---|---|---|
| Setup | Threat model | Define the risks that matter |
| Setup | Sandbox | Isolate and reproduce execution |
| Loop | Discovery | Find candidate vulnerabilities |
| Loop | Verification | Establish exploitability |
| Loop | Triage | Prioritize engineering attention |
| Loop | Patching | Repair and validate |
The sandbox lets an agent deliberately execute a proof of concept, or PoC, to test whether a suspected exploit works. Triage then prevents a different failure: handing a developer a hundred findings when the useful assignment is the ten or twenty most consequential high and critical issues.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the model the system, not just the code
The running example is a hypothetical order service: submit an ID, retrieve an order. Before asking an agent to find vulnerabilities, define which failures matter. Yan reports teams reaching a 90% true-positive rate with well-documented threat models; the companion guide grounds the 90% figure in one team’s experience, not a controlled comparison. He suggests above 75% as a practical target. These figures concern the usefulness of reported findings, not the fraction of all vulnerabilities discovered.
The underlying gap is between code context and system context. A model can read a handler and understand its control flow. It cannot recover an unwritten design decision, an on-call engineer’s history of fixes, or the reason a service was deployed behind a particular boundary. That knowledge has to become a document the agent can use.
A threat model can be built collaboratively:
- Bootstrap from existing evidence. Give the model documentation, source code, comments, historical patches, and the CVEs those patches addressed. Ask it to identify potentially related issues that remain unpatched.
- Interview system experts. Have the model ask what could happen outside the intended design and which apparent risks already have compensating controls.
- Record the deployment assumptions. VPN-only access or a control elsewhere in the system may change the interpretation of a finding, even when neither appears in the repository.
The model can organize this material, but people must supply the knowledge that was never written down.
For the order service, the key asset is customer personally identifiable information, or PII, and the entry point is the orders API. The initial threat model identifies SQL injection and calls without authentication as possible threat vectors. This small document gives discovery a concrete target: protect sensitive order data across the API’s trust boundary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make exploit testing isolated and reproducible
The sandbox has two jobs: isolation and reproducibility. An agent may run untrusted code or deliberately construct an exploit. Yan recommends a VM without egress and without cloud credentials, limiting the opportunity to exfiltrate data or damage production systems. The ability to execute code is useful only if its effects stay inside the intended test environment.
Reproducibility requires a consistent baseline. A finding may depend on an old dependency rather than the version actually deployed. Starting agents from the same baseline container makes that distinction inspectable and gives the pipeline a repeatable environment. Representative testbeds take work to build, but one team identified live-system sandboxes, where agents could execute PoCs and confirm true positives, as its biggest lever.
The order-service testbed contains three Docker images: the application, a Postgres database, and a Redis cache. The security agent sits outside the target boundary and probes the application over HTTP. That arrangement preserves the distinction between the service under test and the actor attempting to exploit it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover with context, simpler prompts, and tools
With the setup in place, discovery depends on three ingredients. First comes context engineering: give the agent the documented system knowledge that improves coverage and helps rule out false positives. Second comes simpler prompting. Yan says he often cuts prompt size by roughly 50% after a model capability jump, removing increasingly unnecessary prescriptions. Instead of enumerating vulnerability classes, a stronger model can work from an instruction such as “Look for where untrusted data hits the trust boundary.”
Third, give the agent the tools a security engineer would use: API requests, response inspection, logs, and source-code access. Reading code alone cannot resolve every question about runtime behavior. Yan reports that one penetration-testing team reached an almost 100% true-positive rate after enabling tools and verification in the loop. No sample size or evaluation protocol is supplied for that team’s result.
The order-service demonstration reduces discovery to a five-line Python GET handler. Its fourth line constructs SQL using string interpolation, allowing the caller’s input to flow into the query itself. The vulnerable pattern can be expressed as:
python
def get_order(cursor, order_id: str):
query = f"SELECT * FROM orders WHERE id = '{order_id}'"
cursor.execute(query)
return cursor.fetchall()
The problem is the boundary between SQL syntax and data. The database receives one assembled string, with no separate indication that the supplied ID must remain a value. An attacker can alter the query’s meaning and potentially retrieve data beyond the intended order. This is deliberately ordinary code—the kind a non-security specialist might write to get a lookup working—and current models are good at spotting it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let a separate agent try to disprove the finding
Discovery and verification have different objectives:
| Stage | Optimize for | Question |
|---|---|---|
| Discovery | Recall | What might be vulnerable? |
| Verification | Precision | Which findings are exploitable? |
Tool-assisted investigation can improve discovery, but forcing the discovery agent to debate every candidate with itself can cause self-censorship and reduce recall. Separating the stages lets discovery search broadly without making every suspicion pass its own final evidentiary test.
The verifier should be independent and adversarial. It receives the candidate vulnerability without the discovery agent’s reasoning traces or work history. It starts from the assumption that the report may be false and tries to establish whether it actually works. In a representative sandbox, that means building a PoC and executing it, rather than merely producing another persuasive explanation of the same code.
For the order service, the verifier starts in a fresh container and receives the location of the suspected SQL injection: line four. It then sends a request using curl. In Yan’s example, the response exposes customer PII, confirming that the query can be exploited. The important output is reproducible evidence that crosses the intended data boundary, not agreement between two agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A real vulnerability still needs business context
A confirmed finding is not automatically the next thing a product engineer should fix. Some findings are correctness issues; others have little business impact. Teams can lose engineers’ trust by forwarding every true finding, including a volume of medium and low severity work they cannot absorb. Engineering attention is the scarce resource.
Triage first removes duplicates, using inexpensive heuristics or model-assisted clustering. It then considers two separate dimensions:
- Impact: What happens if the exploit succeeds? Yan uses ring-zero versus ring-one consequences as an example of differing magnitude. Data sensitivity matters too: customer PII or healthcare records can make an otherwise moderate finding more consequential.
- Likelihood: How many barriers must an attacker cross? Deployment boundaries and compensating controls can reduce exposure, even when the underlying defect remains.
This is why the threat model remains useful after discovery. A firewall can lower the assessed severity; sensitive database contents can raise it.
In the order-service example, the agent initially assigns high impact and high likelihood. Human review keeps the impact assessment but lowers likelihood: an application firewall blocks SQL injection, and the service is used only for internal warehousing or logistics, never exposed to the internet. The review changes the risk assessment; it does not patch the vulnerable query. Those deployment facts may live in a system design, documentation, or Confluence rather than the source code. Writing them into the context lets the next agent make a better-informed assessment.
The same finding after deployment context is added
Constructed example: The comparison represents the absence of controls from the initial assessment as null; it does not imply the controls were newly deployed.
SQL injection in the order-service orders API
Operation: Human review adds deployment controls to the agent's risk assessment; no code patch occurs in this step.
Finding
SQL injection in the orders API
SQL injection in the orders API
Impact
High
High
Likelihood
High
Low
Application firewall
Not present
Blocks SQL injection
Deployment context
Not present
Internal warehousing/logistics; not internet-facing
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validate the patch and preserve what you learned
Patch validation follows a ladder:
- Replay the original PoC. The exploit must stop working.
- Run the existing tests. The repair must preserve legitimate behavior.
- Attack the patched code again. A fresh discovery agent looks for incomplete fixes or another route to the same failure.
- Require human confirmation before merge. The agent proposes and revises; a person approves the repair.
The repeated generation and verification give the patching agent concrete feedback. A failed exploit alone is insufficient if the patch also breaks the service, and passing existing tests alone is insufficient if a nearby exploit still works.
The order-service fix separates the parameter from SQL construction. Using a Postgres driver with %s parameter binding, the earlier Python example becomes:
python
def get_order(cursor, order_id: str):
cursor.execute(
"SELECT * FROM orders WHERE id = %s",
(order_id,),
)
return cursor.fetchall()
The query structure stays fixed while the driver handles order_id as a value. This expresses the mechanism of the one-line repair Yan describes: remove the user-supplied variable from Python’s SQL string interpolation.
The second diff updates the system’s knowledge, not just its implementation. It records the application firewall and internal-only deployment so later scans do not repeat the same mistaken assumptions. Yan frames this as turning repeated operational expense into an accumulating investment: each pass should leave behind better code, better context, or both.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale the decisions around the harness
The technical workflow resembles an ML pipeline: context and reproducibility provide the foundation, followed by recall, precision, ranking, and feedback. But increasing its input by one or two orders of magnitude exposes constraints that additional compute cannot resolve. More scanning can be purchased; more human attention is harder to obtain. Product and security engineers need written agreement on high and critical severity, and experts need to externalize the threat models held in their heads. Yan says he knows few companies using fully automated security patch review.
Three operating changes help the organization absorb the output:
- Route by ownership. Curated emails and manually assigned Jira tickets may work for a dozen findings a month, but not hundreds. Route directly to the code owner or service owner; this step does not need an LLM.
- Calibrate severity together. Product engineers, security engineers, red teams, and blue teams need a shared rubric grounded in the business. Discuss disagreements, document the rules, and give those rules to the agents.
- Supply patches, not just reports. Even with coding assistance, a bare vulnerability leaves substantial work for the recipient. Move toward AI-generated patches, attack them again, and retain human verification.
These changes connect a technically valid finding to an owner who understands its priority and can review a proposed repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start interactively before automating
Begin with open-source dependencies or another scope you understand and feel comfortable investigating. Work interactively in Claude Code or your preferred IDE before trying to automate the whole process. That hands-on pass reveals which context is missing, where precision falls short, and what the harness needs to make its output useful. Budget for verification, triage, patching, and the organizational work around them—not only the scan.
Yan points to Claude Security for scanning codebases, validating findings, and suggesting patches, alongside a companion guide and open-source materials with interactive skills and autonomous harnesses. The Defending Code Reference Harness provides a concrete starting point. The talk’s quick-start walkthrough places a customizable harness at step five; the current repository organizes customization under Step 3 and detection and response under Step 5. Follow its current instructions, then adapt the harness using what the interactive work taught you about your own system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Eugene Yan and Henna Dattani explain threat modeling, sandboxing, discovery, verification, triage and patching.
Mozilla details its vulnerability discovery pipeline, April 2026 fixes and the distinction between security bugs and rollup CVEs.
Anthropic's May 2026 snapshot separates candidate findings, assessed vulnerabilities, disclosures and patches.
The historical launch announcement describes code scanning, finding validation and proposed fixes for Enterprise customers.
Further reading
Interactive security skills and an autonomous reference pipeline with sandboxing, independent verification and patch validation.
AISI explains cybersecurity task horizons, token budgets, human baselines and uncertainty in capability trends.
Full transcript
Loading transcript…