# How Anthropic Merges 30,000 Code Changes a Day - Manish Kapur, Sonar & Addy Osmani, Anthropic

Manish Kapur, Sonar · Addy Osmani, Anthropic

[Talk page](https://ai.engineer/talks/AnsXWH2_EBU-anthropic-merges-30-000-code-changes-day) · [Video](https://www.youtube.com/watch?v=AnsXWH2_EBU)

## How Anthropic Merges 30,000 Code Changes a Day

Manish Kapur and Addy Osmani explain how agent review, browser evidence, layered checks and human ownership support software delivery when generation outpaces manual review.

![Recording frame at 18:17: How Anthropic Merges 30,000 Code Changes a Day](https://ai.engineer/api/talk-essay-media/2b0fe07a6d262f0b8493a679b161cbccf7e738090aa3e14d2f4425aeb02601cd)

How Anthropic Merges 30,000 Code Changes a Day — recording at 18:17

### Generation outpaces line-by-line review

[0:12](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=12s)

Almost everyone raises a hand when Sonar’s Manish Kapur asks whose coding agent opened a pull request in the past week. Far fewer keep their hands up when he asks who reviewed every line before merging. That gap sets up his conversation with Addy Osmani, who works on the developer experience for **Claude Code** at Anthropic: how can verification keep pace with software that agents produce?

Kapur introduces the delivery problem with two figures he attributes to CircleCI’s analysis of 28 million CI workflows: throughput rose about 59%, while three in ten attempts to merge code into production failed. What does faster output buy if changes still cannot get through delivery? The comparison separates the volume of work moving through a system from its ability to accept that work successfully.

![Recording frame at 2:38: Generation outpaces line-by-line review](https://ai.engineer/api/talk-essay-media/72c68374d97a344a0956502710d074bd91fe7f1d322edd33589a006d5255d850)

Generation outpaces line-by-line review — recording at 2:38

Osmani’s phrase for a software factory is “loops at scale.” It shifts attention from an individual agent writing an implementation to the repeated work around it: planning, building, checking and shipping. Scaling that process requires deciding where people still exercise judgment and what evidence lets automation proceed.

### Light and dark factories move human judgment

[2:55](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=175s)

The factory metaphor describes a spectrum of human involvement:

- **Lights-on factory:** Agents repeatedly help with planning, building, checking and shipping. People apply judgment and taste within that process.
- **Dark factory:** Agents scope, build and check work with much less intervention. People concentrate on intent and enter execution when exceptional cases require them.

The question is how much of the process a team can trust to run without continuous human attention.

![Recording frame at 4:07: Light and dark factories move human judgment](https://ai.engineer/api/talk-essay-media/d745589aa79e21cd742f42b7e7b426104295b8e0bd71ae79e4e5db590b6a9858)

Light and dark factories move human judgment — recording at 4:07

Kapur provocatively calls AI coding a solved problem, then identifies the harder constraint. Sonar’s survey slide reports that 76% of developers generate AI code faster than they can verify it. It also reports that 85% of organizations already use coding agents to write code.

Confidence needs something the agent can check. Tests, benchmarks and specifications give it criteria for judging whether its implementation is correct and complete. Agents can help create those checks more cheaply, but teams still need to refine them. A task with a usable verification method lets the agent inspect its own result and correct failures before handing the work back.

### Anthropic’s first pass produces evidence for people

[5:27](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=327s)

Kapur brings the scale into focus: Anthropic has grown from roughly 2,500 merged code changes a day to 30,000. Osmani places the earlier figure in May. He also describes a recent refresh of claude.ai involving about 3,000 pull requests from engineers, product designers and other functions, with a reported threefold performance improvement. These are distinct quantities: daily merged changes across Anthropic and pull requests for a particular refresh. The performance claim does not specify a metric here.

**Claude** performs a first-pass review, with tools that let it try the result like a user. For a browser experience, that can include opening the application, exercising it through end-to-end tests, running linting and using other necessary checks. Review therefore includes observing behavior, rather than relying solely on a plausible reading of the diff.

When review exposes a bug or missing functionality, edits trigger hooks that study the diffs and support rapid correction of the PR. Osmani describes Anthropic using its own code-review tooling to automate as much of this work as possible. The useful feedback is tied to changed code: an edit becomes another opportunity to inspect whether the correction closes the gap.

Team judgment also needs a form the reviewer can consult. Shared files such as `CLAUDE.md` and `AGENTS.md` can record best practices and the team’s expectations. Those instructions supply review criteria; people remain responsible for checking the agent’s evidence. For a UI change, Osmani wants browser testing and perhaps a video showing the agent clicking through the experience. If mobile and desktop both matter, screenshots or videos should cover those experiences.

A browser recording lets a reviewer see which interaction Claude tried and how the application responded. Linting supplies a different signal. The team still decides whether the evidence covers the intended experience.

Review criteria enter; verification evidence returns
Shared instructions guide agent review, while browser and tool checks produce evidence that people inspect.
- Team standards — CLAUDE.md and AGENTS.md encode best practices and taste.
- Claude first-pass review — Consults team guidance and tries the experience with available tools.
- Browser and tool checks — End-to-end tests, browser interaction, linting and other necessary checks.
- Verification evidence — For UI work, screenshots or video can show interactions and mobile or desktop behavior.
- Human inspection — People check the agent’s evidence against the intended experience.
- Team standards → Claude first-pass review (Supplies review criteria)
- Claude first-pass review → Browser and tool checks (Uses tools to try and check the result)
- Browser and tool checks → Verification evidence (Records exercised UI behavior)
- Verification evidence → Human inspection (Provides material to inspect)

### “Verify the work” needs a concrete meaning

[8:21](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=501s)

In Sonar’s forthcoming State of Code survey of more than 1,000 developers, 84% say AI gets features and fixes into production faster. The same slide cites Faros AI for two comparisons: 51% larger pull requests and 31% more code reaching production without review. The slide gives no baseline or comparison period for either figure. Kapur sees unreviewed code as a particular concern for regulated or mission-critical applications.

![Recording frame at 10:12: “Verify the work” needs a concrete meaning](https://ai.engineer/api/talk-essay-media/445ffa03269e326a18bcf90c237f792a89b263e2a6c9ae24bd61b4e70196c1b7)

“Verify the work” needs a concrete meaning — recording at 10:12

Osmani expects further gains from both model quality and better verification practices. But “verify the work” leaves the implementer with another question: which checks actually establish confidence? He names end-to-end tests, property tests and visual verification tests as checks Anthropic relies on. They offer different ways to examine behavior, and the choice matters when the goal is eventually to merge changes automatically.

Sonar’s chart puts the share of organizations running software factories with agents working autonomously at 5% today. It projects 34% in a year, with a corresponding projection of 39% for financial services. Someone in the room asks what counts as a software factory. Osmani returns to repeated loops and the degree of human involvement. Kapur describes an assembly line beginning with intent and specifications, then automating development stages at scale.

The path toward a darker factory depends on the agent generating verification that earns confidence. Human involvement can shrink as that evidence becomes dependable. Most teams in this discussion still occupy the spectrum between assisted execution and autonomous operation.

### Different verification layers catch different failures

[11:30](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=690s)

Sonar’s proposed approach puts verification inside each kind of recurring work: the agent’s development loop, the CI verification loop and ongoing code maintenance. These are places to apply checks throughout the system, rather than a requirement to defer all judgment until a final review.

![Recording frame at 12:55: Different verification layers catch different failures](https://ai.engineer/api/talk-essay-media/4d7dfda9619e81781dfebced16b1ae2ca554444f7ae7bc16cea53d2f79bab122)

Different verification layers catch different failures — recording at 12:55

Kapur divides the checks by the failures they target:

- **Deterministic analysis:** Checks for bugs and security issues, including SQL injection and buffer overflows.
- **Agentic review:** Examines logic and intent issues that deterministic analysis is less suited to identify.
- **Software supply-chain security:** Looks for known vulnerabilities, malware and malicious packages.

The layers complement one another because each examines a different class of problem. A favorable agent review does not settle whether a dependency is malicious, just as a deterministic check does not establish that an implementation follows the intended product behavior.

### Know what agents can reach and what they actually do

[13:19](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=799s)

Verification of code does not answer every operational question. Osmani describes companies that allowed teams to experiment with agents, then discovered they had hundreds or thousands of them without a clear view of their activity. Those agents could touch systems, access data and potentially deploy to production. Agent identity and telemetry become necessary for understanding which agent acted and what happened.

Three concerns follow: control, permissions and the sandbox. Permissions determine what an agent may access or do; the sandbox must keep execution within its intended bounds. Kapur calls Sonar’s approach “guide rails”: directing agents along an allowed path and preventing them from going out of bounds. These controls constrain actions while review and tests examine the resulting software.

### Label the author; keep a human owner

[14:47](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=887s)

The lightning round begins with a request for one-sentence answers. “Can these be long sentences?” Osmani asks. His forecast for most teams by the end of 2027 is similarly qualified: “light factories with some dark corners.” He expects pockets of autonomy rather than everyone operating an entirely dark factory.

Both participants favor labeling agent-written PRs. For Osmani, labeling Claude as the author is compatible with caring about the outcome: whether the shipped code and features are right for users. The label can also help identify additional constraints or checks. Kapur favors keeping labels for at least the foreseeable six to nine months while confidence develops, and points to auditability for financial customers.

Authorship and ownership remain separate records. Anthropic still labels who owns a change. The person who starts a Claude run stays accountable for the result and must fix it if something goes wrong. Naming the agent explains where the implementation came from; naming the owner tells the organization who will answer for it.

The skill to build depends on the role:

- **Engineering leaders:** Identify the gap between current testing and the evidence that would make the team comfortable letting an agent sign off on a merge.
- **Junior developers:** Gain experience with different kinds of tests, including end-to-end and property tests, so verification becomes something concrete they can design and assess.

When is an agent-opened PR safe to merge? Osmani’s answer is confidence that things will not go wrong, or that the risks have been mitigated as much as possible. That is a judgment about the work and its consequences, rather than a universal test-count threshold. It brings the discussion back to the evidence people need before they accept a change.

Kapur closes with “Fast to write is not fast to ship.” Teams need to look at what actually gets merged, verify the output and keep agents within their sandbox before changes reach production. His prescription is independent automated verification inside every factory loop. Autonomy is earned by making the output trustworthy enough to operate with less intervention.

## Key takeaways

- Anthropic’s review workflow starts with Claude using tests, browser interaction and other tools; people inspect the resulting verification evidence.
- Team instructions supply review criteria. Deterministic analysis, agentic review and supply-chain checks address different failure classes.
- Agent authorship does not transfer ownership: a named person remains responsible for the change and for fixing failures.
- Greater autonomy depends on trustworthy verification inside each loop, supported by controls, permissions and sandbox limits.

## Resources

- [Software Factories, Light and Dark](https://addyosmani.com/blog/software-factories/) — Osmani’s factory framing provides context for the discussion of repeated software work and degrees of human involvement.
- [Human judgment doesn't leave the software factory. It relocates.](https://addyosmani.com/blog/human-judgment-doesnt-leave-the-software/) — Develops the practical tradeoffs behind verification budgets, human handoffs and granting autonomy as checks earn trust.
- [Agent Skills](https://github.com/addyosmani/agent-skills) — Reusable engineering workflows offer concrete examples of encoding review standards, testing practices and quality gates for coding agents.

## Related talks

- [In the Land of AI Agents, the Verifiers Are King](https://ai.engineer/talks/VrpEyglYgeU-in-land-ai-agents-verifiers-are-king) — Expands Sonar’s layered verification approach and its placement throughout the development lifecycle.
- [Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, Sonar](https://ai.engineer/talks/7b9tov5Uc5s-rebuilding-ci-ai-scale-commit-volume-gautam) — Explains the CI capacity, integration and approval problems that arise as agent-generated changes multiply.

## Slides

### Who verifies the software factory?

[0:12](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=12s) [The opening slide identifies Addy Osmani of Anthropic and Manish Kapur of Sonar.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f53e6643bd887a240a2c8b42eb590f2f1eac45bf1c9407d95ea68de870d32c9.webp)

Who verifies the software factory?

### Checking code becomes the constraint

[1:57](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=117s) [Two callouts compare growth in CI throughput with the proportion of attempted production merges that fail.](https://assets.ai.engineer/sites/aiecode2025/sha256/05db1a99623fd99c2c9eb7ea19e2dcfb327093fcb3b0ee27464e18d23e066edb.webp)

+59% CI throughput · About 3 in 10 attempts to merge into production now fail

### Software factories and feedback loops

[2:28](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=148s) [A quotation describes a software factory in terms of the loops it can run at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/64b538d74958ceb5a311e97d98d5c02d7ccf97b73d17ab9d2f5956f1b82e2988.webp)

A software factory is harnessing loops at scale.

### The verification gap

[3:51](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=231s) [Four percentage columns summarize Sonar’s survey on coding agents, generated code and the time spent reviewing it.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f0f79d83d6c0580c59e984266644f07bc6b7b6f83168affb44cae33212b0f5a.webp)

Coding is solved. Verification is not. · 85% · 95% · 76% · 53%

### Writing faster versus shipping faster

[8:42](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=522s) [Verification concerns appear beside three reported measures of speed, pull-request size and code reaching production without review.](https://assets.ai.engineer/sites/aiecode2025/sha256/7580baec2064faa658f110bce79a2ecdd779eb1adc8749a193e20f103ec69d50.webp)

Fast to write is not fast to ship

### How many factories run autonomously?

[9:57](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=597s) [A three-bar chart compares the share today, the share expected in a year and the financial-services estimate for that year.](https://assets.ai.engineer/sites/aiecode2025/sha256/3db7b7b2cb01599b78b63fe3b4ebc4a961609151b68c302f5a715332c3216e57.webp)

Today 5% · In a year 34% · Financial services, in a year 39%

### Two layers of verification

[12:11](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=731s) [The slide places deterministic checks such as static analysis beside agentic checks such as business logic and deep semantics.](https://assets.ai.engineer/sites/aiecode2025/sha256/01945d5b9709c63a62846efee4c1cc2866b498ca3541e613fa3e294d0b1b8e2c.webp)

Deterministic verification · Agentic verification

### Controls for agentic development

[13:17](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=797s) [Two survey metrics compare approved agents with few shared controls and factories with autonomous agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/f885ed971c2f9d82b93aaf15c9eb750ad3335964167f1bae549eccd478116f31.webp)

Most teams have agents. Fewer have guiderails. · 28% · 5%

### Four questions for the lightning round

[14:40](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=880s) [Four cards ask about factories, labels for agent-written code, junior developer skills and safe criteria for merging.](https://assets.ai.engineer/sites/aiecode2025/sha256/aec74272086ac645e8c62bfceef90ff5c43eec04279b883ce64a224937a3d77c.webp)

Lightning round: one sentence each

### Three lessons from the discussion

[17:41](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1061s) [The closing slide pairs speed and autonomy with review, production outcomes and automated verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/df543ec499bcd3e779a7edfc386c7dc27dd5813987c637a46cc224973e24695c.webp)

Writing code is the fastest part · Fast to write is not fast to ship · Autonomy is earned

## Full transcript

[0:12](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=12s) I'm pretty excited to be here and talk about who verifies the software factory. Uh, before we get started, uh, can you give me a quick show of hands, uh, raise your hand if your coding agent has pulled, uh, opened any pull request in the last one week. Almost everyone. Thanks. And keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged.  Oh, I do see a few people. That's excellent. So what I'm going

[0:42](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=42s) to talk about is the gap between the, uh, two, uh, camps, uh, merging requests without reviewing manually and, uh, you know, merging requests, uh, after doing some reviews. So my name is Manish Kapur, and I lead the product and solut-- uh, developer relations, uh, team at Sonar, and I'm excited to be here, uh, with my guest speaker, Addy Osmani from Anthropic. Uh, Addy has been, uh, had been at Google

[1:12](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=72s) for about fourteen years. Most of you probably know him, and he's written lots and lots of content and articles. Uh, I learn a lot from Addy, so I'm, uh, really fortunate to have him here stand next to me and, uh, share his wisdom with all of us here today. Uh, he recently joined Anthropic. And Addy, so congratulations on your, uh, role at-

[1:32](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=92s) Thanks

[1:32](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=92s) ... uh, Anthropic. Would you like to tell the team what you're working on at Anthropic, what your role is?

[1:37](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=97s) Yeah, sure. So I'm responsible for making sure that y'all have a really great developer experience building with Claude Code. And any time that we discover there are opportunities to make it even smoother, we bring that right back into the product. We ship a couple of times every single day, and so we're very interested in trying to make sure that you have the best possible experience as often as possible.

[1:56](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=116s) Cool. Thank you. All right. Uh, s-so let's start with some problem and context setting slide. Uh, there are two numbers on the slide. CircleCI analyzed twenty-eight million CI workflows, uh, this year, and they noticed that the throughput jumped by about fifty-nine percent, uh, but yet three in ten attempts to merge that code into production are failing. And this is a stat from one, uh, independent stat from, uh, CircleCI. Uh, and I think I've seen the

[2:26](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=146s) same trend for most of our customers that I speak to every day. Uh, this is a quote from Addy: "A software factory is harnessing loops at scale." This resonated quite a bit for me. And if you've not read that article, you can look it up. It's p-pretty, uh, uh, educational article and talking about f-future, how the future will look like in terms of soft- uh, software factories. Uh, so Addy, would you like to expand on what you were thinking when you wrote this article about soft loops-

[2:53](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=173s) Yeah

[2:53](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=173s) ... and harnessing the loops at scale?

[2:55](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=175s) Yeah, absolutely. So a lot of folks here have probably heard this term software factories, and there's two types of factories that are at least worth keeping in mind. W-- You know, software factories harnessing loops at scale. A lights-on factory is one where you have agents that are helping you, um, with planning, building, checking, and shipping code again and again, but you as a person still apply your judgment and taste inside the picture. And then we have a dark software factory, which is a little bit closer to how, you know, a car factory with

[3:25](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=205s) robotics might be working, where it's constantly doing the work, so it's scoping, building, checking, and you as the person move down, um, further to thinking about intent and you're pulled in when they're really the greatest of exceptions to the case. But that's sort of what I was thinking. I think a lot of teams are somewhere on this spectrum, um, and are on this journey to figure out, like, how, how do I get further and further into automation in a way that I can trust it?

[3:50](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=230s) Cool. So, uh, AI coding is now basically a solved problem. We all know, uh, agents are there. Eighty-five percent of the organizations are already using coding agents. It'll go to ninety-five percent and very soon to hundred percent. Uh, seventy-five percent of the, uh, dev-- seventy-six percent of the developers that generate AI code, uh, they are saying that it's being, uh, code is being produced at a speed that is impossible for them to verify it. So they are spending more and more time reviewing the code. So what do you think is the most pressing question

[4:20](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=260s) that engineering leaders have right now?

[4:23](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=263s) So I think that teams have gotten better at working with agents over the last year. Um, models, harnesses have also gotten better. And what I see a lot of leaders being interested in is how do we enable you to ship code faster, um, with confidence? And that confidence piece is very important. Agents can now generate ways for you to check your code much more cheaply than ever before. And if you think about the things that enabled us to call good code, you know, high quality, uh, those were things like good

[4:53](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=293s) tests, benchmarks, uh, specs, anything that gives us confidence. So agents are able to help us generate that a little bit more easily these days, a little bit more quickly. We can, of course, refine those things, but I think that those checks are one of the most important things we should be spending our time on these days. Like, if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct. If you've got that in place and y-you can make sure that the code is both correct

[5:23](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=323s) and complete, that can easily two to three by, like, the quality of your outcomes.

[5:27](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=327s) Yep. Yep, absolutely right. Spot on. I think code confidence, uh, we have been running an ad, code confidence is sc-scarce, and you really need to build it up now that the coding problem is solved. So I, uh, actually I, yesterday night, so this might be a surprise question for you. Uh, yesterday night, I read a post, uh, from Anthropic, and I encourage all of you to read it. It talks about the scale at which Anthropic is operating. Uh, they are merging about thirty thousand code changes every single day. They used to do about three thousand or so, twenty-five

[5:57](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=357s) hundred or so code changes just a few months back. They have scaled from 2,500 code changes to 3,000-- uh, 30,000 code changes per day. That is mind-boggling. So Addy, what does code review look like at, uh, Claude Code at, at Anthropic and with the Claude Code team that you're working with?

[6:13](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=373s) Yeah, these are great questions. So we were s- we were merging about 2,500 changes all the way back in May. Um, and even for our recent refresh of claude.ai, we put out a case study saying that we kind of 3x'd the performance of it. That was accomplished by merging about 3,000 pull requests in a very short space of time with engineers, product designers, a number of different functions contributing to it. So our workflow for being able to get verification and code review in place is that we first of all have Claude do a first-pass

[6:43](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=403s) review. What's very important in there is making sure that it has a way to actually try the experience out, similar to a real user. So that can mean end-to-end tests, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence. Um, people usually ask us, "Okay, well, what happens if, you know, you discover a bug or a gap in functionality?" Well, we have hooks that will then go and run on any edits. They will study the diffs and allow

[7:13](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=433s) us to get this loop in place where we're correcting the PR as quickly as possible. So we have, uh, Claude Code doing the review. We have a, um, code review product, uh, that you can also check out that's pretty good, and the idea there is we're trying to automate code review, um, as much as possible. This is something that we use ourselves. Um, and increasingly what's important is for your teams to be thinking about how can you cod-- We talk about taste and judgment so much these days-

[7:39](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=459s) Yes

[7:39](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=459s) ... right? It's important for you to be trying to codify what you consider best practice and taste in your team's shared, like CLAUDE.md, AGENTS.md, anywhere where the agent can then consult during code review. And then people are responsible for checking on the agent's evidence. So how do we know? Some-- If I'm building out, um, a piece of UI that's gonna be, you know, used by a user in the browser, I wanna make sure that it's tested in the browser, maybe generated me a video of that experience where it's been clicking around

[8:09](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=489s) and gives me confidence, well, okay, well, this works. If I care about the mobile or desktop experience, maybe it generates me screenshots or videos of that. But basically you wanna get to a place where the agent's giving you as much verification evidence as possible.

[8:21](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=501s) Cool. Yeah, verification is key. I think you need to verify everything that the, uh, software factories will produce, and that's what we are going to talk about today. Uh, so, uh, another s- uh, these stats, by the way, are from, uh, the State of Code survey report that will be coming out very soon from Sonar. Uh, we surveyed about a thousand-- more than thousand developers, and we found some interesting data which will give us insights as to what's happening in the industry today and where we are headed. So 84% say AI features, uh, uh, gets, uh, gets features and fixes

[8:51](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=531s) p-to production faster. The pull requests are getting larger and more complex. Uh, and, you know, yet, uh, only th- uh, 31% more code is reaching, reaching the production without review. So that's a little, little bit of a scary fact, particularly for, uh, production, uh, for applications in the regulated industries, in the mission critical applications where you absolutely ma- need to make sure, uh, that, you know, uh, you have to verify that code and you're confident in the code that is being shipped. So, uh, what are

[9:21](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=561s) the-- where are the next gains going to come from, Addy?

[9:23](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=563s) I think the next gains are gonna come from a combination of us really nailing verification, right? As well as model quality. But models are already, like if you take a look at Opus 5.5, model quality has already gotten to a place where it's pretty good. Um, and I would say that us being able to share best practices about that verification step is gonna become increasingly more important. Because if you tell people, "Yeah, just like verify the work," you know, their next question is, "What does that even mean?"

[9:47](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=587s) Yeah.

[9:47](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=587s) Right?

[9:48](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=588s) Yeah.

[9:48](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=588s) And so I think that the conversation needs to evolve into specifically what kinds of tests are useful. We lean a lot on end-to-end tests, property tests, any visual verification tests, things that can help us just get that confidence to merge those changes in automatically.

[10:02](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=602s) Cool. You mentioned light factories wanna go dark, and that's the way-- uh, direction we are headed. Uh, our survey results show that, you know, today 5% of the co- companies are using, uh, software factories. It will get to 34%, and surprisingly 39% in the financial services. How many of you are using software factories today? Just curious.

[10:22](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=622s) A handful of you. And how many are planning to use software factories by next year?

[10:27](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=627s) How do you define a software factory?

[10:29](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=629s) Yeah, that's a good question. How do you define a software factory? Uh, would you like to take that question or should-

[10:33](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=633s) Yeah, yeah, sure. So we, we talked about, uh, definition of software factories just a little bit earlier. A software factory is basically loops at scale, and when we think about this, it's all about, like, what end of the spectrum are you at in terms of how much people are still involved in the process? Um, the more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark. And I think that, yeah, most people are still somewhere in that spectrum and are not quite, as this data shows-

[10:59](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=659s) Yeah

[10:59](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=659s) ... at that dark point just yet.

[11:00](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=660s) Yep. Just to add to what Addy said, I think software factory, think of it like a production line where you start defining the, uh, intent and your specifications, then you start writing the code. So it's an assembly line where each of these software development lifecycle, uh, steps happen, and automating them at scale where there's, uh, pre-pretty much in a dark fa- uh, factory there would be no, uh, lights on. It's autonomously working, fully autonomously working. A light factory has, uh, uh, is where people are involved in the process at

[11:30](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=690s) every step or most steps. Uh, whereas a dark factory, the goal would be to turn off the lights and let the factory run independently, uh, twenty-four by seven. Uh, all right. So you talked about verification. Uh, I'll talk a little bit about what we think at Sonar about verification. If you saw our keynote earlier today from our CEO, he talked about, uh, verification should be multi-layered. Uh, we absolutely 100% believe that each company should adopt a zero-trust multi-layered verification approach, and verification should be

[12:00](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=720s) done in every loop. Whether you're operating inside the agentic loop or whether you're working inside the CI verification loop, or whether you're doing code maintenance. In each and every loop, it is important to verify the output of the factories or the agents that, uh, code that the agents are producing. So we look at it in, uh, terms of layers, and, uh, think of it like, you know, the f- this first layer, which is a deterministic layer, which catches, uh, bugs, security issues, things like SQL injection,

[12:30](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=750s) uh, buffer overflow bugs, and all of those are caught. And then there's a second layer where agentic review happens, where agents are used to like, uh, Addy was talking about, agents review, review that code. And in that agentic review they catch logic and intent issues, which the deterministic, uh, layer is not well-suited to catch. So we, we use each layer is targeted at catching a specific types of issues. And then the third layer could be a software supply chain security layer, where you look at all the CVEs, all your, uh,

[13:00](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=780s) malware, uh, uh, detection happens, all your, uh, malicious package detection happens. All of that happens in the third layer. So it's a layered approach where each layer is tuned and optimized to catch a certain type of issue. So, uh, we absolutely recommend doing a multi-layered verification. Oops.

[13:19](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=799s) So there's another stat here. 28%, uh, you know, uh, companies have approved agents and shared controls, and very few shared controls. And, uh, 5% are operating, uh, software factories autonomously. What are your thoughts there, Addy, uh, in terms of-

[13:37](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=817s) So I've talked to so many companies where a couple of months ago, one of their biggest worries was they were allowing their engineering teams to go off and play around with agents, and they got to a place where there were hundreds, thousands of agents in some cases, and they just didn't know what they were doing. Like, what are these agents actually doing? They're touching all of our systems, accessing all kinds of data. Maybe they're deploying to production. And so a lot of enterprises think about concerns like, uh, you know, agent identity and telemetry and being able to track what's actually happening.

[14:05](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=845s) Yeah.

[14:06](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=846s) And I'd say that the three biggest things for a lot of companies are: How do we think about control? How do we think about permissions? And how do we think about the sandbox? Like, how are we making sure that the agent isn't jumping out of the sandbox and doing things that it shouldn't be doing?

[14:17](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=857s) Yeah. Yeah. We, we, uh, at Sonar, we call it guide rails. So instead of guard rails, we're thinking of calling it as guide rails, where we guide the agents to follow the rail tracks, right? Uh, a specific path, uh, to take and prevent them from, uh, going out of bounds or so to say, going out of the sandbox, like, uh, Addy was saying. So it's very, very important to have guide rails for our agent and govern the agents that you're working with. Uh, so let's, uh, we're coming to a close now, uh, as my timer is telling me we just have a few minutes left.

[14:47](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=887s) Let's go get into a lightning round. Uh, one sentence response for each of these questions. As, and you have seen the questions already on the slide, so-

[14:54](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=894s) Can these be long sentences?

[14:56](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=896s) They can be as long as the time permits.

[14:58](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=898s) Perfect.

[14:59](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=899s) Uh, light or dark factory for most teams by end of 2027, yes or no?

[15:03](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=903s) I think we're gonna see a light factories with some dark corners. I don't think we're going to get to everybody in this room using dark factories entirely, but it's gonna be something close to that.

[15:12](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=912s) I would guess, uh, Anthropic would do it this year or next year.

[15:20](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=920s) All right. Second question, should agent-written code be labeled in the PR?

[15:26](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=926s) I personally think it should. Historically, we have, uh, felt a great deal of ownership around our code, pride around the fact that, hey, I wrote this thing. I'm putting my name on it. And at Anthropic, like we, we think about this a little bit differently. We've evolved to the point where we don't really mind so much if Claude is like labeled as, "Hey, I'm, I'm the one that wrote this thing," because we're a little bit more focused on the outcomes and making sure that the code, the features, like what we're shipping to users, um, is the right thing. And so I personally think it's fine for the code to be written,

[15:56](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=956s) you know, labeled as being coming from an agent, and that can also help us with things like figuring out, okay, well, if the agent wrote it, what are some of the constraints and extra checks that we need to keep in place.

[16:04](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=964s) I, I, I have the same point of view, Addy. I think in the short run, at least in the foreseeable, uh, you know, six to nine months, we should label the PRs till we are confident enough in our factories. We do want to know and track it, and it's also important for the financial customers here in terms of auditability. If you, uh, you know, if you, uh, label it, uh, you know who's writing the code-

[16:24](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=984s) Yeah

[16:24](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=984s) ... because we are, at the end of the day, still accountable for it.

[16:26](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=986s) Yeah, absolutely. Accountability is the most important part here. Like, we still label, like, who is the owner for this change? I might have kicked off a Cloud R- a Claude run to complete a piece of work, but I'm, I'm still on the hook.

[16:37](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=997s) Yep.

[16:37](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=997s) If something goes wrong, I have to go and fix it.

[16:39](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=999s) Yep, yep. Key point. Yep. Uh, the next question, actually, I'll improvise it here because this is a leadership ses- session. O- one skill a junior developer or leaders should, uh, build on this year.

[16:51](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1011s) I think that you should push your teams... If you're a leader, you should push your teams to really think hard about what verification means. Um, how are you already approaching things like testing? What is the gap between that and you getting to a place where you're very comfortable with the agents being able to sign off on the code being good enough to merge on their own? And so figuring that out. If you're a junior, it's the other side of that, like actually getting more experience in the different flavors of tests, whether it's like end-to-end tests, property tests, et cetera.

[17:18](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1038s) Yep. And the last, uh, you, you actually not following the rules, you're going beyond one sentence, but that's cool. Uh, the last one-

[17:26](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1046s) We have 30 seconds left. I'm fine.

[17:28](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1048s) We can go a minute over, I'm sure. Uh, the agent opened the PR. It is safe to merge when?

[17:35](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1055s) You have full confidence nothing's gonna go wrong or that you've mitigated that-

[17:38](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1058s) Love it

[17:38](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1058s) ... as much as possible.

[17:39](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1059s) Love it. All right. Uh, here's my last slide, and then we'll wrap it up. Uh, so what separates, uh, teams that are shipping, uh, fast but not shipping blind? I think that three key takeaways for you. Uh, writing code is the fastest part, and that is solved. Strong review and verification for all teams, uh, is what we- will help you maintain that speed. Fast to write is not fast to ship. Uh, you have to basically count what is getting merged, look at what is getting merged, and make sure it doesn't escape the sandbox and the things like

[18:09](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1089s) that before it gets into the production. Verify everything. And autonomy is earned. Uh, you-- We will get to software factories very soon, if not next year. Uh, you need to put an independent automated verification inside each loop of your factory. Make sure that you can trust the output that the factory is producing. Uh, with that, I'll wrap it up. Thanks for attending our talk today, uh, a conversation with Addy. Appre- again, Addy, appreciate you taking the time and joining us here in this conversation.

[18:37](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1117s) Of course. Thanks for having me.

[18:38](https://www.youtube.com/watch?v=AnsXWH2_EBU&t=1118s) Uh, visit us at the big red booth, uh, the Sonar booth in the expo hall if you have any further questions or would like to, uh, talk about it. Thank you again.
