AI Engineer Code 2025

What Data from 20 Million Pull Requests Reveal About AI Transformation

Nicholas Arcolano· Head of Research, Jellyfish17:57

Loading slides…

Read the talk

What 20 Million Pull Requests Reveal About AI Transformation

AI adoption correlates with faster software delivery, but repository structure helps explain why some teams see much larger gains than others.

From a talk by Nicholas Arcolano

Before you start: Familiarity with pull requests, commits, and repository organization will help you follow the measurements.

What should AI transformation deliver?

What does good adoption of AI coding tools look like? How much more productive should a team become? What changes besides speed—and what should you investigate when the gains fail to appear? These are practical questions for both new AI-native companies and existing engineering organizations changing how they work.

Speaker beside a slide listing four questions: What does good adoption look like? What productivity gains should I expect? What are the side effects? What if expectations ≠ reality?Open full source frame
Four questions about AI transformation: adoption, productivity, side effects, and expectations versus reality.

Nicholas Arcolano, head of research at Jellyfish at the time of the talk, approaches those questions by connecting tool activity to engineering work. Usage of Copilot, Cursor, and Claude Code, autonomous agents such as Devin and Codex, and PR review bots supplies one part of the picture. GitHub supplies codebase and source-control context; Linear and Jira supply the goals behind the work. Connecting these sources makes it possible to examine both whether developers use AI and what happens to the work they deliver.

The talk draws on approximately 20 million pull requests written and merged by about 200,000 developers at roughly 1,000 companies. The observations span June 2024 through the time of the talk, covering more than a year. This is a customer dataset describing engineering activity in working organizations.

0:000:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Measure behavior, not just generated code

Lines of code offer a striking first view of adoption, even though Arcolano considers them a weak metric. In the tracked cohort, the share of companies generating at least half their code with AI rose from about 2% in summer 2024 to nearly half by the month before the talk. The comparison follows the same companies, rather than comparing an early group with a different later population.

Developer adoption measures the behavior change behind the output. Arcolano describes an individual's adoption rate as the fraction of coding time in which they use AI, and company adoption as the average across individuals. The companion productivity study operationalizes this using active coding days with AI usage, rather than continuously measuring minutes spent coding. At 100% company adoption, every developer uses AI whenever they code; the measure does not mean that AI writes every line.

The weekly adoption chart tracks the 25th, 50th, and 75th percentiles of company adoption rates. Median company adoption rose from approximately 22% in summer 2024 to nearly 90% at the time of the talk. Using several tools concurrently, including synchronous and asynchronous workflows, still corresponds to 100% adoption rather than a rate above 100%. Technical, organizational, and cultural barriers explain why consistent usage remains difficult for some teams even as it becomes routine for others.

2:132:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:13 · section reference included

Interactive adoption is not autonomous deployment

The adoption results overwhelmingly concern interactive coding tools. Copilot, Cursor, and Claude Code can have agentic modes while still participating in an interactive workflow. The talk separately considers fully autonomous workflows such as those associated with Devin and Codex at the time. The relevant distinction is how work proceeds, not simply whether a product has an agent feature.

About 44% of companies in the dataset showed autonomous-agent activity during the preceding three months, mostly trials and experimentation rather than full-scale production. That activity accounted for less than 0.2% of the millions of PRs merged during the same period. The companion autonomous-agent analysis detects agent involvement through PR actions and commit authorship, so AI work committed by a human can escape that measurement. These figures describe detectable participation, not every possible use of an agent.

Speaker beside a slide stating that only 44% of companies used a fully autonomous agent in the past three months, largely for trialing and experimentation rather than full-scale production, accounting for less than 0.2% of PRs.Open full source frame
Autonomous coding agents: 44% of companies have tried them, largely through experimentation, with less than 0.2% of PRs.
4:304:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:30 · section reference included

More PRs, with less time to merge

Limited autonomous deployment does not mean limited productivity gains: the next analysis primarily concerns interactive coding tools. Its starting metric is PR throughput, the average number of pull requests an engineer merges per week. Absolute throughput depends on how a team scopes work and how its code is organized. Tracking a team's change in throughput, with those factors held as constant as possible, is more useful than treating another company's PR count as a target.

The second metric is cycle time, defined here as the interval from a PR's first commit to its merge. Although cycle time can refer more broadly to delivery latency, this definition stops at merge, not deployment. A small TypeScript helper makes that measurement boundary explicit:

typescript

type MergedPullRequest = {
  firstCommitAt: string;
  mergedAt: string;
};

function cycleTimeHours(pr: MergedPullRequest): number {
  const start = Date.parse(pr.firstCommitAt);
  const end = Date.parse(pr.mergedAt);

  if (!Number.isFinite(start) || !Number.isFinite(end) || end < start) {
    throw new Error("Invalid first-commit-to-merge interval");
  }

  return (end - start) / 3_600_000;
}

This computes an individual PR's interval; the chart below conceptually aggregates those intervals for the PRs merged in each company-week.

Each point in the throughput chart represents one company in one week. The horizontal axis is company AI adoption; the vertical axis is average PRs per engineer that week. Arcolano reports an approximately 2× average PR-throughput trend between 0% and 100% AI adoption. He frames that as an expectation for transformation, but the analysis establishes an observational association, not a guaranteed causal doubling for an individual team.

The cycle-time chart keeps the company-week structure but uses the median cycle time of PRs merged that week. Its horizontal bands are revealing: one cluster corresponds to tasks taking less than a day, another to tasks taking about two days, with a valley between them. A long tail of slower merges extends upward beyond the truncated display. Arcolano reports a 24% average decrease in first-commit-to-merge cycle time between 0% and 100% AI adoption. Together, the two measures associate greater adoption with both more merged work and shorter waits to merge.

5:385:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:38 · section reference included

Larger changes do not yet show a quality penalty

More frequent, faster merges raise a second question: what is changing inside the PRs? In the talk, Arcolano describes fully adopting teams' PRs as averaging 18% larger by net lines of code added. There is a metric distinction in the companion PR-size analysis: its approximately 18% result concerns additions per PR, not net additions. The spoken result and the companion measure should therefore not be treated as interchangeable.

The increase comes more from additions than deletions, suggesting growth in new code rather than primarily rewriting existing code. Meanwhile, the average number of files touched stays approximately unchanged. The extra code is largely within the same files, which could mean more thorough implementations or simply more verbose ones. PR size alone cannot distinguish those explanations.

Jellyfish found no statistically significant relationship between AI adoption and either bug tickets created or PR revert rates. These are useful signals of defects and rolled-back work, but they do not establish that all aspects of quality or long-term maintenance cost are unchanged.

Bug-resolution rates did increase. Arcolano attributes that pattern to teams disproportionately using AI to address bug tickets already in their backlogs: more bugs are being resolved, without a corresponding detected increase in bugs being created. That task selection makes sense because a bug can offer a well-scoped problem and a verifiable result. The quality finding remains provisional, especially as asynchronous-agent usage grows.

Speaker beside the complete quality slide: bug tickets created and PR reverts were examined; no relationship to rate of AI usage; increase in bugs resolved; no smoking gun on quality yet.Open full source frame
Quality findings: no relationship to AI usage in the tracked signals, more bugs resolved, and no smoking gun yet.
8:308:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:30 · section reference included

When adoption is high but gains are low

When an organization's results look different, the first diagnostic is adoption: tools cannot change delivery at scale if people rarely use them. But high adoption does not explain everything. If usage is already widespread and throughput has barely moved, code architecture becomes a useful next variable to inspect.

Here, architecture means how product and service code is organized across repositories. Monorepos and polyrepos can reflect monolithic services or microservices, or a centralized versus federated product strategy. They are indicators of structure, not interchangeable labels for it. Arcolano introduces active repositories per engineer, describing it as the number of distinct repositories a typical engineer pushes code to in a week.

Arcolano reports that normalizing repository activity per engineer removes its correlation with company or team size in this analysis. The resulting distribution describes the shape of the code people work across: centralized architectures occupy the left side, balanced and lightly distributed organizations lie between the extremes, and highly distributed architectures form a long tail to the right.

Splitting the company-week observations into four repository regimes changes the adoption-throughput picture substantially:

Repository regimeThroughput trend across the adoption range
CentralizedApproximately 4×
BalancedApproximately 4×
DistributedApproximately 2×
Highly distributedEssentially no correlation; weak, slightly negative trend

These are group-level trends. Centralized and balanced organizations show a stronger association than the overall result; distributed organizations resemble the overall trend; the most distributed group shows little relationship between additional adoption and throughput.

10:5811:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:58 · section reference included

Repository boundaries create context problems

Why might highly distributed teams struggle to turn AI usage into greater throughput? Arcolano's proposed explanation is cross-repository context. The tools he discusses generally work best with one repository at a time: select a repository, load its context, and begin. Work that spans repositories requires assembling relationships that are difficult for both people and agents to reconstruct.

Those relationships may not be written down clearly. Senior engineers can know how repositories fit into systems and products without that knowledge being accessible to a coding tool. Context engineering therefore means making the relationships needed for a task available to the tool, rather than assuming that access to one repository supplies the whole system picture. This is a proposed explanation for the observed pattern, not a demonstrated causal mechanism.

It also leaves room for the pattern to change. Claims that microservices are well suited to AI-native development could become compatible with these findings if teams solve the context problem and deploy autonomous agents effectively at scale. Arcolano imagines a future in which highly distributed architectures become the most productive category; he presents that reversal as a possibility, not an outcome already measured.

Architecture also changes the baseline against which productivity is measured. As repositories become more distributed, migrations and cross-repository coordination can require more PRs to accomplish the same broader piece of work. That pushes absolute PRs per engineer upward even before considering AI. A higher PR count does not necessarily mean more useful work; changes in throughput are the more informative comparison.

14:4214:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:42 · section reference included

Set expectations, then investigate the constraint

Teams do not need to wait for fully autonomous development to benefit from AI: the gains discussed here largely accompany interactive tools already in widespread use. In closing, Arcolano reiterates a 2× increase in PR throughput or more as an expectation, drawing on the earlier zero-to-full-adoption trend. That expectation comes with larger PRs and a reason to continue monitoring quality, even though the tracked signals have not shown major problems so far.

When that expectation does not match reality, examine how code architecture shapes the context a developer or agent needs to complete a task. Repository boundaries can separate the code from the system knowledge required to change it safely. Making that knowledge accessible is a concrete place to begin while other explanations for uneven gains remain under investigation.

Speaker beside four takeaways: AI coding tools have shown up in a big way, agents less so; expect a 2x change in PR throughput; expect bigger PRs but perhaps acceptable quality; consider code architecture when concerned.Open full source frame
Takeaways: widespread coding tools, a 2x PR throughput expectation, bigger PRs, and attention to code architecture.
16:3816:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:38 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…