Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

Loading slides…

Read the talk

Better AI Models Won’t Fix Your Engineering Team

Recording frame at 0:19: Better AI Models Won’t Fix Your Engineering TeamOpen full source frame
Better AI Models Won’t Fix Your Engineering Team — recording at 0:19

Nicholas Arcolano uses engineering telemetry to separate model choice from skill use, token consumption and workplace conditions—and explain why stronger models can coexist with stubborn differences in delivery.

From a talk by Nicholas Arcolano

At a glance

Ideas worth remembering

  • The share of an engineer’s Cursor requests that used Grok after July showed no statistical correlation with their change in merged-PR throughput relative to colleagues at the same company.

  • Across its analyses, Jellyfish attributes less than 1% of variation in throughput to tool choice. The outcome counts merged PRs and excludes quality.

  • Thirty or more skill runs per engineer per week are associated with roughly 30% higher throughput after adjustment for the other modeled factors, while frequent skill use remains uncommon.

  • The consumption curve associates roughly ten times the tokens with twice the throughput from median to top-decile consumption. Company differences also span about twofold at fixed consumption.

  • Among engineers who started in the top quartile for PR throughput, 44% cycled out over a normal half-year and 41% cycled out across the switch to AI. The results favor investing in existing engineers and their working environment over assuming AI has reordered performance.

Better clubs really do help

A bad round of golf makes better clubs look very attractive. Nicholas Arcolano, introducing himself as Jellyfish’s head of AI and research, knows the temptation: buy equipment and make the pain go away. An instructor’s answer is less comforting—work on the fundamentals. But the data complicates that advice. Equipment really has improved performance.

Recording frame at 1:47: Better clubs really do helpOpen full source frame
Better clubs really do help — recording at 1:47

The PGA Tour driving-distance chart starts at an average of 257 yards in 1980 and rises roughly 20% over the period shown. Arcolano points to an approximately 15-yard increase around the transition to solid-core balls near 2000. He attributes that short, roughly three-year jump to technology: golfers did not suddenly acquire that much additional ability together.

The analogy gives model progress its due. An Artificial Analysis chart shows the model frontier advancing, and AI can now handle work that was out of reach a year earlier. The question is how much those capability improvements explain the differences in output among engineers already using AI. Better equipment and better results are related questions, but measuring one does not settle the other.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

A sudden Grok switch, with little movement in throughput

Jellyfish connects AI coding-tool and agent telemetry with engineering workflow data. That connection lets researchers compare what engineers use with what they merge. Within that data, Grok usage suddenly increased much faster than the usual drift in engineers’ model preferences.

The immediate explanation was product behavior: Cursor promoted Grok in its interface and changed how Auto mode worked. Engineers were being pushed toward a different model without each independently deciding that it was best for their work. Arcolano treats this external shock as a natural experiment and asks whether the change in model use accompanied a change in output.

What would a consequential model switch look like in the results? Each point represents one of 12,310 engineers. The horizontal axis shows the share of that engineer’s Cursor requests that used Grok after July, from 0% to 100%. The vertical axis shows the change in their merged-PR throughput relative to colleagues at the same company. Individual outcomes varied widely, but the trend line stays near zero across the range of Grok use. Arcolano reports no statistical correlation. That is the puzzle behind the rest of the talk.

Recording frame at 4:25: A sudden Grok switch, with little movement in throughputOpen full source frame
A sudden Grok switch, with little movement in throughput — recording at 4:25
2:423:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:42 · section reference included

Separate the tool from the way work happens

The main analysis uses a curated subset of approximately 50,000 engineers at 447 companies, with 2026 data for Cursor and Claude Code users. Its outcome is merged pull requests per engineer per week. Restricting the dataset supplies the pieces needed to examine several factors together, rather than treating every difference between engineers as a consequence of their chosen model.

The analysis separates five kinds of variation:

  • The tool: The model and its harness—the software environment through which the engineer uses it—are grouped together because they are intertwined.
  • How well it is used: Skill use, Cursor rules and harness customization capture practices intended to make the tool more effective.
  • How much it is used: Token consumption captures the amount of AI use separately from those practices.
  • Where the work happens: Company and team capture differences in the engineering environment.
  • The individual engineer: Differences remain between people after the other factors are held constant.

Which factors account for the largest shares of throughput variation? Arcolano gives rounded figures from Jellyfish’s analyses of Cursor and Claude Code use from March through September 2026. The tool accounts for less than 1%, how well engineers use it about 3%, consumption about 9%, and company and team about 12%. The individual accounts for roughly three-quarters. These percentages describe variation in throughput. The later skill-use comparison measures how much higher throughput is among frequent users after adjustment. The talk does not develop the statistical specification, so the adjusted relationships should not be read as guaranteed causal gains from changing one factor.

Recording frame at 7:06: Separate the tool from the way work happensOpen full source frame
Separate the tool from the way work happens — recording at 7:06

An audience question establishes another important limit: this analysis has no quality metric. Merged-PR throughput counts completed changes, but it does not establish their quality, difficulty or value. Arcolano says Jellyfish studies quality separately; those results are outside this analysis.

4:425:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:42 · section reference included

Changing models, harnesses or modes

The Grok episode is only one way to examine tool choice. Engineers also switch tools voluntarily, making it possible to compare an engineer with themselves and examine what happens to weekly merged PRs around a switch. That comparison follows the same person instead of comparing two groups whose members may already differ.

Recording frame at 7:36: Changing models, harnesses or modesOpen full source frame
Changing models, harnesses or modes — recording at 7:36

The chart compares weekly merged PRs with the same engineer’s other weeks across three kinds of change:

  • Model choice: Weeks described as “all in” on one model family, shown separately for Opus, Sonnet and GPT / Codex.
  • Harness choice: Moving from Cursor to Claude Code.
  • Interaction mode: Moving from autocomplete to an agentic mode.

Across these comparisons, Arcolano reports little meaningful change in PR throughput. His description is a statistical “nothing burger.” The finding concerns this delivery measure across the observed switches; it does not measure every capability that might make an engineer prefer one tool.

7:197:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:19 · section reference included

Skills have a larger effect than their adoption suggests

The 3% share attributed to skill use initially looks modest. Arcolano’s explanation is adoption: a practice can have a substantial relationship with output among frequent users while explaining relatively little variation across a population that rarely uses it. Here, skills sit within the broader category of teaching and customizing the harness, alongside rules and other working practices.

The adoption chart groups engineer-weeks by zero, low, medium and high skill use. As of September, almost half of the observed weeks had essentially no skill use. Only 12% reached the level Arcolano considers appropriate for modern AI engineering. Conference habits, in other words, are a poor guide to what most engineers routinely do.

How does frequent skill use compare with choosing a different model? The effect comparison places model-family changes alongside skill-use levels. At 30 or more skill runs per engineer per week, the analysis estimates approximately 30% higher PR throughput while holding the other modeled factors constant. This is why the adoption gap matters: a comparatively strong relationship appears in a behavior that is still uncommon.

5:128:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:20 · section reference included

More tokens accompany more output, with diminishing returns

Consumption asks a separate question from skill: how much does an engineer use AI? Arcolano introduces tokens and active-use days, then summarizes an earlier World’s Fair analysis covering approximately 500,000 developer-weeks. Each point in its chart represents a two-percentile consumption bin, plotting average token spend against merged PRs per developer per week.

What does additional consumption buy along that curve? Throughput rises with tokens, but the relationship flattens. Moving from median consumption to the top 10% corresponds to roughly twice the output at about ten times the token consumption. The curve makes the tradeoff visible: substantial additional usage accompanies further gains, while each proportional increase in tokens returns a smaller proportional increase in throughput.

Recording frame at 11:06: More tokens accompany more output, with diminishing returnsOpen full source frame
More tokens accompany more output, with diminishing returns — recording at 11:06

The rocket-fuel analogy captures both parts. More fuel can take you farther, but the conversion becomes increasingly expensive. For a leader deciding how to allocate an AI budget, the ten-to-one consumption change matters just as much as the twofold throughput difference. Token volume alone is an incomplete target.

10:0410:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:04 · section reference included

The same AI consumption can produce different company results

Company and team together account for about 12% of the statistical breakdown. Within that workplace component, approximately one-third is attributed to the team and two-thirds to the company. Codebase, review process and culture are the concrete examples Arcolano gives of the environment in which AI-assisted work must succeed.

How much difference remains when consumption is held fixed? The company distribution shows an approximately twofold throughput spread between its top and bottom 10%, after holding the other modeled factors constant. Compare that with the consumption curve: its roughly twofold output difference accompanies ten times the tokens. The endpoints differ between the two comparisons, but together they show that workplace variation is substantial even at fixed AI consumption.

This makes the environment especially relevant to engineering leadership. Leaders can change review practices, the codebase and organizational conditions even when the token budget is constrained. The company comparison does not isolate which particular change will pay off, but it gives a reason to investigate those conditions before assuming that another tool purchase will solve the problem.

6:4411:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:14 · section reference included

AI has not greatly reshuffled the throughput rankings

The individual is the largest component: roughly three-quarters of the statistical model. The corresponding engineer distribution is wider than the company distribution, with an approximately 3.5-fold throughput spread between the bottom and top 10%, compared with about twofold for companies. That identifies substantial differences between people without unpacking every ability, assignment or habit behind them.

A more focused question is whether AI changes who occupies the top of the distribution. If AI substantially rearranges engineering performance, previously high-throughput engineers might leave the top group more often. Jellyfish examines that possibility in a longitudinal study of about 10,000 engineers, ranking each against their own colleagues by PR throughput.

Do more engineers leave the top quartile when they start using AI? Over a normal half-year, 44% of engineers who started in the top throughput quartile cycled out. Through the switch to AI, 41% cycled out. Turnover was three percentage points lower across the switch.

Recording frame at 14:36: AI has not greatly reshuffled the throughput rankingsOpen full source frame
AI has not greatly reshuffled the throughput rankings — recording at 14:36

Arcolano reads this as evidence that craft still matters: engineers who ranked highly before AI largely continue to rank highly afterward. The result gives little support to a sweeping leveling of this throughput measure. It also keeps the meaning of “top engineer” specific here—the ranking tracks merged PRs, rather than every contribution an engineer makes to a team.

12:4213:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:42 · section reference included

Leaders can change the course

New models attract attention because the technology is exciting. Arcolano shares that enthusiasm, while pointing out a practical constraint: most engineering leaders cannot improve a foundation model themselves. Tighter budgets may also reduce their freedom to keep buying different tools. People and organizational conditions offer more direct opportunities to act.

The golf analogy earns its final turn here. Unlike a golfer, an engineering leader can change the course. You can fill in the water hazards: change the environment in which engineers and their tools have to work. The findings give that advice substance—skill use, consumption, company conditions and persistent differences between engineers all deserve attention when delivery disappoints.

“Maybe it’s not the model” is a useful diagnostic question, rather than a reason to stop caring about models. The closing invitation is to examine the people, practices and environment within your control. For continued comparison, Arcolano points to the monthly Jellyfish AI Engineering Trends benchmarks and the team’s research Substack.

15:0215:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:02 · section reference included

Resources

From the talk

  • The monthly benchmarks Arcolano recommends for following AI usage and engineering outcomes beyond the recording.

  • The research Substack named at the close, for following the analyses behind Jellyfish’s engineering measurements.

Transcript download
Full transcript

Loading transcript…