# Does AI Actually Boost Developer Productivity? (Stanford / 100k Devs Study)

Yegor Denisov-Blanch, Stanford University

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/tbDDYKRFjhk-ai-developer-productivity) · [Video](https://www.youtube.com/watch?v=tbDDYKRFjhk)

## Does AI Actually Boost Developer Productivity? (100k Devs Study)

Yegor Denisov-Blanch presents evidence from enterprise Git histories showing that AI coding tools increase output on average, but rework, task complexity, project maturity, language popularity, and codebase size determine how much becomes useful progress.

### Replace the replacement story with a measurement question

[0:15](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=15s)

The talk opens with executive pressure to treat AI adoption as a race toward replacing developers. Denisov-Blanch offers a more measured starting point: [AI coding assistance can increase developer productivity, but it can also decrease it](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=76s). The useful question is whether generated code helps a particular team deliver useful functionality faster.

![Study overview slide provides scale and methodology context, including longitudinal Git history and cross-company comparisons involving more than 100,000 engineers.](https://ai.engineer/api/talk-essay-media/b9216256f904749f8cb338ec571fe577a499f8ea6d22083bccb2353402ee58ff)

Study overview slide provides scale and methodology context, including longitudinal Git history and cross-company comparisons involving more than 100,000 engineers.

The research group approaches that question with longitudinal and cross-sectional data. Historical Git records let it examine changes over time, including work recorded before a company joined the study. Comparisons across more than 600 enterprises, midsized companies, and startups provide the second dimension. The dataset described in the talk contains more than 100,000 software engineers, dozens of millions of commits, and billions of lines of code.

Most observations come from private repositories. Denisov-Blanch argues that this makes team-level measurement more self-contained than public-repository activity, which may represent occasional or weekend work. That choice improves coverage of organizational coding activity, but the method still centers on changes visible in Git; the talk does not claim that repository history captures every valuable activity performed by an engineer.

### More activity can conceal less useful progress

[4:37](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=277s)

The [first measurement trap](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=277s) is counting commits, pull requests, completed tasks, or shorter intervals between commits. Tasks differ in size, so a higher count does not establish that a team delivered more functionality. AI compounds the problem when generated code creates bugs and therefore creates additional repair tasks. Both the original change and its fix increase activity, even if the team has mostly spun its wheels.

![Self-assessment experiment slide supports the claim that developer surveys poorly predict measured productivity.](https://ai.engineer/api/talk-essay-media/bc6f47024f8853f9501b654354964ce935056b7ed5bf3ba0e285055c07d48823)

Self-assessment experiment slide supports the claim that developer surveys poorly predict measured productivity.

The second trap is generalizing from clean-slate experiments. When one group receives AI assistance and another does not, the assigned work is often a Greenfield task with little prior context. Models are especially effective at producing boilerplate in that setting. Established software work is usually Brownfield: engineers must understand existing code, dependencies, conventions, and domain logic. A result from an isolated new project does not automatically transfer to that environment.

The third trap is self-reporting. In a small experiment with 43 developers, participants placed themselves in five-percentile buckets relative to a global mean or median. Their estimates missed measured productivity by about 30 percentile points, and only one in three placed themselves within the correct quartile. Surveys can still reveal morale, satisfaction, and other experiences that code metrics cannot see. Denisov-Blanch’s narrower judgment is that they should not serve as the productivity measure itself.

### Measure delivered functionality, then account for rework

[7:10](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=430s)

The [proposed method](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=430s) begins with an expensive ideal: ask a panel of 10 to 15 independent engineering experts to evaluate a code change for qualities such as maintainability, output, quality, and the effort required to produce it. Denisov-Blanch says these panels agree with one another and that their aggregate assessments predict real-world outcomes. Repeating that process for every commit, however, would be too slow and costly.

![Headline productivity slide contrasts roughly 30–40% apparent output growth with the smaller 15–20% net gain after corrective work.](https://ai.engineer/api/talk-essay-media/4c62c927abb258bd2829b0b2a15a0f5ccc61ccf1172a259f7baf6cbf52f5a5dc)

Headline productivity slide contrasts roughly 30–40% apparent output growth with the smaller 15–20% net gain after corrective work.

The group therefore built a model intended to automate the panel’s assessment. It connects to Git, examines the source-code changes in each commit, and scores the change across similar dimensions. Authors, SHAs, and timestamps locate those assessments by person and time. In this framework, team productivity is the functionality delivered over time—not the number of commits or lines written. The model is an approximation that the speaker says correlates well with expert review; the talk does not provide the correlation statistic or a full validation protocol.

A [pilot involving about 120 developers](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=550s) shows why the distinction matters. Monthly output is divided into added functionality, removed functionality, refactoring, and rework. Refactoring changes older code and may or may not be wasteful. Rework changes much more recent code, so the framework treats it as corrective work that reduces the value of the initial output. After AI entered the team’s workflow, both apparent production and rework rose.

The headline arithmetic is simple but consequential. AI-assisted coding appears to raise output by roughly 30–40%, yet bug fixes and other rework consume a substantial part of that increase. The aggregate net improvement presented in the talk is about 15–20% across industries and sectors. That average is useful as a baseline, not a forecast for every team—the distribution matters more than the mean.

From generated output to net productivity
AI can increase delivered output while also creating corrective work that reduces the net gain.
- AI-assisted coding — Developers produce more apparent output.
- 30–40% apparent gain — The initial estimate reflects more code and functionality being delivered.
- Bugs and recent-code rework — Developers revisit recent generated changes and repair problems.
- 15–20% average net gain — Corrective work reduces the aggregate improvement.
- AI-assisted coding → 30–40% apparent gain (increases production)
- 30–40% apparent gain → Bugs and recent-code rework (some output needs repair)
- Bugs and recent-code rework → 15–20% average net gain (subtract correction cost)

### Complexity and project maturity move the result

[11:03](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=663s)

The [productivity distributions](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=663s) begin as low as −20%, which immediately rules out a universal positive effect. Simpler work performs better than complex work, and low-complexity Greenfield tasks show the strongest distribution of gains. Denisov-Blanch limits this comparison to enterprise settings; he expects personal Greenfield projects to behave differently and potentially show larger improvements.

![Language-popularity matrix compares lower-popularity languages such as COBOL, Haskell, and Elixir with widely used languages.](https://ai.engineer/api/talk-essay-media/465553306d51cec2be5eef2f5a1b9740263a57be16f45d1dffc2c70e3bd208cf)

Language-popularity matrix compares lower-popularity languages such as COBOL, Haskell, and Elixir with widely used languages.

High-complexity tasks produce smaller gains and sometimes make engineers slower. Brownfield work also benefits less than Greenfield work. The charts summarize both a central estimate and an interquartile range, with the line spanning roughly the 25th to 75th percentiles. This matters because two teams doing nominally similar work can still land on different sides of the average. Denisov-Blanch also says the causes of the negative cases are not yet clear.

A [simplified matrix](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=802s) based on 136 teams across 27 companies makes the interaction concrete: low-complexity Greenfield work shows indicative gains of 30–40%; high-complexity Greenfield work, 10–15%; low-complexity Brownfield work, 15–20%; and high-complexity Brownfield work, 0–10%. The speaker describes these as orientative guidelines. They can help decide where to pilot AI, but they are not precise promises for a project plan.

Language popularity introduces another source of variance. For widely used languages such as Python, Java, JavaScript, and TypeScript, the talk reports gains around 20% on simpler work and 10–15% on complex work. Assistance is less dependable for the lower-popularity examples given—COBOL, Haskell, and Elixir—and complex tasks in those languages can become slower. A tool that helps only two times out of five may be abandoned even if its occasional suggestions are good.

Indicative gains by task and project type
Approximate guidelines from 136 teams across 27 companies; they summarize observed ranges rather than guarantee outcomes.
- Greenfield · low complexity — 30–40% indicative gain
- Greenfield · high complexity — 10–15% indicative gain
- Brownfield · low complexity — 15–20% indicative gain
- Brownfield · high complexity — 0–10% indicative gain

### A large context window does not equal effective codebase understanding

[15:35](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=935s)

The [final analysis](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=935s) is explicitly more theoretical and less empirically established. Denisov-Blanch presents an illustrative curve in which AI productivity gains fall sharply as codebases grow from 1,000 to 10 million lines. He proposes three mechanisms: finite effective context, a worsening signal-to-noise ratio, and the growing number of dependencies and domain-specific rules in larger systems.

![NoLiMa comparison chart shows coding-task performance declining as supplied context grows from about 1,000 to 32,000 tokens.](https://ai.engineer/api/talk-essay-media/85304356c988062646d8a57ad503b3a4375d1eb0184a055bd5d80ff70785cee8)

NoLiMa comparison chart shows coding-task performance declining as supplied context grows from about 1,000 to 32,000 tokens.

An advertised context-window capacity does not show that a model will use every token equally well. Citing a paper comparison, the talk describes coding-task performance declining as context increases from 1,000 to 32,000 tokens; one illustrated model falls from roughly 90% to 50% by 32,000 tokens despite supporting a much larger maximum window. The extrapolation to 64,000 or 128,000 tokens is rhetorical, not a measured result presented here.

The practical conclusion is selective adoption. Use AI in many cases, but evaluate it against the work actually being done. Teams should segment results by task complexity, Greenfield versus Brownfield status, language popularity, codebase size, and supplied context. Most importantly, they should measure useful functionality and the rework needed to preserve it. A universal license rollout may be easy; a universal productivity forecast is not.

Denisov-Blanch closes by directing interested readers to the [Stanford Software Engineering Productivity Research portal](https://softwareengineeringproductivity.stanford.edu). The [official AI Engineer talk page](https://ai.engineer/talks/tbDDYKRFjhk-ai-developer-productivity) provides the recording, chapters, transcript, and a concise reading version.

## Key takeaways

- Count delivered functionality rather than commits, pull requests, tasks, or lines of code; AI-generated repairs can inflate every activity metric without creating equivalent progress.
- The talk’s aggregate estimate falls from a 30–40% apparent output increase to roughly 15–20% net productivity after rework and bug fixing.
- Low-complexity Greenfield work shows the strongest gains; high-complexity Brownfield work shows indicative gains of only 0–10% and can sometimes become slower.
- Widely used languages benefit more consistently than the lower-popularity examples in the talk, where unreliable suggestions can erase gains on complex work.
- Treat the codebase-size relationship as a hypothesis rather than a settled result: the talk labels it illustrative and proposes effective-context limits, noise, dependencies, and domain logic as possible causes.

## Resources

- [Stanford Software Engineering Productivity Research](https://softwareengineeringproductivity.stanford.edu/) — The research portal named at the end of the talk, with information about the group’s engineering-productivity research.
- [Does AI Actually Boost Developer Productivity?](https://ai.engineer/talks/tbDDYKRFjhk-ai-developer-productivity) — The official AI Engineer page for the recording, with chapters, highlights, transcript, and a reading version.
- [Yegor Denisov-Blanch](https://yegordb.com/) — The speaker’s site describes his developer-productivity research, publications, and subsequent work on measuring AI-assisted engineering.
- [How to Quantify AI ROI in Software Engineering](https://ai.engineer/talks/JvosMkuNxF8-quantify-ai-roi-in-software-engineering-stanford) — A later presentation extending the measurement approach with matched teams, usage data, codebase cleanliness, guardrail metrics, and an enterprise case study.

## Related talks

- [How to Quantify AI ROI in Software Engineering (Stanford Study / 120k Devs)](https://ai.engineer/talks/JvosMkuNxF8-quantify-ai-roi-in-software-engineering-stanford) — Denisov-Blanch extends this framework into matched-team analysis, usage measurement, guardrail metrics, and a detailed enterprise adoption case.
- [No Vibes Allowed: Solving Hard Problems in Complex Codebases](https://ai.engineer/talks/rmvDxxNubIg-context-engineering-for-complex-codebases) — Dex Horthy examines why coding agents create rework in established codebases and presents context engineering as a practical response to the Brownfield difficulty described here.
- [Why Agent Hype can fall short of reality – Joel Becker, METR](https://ai.engineer/talks/RhfqQKe22ZA-why-agent-hype-fall-short-reality) — Joel Becker contrasts benchmark gains with a randomized study of experienced developers working on messy, context-dependent software tasks.

## Slides

### Event sponsors

AWS is the innovation partner; six companies are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=10s) [AWS innovation partner and six platinum sponsor logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/24c34e6cae9646cf44466e548e3d828426dbee3257365531132e00e0d3c0de6c.webp)

INNOVATION PARTNER
aws

PLATINUM SPONSORS
Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Does AI Actually Boost Developer Productivity?

Yegor Denisov-Blanch argues that AI can increase developer productivity, but can also decrease it in some cases.

[0:27](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=27s) [Title slide for Yegor Denisov-Blanch’s Stanford University talk at AI Engineer World’s Fair.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6448ea7fd3f1028b915e3a9b4fdba721411aa172764c9e5ee25e9d415d68f03.webp)

05 June 2025

Does AI Actually Boost Developer Productivity?

AI Engineer World’s Fair

Yegor Denisov-Blanch
SWE Productivity Research Group
Stanford University
[redacted]

Stanford University

### Time-series cross-sectional study

The study uses participants’ Git histories to trace trends over time and compares data across 600+ companies, from startups to enterprises.

[1:34](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=94s) [Study design compares Git history over time with data from more than 600 companies.](https://assets.ai.engineer/sites/aiecode2025/sha256/868cfbb31bbd6b612e59c3768621d5be199713a2961cfcfe80c894abf483b288.webp)

Time-series cross-sectional study on software engineering productivity

Time-Series
Git History
Timeline: … 2023 2024 2025

Cross-Sectional
600+ Companies
Enterprise | Mid-Size | Startup

[illegible]ersity
2

### Our current dataset

The dataset spans 100,000+ engineers, millions of commits and billions of lines of code. Its 80% private repos, the speaker argues, offer a more self-contained view of team productivity.

[2:11](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=131s) [Dataset includes over 100,000 engineers, millions of commits, billions of lines of code, and 80% private repositories.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b4f81e7f1c59c3514ae3ee82fbd02891c573adbb4fbe6cb931f70383bb95859.webp)

Our current dataset

100,000+ engineers
Millions of commits
Billions of lines of code
80% private repos

[illegible]ersity
3

### Research Team

Simon Obstbaum brings engineering leadership experience; Yegor Denisov-Blanch studies data-driven decisions in software engineering; Michal Kosinski researches human behavior in digital environments.

[3:22](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=202s) [Profiles of Simon Obstbaum, Yegor Denisov-Blanch, and Michal Kosinski and their research or industry experience.](https://assets.ai.engineer/sites/aiecode2025/sha256/84bf6b09dcc77c06e437e7751ab297f3c731ee5cae8d83c9d579373fcce05186.webp)

Research Team (not exhaustive)

Simon Obstbaum
• Ex-CTO, Crunchyroll & Ellation
  • Portfolio of video streaming services, 100M+ users
  • Hundreds of engineers across 20+ distributed teams
• Founder & ex-CEO, YOPESO (software dev. house, 250+ engineers) (exited)

Yegor Denisov-Blanch
• Stanford Graduate Researcher since 2022
• Research focus: data-driven decision-making in software engineering
• Digital transformation for F100 company with 6,000+ engineers

Prof. Michal Kosinski
• Stanford Professor (top 1% most cited researchers)
• Research focus: human behavior in a digital environment
• Cambridge Analytica whistleblower
• Stanford Computer Science Postdoc

[illegible]ersity
5

### What We’ll Cover

The talk starts with limitations of existing studies, then explains the methodology and spends most of its time on results about AI’s impact on developer productivity.

[4:11](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=251s) [Agenda listing limitations of existing studies, methodology, and results on AI and developer productivity; the first item is highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e2213299abf48334105f2f5c33beb876011bf5a8b65ad9669adb82b1338de8b.webp)

What we’ll cover

1 Limitations of Existing Studies
2 Our Methodology
3 Results: Impact of AI on Dev Productivity

[illegible]ersity
6

### Limitations of Existing Studies

The speaker argues that task counts can include bug fixes, greenfield tests miss existing codebases and dependencies, and surveys poorly predict productivity.

[4:35](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=275s) [Three columns describe limitations of commits and pull requests, greenfield tasks, and surveys as measures of productivity.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab68e06db3f0c6afd551151f8b73e76001202df4937ccce63a92307546b9ab4e.webp)

Most existing studies on the impact of AI on engineering productivity have limitations

Commits / PRs
• Task size varies
• More tasks != more productivity
• Tasks to fix the bugs introduced in the earlier tasks

Greenfield Tasks
• AI is great at greenfield, boilerplate code
• Most of software engineering isn’t greenfield
• Usually an existing codebase with dependencies

Surveys
• Ineffective predictor of productivity

[illegible]ersity
7

### What We’ll Cover: Methodology

The highlighted agenda moves the talk into its methodology, between the limitations of existing studies and results on AI’s impact on developer productivity.

[7:15](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=435s) [Agenda listing limitations, methodology, and results, with methodology highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/51fbc6ce2050c72d23f567d55c94a0d35838a4fa1a6d431a0630dd0489ae8834.webp)

What we’ll cover

1 Limitations of Existing Studies
2 Our Methodology
3 Results: Impact of AI on Dev Productivity

[Code on tablet: illegible]
[illegible]ersity
9

### Automating human code reviews

The model’s evaluations are shown correlating above 0.85 with those of a 15-expert panel. The slide presents the model as a faster, scalable and affordable alternative.

[7:20](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=440s) [Diagram compares expert code reviews with model evaluations and reports a strong correlation.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f906a3881fc5ac771dd84e37e55cfd0c46d6827ab3fd6ffd6d2ee005125f554.webp)

Our model automates human code reviews at scale

Engineer
Writes Code
Is evaluated by

Panel of 15 Independent Experts
Output Evaluation
Slow
Not scalable
Expensive

Exceptional Correlation
r = >0.85

Our model automates human code reviews

Our Model
Output Evaluation
Fast
Scalable
Affordable

[illegible]ersity
10

### Evaluating code changes per commit

The model analyzes each Git commit across code dimensions, then combines the changes with Git metadata to show a productivity metric over time in a dashboard.

[8:26](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=506s) [Pipeline shows how commit changes and Git metadata feed a productivity dashboard, with examples of code dimensions analyzed.](https://assets.ai.engineer/sites/aiecode2025/sha256/f466eee5d90de1c0e29dd6e069a088724c9a66f4f1476b04f94070e608a285a3.webp)

Our model quantitatively evaluates software engineering output by analyzing source code changes on a per-commit basis

1. Git Repository
2. Model
3. Output Dashboard

A. Analyzes the source code changes of every commit
B. Quantifies changes based on key dimensions
C. Combines with Git metadata to form a productivity metric

Cohesion | Complexity | Coupling
Data Structures | Interfaces | Methods
Persistence Layers | APIs Consumed | Architectural Patterns
Dependencies | Dependency Injections | …and more

Effective Output
Jun 24 - Jun 25
Prev Period
Y-axis: Effective Output
Y-axis ticks: 0, 50, 100, 150, 200, 250
Timeline: Jun 24, Dec 24, Jun 25
Excellent
Normal
Below Average
Low

30+ Supported Languages / Frameworks

[illegible]ersity
11

### AI use and rework

A team of about 120 developers piloted AI starting in September 2024. The chart shows a larger orange rework segment from September through December than in the preceding months.

[9:08](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=548s) [Stacked monthly output chart highlights increased rework from September through December 2024.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c46dfdbe7407b57fc13d9b8d90a948d5ebf220af0108513cd4bf6be8e29f0bb.webp)

Using AI in software engineering increases rework

Monthly Output Breakdown
May 2024 - Dec 2024

Y-axis: Output Units
Y-axis ticks: -250, 0, 250, 500, 750, 1000, 1250, 1500, 1750
Bands: Low, Below Average, Normal
Timeline: May 2024, Jun 2024, Jul 2024, Aug 2024, Sep 2024, Oct 2024, Nov 2024, Dec 2024
Legend: Reworked, Refactoring, Added, Removed

[illegible]ersity
13

### Code output, rework and net productivity

The slide estimates 30–40% more new code, offset by 15–25% rework, for a 15–20% net productivity gain (n=2,562; Aug 2024). The speaker describes this as a rough summary with discrepancies.

[10:24](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=624s) [Chart reports 30–40% more new code, 15–25% rework and 15–20% net productivity gains.](https://assets.ai.engineer/sites/aiecode2025/sha256/728f1d6c381cd9e10cf431110834ec522723c058fffd3bcb9460574d4ac2a9c9.webp)

Using AI in software engineering boosts code output but also increases rework, resulting in net productivity gains of 15–20%

Software Engineering Productivity
Increases from AI Use (n=2,562 – Aug 2024)

AI makes it easy to generate & understand code
However, a lot of that new code has bugs and must be re-worked
The Net overall software engineering productivity gains from AI are ~15-20%

New Code: 30-40%
Rework: 15-25%
Net Productivity Gains from AI: 15-20%

[illegible]ersity
[illegible]

### Productivity gains by task complexity

Across 136 teams at 27 companies, low-complexity tasks show larger productivity gains. Greenfield gains vary widely, while some tasks show productivity losses.

[10:59](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=659s) [Violin plots compare productivity gain distributions for low- and high-complexity tasks in greenfield and brownfield projects.](https://assets.ai.engineer/sites/aiecode2025/sha256/69f4ad8060f804de00b11306aa99625710520535c6a9d78e5df1889d710e72df.webp)

AI delivers larger gains on easy tasks than complex tasks

KDE - Impact of AI Coding Assistants on Developer Productivity
n = 136 teams across 27 companies

Y-axis: Productivity Gain
Y-axis ticks: -20%, 0%, 20%, 40%, 60%, 80%, 100%, 120%
X-axis: Greenfield, Brownfield
Task Complexity: Low Complexity, High Complexity

2025 - Research at Stanford University
Yegor Denisov-Blanch ([redacted]) softwareengineeringproductivity.stanford.edu

[illegible]ersity
15

### Productivity gains by project maturity and task complexity

Across 136 teams in 27 companies, mean gains were 31% and 12% for low- and high-complexity greenfield work, versus 16% and 7% in brownfield. Error bars show the interquartile range.

[12:43](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=763s) [Bar chart compares productivity gains for low- and high-complexity tasks in greenfield and brownfield projects, with interquartile ranges.](https://assets.ai.engineer/sites/aiecode2025/sha256/64318349a06b7734c107f46bec13f55b0dfc387dc8dce25c2e99f4a9441a5143.webp)

Greenfield projects gain 30–35% on simple tasks and 10–15% on complex ones, versus 15–20% and 5–10% in brownfield projects

Impact of AI Coding Assistants on Developer Productivity (Mean ± IQR)
n = 136 teams across 27 companies
Error bars represent the interquartile range (IQR)

Y-axis: Productivity Gain (%)
Y-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%
X-axis: Greenfield, Brownfield
Legend: Task Complexity; Low Complexity; High Complexity
Bar labels: Greenfield 31%, 12%; Brownfield 16%, 7%
Interquartile Range

Research at Stanford University, Yegor Denisov-Blanch ([redacted]), softwareengineeringproductivity.stanford.edu

[illegible]ersity

### Productivity guidelines by language popularity

The slide estimates 20–25% gains on simple tasks in popular languages, versus 10–15% on complex tasks. For niche languages, complex tasks range from a 5% loss to a 5% gain.

[14:18](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=858s) [Matrix compares estimated productivity gains for popular and niche languages across low- and high-complexity tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5a855ad99962621ec9b10a7aff417a6af0a4afd1af61055c8e3d493ab7b480c.webp)

AI boosts productivity by 10–25% in popular languages, but can decrease productivity in niche languages

Software Engineering Productivity Increases from AI Use
Orientative Guidelines

Based on data from 100s of companies

Task Complexity: High
Language Popularity: Low
(-5)-5%
Poor training on legacy languages

Task Complexity: High
Language Popularity: High
+10-15%
Complex tasks require human judgment

Task Complexity: Low
Language Popularity: Low
0-5%
Limited AI support for niche languages

Task Complexity: Low
Language Popularity: High
+20-25%
Abundant data and model tuning

Language Popularity: Low — e.g. COBOL, Haskell, Elixir
Language Popularity: High — e.g. Python, Java

[illegible]ersity

### Illustrative productivity gains by codebase size

Illustrative curves show AI productivity gains falling as codebases grow, with future AI declining more slowly. The speaker cites context limits, noise, dependencies, and domain-specific logic.

[15:34](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=934s) [Illustrative chart shows estimated AI productivity gains declining as codebase size grows, alongside context, noise, and complexity factors.](https://assets.ai.engineer/sites/aiecode2025/sha256/cfc5928da2479ed0570a3da30f3e880237cc29cf88e657c6905251464fad422b.webp)

As codebase size increases, productivity gain from AI decreases

Productivity Gain from AI vs. Codebase Size: Current & Future Outlook

Y-axis: Estimated Productivity Gain from AI (%)
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70, 80
X-axis: Codebase Size (Lines of Code)
X-axis ticks: 1K LOC, 10K LOC, 100K LOC, 1M LOC, 10M LOC
Legend: Current AI Capabilities; Future AI (Improved Context & Models); Potential Improvement Zone

Illustrative

Context Window Limitations
• Performance gains decrease with larger context windows

Signal:Noise Ratio
• Large codebases have more noise that can mislead the model

Complexity
• Dependencies
• Domain-specific logic

[illegible]ersity
19

### Model performance and context length

NoLiMa scores fall across all seven models as context grows from 1,000 to 32,000 tokens. Gemini 1.5 Pro drops from roughly 90 to 50 despite its listed 2 million-token context window.

[16:36](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=996s) [NoLiMa chart shows model scores declining as context length increases, beside a table of model context windows.](https://assets.ai.engineer/sites/aiecode2025/sha256/f375668805d4fa93885f819ee4328c20d913c032ce124ac2e95870bb19b392ae.webp)

SOTA model coding performance drops with context length

Model Performance vs Context Length
NoLiMa Benchmark Results

Y-axis: Long Context Performance Score (NoLiMa)
Y-axis ticks: 0, 25, 50, 75, 100
X-axis: Context Length (tokens)
X-axis ticks: 1,000, 2,000, 4,000, 8,000, 16,000, 32,000

Legend:
GPT-4o
GPT-4o mini
Llama 3.3 70B
Llama 3.1 405B
Gemini 1.5 Pro
Gemini 1.5 Flash
Claude 3.5 Sonnet

Model | Context Window (K)
GPT-4o | 128
GPT-4o mini | 128
Llama 3.3 70B | 128
Llama 3.1 405B | 128
Gemini 1.5 Pro | 2,000
Gemini 1.5 Flash | 1,000
Claude 3.5 Sonnet | 200

Source: [illegible] A., Deilamsalehy H., Dernoncourt F., Bui T., Rossi R., Yoon S., & Schütze H. (2025). NOLIMA: Long-Context Evaluation Beyond Literal [illegible] arXiv:2502.05167 v2, 26 March 2025.

[illegible]ersity
20

### AI productivity depends on context

AI can improve developer productivity, but gains vary with task complexity, codebase maturity and size, language popularity, and context length.

[17:25](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1045s) [Summary slide listing task complexity, codebase maturity, language popularity, codebase size, and context length as factors affecting AI productivity gains.](https://assets.ai.engineer/sites/aiecode2025/sha256/aec66421f634f6ef93712609df782fdae7abedd541ad13d575cb4d0d9ccdd474.webp)

Takeaway: AI does increase developer productivity, but not always and not equally

Task Complexity
Codebase Maturity
Language Popularity
Codebase Size
Context Length

[illegible]ersity
21

## Full transcript

[0:00](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=0s) [upbeat music] In January of this year, Mark Zuckerberg said that he was gonna replace all of the mid-level engineers at Meta with AI by the end of

[0:25](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=25s) the year. I think Mark was a bit optimistic, and he was probably acting like a good CEO would, would to inspire a vision and also probably to keep the Facebook stock price up.

[0:38](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=38s) But what Mark also did was create a lot of trouble for CTOs worldwide. Why?

[0:46](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=46s) Because after Mark said that, every single CEO in the world, almost, turned to their CTO and said, "Hey, Mark is-- says he's gonna replace all of his developers with AI.

[0:57](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=57s) Where are we in that journey?" And the answer probably was, "Honestly, not very far, and we're not sure we're gonna do that."

[1:05](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=65s) And so I personally think hopefully this is not, um, you know, gonna, gonna change, but I don't think AI is gonna replace developers entirely, at least, at least this year, let alone at, at, at Meta, right?

[1:19](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=79s) But, um... And I do think that AI increases developer productivity, but there's also cases in which it decreases developer productivity. So AI or using AI for coding is not a one-size-fits-all solution, and there are cases in which it shouldn't be used.

[1:35](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=95s) And so for the past three years, we've, uh, been running one of the largest, uh, studies on software engineering productivity at Stanford, and we've done this in a time series and cross-sectional way.

[1:49](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=109s) So time series meaning that even if a participant joins in twenty twenty-five, we get access to their Git history, meaning we can see trends of data across time. We can see COVID.

[1:58](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=118s) We can see AI. We can see all of these trends and, and, uh, things that happened. And then also cross-sectional because we have more than six hundred companies participating, enterprise, mid-sized, and also startups.

[2:11](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=131s) And so this means that we have more than a hundred thousand software engineers in our dataset right now, dozens of millions of commits, and billions of lines of code.

[2:22](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=142s) And most importantly, most of this data is private repositories. This is important because if you use a public repo to measure someone's productivity, that public repo is not self-contained.

[2:33](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=153s) Someone could be working on that repo on the weekend or once in a while, right? Whereas if you have a private repo, it's much more self-contained and much easier to measure the productivity of a team, of a company, of an organization.

[2:49](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=169s) So late last year, there was a huge, um, controversial, uh, thing around ghost engineers. So this came from kind of the ser-- the, the same research group, our research group.

[2:58](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=178s) And here, uh, Elon Musk was kind enough to retweet us. But what we found is that roughly ten percent of software engineers in our dataset, at the time about fifty thousand, were what we called ghost engineers.

[3:12](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=192s) These people collect a paycheck but basically do no work.

[3:16](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=196s) So that was very surprising for some people, very unsurprising for others. And so some of the people in this research team are, for example, Simon from industry. Uh, so he was CTO at a unicorn, which he exited, and he had a team of about seven hundred developers.

[3:34](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=214s) And as CTO, he was always the last person to know when something was up with his engineering team, right? And so he thought, "Okay, how can I change this?"

[3:42](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=222s) Myself, I've been at Stanford since [REDACTED:age], and I focus on what I call data-driven decision-making in software engineering. And in a past life, I was looking after digital transformation for a large company with thousands of engineers.

[3:57](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=237s) Part of the team is also Professor Kosinski, who is at Stanford, and his research focuses on human behavior in a digital environment. And basically, he was the Cambridge Analytica whistleblower back in the day, if you recall that.

[4:11](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=251s) So today, we're gonna be talking about three things.

[4:14](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=254s) We're gonna start off with the limitations of existing studies that seek to quantify the impact of AI on developer productivity. We're gonna showcase our methodology. And lastly, we're gonna spend most of the time looking at some of the results.

[4:27](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=267s) What is the impact of AI on dev productivity, and how are ways we can slice and dice these results to make them more meaningful?

[4:37](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=277s) And so there's lots of research being done on this topic, but a lot of it is led by vendors who themselves are trying to sell you their own AI coding tools, right?

[4:48](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=288s) And so there's a bit of a conflict of interest there sometimes. And the biggest three limitations that I see is that a lot of these studies revolve around commits and PRs and tasks.

[4:58](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=298s) "Hey, we completed more commits, more PRs. The time between commits decreased."

[5:04](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=304s) The problem here is that task size varies, right? And so delivering more commits does not necessarily mean more productivity. And in fact, what we found very often is that by using AI, you're introducing new tasks that are bug fixes to the stuff that the AI just coded before.

[5:22](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=322s) So by that case, like you're kinda spinning your wheels in place, right? So that's kind of funny.

[5:29](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=329s) Secondly, there's a bunch of studies who say, "Well, we grabbed a bunch of developers. We split them into two groups, and we kinda gave one AI, and one of them we didn't."

[5:37](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=337s) And what usually happens there is that these are kinda greenfield tasks, where they're asked to build something with kinda zero context from scratch. And there, of course, AI decimates, uh, the non-AI people, but that's bo- because AI is just really good at greenfield kinda boilerplate code, right?

[5:54](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=354s) But actually, most of software engineering isn't greenfield and isn't always boiler-boilerplate, right? And so there's usually an existing code base. There's usually dependencies. So these studies can't be like applied to, to well to these situations either.

[6:10](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=370s) And then we also have surveys, which we found to be an ineffective predictor of productivity by doing this small experiment with forty-three developers whereby we ask every developer to evaluate themselves relative to the global mean or median in five percentile, uh, buckets from zero to one hundred.

[6:30](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=390s) And then we compare that to their measured productivity. We'll get into what that means later. But what we found is that asking someone how productive they think they are is almost as good as flipping a coin.

[6:40](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=400s) There's very little correlation, right? And so we found that people misjudged their productivity by about thirty percentile points.

[6:49](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=409s) Only one in three people actually estimated their productivity within their quartile, o- one quartile. And I think surveys are great. They're valuable for surfacing, you know, morale and other issues that cannot be derived from metrics, but surveys shouldn't be used to measure developer productivity, much less the impact of AI on developers

[7:10](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=430s) for pr- p- productivity cases. You can measure it to kinda see how happy they are using AI or whatever, I suppose. Um, great. So now let's dive into our methodology.

[7:21](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=441s) So in an ideal world, you would have an engineer who writes code, and this code is evaluated by a panel of ten or fifteen experts who separately, without knowing what every person is, uh, answering, evaluates that code based on quality, maintainability, output, how long would this take me, how good is it, right?

[7:41](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=461s) So k- kind of like a, a bucket of questions. And then, um, what happens is that you aggregate those results, and we found two things. The first one is that this panel actually agrees with one another.

[7:54](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=474s) So it turns out that one engineering expert agrees with the other engineering exp- expert when they're talking about an objective code in front of them. And secondly, and probably most importantly, is that you can use this to predict reality, and reality is predicted by a panel like this.

[8:11](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=491s) The problem then is that this is very slow, it's not scalable, it's expensive. Um, and so what we did is we built a model that essentially automates this, correlates pretty well, it's fast, it's scalable, and it's affordable.

[8:24](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=504s) The way it works is it plugs into Git, and then the model analyzes the source code changes of every commit and quantifies them based on a bunch of these dimensions.

[8:35](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=515s) And then since every commit has a unique author, a unique SHA, a unique timestamp, then you can kinda understand that, uh, okay, the productivity of a team is basically the functionality of the code they deliver across time, not the lines of code, not the whatever commits, but the func-- like what that code is doing, right?

[8:55](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=535s) And so then you can kinda put this in a dashboard and, uh, overlay it across time and get something similar to this. [clears throat]

[9:05](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=545s) Great. So now let's dive into some of our results.

[9:10](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=550s) So here in September is when this company implemented AI. This is a team of about a hundred and twenty developers, and they were piloting whether they wanted to use, uh, you know, AI in their kinda regular workflow.

[9:22](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=562s) And we have here, um, these bars, and every bar is the sum total of the output done in that month

[9:30](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=570s) using our methodology, not lines of code. And we can see that in green it's added functionality, in gray it's removed, in blue is refactoring, and in orange is rework.

[9:41](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=581s) And so rework versus refactoring, they both alter existing code, but rework alters code that's much more recent, meaning it's wasteful. Refactoring could be wasteful, could be not wasteful. And so from the get-go, you see that by implementing AI, you get a bunch more of rework.

[9:59](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=599s) What happens is that you feel like you're delivering more code because there's just like more volume of code being written, more commits, more stuff being pushed. But not all of that is actually useful.

[10:08](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=608s) To be clear, I think there... I mean, based on this chart and overall, there is a productivity boost of about fifteen to twenty percent, but then a lot of the gains you're seeing are, uh, basically this kind of rework, which is a bit, you know, misleading.

[10:24](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=624s) So if I could summarize it into one chart with many discrepancies, it would be something like this.

[10:31](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=631s) So with AI coding, you generate or you increase your productivity by roughly thirty, forty percent. Like, you're delivering more code. However, you gotta go back and kinda fix some of the bugs that code introduced and kinda, you know, fix the, the, the mess that the AI made, which in turn gives you an average productivity gain across all

[10:50](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=650s) industries, all sectors, everything, of roughly about fifteen to twenty percent.

[10:56](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=656s) There's a lot of nuance here, which we're gonna see in just a second.

[11:03](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=663s) So here we have two violin charts, and they plot the distributions of the gains in productivity from using AI. And so kind of like the y-axis is the gains.

[11:15](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=675s) It starts from minus twenty percent, take note, and then it goes up. And here we have kind of four pieces of data being shown. In blue is low complexity tasks,

[11:26](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=686s) and in red is high complexity tasks. And kind of like your left, uh, the, the chart to the left is greenfield tasks, the chart to the right is brownfield tasks.

[11:38](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=698s) So right from the get-go, the first conclusion we have is that, sure, it, it seems like AI performs better in coding with simpler tasks. That's good. It's proven by data.

[11:48](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=708s) That's awesome. The second thing we see is that, hey, it sounds like for low complexity greenfield tasks, there is a much more elongated distribution and a much higher distribution on average.

[12:02](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=722s) Keep in mind that this is for enterprise settings. This doesn't apply for kinda like personal projects or vibe coding something for yourself from scratch. The improvements there would be much bigger.

[12:11](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=731s) This is kind of for like real world working company settings.

[12:16](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=736s) And the third thing we see is that if you look at the high complexity tasks, I mean, they're lower than the low complexity ones on average in terms of the distribution, but also in some cases,

[12:27](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=747s) they are more likely to decrease an engineer's productivity.

[12:32](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=752s) Now, this decrease could be for many things, many reasons, but that's kinda what we see in the data, right? The underlying causes are still not super clear to us.

[12:43](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=763s) If we translate this to a chart like this, which is a bit more digestible, you have, uh, in, i- in the bars and the columns

[12:52](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=772s) kind of like the average or the median gain, and then the line represents the interquartile range. So the bottom of the line is the twenty-fifth percentile, and the top of the line is roughly seventy-fifth percentile.

[13:04](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=784s) And so here it's very clear to see how we have, you know, more gains from low complexity tasks, less gains from, uh, high complexity tasks, and then brownfield, it's harder to leverage AI, um, to make increases in productivity there compared to greenfield.

[13:22](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=802s) So if there is maybe a slide that you could show to your leadership team, it could be this one or it could also be this one. So here I have a matrix really simplifying things.

[13:31](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=811s) You know, reality is a bit more difficult than this, but here we have kinda on one axis task complexity, low and high, on the other one, project maturity, greenfield versus brownfield.

[13:40](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=820s) Kind of we see that, hey, low complexity greenfield, thirty to forty percent gains, right, from AI.

[13:47](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=827s) High complexity, but greenfield, more modest gains, ten to fifteen.

[13:52](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=832s) Brownfield and low complexity, pretty good, fifteen to twenty percent. And most importantly, high complexity brownfield tasks, zero to ten percent.

[14:02](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=842s) These are orientative guidelines based on what is seen... what we see in the data. And I forgot to mention, uh, this slide has a sample size of hundred and thirty-six teams across twenty-seven companies, so pretty representative.

[14:14](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=854s) Um, and then that's gonna derive-- or this chart is derived from that data.

[14:21](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=861s) Then here we have a similar matrix, except at the bottom we have language popularity. So in low we have examples such as COBOL, Haskell, Elixir, really kinda obscure, obscure stuff, and high is things like Python, Java, you know, JavaScript, TypeScript.

[14:38](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=878s) And what we see is that AI doesn't really help even with low complexity, uh, tasks for low popularity languages. It can help a bit, but it's not terribly useful, and what ends up happening is that people just don't use it because if it's only helpful two times out of five, you're just not gonna use it very often.

[14:54](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=894s) What's funny or interesting is that for low language popularity and complex tasks, AI can actually decrease productivity because it's so bad at coding in COBOL or Haskell or Elixir that it just makes you slower, right?

[15:06](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=906s) Granted, this isn't very, uh... Like, this happens, but it may be five or ten percent of the kinda global development work, if that, right?

[15:15](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=915s) Most of the development work is probably somewhere in the lang- in the high language popularity kinda part of the chart. And here you have gains between twenty percent for the low complexity and ten to fifteen percent for the high complexity.

[15:35](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=935s) So now moving into something a bit more theoretical, less empirically proven, but more so kind of like what we're seeing in, in, in the data, right? This is like an illustrative chart which has kinda productivity gain from AI on the Y-axis and a logarithmic scale of the codebase size, right, from one thousand lines of code to ten

[15:54](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=954s) million on the X-axis. And we see that as the codebase size increases, the gains you get from AI decrease sharply, right? And I think most codebases nowadays are kinda somewhere in the...

[16:06](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=966s) depending on, on your use case, right? But they're bigger than a thousand lines of code unless you are a YC startup or something that's like kinda spun out a couple months ago, right?

[16:15](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=975s) And that's because, you know, there's three, three reasons for this really. Context window limitations. We're gonna see in a second how performance decreases even with larger context windows. The signal-to-noise ratio is...

[16:27](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=987s) kinda confuses the, the model, if, if you will. And then of course, larger codebases have more dependencies and more domain-specific logic present.

[16:38](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=998s) And so then borrowing work from this paper called NoLiMa, which shows you on a scale of zero to one hundred how LLMs perform on coding tasks, you see that as context length increases from one thousand to thirty-two thousand tokens,

[16:56](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1016s) performance decreases. And so we see all these models here. For example, Gemini 1.5 Pro has a context window of two million tokens, and you might think, "Whoa, I can just throw my entire codebase into it and it's gonna retrieve and then code perfectly," right?

[17:09](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1029s) And what we see is that even at thirty-two thousand tokens, it's already showing a decrease in performance from ninety percent to about fifty percent, right? So what's gonna happen when you move from thirty-two to sixty-four or one twenty eight, right?

[17:22](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1042s) You're gonna see really, really poor performance here.

[17:26](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1046s) And so in short, AI does increase developer productivity. You should use AI for most cases, but it doesn't increase the productivity of developers all the time and equally. It depends on things like task complexity, codebase maturity, language popularity, codebase size, and also con- context length.

[17:48](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1068s) Thank you so much for listening. If you'd like to learn more about our research, you can access our research portal, which is softwareengineeringproductivity.stanford.edu. You can also reach me, uh, by email or LinkedIn.

[18:01](https://www.youtube.com/watch?v=tbDDYKRFjhk&t=1081s) Super happy to talk about this topic at any time. Thank you so much. [audience applauding] [upbeat music]
