AI Engineer World's Fair 2024

The AI emperor has no DAUs: why most devs still don't use code AI

Quinn Slack· CEO and co-founder, Sourcegraph18:45

Loading slides…

Read the talk

The AI emperor has no DAUs

Code AI can be useful without becoming a habit. Quinn Slack examines the gap between impressive capabilities and daily use, then shows how to design products developers can check, correct, and keep using.

From a talk by Quinn Slack

An enthusiastic room is not the market

A room full of AI engineers is a dangerous place to judge developer adoption. Quinn Slack, Sourcegraph’s CEO and co-founder, approaches the problem as a working programmer: after a stint at Bleacher Report, he has been building code search and code AI since 2013. But writing software is only part of the job. If the product never becomes useful enough for people to adopt, the implementation does not accomplish much. His experience building Cody supplies both the mistakes and the attempts to correct them.

Purple slide listing three topics: most developers don’t use code AI, why this should scare you, and mistakes and tips from building Cody at Sourcegraph.Open full source frame
Most developers don’t use code AI: the problem and lessons from building Cody.

Asked whether they have used any code-AI tool while coding in the past week, the San Francisco audience responds enthusiastically. That is precisely the sampling problem. This room contains unusually committed users; it cannot stand in for the worldwide developer population. Slack says the public adoption numbers he is about to examine broadly match the private information he encounters through his work.

0:160:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

What adoption statistics actually measure

The first impressive number is a GitHub survey reporting 92% use of AI coding tools among 500 US developers. That is a sample of developers at large employers, not a worldwide census. Its descriptions also vary between workplace use and use at work or outside it. The broad definition matters: trying a tool is a different event from depending on it every day. A second figure, 70% seeing benefits from AI coding tools, concerns perceived or anticipated value rather than demonstrated adoption.

Microsoft’s January 2024 earnings call reported over 1.3 million paid GitHub Copilot subscribers. Payment is stronger evidence than interest, but it still does not tell us whether a subscriber uses the product.

Slack then cites roughly 935,000 developers who received a Copilot suggestion in a GitHub study. Receiving ghost text does not require accepting it. He interprets the figure as yearly activity and infers substantially lower monthly and daily use. The matching report concerns Copilot’s first year around its 2023 publication; it does not establish a contemporary 2024 yearly-active total or disclose the MAU and DAU figures inferred here.

SignalWhat it establishesWhat remains unknown
Survey responseReported use or perceived benefitSustained behavior
Paid subscriptionA purchased entitlementActual activity
Suggestion receivedExposure to an outputAcceptance or usefulness
Monthly or daily activityUse within a defined intervalValue delivered by that use

The event and observation window are part of the metric. Without them, an adoption number can sound much stronger than its evidence.

1:492:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:49 · section reference included

A large audience, a narrow habit

The denominator changes the picture again. Slack uses roughly 26 million professional developers, assembled from broadly similar search results, and considers it an undercount. Including students and others who touch code, he loosely estimates 100–150 million people. These are rough population estimates, not a common measurement base with the Copilot figures.

Slack’s best-case estimate is that around 5% of professional developers use code AI. His illustration is a room of twenty developers in which nineteen do not use it. That estimate is his judgment about the market, not a worldwide adoption rate established by the preceding subscription and study counts.

Even among users, the dominant habit is narrow: autocomplete supplies ghost text for the next fragment of code. The slide makes this more specific: 63% use only autocomplete, labeled as Sourcegraph Cody usage data from June 2024. That condition matters; it is not a measurement of every code-AI product. Yet the capabilities already extend to writing whole files, answering questions, and fixing bugs. Getting those capabilities into everyday use requires changing the behavior of developers outside the early-adopter room.

Slide showing “~5% of pro devs use code AI” and “63% use only autocomplete,” with Sourcegraph Cody usage data, June 2024, beneath the latter.Open full source frame
Best-case estimate: about 5% of professional developers use code AI; 63% use only autocomplete.
3:233:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:23 · section reference included

Why developers decline code AI

Sourcegraph’s internal experience with Cody produces a less exotic list of adoption barriers than the technology might suggest. Slack presents them in descending order of prominence:

  • No compelling reason: Some developers have not tried ChatGPT, or have barely encountered it. A new capability is not automatically a reason to change a workflow.
  • Wrong answers: An initial failure can settle the developer’s opinion before they learn how to obtain useful results.
  • Little personal benefit: A developer may try the tool without finding a task where it helps enough to matter.
  • Organizational and price barriers: The company has not adopted the tool, or the cost is difficult to justify.
  • Security, privacy, and legal concerns: Slack reports encountering these objections less often than before.

His response to the middle objections is that learning to use the tool is itself part of making it useful. His joke about widespread adoption creating mutually assured destruction explains a perceived normalization of risk; it does not resolve the underlying legal questions.

5:045:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:04 · section reference included

Where sustainable AI revenue comes from

Weak adoption matters beyond the companies selling coding assistants. Startup funding, enterprise plans, and even investors’ retirement and options accounts depend on expectations of future demand. To understand that dependence, trace the money through the stack. Foundation-model companies such as OpenAI and Anthropic supply models; infrastructure companies serve inference and send some revenue upstream. Sometimes those functions belong to the same company.

Experimentation can generate inference bills and revenue before anyone has established sustained use. Next come individual users, illustrated by someone paying twenty dollars a month, followed by developers using the product at work. The durable business appears when enterprise customers renew because the software continues to earn its place.

Slack illustrates that distinction with a hypothetical JPMorgan contract: a million dollars in the first year, followed by a two-million-dollar renewal. The first sale can express hope; the renewal should reflect value experienced in use. An infrastructure investment may rest on the belief that an enterprise will eventually make payments like that years into the future.

Add Nvidia and other chipmakers above the model and infrastructure layers, and the amount of investment depending on the eventual user grows larger still. The enterprise developer doing ordinary work may be less glamorous than the technology powering the system, but that developer’s continued use is what supports the revenue chain.

6:116:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:11 · section reference included

Sizing the revenue gap

Slack estimates total recurring code-AI revenue at about $300 million ARR, drawing on internal information and GitHub disclosures at the conference. That could support a substantial standalone business. The next question is how much reaches the companies supplying its models and infrastructure.

Slack reports that Sourcegraph spends less than 10% of its revenue on AI inference, despite not having optimized that spending heavily. Applying a 10% share to his $300 million market estimate yields $30 million annually for model and infrastructure providers. This is an extrapolation from Sourcegraph’s economics, not an observed total for upstream suppliers. The distance from that amount to revenue supporting multibillion-dollar valuations is the financial version of the adoption gap.

Using an approximate $36 billion in annual Salesforce revenue, Slack puts his $300 million code-AI ARR estimate at one one-hundred-and-twentieth of Salesforce’s scale. For historical context, Salesforce reported $34.86 billion for fiscal 2024; the talk does not identify the period behind its rounded figure, and annual recognized revenue differs from ARR. The comparison is about order of magnitude.

One possible reading is a hype bubble that will collapse. Slack acknowledges that possibility, but his own daily use points him toward a different conclusion: the utility is real and the market is early. Product builders must therefore plan for the work of growing usage, rather than assume widespread adoption has already happened.

8:268:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:26 · section reference included

Cody’s context advantage

Sourcegraph arrived at coding assistance through code search. After a decade of helping developers navigate company code, it already had access to the material an assistant would need to answer repository-specific questions. Code context turned an existing search product into a foundation for Cody.

The Cody described here combines autocomplete, inline edits, unit-test generation, and chat. Chat is where Slack sees the strongest differentiation: company code can help answer why something is broken, how to change it, or where to begin. Those questions require more than plausible code in isolation; they require connecting the request to the system the developer actually maintains.

Slack says Sourcegraph’s customers include four of the five largest US banks and most of the major technology companies grouped under FAANG or MAÑANA. He also describes Sourcegraph as second in code-AI revenue behind GitHub Copilot. These company claims establish the commercial experience behind the lessons that follow, rather than independently measured product superiority.

Diagram linking Code Search to Cody, with Search, Nav, Batch Changes, Autocomplete, Edit/Commands, and Chat. Below, a yellow callout over product screenshots says “We’re the #2 code AI product in revenue… so we try harder!”Open full source frame
Cody combines code search with autocomplete, edits, and chat.
10:0610:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:06 · section reference included

Test usefulness instead of following hype

Two lessons frame Sourcegraph’s experience: hype can fool builders and customers alike, and autocomplete was an unusually fortunate first feature. To resist the first problem, Slack starts with a demanding test: the people building the product should use it every day. Customer enthusiasm can otherwise conceal the absence of a habit. A compliment costs little; repeatedly choosing the product during real work is stronger evidence.

For about a year, customers repeatedly requested fine-tuned models. Salespeople relayed the request, while competitors advertised the capability as coming soon. Slack’s interpretation is that customers were naming a fashionable mechanism for an underlying need: they wanted the product to work well. The useful response is to understand that need and explain the delivered benefit without relying on the word AI.

Slack claims developer productivity gains of 20–30%, without specifying a measurement method in the talk. He then turns the claim into a harder business question. If the gain is so valuable, why sell the tool instead of buying a software outsourcing firm and capturing the productivity improvement directly? It resembles the question one might ask someone selling stock-picking newsletters: why not invest on those picks?

Sourcegraph was considering that acquisition experiment, not reporting a completed purchase or a measured result. For a well-funded builder, willingness to put a few million dollars behind the experiment becomes a test of confidence in the product’s value.

Public attention is a weaker test. A viral product tweet can collect enormous engagement without producing daily active users. Slack imagines a social-media display that would reveal the promoted product’s DAU alongside the engagement counters. Until such a display exists, builders should mentally separate the two rather than treat competitors’ publicity as evidence of habitual use.

11:2211:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:22 · section reference included

Frequency and verification cost

Sourcegraph evaluates features with a four-box framework: how often is the feature used, and how quickly can the developer tell whether its output is correct? The desirable corner combines frequent use with easy verification. Autocomplete fits unusually well because typing continually creates opportunities to help, while a short suggestion can often be judged at a glance. Slack describes checking a suggestion in seconds or milliseconds, including roughly 100 milliseconds; this is an illustration of low verification effort, not a timed evaluation.

Other interfaces have a harder path:

FeatureOpportunity for useVerification burden
AutocompleteDuring typingInspect a short suggestion
ChatWhen a question arisesRead and assess a longer answer
Inline editsWhen requesting a changeReview the proposed edit

Chat is less frequent, and its long responses take more work to vet. Inline edits—including the Meta+K and Option+K interactions Slack mentions in Cody and Cursor—face similar constraints. A useful future interface may look substantially different from the current one.

Agents raise the same question at a larger scope. Slack asks who has used a code-AI agent to merge a pull request in the previous week. He sees agents as part of the future, but judges their product fit at the time of the talk to be immature. The design task is to move features toward frequent use and inexpensive checking, rather than assume that greater autonomy alone makes them more adoptable.

13:5114:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:51 · section reference included

Searching for the next coding modality

What comes after autocomplete remains an open product question. Slack is skeptical that next-edit suggestions or Copilot++ necessarily supply the answer. He is interested in Steve Yegge’s description of chat-oriented programming, or CHOP: a developer keeps a Cody or ChatGPT conversation open throughout the coding session, treating it as an ongoing exchange rather than a sequence of isolated questions.

The memorable detail is who uses it. Slack sees the pattern among both Gen Z developers and disgruntled programmers, and welcomes its use across generations. That observation suggests a behavior worth understanding across experience levels, not simply a generational replacement of old habits. The next successful mode of working may emerge from patterns that current product categories do not yet capture.

15:2515:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:25 · section reference included

Make explicit workflows work before automating

Autocomplete can teach the wrong lesson because it triggers automatically with every keystroke. Its success does not mean every new feature should begin with hidden decisions and automatic actions. Make the manual, explicit workflow useful first.

For context and agent features, that means a concrete sequence:

  1. Let developers select context. In chat, make explicit @mentions work before trying to infer and insert all relevant context automatically.
  2. Keep correction in the editor. Put an agent where the developer can directly change its incorrect output, rather than requiring a move to Codespaces or another separate interface.
  3. Add automation after the workflow works. First make the result useful and the repair path easy. That work may occupy the team for a long time before additional magic is warranted.

Explicit context selection gives the developer a way to steer the answer. Local correction gives them a way to recover when steering is insufficient. Both lower the cost of continuing to use an imperfect system.

16:1416:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:14 · section reference included

Actively lower inflated expectations

Expectation setting is another part of the interface. People mentally add hype even when a pitch tries to remove it, so Slack recommends deliberately lowering expectations. He uses a Sam Altman post about ChatGPT as the example: the displayed post warns about limitations, reliability, and the misleading impression that a compelling interaction can create.

Slack loosely recalls the post as about a year old and guesses that ChatGPT already had tens of millions of daily users; that is his recollection, not an established DAU disclosure. His prescription for less validated products is rhetorical: de-hype ten times more intensely. A builder should be especially careful about promises when the product has less evidence behind it. Understatement can also be effective marketing because it makes the useful result easier to believe.

Sam Altman post over a Sourcegraph slide says ChatGPT is incredibly limited, warns against relying on it for anything important, and notes work remaining on robustness and truthfulness.Open full source frame
A Sam Altman post warns that ChatGPT can create a misleading impression of greatness.
17:0017:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:00 · section reference included

Become a daily user

Slack’s final product test is blunt: become a daily active user yourself, or kill the product. This reconnects the builder’s own behavior to the economics of the entire stack. Sourcegraph’s payments to Anthropic, OpenAI, Mistral, and Fireworks depend on Cody generating substantial revenue. Those providers, in turn, have a stake in whether applications become useful businesses.

Applications, infrastructure, and model providers therefore share an interest in reducing hype and increasing actual use. The work starts close to home: build something that earns a place in your own daily development, make it easy to check and correct, and give other developers a reason to return to it during theirs.

17:3717:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:37 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…