AI Engineer Summit 2023
The AI Evolution
Loading slides…
Read the talk
The AI Evolution: From Code Completion to Shared Software Creation
Copilot’s early lessons connect interface design, latency and evaluation to a larger question: how should programming change when AI becomes a collaborator?
From a talk by Mario Rodriguez
Before you start: Familiarity with code editors, pull requests and basic software delivery workflows is helpful; no machine-learning background is required.
What would an automated programmer actually look like?
What would make an automated programmer useful inside a developer’s working day? At the time of this talk, Mario Rodriguez oversaw GitHub’s productivity products: repositories, pull requests, Issues, Projects, mobile and the CLI. He had also taken responsibility for Copilot and GitHub’s AI strategy that January, while crediting the people who created Copilot before him. That breadth matters: generating code was only one part of the workflow the product had to fit.
Rodriguez traces a catalyst for Copilot to an internal paper circulated in August 2020, An Automated AI Programmer: Fact or Fiction?, written by Oege, Albert and other colleagues. The question preceded the ChatGPT boom. It left room for both ambition and skepticism about what an AI programmer could become.
He calls that tension polarity: alternatives that are interconnected, rather than a simple choice between A and B. A dinosaur riding a train supplies the playful illustration. An adult sees fiction; his young children treat the dinosaur as real. Inside GitHub, the less playful version was a continuing debate about the product’s shape. An early attempt to put it in the pull-request flow worked poorly. Traction came later, leading to the 2021 technical preview; general availability followed in June 2022. Rodriguez describes that initial release as the world’s first AI programmer operating at scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A growing business, with flow as the product goal
Rodriguez reports adoption by over 20,000 organizations and more than one million developers; the contemporary public developer figure counted activations, not established monthly active or paying users. GitHub’s reported 46% average code contribution concerned Copilot usage across languages, with Rodriguez describing tab-accepted completions and noting that multiline effectiveness varies by language. That percentage measures contributed code, not a productivity speedup or a share of all software written.
He also rejects a recent report that Copilot was losing money. After asking the audience whether annualized revenue exceeded $25 million and then $50 million, Rodriguez reports that Copilot had exceeded $100 million in ARR. He withholds a more current figure, citing disclosure constraints associated with the SEC. The revenue claim and denial of losses are his account; the talk does not provide a profitability calculation.
Commercial success is not his definition of product success. GitHub is building for developers, and Rodriguez connects that purpose to developer happiness, including a brief callback to the previous day’s Lindy, Flo and Milton discussion. Accepted lines of code are a useful signal, but the deeper objective is to remove boilerplate and the work that breaks concentration.
The desired experience is flow. Completion, inline chat and other possible interaction modes serve that goal; a chat box is only one option. The product should let a developer keep thinking about the software instead of repeatedly switching attention to the assistant. Rodriguez’s premise is that happier developers write better software.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why completion worked
The first ingredient in Copilot V1 was ghost text. With Codex, the original shipping model, a suggestion could emerge from a natural-language comment or from code already being typed. Showing the possible continuation inside the editor made the interaction a small decision: is this what I want? The developer could inspect a proposal without leaving the code.
For example, a JavaScript comment can express a local intent before an implementation exists:
javascript
// Return the total price of the items in the cart.
A proposed continuation could be:
javascript
// Return the total price of the items in the cart.
function cartTotal(items) {
return items.reduce((total, item) => total + item.price * item.quantity, 0);
}
The implementation remains a suggestion until the developer accepts it. The comment supplies intent, while the placement keeps the proposed code beside the work it would change.
The second ingredient was speed: waiting seconds for a suggestion defeats the attempt to preserve flow. Rodriguez says the completion backend had recently moved from Codex to GPT-3.5 Turbo, describing the historical system at the time of the talk. He reports that Codex latency was below 100 milliseconds many times. The third ingredient was model capability: ghost text and low latency became useful together because the model could produce substantially better continuations than GitHub’s earlier experiments.
The fourth ingredient was prompt engineering, important enough that Rodriguez turns the point into a hiring invitation. Producing a quick demo and engineering a product that companies will buy and millions of people can use are different tasks. His four ingredients reinforce one another:
| Ingredient | Contribution |
|---|---|
| Ghost text | Put a proposal at the point of work |
| Low latency | Preserve the developer’s momentum |
| Model capability | Make the continuation worth considering |
| Prompt engineering | Shape the model’s behavior for the product |
Interaction design can make or break an AI application. Choosing a modality is part of building the product, not a cosmetic step after selecting a model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Semantics, global serving and quality scorecards
Product-market fit increases the pace rather than easing the workload. Internal questions, external demands and pressure to ship all accelerate. The first technical lesson is that syntax is not software: being able to read Java does not make someone a Java developer. Likewise, recognizing valid syntax does not give a model the semantic understanding needed to generate the right code. That understanding requires additional engineering.
A global product also needs a global serving footprint. Rodriguez describes deployments across Japan, Europe and North America, with multiple data centers supporting responsiveness. He associates that deployment footprint with time to first byte of roughly 150 milliseconds many times. This is distinct from the earlier Codex latency figure: neither claim specifies a workload, percentile or measurement window, so they do not establish a combined end-to-end guarantee.
Quality needs an equally deliberate feedback loop. A change can pass offline evaluations and then perform poorly in production. Offline and online scorecards make that disagreement visible: offline tests assess a candidate before deployment, while online evaluations reveal what happens in actual use. Maintaining both gives the team a way to iterate quickly without treating a good offline result as proof of production quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trust becomes a product requirement
Trust accumulates slowly and can disappear suddenly—the difference between a coconut tree growing and a coconut falling. For an AI product sold to businesses, Rodriguez identifies three obligations:
- Security and data handling. He advises against storing prompts or customer data at rest and against training on customer data without approval. These are his recommendations for builders, rather than a statement of a universal GitHub retention policy.
- Legal protection. He points to customer indemnification and ongoing discussions about regulation. The contemporaneous Copilot Copyright Commitment provided conditional commercial copyright protection, including requirements to use guardrails and content filters—not blanket protection against every possible problem.
- Responsible AI. The system must behave in ways that earn users’ trust and avoid harm. Rodriguez introduces this point through a secondhand account of a prior-day OpenAI demo that he understood to have failed on trustworthiness and responsible-AI grounds.
GitHub’s own investment in responsible AI follows the same purpose: helping users without harming them. For Rodriguez, advancing human progress is a constraint on how the product should work, not merely a reason to build it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From procedures to goals and constraints
The next question is whether better completion is enough to transform programming. Rodriguez points to Bret Victor’s The Future of Programming as a way to revisit problems that remain unresolved. Today’s dominant interaction still revolves around procedures, text files and sequential instructions: the programmer specifies how to do things.
The GitHub CEO’s Octo Arcade snake game illustrates that pattern. It was written in JavaScript; Rodriguez jokes about having Copilot convert it to TypeScript. AI can help produce or translate those instructions, but the programmer is still working within the same procedural environment. His proposed next step is to operate on goals and constraints, which would require changes to the REPL and the surrounding programming environment.
The desired request is at the application level: create an app using Remix, Redis and Postgres; include a state machine and a chosen UI library; host it in the user’s cloud. That combines an outcome with architectural and deployment constraints, leaving more implementation work to the system. Once that level of assistance exists, the developer could turn to evaluating vector databases and embedding models for the AI product. This is a proposed direction for programming, not a demonstrated app-generation result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A semantic workspace needs retrieval and summarization
Rodriguez calls this direction a semantic workspace, then asks what it would mean for AI to reason about code. Consider a concrete task: add a feature flag to a codebase you do not know. Before changing anything, what would you search for? Forming that query requires bringing together prior experience, the task’s meaning and a guess about how the repository organizes the relevant behavior.
His analogy is that the audience has just performed retrieval-augmented generation—RAG—inside its own heads, assembling relevant knowledge into a query string. He considers contemporary AI far behind people at this task. The engineering challenge is therefore not simply generating more text: it is producing useful summaries quickly enough, and with enough fidelity, to support reasoning about the code.
The next examples concern perception. Rodriguez describes a gray car that viewers perceive as blue, then invokes the familiar redness of strawberries. His explanation is that the brain supplies expected context instead of merely reporting isolated visual inputs. The connection to programming is the role of interpretation: useful understanding depends on organizing what is present and relating it to what is already known.
He then points to a project Amelia developed at GitHub, asking the audience to consider breaking code into parts that can be reasoned about and summarized rapidly. The proposed research direction combines model architecture, advances in RAG and visualizations that help developers reason over code. Visualization is part of the reasoning environment, alongside retrieval and summarization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An immersive experience across the platform
The closing collaboration vision is an immersive experience that people can enter and leave together with AI. Rodriguez gestures toward making meetings more transparent, but offers no product sneak peek or specific meeting capability. He skips a planned fourth prompt on security while keeping it a stated priority, leaving further possibilities undeveloped.
That vision reaches beyond GitHub as a home for open source or a version-control service. Actions supplies CI/CD; Packages, Copilot and Codespaces cover other parts of development. Rodriguez closes with the ambition to infuse that end-to-end platform with AI. The scope is the full path from deciding what to build to collaborating, checking and delivering it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Bret Victor's companion references connect his talk to earlier ideas about programming languages and computing environments.
Further reading
Amelia Wattenberger's interactive exploration of repository structure uses visual fingerprints to help readers navigate unfamiliar code.
- Code AtlasArticle
A GitHub Next exploration by Amelia Wattenberger of combining flexible LLM reasoning with structured computation.
Microsoft's announcement explains commercial copyright protection for Copilot customers and the conditions attached to that protection.
Full transcript
Loading transcript…