AI Engineer Summit 2023

The AI Evolution

Mario Rodriguez· VP of Product, GitHub19:32

Loading slides…

Read the talk

The AI Evolution: From Code Completion to Shared Software Creation

Copilot’s early lessons connect interface design, latency and evaluation to a larger question: how should programming change when AI becomes a collaborator?

From a talk by Mario Rodriguez

Before you start: Familiarity with code editors, pull requests and basic software delivery workflows is helpful; no machine-learning background is required.

What would an automated programmer actually look like?

What would make an automated programmer useful inside a developer’s working day? At the time of this talk, Mario Rodriguez oversaw GitHub’s productivity products: repositories, pull requests, Issues, Projects, mobile and the CLI. He had also taken responsibility for Copilot and GitHub’s AI strategy that January, while crediting the people who created Copilot before him. That breadth matters: generating code was only one part of the workflow the product had to fit.

Rodriguez traces a catalyst for Copilot to an internal paper circulated in August 2020, An Automated AI Programmer: Fact or Fiction?, written by Oege, Albert and other colleagues. The question preceded the ChatGPT boom. It left room for both ambition and skepticism about what an AI programmer could become.

Dark slide with a Copilot icon above the text “An automated AI pair Programmer, Fact or Fiction” and a magenta wave along the bottom.Open full source frame
An automated AI pair programmer: fact or fiction?

He calls that tension polarity: alternatives that are interconnected, rather than a simple choice between A and B. A dinosaur riding a train supplies the playful illustration. An adult sees fiction; his young children treat the dinosaur as real. Inside GitHub, the less playful version was a continuing debate about the product’s shape. An early attempt to put it in the pull-request flow worked poorly. Traction came later, leading to the 2021 technical preview; general availability followed in June 2022. Rodriguez describes that initial release as the world’s first AI programmer operating at scale.

0:541:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:54 · section reference included

A growing business, with flow as the product goal

Rodriguez reports adoption by over 20,000 organizations and more than one million developers; the contemporary public developer figure counted activations, not established monthly active or paying users. GitHub’s reported 46% average code contribution concerned Copilot usage across languages, with Rodriguez describing tab-accepted completions and noting that multiline effectiveness varies by language. That percentage measures contributed code, not a productivity speedup or a share of all software written.

He also rejects a recent report that Copilot was losing money. After asking the audience whether annualized revenue exceeded $25 million and then $50 million, Rodriguez reports that Copilot had exceeded $100 million in ARR. He withholds a more current figure, citing disclosure constraints associated with the SEC. The revenue claim and denial of losses are his account; the talk does not provide a profitability calculation.

Commercial success is not his definition of product success. GitHub is building for developers, and Rodriguez connects that purpose to developer happiness, including a brief callback to the previous day’s Lindy, Flo and Milton discussion. Accepted lines of code are a useful signal, but the deeper objective is to remove boilerplate and the work that breaks concentration.

The desired experience is flow. Completion, inline chat and other possible interaction modes serve that goal; a chat box is only one option. The product should let a developer keep thinking about the software instead of repeatedly switching attention to the assistant. Rodriguez’s premise is that happier developers write better software.

3:313:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:31 · section reference included

Why completion worked

The first ingredient in Copilot V1 was ghost text. With Codex, the original shipping model, a suggestion could emerge from a natural-language comment or from code already being typed. Showing the possible continuation inside the editor made the interaction a small decision: is this what I want? The developer could inspect a proposal without leaving the code.

For example, a JavaScript comment can express a local intent before an implementation exists:

javascript

// Return the total price of the items in the cart.

A proposed continuation could be:

javascript

// Return the total price of the items in the cart.
function cartTotal(items) {
  return items.reduce((total, item) => total + item.price * item.quantity, 0);
}

The implementation remains a suggestion until the developer accepts it. The comment supplies intent, while the placement keeps the proposed code beside the work it would change.

The second ingredient was speed: waiting seconds for a suggestion defeats the attempt to preserve flow. Rodriguez says the completion backend had recently moved from Codex to GPT-3.5 Turbo, describing the historical system at the time of the talk. He reports that Codex latency was below 100 milliseconds many times. The third ingredient was model capability: ghost text and low latency became useful together because the model could produce substantially better continuations than GitHub’s earlier experiments.

The fourth ingredient was prompt engineering, important enough that Rodriguez turns the point into a hiring invitation. Producing a quick demo and engineering a product that companies will buy and millions of people can use are different tasks. His four ingredients reinforce one another:

IngredientContribution
Ghost textPut a proposal at the point of work
Low latencyPreserve the developer’s momentum
Model capabilityMake the continuation worth considering
Prompt engineeringShape the model’s behavior for the product

Interaction design can make or break an AI application. Choosing a modality is part of building the product, not a cosmetic step after selecting a model.

6:266:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:26 · section reference included

Semantics, global serving and quality scorecards

Product-market fit increases the pace rather than easing the workload. Internal questions, external demands and pressure to ship all accelerate. The first technical lesson is that syntax is not software: being able to read Java does not make someone a Java developer. Likewise, recognizing valid syntax does not give a model the semantic understanding needed to generate the right code. That understanding requires additional engineering.

A global product also needs a global serving footprint. Rodriguez describes deployments across Japan, Europe and North America, with multiple data centers supporting responsiveness. He associates that deployment footprint with time to first byte of roughly 150 milliseconds many times. This is distinct from the earlier Codex latency figure: neither claim specifies a workload, percentile or measurement window, so they do not establish a combined end-to-end guarantee.

Quality needs an equally deliberate feedback loop. A change can pass offline evaluations and then perform poorly in production. Offline and online scorecards make that disagreement visible: offline tests assess a candidate before deployment, while online evaluations reveal what happens in actual use. Maintaining both gives the team a way to iterate quickly without treating a good offline result as proof of production quality.

Slide headed “Get used to a fast pace ...” with six boxes: “Syntax is not software,” “Global Presence,” “Set up scorecards for quality,” “RAG for context,” “Conversational changes everything,” and “Keep shipping.”Open full source frame
Engineering lessons at a fast pace.
8:488:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:48 · section reference included

Trust becomes a product requirement

Trust accumulates slowly and can disappear suddenly—the difference between a coconut tree growing and a coconut falling. For an AI product sold to businesses, Rodriguez identifies three obligations:

  • Security and data handling. He advises against storing prompts or customer data at rest and against training on customer data without approval. These are his recommendations for builders, rather than a statement of a universal GitHub retention policy.
  • Legal protection. He points to customer indemnification and ongoing discussions about regulation. The contemporaneous Copilot Copyright Commitment provided conditional commercial copyright protection, including requirements to use guardrails and content filters—not blanket protection against every possible problem.
  • Responsible AI. The system must behave in ways that earn users’ trust and avoid harm. Rodriguez introduces this point through a secondhand account of a prior-day OpenAI demo that he understood to have failed on trustworthiness and responsible-AI grounds.

GitHub’s own investment in responsible AI follows the same purpose: helping users without harming them. For Rodriguez, advancing human progress is a constraint on how the product should work, not merely a reason to build it.

10:3510:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:35 · section reference included

From procedures to goals and constraints

The next question is whether better completion is enough to transform programming. Rodriguez points to Bret Victor’s The Future of Programming as a way to revisit problems that remain unresolved. Today’s dominant interaction still revolves around procedures, text files and sequential instructions: the programmer specifies how to do things.

The GitHub CEO’s Octo Arcade snake game illustrates that pattern. It was written in JavaScript; Rodriguez jokes about having Copilot convert it to TypeScript. AI can help produce or translate those instructions, but the programmer is still working within the same procedural environment. His proposed next step is to operate on goals and constraints, which would require changes to the REPL and the surrounding programming environment.

The desired request is at the application level: create an app using Remix, Redis and Postgres; include a state machine and a chosen UI library; host it in the user’s cloud. That combines an outcome with architectural and deployment constraints, leaving more implementation work to the system. Once that level of assistance exists, the developer could turn to evaluating vector databases and embedding models for the AI product. This is a proposed direction for programming, not a demonstrated app-generation result.

12:0812:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:08 · section reference included

A semantic workspace needs retrieval and summarization

Rodriguez calls this direction a semantic workspace, then asks what it would mean for AI to reason about code. Consider a concrete task: add a feature flag to a codebase you do not know. Before changing anything, what would you search for? Forming that query requires bringing together prior experience, the task’s meaning and a guess about how the repository organizes the relevant behavior.

His analogy is that the audience has just performed retrieval-augmented generation—RAG—inside its own heads, assembling relevant knowledge into a query string. He considers contemporary AI far behind people at this task. The engineering challenge is therefore not simply generating more text: it is producing useful summaries quickly enough, and with enough fidelity, to support reasoning about the code.

The next examples concern perception. Rodriguez describes a gray car that viewers perceive as blue, then invokes the familiar redness of strawberries. His explanation is that the brain supplies expected context instead of merely reporting isolated visual inputs. The connection to programming is the role of interpretation: useful understanding depends on organizing what is present and relating it to what is already known.

He then points to a project Amelia developed at GitHub, asking the audience to consider breaking code into parts that can be reasoned about and summarized rapidly. The proposed research direction combines model architecture, advances in RAG and visualizations that help developers reason over code. Visualization is part of the reasoning environment, alongside retrieval and summarization.

14:2514:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:25 · section reference included

Designing collaboration beyond the sidebar

Most development interfaces were not designed with an AI collaborator in mind. A chat sidebar fits into the space those interfaces already provide, which helps explain its popularity. But an assistant that participates in getting work done raises a larger design problem than where to put a conversation.

The familiar workflow remains:

  1. Receive an assigned issue.
  2. Meet to discuss the work.
  3. Code with Copilot.
  4. Open a pull request.
  5. Pass the check suites.
  6. Merge and deploy.

Rodriguez describes GitHub’s many check suites with mixed affection. Deployment supplies the joke at the end of the sequence: finish before Friday, unless you embrace the Vercel-and-Guillermo version of deploying whenever you like. Beneath the humor is a workflow in which completion helps one stage while the surrounding collaboration remains largely unchanged.

An interface built for humans and AI together would need to let participants steer the work: the UI, the AI and the other people collaborating on the code. It would also need to distinguish suggestions that can be ignored from changes that must be verified. Steering and verification are central interaction requirements, not chores left outside the assistant. That ambition extends beyond completion, chat or training a model; it concerns how people build software together.

Slide asks about inviting others to participate and enabling suggestions to be easily ignored, steered, and verified, with a speaker inset at lower left.Open full source frame
Collaboration with suggestions that can be ignored, steered, and verified.
16:2516:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:25 · section reference included

An immersive experience across the platform

The closing collaboration vision is an immersive experience that people can enter and leave together with AI. Rodriguez gestures toward making meetings more transparent, but offers no product sneak peek or specific meeting capability. He skips a planned fourth prompt on security while keeping it a stated priority, leaving further possibilities undeveloped.

That vision reaches beyond GitHub as a home for open source or a version-control service. Actions supplies CI/CD; Packages, Copilot and Codespaces cover other parts of development. Rodriguez closes with the ambition to infuse that end-to-end platform with AI. The scope is the full path from deciding what to build to collaborating, checking and delivering it.

18:1418:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:14 · section reference included

Resources

From the talk

  • Amelia Wattenberger's interactive exploration of repository structure uses visual fingerprints to help readers navigate unfamiliar code.

  • Code AtlasArticle

    A GitHub Next exploration by Amelia Wattenberger of combining flexible LLM reasoning with structured computation.

  • Microsoft's announcement explains commercial copyright protection for Copilot customers and the conditions attached to that protection.

Transcript download
Full transcript

Loading transcript…