AI Engineer World's Fair 2024
The Making of Devin
Loading slides…
Read the talk
The Making of Devin
A name-game website, a production search bar, and a shared development machine show how coding agents turn instructions into software—and where human judgment still matters.
From a talk by Scott Wu
Before you start: Basic familiarity with React state, Git pull requests, and development environments will help with the examples.
A name game built through feedback
Remembering names and faces is hard; a small practice game could help. Preparing for the AI Engineer World's Fair that morning, Scott Wu asked Devin to build a mobile-friendly website from a TSV containing the event speakers’ names and photographs. Each round should show two different random faces and one person’s name, then ask the player to choose the matching face.
The request starts a development process, not just a code response. Devin has the tools a human engineer would use and first produces a plan. That plan can change as new information or feedback arrives. In the demonstration, Devin creates a directory, starts a React application, reads the TSV, and builds the website.
After some minutes, Devin deploys a first version. It is close, but the photographs still display their names, giving away the answer. Wu acknowledges that he may not have specified the requirement precisely enough. Clicking an answer produces the expected correct-answer feedback, but the interface needs another pass.
Wu supplies that pass in plain English: hide both photograph labels until the player answers, restyle the Play Again button, and add a streak counter that returns to zero after a wrong answer. These requests refine both presentation and application state. In React, the requested answer and streak behavior can be expressed with a small state transition:
jsx
import { useState } from "react";
export function NameGameRound({ faces, target, onNext }) {
const [answerId, setAnswerId] = useState(null);
const [streak, setStreak] = useState(0);
const answered = answerId !== null;
function choose(face) {
if (answered) return;
setAnswerId(face.id);
setStreak(count => face.id === target.id ? count + 1 : 0);
}
function playAgain() {
setAnswerId(null);
onNext();
}
return (
<section>
<h1>Which person is {target.name}?</h1>
<p>Streak: {streak}</p>
<div>
{faces.map(face => (
<button
key={face.id}
disabled={answered}
onClick={() => choose(face)}
>
<img
src={face.photoUrl}
alt={answered ? face.name : "Choose this person"}
/>
{answered && <span>{face.name}</span>}
</button>
))}
</div>
{answered && (
<>
<p>{answerId === target.id ? "Correct!" : "Incorrect"}</p>
<button onClick={playAgain}>Play Again</button>
</>
)}
</section>
);
}
Here the parent supplies two distinct faces and a target drawn from that pair; onNext supplies the next round while keeping this component mounted. The target name remains visible throughout, while the photograph labels appear only after an answer.
The final deployed version demonstrates those rules. Wu identifies Justine, builds up a streak, then deliberately chooses a wrong answer to show the counter resetting to zero. He says playing the game helped him learn everyone’s names. Wu estimates that the game’s dataset contains approximately 170 World's Fair speakers.
This is a toy application, but its development loop is consequential: describe a desired result, inspect working software, and refine the request. Wu also had Devin build the QR-code website used to share the game. The same tool was already being used inside Cognition to help build Devin itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A search bar in Devin’s own repository
The next example moves from a new application to an existing production codebase. Bryce, a Cognition teammate, asks Devin to create a search bar for the Devin sessions list. The task begins with more context than the name game: a prepared machine snapshot provides the repository environment, a playbook supplies repository-specific knowledge, and Git tooling lets Devin create a pull request.
After the first pull request, Bryce asks for a magnifying-glass icon and an implementation idiomatic to the project, naming Phosphor as an option. The interaction remains conversational, but the artifact under review is now a repository change rather than a standalone deployment.
The exchange also exposes a practical limit. Bryce says there is no need to test, while Devin reports trouble with the login process. Wu confirms that the search-bar pull request was merged; this example establishes a reported production contribution, not successful end-to-end verification. He also credits Devin with building API integrations, internal dashboards, and metrics tracking used by the company.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a hacker house to an agent
Work began in November 2023, roughly seven months before this June 2024 talk, in a hacker house in Burlingame. Several teammates already lived together, each brought experience in AI, and they shared an intention to build a coding agent. The team subsequently moved between hacker houses, including one in New York, renting progressively larger Airbnbs as it grew. At the time of the talk, the plan was to settle in one place.
The product direction starts with a distinction between completing text and deciding what to do. A language model takes a prefix and generates a suffix. Wu uses ChatGPT, question-answering systems, marketing copy, customer support, GitHub Copilot, and Cursor as examples of valuable products built around that interaction. This is his framing of the product landscape at the time of the talk.
An agent adds autonomous decisions to the generation process. That creates new possibilities, but also raises the required level of consistency: the system must choose useful actions, not merely provide useful text. Building Devin therefore involves two distinct problems—improving its ability to complete work and designing an interface through which people can direct and understand that work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why code gives agents a useful feedback loop
The team’s enthusiasm for teaching AI to code is one motivation, but software engineering also fits the agent pattern unusually well. Much of the job happens outside the act of typing code: inspecting files, running commands, consulting documentation, opening a front end to reproduce a bug, making an edit, and trying again. An agent can participate in that sequence because each step creates information that helps determine the next action.
Consider Wu’s hypothetical assignment: fix one bug in a production system containing thousands of files and hundreds of thousands of lines. Reading the whole system and immediately producing the correct change would be difficult for a human or an AI. The practical route is investigation:
- Add print statements to expose relevant runtime behavior.
- Inspect logs and monitoring to see what actually happened.
- Move between the implicated files to narrow the diagnosis.
- Run code after a decision and use the result to choose the next step.
Execution supplies evidence that static inspection alone does not. The agent can revise its understanding rather than committing to its first guess.
Wu also expects the underlying capabilities to improve quickly. In his assessment, even the name-game demonstration would have been almost unthinkable two years earlier. Looking ahead another two years, he expects better data and training to keep expanding what agents can accomplish.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Supervision needs more than a chat window
The obvious interface models are familiar software and human collaboration. Showing Devin’s computer and commands resembles looking over an intern’s shoulder. But neither analogy fully captures an agent’s needs: parallel work, information gathering, and context management behave differently enough to require their own product decisions.
The resulting feature set extends well beyond shell access, code editing, and web browsing:
- Session control: Fork work into another session or roll it back.
- Team integration: Connect work to Slack and GitHub.
- Prepared context: Supply playbooks and stored machine snapshots.
- Operational support: Manage secrets and provide appropriate verification tools.
These controls are part of making the agent usable, alongside improving its ability to write and debug software.
A recently shipped feature at the time made Devin’s machine directly accessible through VS Code Live Share. Instead of describing every edit in chat, a human could enter the shared environment, change a few lines, and tell Devin what had changed. Collaboration could therefore alternate between natural-language direction and direct intervention in the files.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Delegating work asynchronously
Cognition’s own workflow had become more asynchronous. Wu describes an engineer with four tasks assigning them to four Devin sessions running in parallel. The engineer then coordinates the work much like an engineering manager. His analogy is “enthusiastic interns”: the agents try hard, but lack knowledge, make mistakes, and ask questions. Parallelism still requires supervision and iteration.
A same-day example makes that workflow concrete. During a Slack discussion about a color change, a teammate can mention @Devin and ask it to make the change. Devin produces a pull request; a human merges it. The slide shows the request in the conversation where the work originated, with the relevant interface context attached.
Wu says teammates have made requests from the gym or a car, without their development computer available, and reviewed the code afterward. The change is not simply a different way to enter code: specifying a task and reviewing its implementation no longer have to happen in the same sitting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deciding what to build remains engineering work
What does this leave for software engineers? Wu starts with a boundary: Devin does not decide what should be built. He divides engineering into two kinds of work that currently occupy the same person’s day.
| Work | Decisions and activities |
|---|---|
| Solution design | Architecture, user flows, details, edge cases |
| Implementation | Debugging, functions, unit tests |
The first establishes the solution to pursue; the second turns that solution into working software. Wu estimates that an average engineer currently spends 10–20% of their time on thinking and solution design and 80–90% on implementation. He presents this as an estimate, not a measured time-use study.
His expectation is that better agents will free engineers to spend more time on the first category. Kubernetes setup, debugging broken APIs, version changes, and migrations are examples of implementation work he hopes to delegate. The remaining role would put more emphasis on understanding problems and designing solutions, combining aspects of a technical architect and a product manager. This is a direction for an early technology, not a claim that those responsibilities have already been automated away.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cheaper implementation could mean more software
Wu predicts that the job will change while the number of software engineers grows. His historical analogy runs from punch cards to assembly to C: the mechanics of programming changed, yet programming became more widespread. The argument depends on how much useful software remains unbuilt, rather than assuming that today’s workload is fixed.
He describes software as the leading driver of world progress over the preceding 40–50 years, while arguing that its potential remains far from exhausted. Wu estimates that demand for software is substantially more than 10 times current output. Wu projects that engineers could become 5–10 times more effective while the total amount of software engineering increases. These are economic expectations and a productivity forecast, not measured results from the demonstrations. In that view, easier implementation both expands what existing engineers can attempt and lets more people participate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Access and executable verification
The first audience question is practical: how can people get access? At the time of the talk, Cognition was admitting more users each week and expanding its enterprise customer rollout, but still had substantial waitlists. Wu gives no firm access date.
The next question asks how Devin checks generated code and whether it can actually run the application. In Cognition’s repository, Wu explains, Devin receives an instantiated machine configured to run the development environment, server, and front end. For a debugging task, it can open the application, reproduce the reported problem, debug it, and try again. When the audience member characterizes the environment as fully executable, Wu confirms it. The verification mechanism is access to the running system and its behavior, not just another reading of the generated code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How do junior engineers learn?
Automating simpler tasks raises a different concern: those tasks are also how interns and junior engineers learn. Asked what happens to that training path, Wu expects demand to rise with supply and the training process to change.
His answer centers on what makes an engineer strong. Typing speed is less important than understanding a problem, knowing alternative architectures, and recognizing edge cases. He expects junior engineers to practice those fundamentals earlier as agents take on more implementation. That identifies the skills he believes will remain valuable, though it does not supply a replacement curriculum for learning them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Progress has to come across the stack
The final question asks for the biggest challenges to realizing this future. Wu’s answer spans speed, consistency, access, integrations, and product UX. No single capability improvement resolves all of them: an agent must do useful work reliably, fit the systems around it, and give people workable ways to collaborate.
He expects progress from complementary sources—new hardware, stronger foundation models, and agentic research—with many optimizations improving different parts of the stack. His optimism is about their combined effect. A better coding agent will not arrive through one small breakthrough; it will emerge as the execution environment, underlying capabilities, and collaboration product improve together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
An icon family for interfaces, including the kind of search icon discussed in the production demo.
Microsoft's collaborative development tool supports shared editing, debugging, terminals, and servers.
Further reading
- Introducing DevinArticle
Scott Wu's original announcement explains Devin's tools, planning, collaboration, and early demonstrations.
- Devin's 2024 release notesDocumentation
A dated record of Devin's early playbooks, snapshots, Slack integration, and shared-machine controls.
Updates since the talk
The December 2024 announcement records Devin's expansion beyond its early-access rollout.
Full transcript
Loading transcript…