AI Engineer World's Fair 2025
#define AI Engineer
Loading slides…
Read the talk
#define AI Engineer: From a Sorting Widget to Systems That Learn
Greg Brockman traces the work of AI engineering from making an idea usable to building research systems, coding agents, recoverable workflows, and responsible domain applications.
From a talk by Greg Brockman, swyx . and Jensen Huang
Before you start: Familiarity with software modules, automated tests, and basic machine-learning terminology will help with the systems discussion.
Making an idea work in the world
How do you turn a chemistry textbook into something people can use when no publisher will take it? That problem drew Greg Brockman toward programming. In his conversation with swyx, the biographical introduction includes theater, chemistry, mathematics, and a calendar scheduling app. Brockman’s own account of discovering programming starts with the textbook and ends with a table that sorts when you click a column.
Brockman had imagined becoming a mathematician in the tradition of Galois and Gauss, working on ideas whose usefulness might emerge centuries later. After high school, he wrote a chemistry textbook and sent it to a friend who had done something similar in mathematics. The friend suggested either self-publishing or making a website. Self-publishing sounded expensive and laborious, so Brockman learned PHP through W3Schools. His first widget translated a precise picture in his head into a working interaction: click a column, and the table sorts by that column.
The essential operation is small. In PHP, sorting a few chemistry records by a selected column can look like this:
php
<?php
$elements = [
['name' => 'Oxygen', 'atomic_number' => 8],
['name' => 'Hydrogen', 'atomic_number' => 1],
['name' => 'Carbon', 'atomic_number' => 6],
];
$column = 'atomic_number';
usort($elements, fn($a, $b) => $a[$column] <=> $b[$column]);
foreach ($elements as $element) {
printf("%s: %d\n", $element['name'], $element['atomic_number']);
}
The appeal lies in what that operation makes available to someone else. A proof and a program might each have only a few readers who understand their internals. But people can benefit from a program without reading it. The idea becomes a usable object in the world. Brockman’s ambition shifted from waiting for distant usefulness to building things now.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finding people to build with
Stripe found Brockman through mutual contacts. The company had about three people, and recommendations from both Harvard and MIT led to his name: he had already left Harvard to attend MIT. When he flew out to meet Patrick Collison, he arrived late at night during a storm. They immediately started talking about code. That technical affinity made the opportunity concrete—these were people he wanted to work with—and he left MIT to join them.
A guest question from his brother Matthew Brockman, introduced as Julius AI’s CTO, asks when their parents will abandon the hope that Greg finishes his degree, perhaps at Harvard or the University of North Dakota. Brockman’s answer is affectionate: probably never. Leaving Harvard was difficult to explain; leaving school altogether was harder. His parents nevertheless trusted that he could see something from his position that they could not see from across the country. As Stripe became a real company and a place where he was learning, their concerns eased.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separating real constraints from waiting
Swyx frames Brockman’s Stripe tenure as growth from four people to 250, with Brockman becoming its first CTO. The famous story of the Collisons immediately installing Stripe for prospective customers receives a qualification: Brockman tentatively agrees that it happened only a handful of times. The less celebrated practice was more continuous. Stripe added customers on G Chat and stayed in close contact, creating a short path from a user’s problem to an engineer who could fix it.
That same short feedback loop mattered when Stripe discovered its payment backend would not scale. Brockman recalls a nine-month estimate for the Wells Fargo technical integration and says the team completed it in 24 hours. The work proceeded like an intense college problem set:
- Brockman implemented the integration.
- John tested from the top of the test script while Dara worked upward from the bottom.
- During certification the following morning, an error appeared. The certifier proposed returning the next week.
- Patrick kept the conversation going while Brockman edited the code. About five attempts still ended in failure.
- The certifier agreed to return two hours later, and the integration passed.
Brockman characterizes the compressed certification loop as six weeks of normal development work. The speed came partly from implementation effort and partly from keeping the people who could diagnose, fix, and certify the system in the same feedback loop.
Question the reason for a delay before accepting its duration. Brockman explicitly rejects a general rule of ignoring constraints. Some work needs to be slow or carefully sequenced. The useful question is which overhead exists because of assumptions that no longer apply to your circumstances. AI acceleration makes that distinction more consequential: faster implementation only helps if the surrounding process can use it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Independent study that compounds
Swyx connects several periods of independent study: school, Recurse Center, and a later sabbatical. Brockman traces the habit to his father teaching him algebra in sixth grade. His seventh-grade teacher initially refused to let him skip ahead, then relented after roughly a month of seeing that he could answer the questions despite paying little attention in class. By eighth grade, he had exhausted the mathematics available in his middle school. Without a car, he turned to online courses.
Brockman says he completed three years of high-school mathematics in one year through online study. After finishing the remaining high-school mathematics the next year, he could drive to the University of North Dakota for courses during grades ten through twelve. Removing one pacing constraint opened the next opportunity, and the benefits compounded. Programming followed the same pattern: self-study, building, and contact with real problems. His advice is to go deeply into something that excites you, while recognizing that sustained interest still includes boredom and difficult stretches worth pushing through.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning AI through people and hardware
Asked about learning machine learning, Brockman confirms that he spoke with Geoff Hinton, but his account centers on a broader chain of introductions. While at Stripe in 2013 and 2014, he kept seeing Hacker News stories about deep learning applied to another problem. He knew one person in the field, who introduced him to others. Those introductions repeatedly led back to some of his strongest college friends. Alongside the people, he saw a substantive technical signal: computers were beginning to do things they had not been able to do before.
After Stripe, he wanted to start an AI company but did not yet know where his skills would fit. In New York, he chose a practical entry point: assemble a GPU rig and try Kaggle competitions. He bought Titan X cards through Newegg and built the machine himself. He recalls a 2015 tweet showing its green lights and spinning fans when it powered on. Learning the field meant finding people, investigating new capabilities, and getting his hands on the hardware that made experiments possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning rules instead of writing them all
What made artificial general intelligence seem plausible? Swyx mentions an unsuccessful chatbot experiment, but Brockman reaches further back, to reading Alan Turing’s Computing Machinery and Intelligence around 2008. The imitation test was only part of its attraction. The deeper question was how to build such a machine when a programmer could never enumerate every rule it would need. Turing’s child-machine proposal offered another route: build a system that learns, then educate it through rewards and punishments.
Ordinary programming requires the programmer to understand how to solve the problem well enough to specify the solution. A learning machine raises a more ambitious possibility: solving problems its programmer does not know how to solve. Brockman asked an NLP professor about doing research and was offered parse trees; swyx adds WordNets to the picture of the period. The work did not resemble the general learning system that had excited him in Turing.
AlexNet changed that picture with its 2012 ImageNet result. Brockman emphasizes the combination of a general learning procedure and an architectural prior—convolutions—rather than an exhaustive collection of human-written vision rules. He describes its impact as surpassing decades of computer-vision work. As deep learning subsequently succeeded in machine translation and NLP, boundaries that had seemed to separate research disciplines began to weaken. The same broad approach could travel between problems.
He places that progress in a longer lineage, recalling the McCulloch–Pitts neuron paper from around 1943 and diagrams that resemble familiar networks of neurons. He also recalls an unnamed paper from the 1990s criticizing neural-network researchers for merely wanting bigger computers. For Brockman, that criticism identified something worth pursuing. More capable learning systems could both address problems beyond human understanding and change the interface to computing: a person could interact naturally with a machine instead of accommodating it through keyboards or assembly language.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The idea needs an engineered system
Swyx returns to Brockman’s 2022 essay, It’s time to become an ML engineer, and its claim that great engineers can contribute as much as great researchers. He mentions a friend who read it, cold-emailed Brockman, and joined OpenAI. Brockman considers the premise even more applicable as the field develops.
AlexNet supplies the concrete partnership. Alex Krizhevsky built fast convolution kernels for GPUs; in Brockman’s telling, some lab peers underestimated the value of that work. Ilya Sutskever saw the opportunity to apply it to ImageNet. The engineering made the experiment possible, and the research judgment selected a consequential use for it. Today, Brockman points to systems on the scale of a hundred thousand GPUs and elaborate reinforcement-learning orchestration as examples of the engineering ambition involved. These are examples of the work required, not a specification of a particular disclosed training run.
Ideas and implementation are joint requirements. Without a useful idea, there is no experiment to build. Without engineering, the idea never becomes a working system. Brockman describes valuing research and engineering as partners as an OpenAI founding ethos—and something the organization must keep practicing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When an interface does not hide the risk
That partnership creates friction because the two disciplines may assign different meanings to an abstraction. Brockman describes a recurring disagreement:
| Perspective | What an interface should permit | Main concern |
|---|---|---|
| Engineering | Change internals behind an agreed contract | Preserve the contract |
| Research | Ignore internals only after deep trust | Detect silent performance loss |
A bug in a research system may produce slightly worse model performance rather than an exception. The researcher therefore feels responsible for understanding every component that could affect the result. An interface can hide implementation only when its behavior is trustworthy enough to stop investigating.
In one early project, engineers wrote code and researchers disputed it line by line. Progress stalled. Brockman changed the interaction by offering several ideas at once and welcoming the rejection of most of them. A researcher saying four of five ideas were bad supplied useful information quickly; it did not require a prolonged defense of each implementation choice.
He calls the necessary habit technical humility. Engineering skills remain valuable, but intuitions from a web startup do not automatically transfer to research. First read, listen, and understand why the system works as it does. Then change its architecture or abstractions. The relationship is never permanently solved: each new level of system sophistication exposes another version of the problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Launch demand borrows from the future
Swyx introduces the launch comparison with two audience-growth figures: ChatGPT reached one million users in five days, while GPT-4o image generation reached 100 million users in five days. Those are the interview’s framing figures; the discussion does not establish matching definitions of users for the two launches. Brockman’s comparison focuses on the operational experience. ChatGPT was intended as a relatively low-key research preview, yet demand overwhelmed the systems serving it.
The team had expected GPT-4 to be necessary for that level of popularity. Exposure to stronger internal models had already shifted its expectations. Brockman describes how quickly amazement turns into a request for the next capability: after seeing something extraordinary, people soon want it to merge a stack of pull requests for them. Image generation produced a similar surge of enthusiasm and demand.
For both launches, Brockman says OpenAI reassigned compute from research to serving users. He describes that choice as “mortgaging the future”: capacity used to meet present demand cannot simultaneously support the experiments that produce the next advance. The decision was to preserve a moment when many people wanted to experience a materially new capability, even at that research cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a robust demo to a cloud coworker
The GPT-4 joke-website demonstration began with a drawing made by Brockman’s wife; he jokes that his own handwriting was too difficult even for the model. He had tested the general shape of the demo, while leaving room for improvisation. That is a deliberate standard for demonstrations: a slight variation or a single mistyped character should not destroy the result. A robust capability permits variation in what happens onstage.
Swyx retrospectively describes the website demo as an early example of vibe coding. Brockman sees interactive coding as an empowering starting point, with a broader form of delegation ahead. The Codex under discussion is the cloud-agent research preview introduced in May 2025. His vision extends from a few copies to hundreds, thousands, and eventually a hundred thousand agents; that scale is a future ambition. These agents would behave more like coworkers, connect to tools, and continue working in the cloud while the user sleeps or closes the laptop.
He expects interactive coding to keep growing while autonomous agent work eventually overtakes it. The larger opportunity also moves beyond creating a new app from scratch:
- Library upgrades: Maintain existing software as its dependencies change.
- Migrations: Transform systems whose accumulated history makes manual changes difficult.
- Legacy-language replacement: Tackle work such as moving COBOL applications to another language.
These tasks extend the appeal of quick app creation into the less glamorous work that determines how quickly an established company can change its software.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the repository easy to work in
When swyx calls Codex something like Brockman’s personal project, he redirects the credit to the team and its vision. His practical observation is about the surrounding codebase: repository structure determines how much useful work a coding model can do. Human-oriented repositories often depend on developers holding extensive conceptual context in their heads. Models have a different, uneven capability profile: they can handle a wide variety of material, but at the time of the conversation they are less reliable at connecting deep ideas.
The resulting design advice is familiar software engineering with stronger incentives:
- Small, documented modules: Give a task a comprehensible boundary.
- Strong tests: Make expected behavior explicit so the model can check its implementation.
- Fast test execution: Keep repeated implementation-and-test cycles cheap.
- Clear component connections: Separate architectural relationships from the detailed work inside each component.
The agent can fill in implementation details and run the tests itself. Humans sometimes skip this structure because they can mentally compensate for what is missing. A model benefits when that context becomes explicit in the repository.
Brockman suggests a model may run tests 100 or 1,000 times more often than a human, making test speed especially consequential. This is his motivating estimate, not a measured comparison presented in the interview. His shorthand is to build codebases that are accessible to junior developers. Stronger models may change which accommodations matter, but small modules, explicit expectations, and maintainability also help the humans working alongside them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the early adoption figures show
Asked for productivity numbers, Brockman says he does not know the latest figures. He recalls a low-double-digit percentage of OpenAI’s internal pull requests being written entirely by Codex. OpenAI also uses other coding systems internally, and he describes adoption as early. He also recalls approximately 24,000 pull requests merged in public GitHub repositories in the preceding day during the Codex discussion. These are adoption indicators; without a defined reporting population, attribution method, or comparison of effort and outcomes, they do not establish a productivity gain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recovery must include the tools
Dylan Patel’s guest question moves from coding productivity to failures during long-running agent training. A GPU failure can interrupt a partially completed trajectory whose continuation is not deterministic. Restarting may be tolerable when a rollout lasts thirty seconds; when it lasts days, discarded work becomes a much larger concern. Brockman describes this as another problem that must be solved again as model capability increases.
Checkpointability is a property of the whole system. Pretraining already requires careful recovery design, and checkpointing every step makes copying and blocking costs important. More complex reinforcement-learning systems add other state worth preserving:
| State | Recovery concern |
|---|---|
| Training checkpoint | Save frequently without costly copying or blocking |
| RL cache | Preserve work that would otherwise be recomputed |
| Language-model state | Store the explicit state needed to continue |
| Attached tool state | Determine whether the tool can restart or recover |
The last row can break an otherwise sound recovery design. A model’s state may be explicit and storable while the external tool it uses has changed in a way that cannot be restored. Saving the model alone does not settle what happens when the entire workflow resumes.
Exact recovery is not always necessary. Brockman allows that restarting and accepting a small perturbation in the training curve may be adequate, depending on the system. Swyx adds a preview of a next-day launch involving taking over a VM, checkpointing its state, and restarting it. The exchange identifies that recovery approach without naming a product or specifying an implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One fleet, very different workloads
In a recorded question introduced as coming from Paris, Jensen Huang asks how infrastructure can support two increasingly different kinds of AI work. Deep-research agents reason, plan, collaborate with other agents, and operate over large contexts and substantial memory. Their balance of prefill, which processes input context, and decode, which generates output, can vary. An always-available vision-and-speech companion has another priority: respond immediately.
| Workload | Dominant requirement | Infrastructure pressure |
|---|---|---|
| Extended research and reasoning | Sustained computation | Throughput and capacity |
| Live multimodal companion | Immediate interaction | Low latency |
Huang’s question is how to serve both well, including workloads between those extremes. Brockman describes the answer as a hardware–software co-design problem that takes an organization from writing software into building enormous physical infrastructure.
The straightforward proposal is two kinds of accelerator: one optimized for computation and another for latency, with different allocations of compute and high-bandwidth memory. But specialization creates a fleet-planning problem. If demand does not match the purchased ratio, a substantial part of the fleet can become unusable for the work that needs doing. Engineers may adapt algorithms to exploit an imbalance, but doing so can be painful.
Brockman uses mixture of experts as a partial example of that adaptation: spare memory can hold additional parameters while only a subset participates in a particular computation. His description of this as adding parameters without compute cost is best understood as increasing capacity without a proportional increase in active computation, rather than making the system literally free to run. The broader point is that model architecture can respond to the balance of hardware resources.
A homogeneous accelerator fleet is therefore an attractive default because it avoids committing too early to a precise workload mix. Purpose-built accelerators can still make sense when infrastructure spending becomes large enough to justify aggressive optimization. Brockman leaves the choice open: research is moving quickly enough to change the workloads that the hardware will eventually need to serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Algorithms become a bottleneck again
Swyx asks Brockman to rank compute, data, algorithms, power, and money as bottlenecks for GPT-6. Brockman gives neither a concrete ranking nor specifications for that model. Instead, he describes a return of basic research. Scaling a relatively unchanged transformer from Attention Is All You Need offered a well-defined engineering challenge, but pushing compute and data far enough makes algorithmic limitations increasingly consequential. The limiting resource can shift from day to day; progress depends on keeping the pieces in balance.
OpenAI’s deliberate investment in reinforcement learning provides his example. When the team first interacted with GPT-4, its fluency prompted the question of whether it might be AGI. Brockman explicitly says it was not: the system could seem smooth and capable, then fall off the rails. He describes the reliability gap as the difference between observing the world and experiencing it—like someone who has read all the books but has only watched events through glass. That diagnosis motivated sustained work on a different learning paradigm. He sees other missing capabilities still requiring the same kind of research effort.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Domain expertise remains part of the system
Huang’s second question asks what happens to development when increasingly capable foundation systems support domain-specific agents. Those agents still need workflows, pipelines, feedback loops, tools, and short- and long-term memory. How much of that work remains for engineers as the underlying models improve?
Brockman leaves several futures open: AI could write all the code, one central system could dominate, or many specialized agents could require substantial domain work. He sees evidence moving toward a diverse collection of models. Inference costs differ, distillation creates useful trade-offs, and one model can use another. That creates room for systems whose components have different capabilities and costs, rather than requiring every task to use the same model. An economy powered broadly by AI is his anticipated direction, not a condition he says has already arrived.
The economy itself also changes as new capabilities become available. Brockman frames tenfold increases in activity, economic output, and benefit as an aspiration, rather than a forecast established by the discussion. The opportunity extends beyond dividing today’s fixed set of tasks between people and AI: lower barriers can make additional work possible.
That expansion still requires people who understand the setting in which a system will operate. Healthcare demands responsibility in deciding how to deploy a capability. Education involves several stakeholders—parents, teachers, and students—whose needs must be considered together. Stronger models do not remove those obligations. They give engineers more to build with, while domain expertise, careful thought, and substantial implementation work remain essential to making the result useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Brockman's essay on engineering contributions to AI research and the technical humility needed to enter the field.
Turing's 1950 paper introduces the imitation game and explores how a child machine might learn.
The historical May 2025 introduction to Codex's cloud tasks, repository environments, test execution, and review workflow.
Further reading
- My path to OpenAIArticle
Brockman's account of college, Stripe, early AI experiments, and the conversations that led to OpenAI.
A foundational sparse mixture-of-experts paper explaining how conditional computation expands parameter capacity without proportionally increasing computation.
Full transcript
Loading transcript…