AI Engineer World's Fair 2026
First Steps Toward Automated AI Research
Loading slides…
Read the talk
First Steps Toward Automated AI Research
Richard Socher’s Eureka machine begins with a practical loop: propose an idea, implement it, and test it—then use that loop to improve the machinery of AI research itself.
From a talk by Richard Socher
Before you start: Basic familiarity with language-model training, attention and GPU computation will help with the three research examples.
A machine that invents
Could we build a machine that eventually invents humanity’s future inventions? Richard Socher calls this ambition the Eureka machine, his equivalent of going to Mars. The starting point is evolution: a process that has already produced extraordinary inventions without a designer specifying every intermediate step. His proposed path toward automated discovery draws on joint work at Recursive, You.com and AIX Ventures, with some slides inspired by Recursive co-founder Tim Rocktäschel.
The connection runs from biological evolution through science and technology to AI. If increasingly capable systems can take over the work of designing AI, engineers may spend more time directing research than implementing each experiment themselves. Socher introduces that possibility with an old machine-translation saying: “Whenever I fire a linguist, my accuracy goes up.” His provocative extension is that AI engineers could become managers of AI researchers—systems that work on AI rather than merely answer questions about it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Successive waves of invention
The first scale is biological: roughly three and a half billion years of evolution, from simple organisms to the branching diversity of plants and animals, including humans. That history establishes the broad possibility Socher wants to borrow: an open-ended process can produce capabilities far beyond its starting point.
Zooming into technological history changes the timescale and introduces deliberate human action. Hunting, farming, the scientific method and the Industrial Revolution each expanded what people could produce. Socher describes world-product history as a succession of exponentials that eventually flatten into S-curves, with new technologies opening another period of growth. Over the last several thousand years—and especially recent centuries—technology also increased the number of people the world could sustain. Humans now consciously guide much of this evolutionary process, which makes it a useful source of ideas for guiding AI development.
This leads to the philosophical premise behind the project. Drawing on Marc Andreessen’s The Techno-Optimist Manifesto, Socher treats technology as the economy’s perpetual source of growth and expects AI to expand the economy despite concerns about displaced jobs. He extends that optimism specifically to material problems, rather than psychological ones: starvation motivates the Green Revolution; darkness, lighting; cold, indoor heating; and heat, air conditioning. These examples motivate his belief that further invention can support more human flourishing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From powered flight to the Moon
A person born in 1900 would have been three when the Wright brothers achieved sustained powered flight, and could have watched humans reach the Moon in 1969. The comparison makes the scale of technological change tangible: within one lifetime, a capability can move from a fragile first demonstration to something that once seemed unattainable.
Socher sees a possible equivalent for intelligence. Our generation may be too late to explore Earth and too early to explore the stars, but it can participate in building AI that transforms what intelligence can accomplish. He speculates that AI could move from worse than humans at their tasks to better at any specific task over a comparable sixty-year period, perhaps thirty years if progress accelerates. That is the horizon of the ambition, not the status of the systems demonstrated later.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Science needs both selection and researchers
Science and technology do not advance in a single direction. Sometimes a working application arrives before the theory explaining it; sometimes theory enables a technology that did not previously exist. In either direction, a research system needs a way to choose among competing explanations. Socher invokes Karl Popper’s view of science as an evolutionary process: theories compete, and the ones that withstand testing survive. For an LLM-based researcher, even finding the competing theories requires tools such as web search.
The operative mechanism is to propose a hypothesis, explanation or description and subject it to rigorous empirical testing. Testing supplies the selection pressure. If this process produces more science, and science enables technology, growth and flourishing, scaling scientific discovery becomes an attractive goal.
But scaling by adding human researchers encounters a bottleneck. Socher draws on Stanisław Lem’s Summa Technologiae: as science expands into more specialties, the people available to work on each specialty become a limiting resource. A growing scientific enterprise does not mean a million researchers can be concentrated on every niche problem. The widening scope of knowledge can instead leave each individual area with a shrinking share of scientific attention. Automating discovery is the proposed response to that constraint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four pillars for automated discovery
The Eureka machine is intended to automate discovery across physics, chemistry, biology, neuroscience, medicine, economics and astrophysics. Socher also mentions a forthcoming book on the idea, expected a couple of months after the talk. The architecture he sketches brings together four complementary foundations.
| Pillar | Role in discovery |
|---|---|
| Existing knowledge | Establish what humanity already knows and has invented. |
| Measurement data | Supply observations of the world. |
| Simulations | Explore cases that cannot yet be measured directly. |
| Physical laboratories | Run experiments in the real world. |
The order matters. A researcher first needs existing knowledge and measurements; where direct measurement is unavailable, simulation offers a way to test possibilities. Socher’s premise is that what can be simulated can be verified and made tractable for AI. Physical experiments remain necessary when those approaches fail or when a result must finally be tested in the world.
An agent swarm sits above these foundations, working with the different sources of knowledge, data, experiments and rewards. The machine is therefore more than a model generating hypotheses: it needs access to the evidence and experimental environments that can determine whether those hypotheses survive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tools designed for AI users
Supporting that research process means reconsidering infrastructure originally built for people. Socher includes the internet, browsers, GPUs and systems for accessing knowledge among the layers that could be redesigned as infrastructure for superintelligence. Each layer creates an opportunity to build tools around what an AI researcher can consume and do.
Search provides a concrete comparison. At You.com, the focus includes web search for LLMs and agents. A human search interface commonly presents ten blue links with short snippets; an agent can potentially read thousands of much longer snippets. That changes the useful shape of the interface and its output. The purpose of rebuilding these tools is to support superintelligence that can automate science.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let the system improve how it does research
How could such a system be built? Socher’s answer follows a recurring pattern in AI: replace a manually designed process with a learned one. In natural language processing, linguists once supplied features for statistical models. Word vectors, neural architectures, end-to-end learning and backpropagation moved more of that work into learning. Architecture engineering followed, and even efforts toward a unified architecture still leave many manual decisions in place.
The next candidate for automation is the process that improves AI itself. Socher’s concise rationale is that “AI is code and AI can code now.” He attributes the enabling change to improvements in coding over longer time horizons during the six to eight months preceding the talk. A system could identify its own shortcomings and modify itself to address them—a functional sense of self-awareness in this proposal, centered on diagnosing and repairing limitations. The intended sequence is to become good at AI research first, then apply that research capability to other scientific fields.
At the highest level, the research cycle has three steps:
- Ideation: propose a change or hypothesis.
- Implementation: turn it into an experiment.
- Validation: determine whether the experiment supports the idea.
This is the same basic cycle whether the target is a learning algorithm or a problem in another scientific field.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Autoresearch is a first step, not full RSI
The first experiments target three accessible parts of AI development: better training, faster training and better NVIDIA GPU kernels. These are bounded tests of an automated research system, chosen to provide initial proof points for the larger machinery.
The distinction between autoresearch and recursive self-improvement, or RSI, is crucial. nanochat supplies a small-model setting associated with Karpathy’s autoresearch, but improving a separate model is not the same as improving the researcher itself. Socher reserves full RSI for a system that recognizes its own shortcomings, has access to pretraining, reinforcement learning, harnesses and the rest of its machinery, and can update that whole system in its next version.
| Research mode | What changes? | Scope |
|---|---|---|
| Autoresearch | Another model or process | A bounded experiment, such as a short nanochat training run |
| Full RSI | The research system itself | Its training, harnesses and other components |
A system that improves a small nanochat run is an important milestone, but it does not establish full RSI. Socher reports that the initial autoresearch system quickly outperformed multiple teams, including teams using other AI research tools; the following examples show what kinds of improvements he means.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
nanochat: improving quality through architecture
The nanochat experiment asks how much model quality can be obtained from a very short training run, measured in bits per byte: lower is better. Socher describes training a small model in less than five minutes. In the autoresearch setup, the five-minute budget covers training, excluding startup and compilation; it is not the total time to produce a finished chat assistant.
Socher reports reducing nanochat’s bits per byte from about 0.93 to 0.91 after roughly a day or two of automated research. The companion report gives the more precise validation figures: 0.9372 to 0.9109 on B200, following search on H100, with a community baseline cleaned of reward hacks and evaluation over ten seeds. Those conditions matter: the short training budget applies to an individual candidate, while the research process explores candidates over a much longer period.
The interesting result is the kind of change discovered. Instead of only tuning hyperparameters, the system introduced hashed bigram and trigram embeddings, stored in tables, and mixed those features into attention value paths through learned gates. A bigram feature depends on a pair of adjacent tokens; a trigram depends on three. Hashing maps these combinations into a bounded table, and a learned gate controls their contribution to the value representation. Socher presents this as architectural discovery beyond parameter tuning; the companion report connects the approach to existing techniques, so the result does not establish that the underlying idea was invented for the first time.
A compact PyTorch illustration makes the described operation concrete. Given token_ids and an existing attention value tensor values, this module looks up hashed n-gram features and adds a gated contribution while preserving the tensor’s shape. The hash constants and scalar gate below are illustrative choices, not the experiment’s exact implementation.
python
import torch
from torch import nn
class NgramValueMix(nn.Module):
def __init__(self, width: int, buckets: int = 4096):
super().__init__()
self.buckets = buckets
self.bigrams = nn.Embedding(buckets, width)
self.trigrams = nn.Embedding(buckets, width)
self.gate = nn.Linear(width, 1)
def forward(self, token_ids, values):
# token_ids: [batch, time]; values: [batch, time, width]
previous = torch.zeros_like(token_ids)
earlier = torch.zeros_like(token_ids)
previous[:, 1:] = token_ids[:, :-1]
earlier[:, 2:] = token_ids[:, :-2]
bigram_ids = (previous * 31 + token_ids) % self.buckets
trigram_ids = (
earlier * 31 * 31 + previous * 31 + token_ids
) % self.buckets
positions = torch.arange(token_ids.size(1), device=token_ids.device)
bigram_features = self.bigrams(bigram_ids)
trigram_features = self.trigrams(trigram_ids)
features = (
bigram_features * (positions >= 1)[None, :, None]
+ trigram_features * (positions >= 2)[None, :, None]
)
return values + torch.sigmoid(self.gate(values)) * features
The masks omit positions without enough preceding tokens. The architectural change is the additional learned feature path into values; deciding whether that path helps still requires training and validation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
nanoGPT: reaching the target faster
The second experiment changes the objective from quality within a short budget to training speed. The nanoGPT speedrun is a community effort to reach a specified validation loss as quickly as possible. Socher describes rapid progress against a benchmark that people, often working with AI, had already spent more than a year improving.
In the talk, Socher describes the nanoGPT run as roughly seventy seconds, more than two seconds faster than the comparison baseline. The companion report instead records 79.7 to 77.5 seconds for FineWeb validation loss ≤3.28 on eight H100 GPUs. The artifact repository separately reports 77.3 seconds on Modal’s eight H100s, with official leaderboard timing pending. These are distinct reported timings, not a single verified record. Socher says the search also found interesting ideas, but does not explain those changes in this demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
CUDA kernels: optimize, then check the reward
The third experiment moves below model architecture and training schedules to CUDA kernels, the GPU operations used during training and testing. The motivation is GPU efficiency: small improvements can matter when computation is expensive. Socher cites roughly 30% utilization for some mixture-of-experts deployments on billion-dollar clusters, although he does not specify a workload or utilization definition. That estimate motivates the optimization work; it is not the metric used to evaluate the kernels.
Socher reports that, after a couple of days, the system found kernels that exceeded NVIDIA benchmark leaderboard leaders by sizable margins across categories. The companion report identifies the benchmark as SOL-ExecBench and reports a mean score increase from 0.699 to 0.754 across 235 B200 kernels. The slide shows per-kernel and category charts alongside overall score cards for Cursor, doubleAI, the leaderboard best and Recursive. The aggregate gain does not imply that every individual kernel won, or that whole-cluster utilization increased by the same amount.
The team did not have specialists who had spent their careers writing CUDA kernels. That makes automated search attractive, but it also raises the importance of checking what the system optimized. A higher benchmark reward is useful only if it reflects a valid improvement rather than an exploited weakness in the evaluation. Socher says the team worked with NVIDIA to check for reward hacks and other issues, and that pretty much all the kernel results checked out. The validation is part of the research loop, not a step that can be replaced by the leaderboard score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The next S-curve depends on what intelligence can become
The proposed next step is to turn these bounded improvements into a system that improves its own research capabilities. Socher sees RSI as another S-curve layered over earlier technological advances: better AI enables more science, which enables more technology and, ultimately, more people to flourish. The three experiments are early steps toward that trajectory, not demonstrations that the full cycle has already been achieved.
Every exponential eventually flattens, so how much room remains? Socher closes by questioning what exponential growth in AI even means. Intelligence does not have a single obvious axis. He proposes ten spaces of intelligence, each with multiple dimensions, together forming a more complex, volumetric conception of capability. He does not enumerate those spaces here, but argues that current systems remain astronomically far from upper bounds across nearly all of them. The invitation to help build the Eureka machine rests on that remaining headroom: a research system could improve not just one score, but the range of problems it can formulate, investigate and solve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A small agent-driven research setup that optimizes nanochat training within a fixed five-minute GPU budget.
Community benchmark for reaching a fixed FineWeb validation loss as quickly as possible on eight H100 GPUs.
Web search and content services designed for AI agents and language models.
Marc Andreessen's argument for technological growth and material abundance.
Further reading
Recursive's technical account of its three benchmark experiments and discovered optimizations.
Training scripts, validation trajectories and selected GPU kernels from the reported experiments.
Lem's exploration of technological evolution and the limits of human scientific capacity.
Full transcript
Loading transcript…