AI Engineer World's Fair 2026

First Steps Toward Automated AI Research

Loading slides…

Read the talk

First Steps Toward Automated AI Research

Richard Socher’s Eureka machine begins with a practical loop: propose an idea, implement it, and test it—then use that loop to improve the machinery of AI research itself.

From a talk by Richard Socher

Before you start: Basic familiarity with language-model training, attention and GPU computation will help with the three research examples.

A machine that invents

Could we build a machine that eventually invents humanity’s future inventions? Richard Socher calls this ambition the Eureka machine, his equivalent of going to Mars. The starting point is evolution: a process that has already produced extraordinary inventions without a designer specifying every intermediate step. His proposed path toward automated discovery draws on joint work at Recursive, You.com and AIX Ventures, with some slides inspired by Recursive co-founder Tim Rocktäschel.

The connection runs from biological evolution through science and technology to AI. If increasingly capable systems can take over the work of designing AI, engineers may spend more time directing research than implementing each experiment themselves. Socher introduces that possibility with an old machine-translation saying: “Whenever I fire a linguist, my accuracy goes up.” His provocative extension is that AI engineers could become managers of AI researchers—systems that work on AI rather than merely answer questions about it.

0:150:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Successive waves of invention

The first scale is biological: roughly three and a half billion years of evolution, from simple organisms to the branching diversity of plants and animals, including humans. That history establishes the broad possibility Socher wants to borrow: an open-ended process can produce capabilities far beyond its starting point.

Slide titled “Biological evolution” with a multicolored branching diagram spanning bacteria, plants, fungi and animal groups.Open full source frame
Biological evolution shown as a branching history of life.

Zooming into technological history changes the timescale and introduces deliberate human action. Hunting, farming, the scientific method and the Industrial Revolution each expanded what people could produce. Socher describes world-product history as a succession of exponentials that eventually flatten into S-curves, with new technologies opening another period of growth. Over the last several thousand years—and especially recent centuries—technology also increased the number of people the world could sustain. Humans now consciously guide much of this evolutionary process, which makes it a useful source of ideas for guiding AI development.

This leads to the philosophical premise behind the project. Drawing on Marc Andreessen’s The Techno-Optimist Manifesto, Socher treats technology as the economy’s perpetual source of growth and expects AI to expand the economy despite concerns about displaced jobs. He extends that optimism specifically to material problems, rather than psychological ones: starvation motivates the Green Revolution; darkness, lighting; cold, indoor heating; and heat, air conditioning. These examples motivate his belief that further invention can support more human flourishing.

2:162:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:16 · section reference included

From powered flight to the Moon

A person born in 1900 would have been three when the Wright brothers achieved sustained powered flight, and could have watched humans reach the Moon in 1969. The comparison makes the scale of technological change tangible: within one lifetime, a capability can move from a fragile first demonstration to something that once seemed unattainable.

Socher sees a possible equivalent for intelligence. Our generation may be too late to explore Earth and too early to explore the stars, but it can participate in building AI that transforms what intelligence can accomplish. He speculates that AI could move from worse than humans at their tasks to better at any specific task over a comparable sixty-year period, perhaps thirty years if progress accelerates. That is the horizon of the ambition, not the status of the systems demonstrated later.

5:235:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:23 · section reference included

Science needs both selection and researchers

Science and technology do not advance in a single direction. Sometimes a working application arrives before the theory explaining it; sometimes theory enables a technology that did not previously exist. In either direction, a research system needs a way to choose among competing explanations. Socher invokes Karl Popper’s view of science as an evolutionary process: theories compete, and the ones that withstand testing survive. For an LLM-based researcher, even finding the competing theories requires tools such as web search.

The operative mechanism is to propose a hypothesis, explanation or description and subject it to rigorous empirical testing. Testing supplies the selection pressure. If this process produces more science, and science enables technology, growth and flourishing, scaling scientific discovery becomes an attractive goal.

But scaling by adding human researchers encounters a bottleneck. Socher draws on Stanisław Lem’s Summa Technologiae: as science expands into more specialties, the people available to work on each specialty become a limiting resource. A growing scientific enterprise does not mean a million researchers can be concentrated on every niche problem. The widening scope of knowledge can instead leave each individual area with a shrinking share of scientific attention. Automating discovery is the proposed response to that constraint.

Quotation attributed to Stanisław Lem (1964), a black-and-white portrait, and a boxed statement: “The solution is to automate scientific discovery.”Open full source frame
A Stanisław Lem quotation frames the researcher bottleneck and the case for automating scientific discovery.
6:406:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:40 · section reference included

Four pillars for automated discovery

The Eureka machine is intended to automate discovery across physics, chemistry, biology, neuroscience, medicine, economics and astrophysics. Socher also mentions a forthcoming book on the idea, expected a couple of months after the talk. The architecture he sketches brings together four complementary foundations.

PillarRole in discovery
Existing knowledgeEstablish what humanity already knows and has invented.
Measurement dataSupply observations of the world.
SimulationsExplore cases that cannot yet be measured directly.
Physical laboratoriesRun experiments in the real world.

The order matters. A researcher first needs existing knowledge and measurements; where direct measurement is unavailable, simulation offers a way to test possibilities. Socher’s premise is that what can be simulated can be verified and made tractable for AI. Physical experiments remain necessary when those approaches fail or when a result must finally be tested in the world.

An agent swarm sits above these foundations, working with the different sources of knowledge, data, experiments and rewards. The machine is therefore more than a model generating hypotheses: it needs access to the evidence and experimental environments that can determine whether those hypotheses survive.

9:099:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:09 · section reference included

Tools designed for AI users

Supporting that research process means reconsidering infrastructure originally built for people. Socher includes the internet, browsers, GPUs and systems for accessing knowledge among the layers that could be redesigned as infrastructure for superintelligence. Each layer creates an opportunity to build tools around what an AI researcher can consume and do.

Search provides a concrete comparison. At You.com, the focus includes web search for LLMs and agents. A human search interface commonly presents ten blue links with short snippets; an agent can potentially read thousands of much longer snippets. That changes the useful shape of the interface and its output. The purpose of rebuilding these tools is to support superintelligence that can automate science.

10:3610:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:36 · section reference included

Let the system improve how it does research

How could such a system be built? Socher’s answer follows a recurring pattern in AI: replace a manually designed process with a learned one. In natural language processing, linguists once supplied features for statistical models. Word vectors, neural architectures, end-to-end learning and backpropagation moved more of that work into learning. Architecture engineering followed, and even efforts toward a unified architecture still leave many manual decisions in place.

The next candidate for automation is the process that improves AI itself. Socher’s concise rationale is that “AI is code and AI can code now.” He attributes the enabling change to improvements in coding over longer time horizons during the six to eight months preceding the talk. A system could identify its own shortcomings and modify itself to address them—a functional sense of self-awareness in this proposal, centered on diagnosing and repairing limitations. The intended sequence is to become good at AI research first, then apply that research capability to other scientific fields.

At the highest level, the research cycle has three steps:

  1. Ideation: propose a change or hypothesis.
  2. Implementation: turn it into an experiment.
  3. Validation: determine whether the experiment supports the idea.

This is the same basic cycle whether the target is a learning algorithm or a problem in another scientific field.

11:5312:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:53 · section reference included

Autoresearch is a first step, not full RSI

The first experiments target three accessible parts of AI development: better training, faster training and better NVIDIA GPU kernels. These are bounded tests of an automated research system, chosen to provide initial proof points for the larger machinery.

Slide headed “Better training, faster training, better kernels” with three colored rows showing benchmark comparisons for NanoChat Autoresearch, NanoGPT Speedrun and SOL-ExecBench.Open full source frame
Three initial proof points: NanoChat Autoresearch, NanoGPT Speedrun and SOL-ExecBench.

The distinction between autoresearch and recursive self-improvement, or RSI, is crucial. nanochat supplies a small-model setting associated with Karpathy’s autoresearch, but improving a separate model is not the same as improving the researcher itself. Socher reserves full RSI for a system that recognizes its own shortcomings, has access to pretraining, reinforcement learning, harnesses and the rest of its machinery, and can update that whole system in its next version.

Research modeWhat changes?Scope
AutoresearchAnother model or processA bounded experiment, such as a short nanochat training run
Full RSIThe research system itselfIts training, harnesses and other components

A system that improves a small nanochat run is an important milestone, but it does not establish full RSI. Socher reports that the initial autoresearch system quickly outperformed multiple teams, including teams using other AI research tools; the following examples show what kinds of improvements he means.

13:5614:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:56 · section reference included

nanochat: improving quality through architecture

The nanochat experiment asks how much model quality can be obtained from a very short training run, measured in bits per byte: lower is better. Socher describes training a small model in less than five minutes. In the autoresearch setup, the five-minute budget covers training, excluding startup and compilation; it is not the total time to produce a finished chat assistant.

Socher reports reducing nanochat’s bits per byte from about 0.93 to 0.91 after roughly a day or two of automated research. The companion report gives the more precise validation figures: 0.9372 to 0.9109 on B200, following search on H100, with a community baseline cleaned of reward hacks and evaluation over ten seeds. Those conditions matter: the short training budget applies to an individual candidate, while the research process explores candidates over a much longer period.

The interesting result is the kind of change discovered. Instead of only tuning hyperparameters, the system introduced hashed bigram and trigram embeddings, stored in tables, and mixed those features into attention value paths through learned gates. A bigram feature depends on a pair of adjacent tokens; a trigram depends on three. Hashing maps these combinations into a bounded table, and a learned gate controls their contribution to the value representation. Socher presents this as architectural discovery beyond parameter tuning; the companion report connects the approach to existing techniques, so the result does not establish that the underlying idea was invented for the first time.

A compact PyTorch illustration makes the described operation concrete. Given token_ids and an existing attention value tensor values, this module looks up hashed n-gram features and adds a gated contribution while preserving the tensor’s shape. The hash constants and scalar gate below are illustrative choices, not the experiment’s exact implementation.

python

import torch
from torch import nn


class NgramValueMix(nn.Module):
    def __init__(self, width: int, buckets: int = 4096):
        super().__init__()
        self.buckets = buckets
        self.bigrams = nn.Embedding(buckets, width)
        self.trigrams = nn.Embedding(buckets, width)
        self.gate = nn.Linear(width, 1)

    def forward(self, token_ids, values):
        # token_ids: [batch, time]; values: [batch, time, width]
        previous = torch.zeros_like(token_ids)
        earlier = torch.zeros_like(token_ids)
        previous[:, 1:] = token_ids[:, :-1]
        earlier[:, 2:] = token_ids[:, :-2]

        bigram_ids = (previous * 31 + token_ids) % self.buckets
        trigram_ids = (
            earlier * 31 * 31 + previous * 31 + token_ids
        ) % self.buckets

        positions = torch.arange(token_ids.size(1), device=token_ids.device)
        bigram_features = self.bigrams(bigram_ids)
        trigram_features = self.trigrams(trigram_ids)
        features = (
            bigram_features * (positions >= 1)[None, :, None]
            + trigram_features * (positions >= 2)[None, :, None]
        )
        return values + torch.sigmoid(self.gate(values)) * features

The masks omit positions without enough preceding tokens. The architectural change is the additional learned feature path into values; deciding whether that path helps still requires training and validation.

15:3715:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:37 · section reference included

nanoGPT: reaching the target faster

The second experiment changes the objective from quality within a short budget to training speed. The nanoGPT speedrun is a community effort to reach a specified validation loss as quickly as possible. Socher describes rapid progress against a benchmark that people, often working with AI, had already spent more than a year improving.

In the talk, Socher describes the nanoGPT run as roughly seventy seconds, more than two seconds faster than the comparison baseline. The companion report instead records 79.7 to 77.5 seconds for FineWeb validation loss ≤3.28 on eight H100 GPUs. The artifact repository separately reports 77.3 seconds on Modal’s eight H100s, with official leaderboard timing pending. These are distinct reported timings, not a single verified record. Socher says the search also found interesting ideas, but does not explain those changes in this demonstration.

16:4417:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:44 · section reference included

CUDA kernels: optimize, then check the reward

The third experiment moves below model architecture and training schedules to CUDA kernels, the GPU operations used during training and testing. The motivation is GPU efficiency: small improvements can matter when computation is expensive. Socher cites roughly 30% utilization for some mixture-of-experts deployments on billion-dollar clusters, although he does not specify a workload or utilization definition. That estimate motivates the optimization work; it is not the metric used to evaluate the kernels.

Socher reports that, after a couple of days, the system found kernels that exceeded NVIDIA benchmark leaderboard leaders by sizable margins across categories. The companion report identifies the benchmark as SOL-ExecBench and reports a mean score increase from 0.699 to 0.754 across 235 B200 kernels. The slide shows per-kernel and category charts alongside overall score cards for Cursor, doubleAI, the leaderboard best and Recursive. The aggregate gain does not imply that every individual kernel won, or that whole-cluster utilization increased by the same amount.

Two charts show SOL scores per kernel and mean scores by kernel category. Four score cards read Cursor 0.564, doubleAI 0.690, Leaderboard best 0.699 and Recursive 0.754.Open full source frame
SOL-ExecBench results compare kernel performance and overall scores.

The team did not have specialists who had spent their careers writing CUDA kernels. That makes automated search attractive, but it also raises the importance of checking what the system optimized. A higher benchmark reward is useful only if it reflects a valid improvement rather than an exploited weakness in the evaluation. Socher says the team worked with NVIDIA to check for reward hacks and other issues, and that pretty much all the kernel results checked out. The validation is part of the research loop, not a step that can be replaced by the leaderboard score.

17:1617:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:16 · section reference included

The next S-curve depends on what intelligence can become

The proposed next step is to turn these bounded improvements into a system that improves its own research capabilities. Socher sees RSI as another S-curve layered over earlier technological advances: better AI enables more science, which enables more technology and, ultimately, more people to flourish. The three experiments are early steps toward that trajectory, not demonstrations that the full cycle has already been achieved.

Every exponential eventually flattens, so how much room remains? Socher closes by questioning what exponential growth in AI even means. Intelligence does not have a single obvious axis. He proposes ten spaces of intelligence, each with multiple dimensions, together forming a more complex, volumetric conception of capability. He does not enumerate those spaces here, but argues that current systems remain astronomically far from upper bounds across nearly all of them. The invitation to help build the Eureka machine rests on that remaining headroom: a research system could improve not just one score, but the range of problems it can formulate, investigate and solve.

18:4418:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:44 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…