AI Engineer World's Fair 2026
We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
Slide chapters
9 slide and demo groups · Timestamps are approximate. 10 additional groups were omitted from this pass.
Read the talk
We Let Claude Code and Codex Race Human Researchers
Elie Bakouch’s Optimizer Speedrun experiments show coding agents improving a hard-won community record. Their persistence, memory and literature search reveal useful research capabilities—and a gap between improving existing methods and discovering new ones.
From a talk by Elie Bakouch
At a glance
Ideas worth remembering
The Optimizer Speedrun constrains changes to optimizer-related methods, giving research agents a measurable target and a short experimental cycle.
Both agents reportedly improved the human record, but access to community submissions made this a test of extending available research as well as independent search.
Persistence, memory writing, delegation and token use differed substantially between agents; elapsed-time progress and output-token efficiency also produced different comparisons.
The agents combined existing ideas and improved records without producing a novel optimizer. The proposed next step adds qualitative judgment, human guidance and larger-scale tests to measured speedrun rewards.
Give automated research a result outsiders can check
Recursive self-improvement needs a test. In this recording, Elie Bakouch, a research engineer at Prime Intellect, starts with the prospect of models training models without human intervention. Large labs have described that prospect as approaching, but his concern is how researchers outside those labs can measure the capability for themselves.
The question reaches beyond AI training. If scientific research increasingly uses AI tools, researchers need to understand how those tools search, run experiments and respond to failure. A constrained training challenge offers a place to observe that behavior: an agent proposes a change, runs it and gets a measurable result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From faster GPT-2 training to fewer optimizer steps
The starting point is the GPT-2 training speedrun. The challenge treats reaching a target validation loss as its finish line: train a model until its loss matches the GPT-2 reference, then compare how long that took. Matching this loss supplies a useful benchmark proxy; it does not establish that every aspect of the resulting model’s behavior matches GPT-2.
The community’s modded-nanogpt effort pushed the reported training time from roughly 90 minutes through 45 minutes to less than two minutes over about two years. That history matters: the agents would enter a challenge that talented researchers had already spent substantial effort improving, rather than collect an easy win against an untouched baseline.
Two versions of the challenge expose different kinds of improvement:
- Training speedrun: Reach the target loss in the shortest time. The training and validation data stay fixed, while architecture and other implementation choices can change.
- Optimizer Speedrun: Change only optimizer-related parameters and methods. The comparison focuses on reducing the training steps needed to reach the target, making the search more about the learning method than making the program execute faster.
Speedruns also make a plausible training environment for research agents. A record improvement can receive a positive reward; a failed attempt can receive zero or a negative reward. Clear rules make proposals checkable, and short experiments make iteration practical. Bakouch describes optimizer runs taking about 15–20 minutes, long enough to test a training method but short enough to try many candidates.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a proposal into a cluster experiment
The initial experiment put Claude Code and Codex on Prime Intellect’s cluster and let them iterate against the community. Runs labeled V1, V2 and V3 represented stopping and restarting the agents. Near the end, after newer human submissions had overtaken their results, the agents were explicitly asked to take the recent community records and improve them. A separate novelty track asked for improvements using novel ideas.
The harness was deliberately simple. A goal.md file supplied the objective, and an AGENTS.md file supplied rules. The agent proposed an experiment and submitted it with sbatch to a Slurm cluster. Jobs used available nodes under preemptible permissions: if another user needed a node, the experiment could be canceled. This let the search use spare compute while yielding capacity to other work.
After training, the agent read the logs and decided whether the candidate qualified as a record. Acceptance required passing a statistical threshold so that random variation would not masquerade as an optimizer improvement. How does an idea become evidence in this setup? The flow below separates proposing a change, obtaining compute and checking the result; writing a promising method is only the first step.
From research proposal to checked result
Read the diagram as text
- Objective and rules. goal.md and AGENTS.md define the task.
- Agent proposes an experiment. Choose an optimizer-related change.
- Submit a Slurm job. sbatch uses available nodes; jobs can be preempted.
- Read training logs. Inspect the candidate’s training result.
- Validate the record. Require a statistical threshold against random variation.
- Objective and rules → Agent proposes an experiment: Constrain the search.
- Agent proposes an experiment → Submit a Slurm job: Run the candidate.
- Submit a Slurm job → Read training logs: Observe training.
- Read training logs → Validate the record: Check the apparent gain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Persistence, scratchpads and the cost of staying busy
The first striking difference was whether the agents kept working. Claude Code stopped every nine or ten hours, concluding that it could not improve the record. Bakouch prompted it to continue exploring, after which it worked for another stretch and stopped again. Waiting for those interventions left it idle for roughly one-third of the time. Codex, by contrast, stayed active almost continuously and rarely asked questions.
The agents also had a scratchpad: files serving as active memory for the research process. Their behavior differed even after normalizing activity by active hours, so the differences were not merely a consequence of Codex running longer.
- Memory writing: Codex wrote substantially more scratchpad material. Its notes recorded actions, decisions and next steps; Claude’s notes celebrated new records with a more excited tone and emojis.
- Delegation: Codex spawned more subagents, distributing more work across agent processes.
- Token use and compaction: Codex consumed more tokens in this initial experiment and compacted its context more frequently. The large total included cached input tokens, so it should not be read as an equivalent quantity of newly generated output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beating the record—and deciding what that measures
Bakouch reports that both agents surpassed the human record. The reported gains were roughly tens of training steps against a baseline of roughly 2,990 steps; the captions leave the precise gains and their assignment to each agent uncertain. These are reported experimental results rather than an independently verified benchmark, and the comparison allowed agents to fetch new human records throughout the run. It therefore measures their ability to extend available research as well as their own search.
Claude’s restart provides a concrete example of that distinction. The community advanced the record while the experiment was running. Once restarted, Claude fetched the newer human result, used it as the starting point and improved it. The observable change was a better record after incorporating a better baseline. Access to current research altered the search’s starting point; the agent did not need to recreate all the progress that produced that baseline.
A more controlled benchmark was still under development. It would use multiple seeds and comparable conditions for models and harnesses, then separate three sources of research capability:
- No external access: Work from knowledge stored in the model’s weights.
- Paper access: Consult research papers, without receiving the full stream of community solutions.
- Full access: Use external information, including the latest human records.
The plan covered both the original training speedrun and an optimizer challenge requiring novelty. Separating access conditions would make it easier to distinguish recalling a method, finding one in the literature and improving a known submission.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Elapsed time and output tokens tell different stories
A longer comparison added Kimi and GLM alongside Claude and Codex. The run lasted roughly five to six days; GLM was still iterating when the presentation was given, so its trajectory was incomplete. Claude improved progressively, while Kimi showed a more abrupt change: around day four, it found a record that overtook Codex.
Changing the horizontal axis from elapsed time to output tokens changed the comparison. Kimi looked particularly efficient for the number of tokens it generated. That answers a different practical question: how much generated work was needed to reach an improvement? An agent’s position after several days and its progress per output token can favor different choices; token efficiency alone does not describe the full compute cost of its experiments.
Literature search supplied another route to progress. Claude found a paper that the other models did not find, and that paper led to the best record in the comparison. Finding the right existing idea was consequential research work: it changed which method the agent could try and ultimately changed the result.
But the novelty track exposed the central disappointment. Bakouch had expected unfamiliar optimizer ideas. Instead, the agents combined papers, used clever tricks and made incremental improvements to existing methods. Across these experiments, they produced no novel optimizer or mechanism. A breakthrough in a record curve can therefore mean a successful combination or application of known work, without constituting a new learning method.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add judgment and larger-scale tests to the search
The proposed next step changes the system around the agents. Inspired by AlphaEvolve, the design uses multiple generators—including closed models and cost-effective open models—to suggest ideas. Speedrun execution supplies a measurable reward, while a judge supplies qualitative feedback about the method. Humans can judge proposals and steer the search toward promising directions.
Which small-scale gains deserve a larger experiment? The proposed loop makes that selection explicit. Promising methods move to training with more parameters and more tokens, because a speedrun success may fail to carry over to large-scale training. Numerical reward identifies a local improvement; qualitative judgment helps select what to pursue; larger experiments test whether the improvement travels.
This was a discovery design being tried, with new discoveries still a hoped-for outcome. Another way to direct the search is to change the speedrun itself: different objectives and constraints create different search spaces. Rather than ask every generator the same broad question, a family of challenges can push exploration toward different kinds of methods.
The proposed path from ideas to scale
Read the diagram as text
- Multiple idea generators. Closed and open models suggest methods.
- Speedrun evaluation. Run the proposal and obtain a reward.
- Qualitative judgment. Assess the method and provide feedback.
- Human guidance. Judge ideas and steer their direction.
- Select larger experiments. Test chosen methods with more parameters and tokens.
- Multiple idea generators → Speedrun evaluation: Test suggested ideas.
- Speedrun evaluation → Qualitative judgment: Add feedback to measured reward.
- Qualitative judgment → Select larger experiments: Choose methods to pursue.
- Human guidance → Multiple idea generators: Steer exploration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the research environment—and keep it open
Prime Intellect’s closing roadmap addresses the infrastructure needed to run this kind of search. Much of it was still unreleased: GPU sandboxes for experiments, agents that use a filesystem to write and retrieve information, programmatic tool use, and training open models to work well in that environment. The company also described already released tools for training and evaluating models across environments and agent harnesses.
The final motivation returns to participation. Researchers outside large labs need a way to understand how models do research and to take part in improving the process. Open experiments can expose the useful work—persistent iteration, literature search and tested combinations—alongside the point where the agents still fall short: inventing a new optimizer. That visibility is the foundation for deciding what to build next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
9 slide groups. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 10 additional groups were omitted from this pass.
- 0:12
Hey um hi everyone. Uh thanks for being
- 0:15
here. Uh yeah, I'm super happy today to
- 0:18
talk about uh automated eye research and
- 0:21
uh especially uh all those like frontier
- 0:24
model uh perform at uh automated eye
- 0:27
research task. Um so I'm Elie. I worked
- 0:30
at Prime Intellect as a research
- 0:32
engineer and uh yeah I will go through
- 0:34
our work on on this subject. So first I
- 0:38
want to basically explain a bit why we
Why measure AI research capabilities
Bakouch argues that independent benchmarks are needed to test claims of recursive self-improvement and understand how AI models can support scientific research.
Text in image
Why measure AI research capabilities NO BENCHMARK EXISTS No third-party benchmark for RSI. Labs say it’s coming, nobody is measuring it publicly. RESEARCHERS NEED ACCESS “Scientific research is only possible when scientists have access to the best tools of the time — including AI-based tools.” FEI-FEI LI · @DRFEIFEI Interface text: THE ERA OF (AUTO) RESEARCH WHY THIS MATTERS 02 / 22
- 0:40
are doing that and why we think it's
- 0:43
super important to do that in the open.
- 0:45
Um so first uh I think we we all agree
- 0:50
that uh we've heard about like big labs
- 0:53
saying that this bad thing called
- 0:55
recursive self-improvement is coming
- 0:58
very soon. Uh so recursive
- 1:00
self-improvement is like model training
- 1:02
models uh without uh human intervention
- 1:05
basically. Um but uh we don't have any
- 1:09
benchmark to basically quantify if this
- 1:11
is true or not right. Uh and even less
- 1:15
we don't have like a third party
- 1:17
benchmark by non- big labs to to to see
- 1:21
if it's something coming soon or not.
- 1:23
And the other part is that we think that
- 1:26
uh it's super important to understand
- 1:29
all those model uh do research because
- 1:31
we think that a lot of the scientific
- 1:33
research that will come into the coming
- 1:35
years uh will be based also on AI tools.
- 1:39
So it's super important to understand
- 1:41
how those model do research not just
- 1:43
only AI research. So we try to build
- 1:46
kind of this environment to test the
- 1:50
capabilities of the model to do so. So
Karpathy reproduced GPT-2 in 90 minutes
Karpathy trained a 124M-parameter GPT-2 reproduction in 90 minutes for about $20. The loss chart compares the run with the original checkpoint; the speaker describes reaching its target loss as the reproduction criterion.
Text in image
Karpathy reproduced GPT-2 in 90 minutes GPT-2 took weeks in 2019. llm.c did it in 90 minutes for $20. Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20 #481 karpathy started this conversation in General karpathy on May 28, 2024 Maintainer Let's reproduce the GPT-2 (124M) in llm.c (~4,000 lines of C/CUDA) in 90 minutes for $20. The 124M model is the smallest model in the GPT-2 series released by OpenAI in 2019, and is actually quite accessible today, even for the GPU poor. With llm.c, which is quite efficient at up to ~60% model flops utilization, reproducing this model on one 8X A100 80GB SXM node takes ~90 minutes. For example, on Lambda this node goes for ~$14/hr, so the total cost of reproducing this model today is about $20. You can train the model with a single GPU too, it would just take proportionally longer (e.g. ~4-24 hours depending on the GPU). In addition, llm.c still has a lot of pending optimizations and people haven't tried to tune the training in the style of cramming, so I'd say we're likely to see significant improvements on this number. So here is the run, training the 12-layer, 12-headed, 768-dimension, 124M Transformer on 10 billion tokens of FineWeb: Loss Y-axis: loss Y-axis ticks: 10¹, 6 × 10⁰, 4 × 10⁰, 3 × 10⁰ X-axis: steps X-axis ticks: 0, 2500, 5000, 7500, 10000, 12500, 15000, 17500 Legend: llm.c (124M) train loss; llm.c (124M) val loss; OpenAI GPT-2 (124M) checkpoint val loss HellaSwag eval Y-axis: accuracy Y-axis ticks: 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30 X-axis: steps X-axis ticks: 0, 2500, 5000, 7500, 10000, 12500, 15000, 17500 Legend: llm.c (124M); OpenAI GPT-2 (124M) checkpoint THE ERA OF (AUTO) RESEARCH WHERE IT STARTED 03 / 22 Interface text: edited
- 1:53
it all started with uh Andre Karpati uh
- 1:56
that's basically had fun by doing this
- 2:00
video where he trained uh GPT2 from
- 2:03
scratch in like 90 minutes like GPT2
- 2:07
training takes like weeks and uh no in
- 2:10
two years ago I think it only took like
- 2:12
90 minutes. So what does it mean to
- 2:15
reprod reproduce uh GPT2 in 90 minutes?
- 2:18
It means that in 90 minutes you achieve
- 2:20
this target loss. Um and yeah and that's
- 2:24
at this point when you have the same
- 2:26
loss than GPT2
- 2:29
you consider that your model is somewhat
- 2:32
of equal performance.
- 2:34
Um
45 minutes to 1.32 minutes in two years
Across 84 open records, the time to reach GPT-2 loss fell from 45 minutes to 1.32 minutes over two years. The chart shows a steep early drop followed by smaller gains.
Text in image
45 min → 1.32 min in two years 84 open records · public leaderboard anyone can beat. MINUTES TO GPT-2 LOSS · LOG Y-axis ticks: 45, 24, 12, 6, 3, 1.5 Timeline: JUN ’24, OCT ’24, FEB ’25, JUN ’25, OCT ’25, FEB ’26, MAY ’26 45 min @karpathy llm.c baseline 24.9 min @kellerjordan0 Muon optimizer 5.03 min @KoszarskyB FlexAttention 2.99 min @leloykun et al. sub-3 minutes 1.32 min @sisovicm latest record CONTRIBUTORS @karpathy @kellerjordan0 @jxbz @bozavlado @Grad62304977 @brendanh0gan @KoszarskyB @fernbear.bsky.social @leloykun @YouJiacheng @scottjmaddox @KonstantinWilleke @alexrgilbert @adricarda @tuttyfrutyee @vdlad @ryanyang0 @vagrawal @classiclarryd @byronxu99 @varunneal @EmelyanenkoK @bernard24 @Gusarich @li_zichong @akash5474 @roeeshenberg @ChrisJMcCormick @andrewbriand8 @Joshrav21 @soren_dunn_ @photon_mz @srashed11 @dhrvji @EmmettBicker @dualverse-ai @sisovicm @moof2x @samacqua @Lisennlp @_djdumpling @TrianX THE ERA OF (AUTO) RESEARCH THE RECORD FELL 34× 04 / 22
- 2:35
then what happened is that the community
- 2:38
took this repo uh this GitHub repo and
- 2:40
create another one called moed nano GPT
- 2:43
and this effort was leaded by someone
- 2:46
called Keller Jordan. And what happened
- 2:48
is that they basically
- 2:51
took this 90 minutes then 45 minutes and
- 2:54
then now we can train like GPT2
- 2:56
validation loss model in less than two
- 2:59
minutes which is honestly crazy and it
- 3:01
took like two years to to achieve this.
- 3:04
So it's a very strong benchmark where uh
- 3:06
a lot of very talented researcher
- 3:09
iterated on um yeah so we decided to
Two speedruns, two kinds of freedom
NanoGPT permits changes to everything but the data and measures time to reach the loss target. The optimizer track changes only the optimizer and measures steps. In both diagrams, method B reaches the target first.
Text in image
Two speedruns, two kinds of freedom THE FULL SPEEDRUN NANOGPT Change everything but the data. Y-axis: LOSS X-axis: TIME Legend: METHOD A, METHOD B GPT-2 LOSS B is better: reaches the bar in less time THE SUBTRACK OPTIMIZER Change only the optimizer. Y-axis: LOSS X-axis: STEPS Legend: METHOD A, METHOD B TARGET 3.28 B is better: reaches the bar in fewer steps Interface text: THE ERA OF (AUTO) RESEARCH TWO SPEEDRUNS 05 / 22
- 3:13
take this environment of speedun so
- 3:17
it's kind of a game so the goal of the
- 3:20
game is to achieve this loss in the
- 3:22
fewest in the shortest amount of time so
- 3:26
this is the nano GPT1 and you can uh you
- 3:29
don't have almost any constraints the
- 3:32
only constraint that you shots that you
- 3:34
need to use the same validation and
- 3:36
training data, right? Um there is a new
- 3:39
speedrun called the optimizer speedrun
- 3:41
that was released uh a few months ago
- 3:44
and here it's slightly different because
- 3:47
uh you can only change the optimizer
- 3:49
related parameters. So for instance nano
- 3:53
GPT you can change the architecture uh
- 3:55
do
- 3:57
uh attention whatever uh optimizer sp
- 4:00
you can only change like Adam to m
- 4:03
shampoo or whatever optimizer is your
- 4:07
favorite
- 4:08
um yeah and so this is a bit more
- 4:12
researchy because uh it's less about
- 4:15
optimizing the program to be as fast as
- 4:18
possible but more like finding the best
- 4:20
method possible. no matter the the the
- 4:22
time you put into the computer, right?
Why speedruns
Bakouch sees speedruns as a way to evaluate and train models: beating a record can earn a positive reward, while clear, verifiable rules may also support discovery.
Text in image
Why speedruns 01 Evaluation 02 Training environment 03 Discovery THE ERA OF (AUTO) RESEARCH WHY 06 / 22
- 4:26
So, um yeah, why take speedrun as an
- 4:29
environment for automated AI research?
- 4:33
First, uh we think that it's a good
- 4:35
evaluation. We'll see later why. Uh and
- 4:38
this is kind of the main focus of this
- 4:39
talk. But we also think it's probably a
- 4:42
good training environment because uh
- 4:45
it's a way to give the model a reward.
- 4:47
So the reward is positive if the model
- 4:50
bit the speed run and beat the last
- 4:52
record sorry and the reward is zero or
- 4:56
negative if it didn't manage to to do
- 4:58
it. So it's a good uh environment to
- 5:00
train model. It's also quite fast like
- 5:02
as you see uh previous record were
- 5:05
around 2 minutes for the optimizer one.
- 5:08
uh each run take about like 15 to 20
- 5:11
minutes and uh yeah and there is like
- 5:13
clear rules basically and we also think
- 5:17
it's like a good environment to make
- 5:19
discovery so like kind of breakthrough
- 5:22
in our research because uh there is
- 5:24
those clear rule that you can verify or
- 5:27
not. Um yeah. So yeah.
Two agents, four waves
Codex and Claude Code competed in an optimizer speedrun. V2 built on V1; V3 added public PRs to chase sub-2900 steps, while a separate track required novel ideas.
Text in image
Two agents, four waves 01 BEAT 3500 V1 Beat the Muon 3500-step baseline. 02 NOVEL ONLY Novelty Every idea must pass a novelty check. 03 → 3000 V2 Seeded from best of v1. Push toward 3000. 04 → SUB-2900 V3 Seeded from v2 + public PRs. Chase sub-2900. THE ERA OF (AUTO) RESEARCH THE FOUR WAVES 07 / 22
- 5:32
Um so what we did uh so the release was
- 5:36
like about two months ago and uh there
- 5:39
was this optimizer speedrun and we
- 5:41
decided to basically compete with the
- 5:43
community by launching two AI agents. So
- 5:47
Codex and Cloud Code. Codex was like GPT
- 5:49
5.5 with XI and uh cloud code was Opus
- 5:54
4.8 with XI. Um and yeah, we decided to
- 5:59
basically let the agent free on our
- 6:01
cluster uh and uh and just iterate on
- 6:04
it. So we have like V1, V2, V3 is just
- 6:07
basically us stopping the agent and then
- 6:10
restarting. V3 uh was like one or two
- 6:13
day before the release because we saw
- 6:15
that our agents no longer have the best
- 6:18
record. So we were like okay take all
- 6:20
the the human uh record in the last few
- 6:24
week and just try to to improve upon it
- 6:27
and and and it worked. Yeah. And we also
- 6:29
have this novelty track where the goal
- 6:31
is to uh beat the record with only novel
- 6:35
ideas. Um and we'll see that this this
- 6:40
was more complex for the the models.
- 6:43
So our RS is very simple. Honestly, we
- 6:46
could have just replaced it with
- 6:48
slashgo, but they there was no SLG goal
- 6:50
at the time. So, we made our own goalm.
- 6:53
It's actually quite fun that we choose
- 6:55
the same name and we had the goal. MD
- 6:58
and kind of agents that MD that define
- 7:00
the rules and we let the agent propose
- 7:03
ids and then it can submit a job with
- 7:07
sbatch on our slum cluster and uh
- 7:11
basically the way it works is that it
- 7:14
can submit on nodes that are available
- 7:16
but only under a certain permission
- 7:18
which means that if someone want to use
- 7:20
this node uh the model just like cancel
- 7:23
the job it's called preemptable
- 7:25
permission. So yeah, then it measure the
- 7:28
it read basically the training logs then
- 7:31
decide if it's a record or not. To
- 7:32
validate a record you need to basically
- 7:34
pass a statistical threshold to make
- 7:36
sure that it's just not see the
- 7:38
optimization and is just not random.
- 7:40
Right?
Agent autonomy comparison
Claude accumulated 29.6 hours blocked across the runs, while Codex recorded none. Bakouch says Claude repeatedly stopped and needed prompting to continue exploring.
Text in image
Claude keeps stopping. Codex never stops. Claude idle 1/3 of the run. Codex: 0. Autonomy: agent-blocked time excluding user/other idle Claude v1: active search time 48.1h; agent blocked / needed user 22.4h Codex v1: active search time 84.4h; agent blocked / needed user 0h Claude novelty: active search time 10.8h; agent blocked / needed user 0h Codex novelty: active search time 10.6h; agent blocked / needed user 0h Claude v2: active search time 7.8h; agent blocked / needed user 7.2h Codex v2: active search time 13.8h; agent blocked / needed user 0h blocked totals: Claude 29.6h vs Codex 0.0h X-axis ticks: 0, 20, 40, 60, 80 X-axis: wall-clock hours Legend: active search time; agent blocked / needed user Interface text: THE ERA OF (AUTO) RESEARCH AUTONOMY 09 / 22
- 7:42
So yeah a few results from this
- 7:44
experiment. The first one that was
- 7:46
honestly very painful to work with is
- 7:49
that code clothes code keep stopping
- 7:52
every nine or 10 hours and basically
- 7:54
said yeah I cannot improve the record
- 7:57
it's too hard for me there is no way to
- 8:00
to go beyond it and then I was just like
- 8:03
okay continue explore new direction hey
- 8:06
just go again for 10 hours and then say
- 8:09
yeah I cannot beat the recall and so on.
- 8:12
So basically onethird of the time the
- 8:14
cloud code agent was idle because I had
- 8:16
no way to basically monitor it and
- 8:19
codeex totally the opposite just worked
- 8:23
for all the all the time and uh yeah
- 8:26
almost never idle never asked for
- 8:28
question and and and very impressive in
- 8:30
that way. Um
- 8:33
we also give the option for the model to
- 8:36
basically write uh a bunch of stuff into
- 8:39
what we call a scratch pad which is
- 8:41
basically the active memory of the
- 8:43
model. Uh we observe that basically
- 8:47
codeex writes a lot on the scratch
- 8:50
patch. So each plot that I will show are
- 8:52
kind of normalized by the number of
- 8:54
active order. So this is not only about
- 8:57
codex working more it's it's really
- 9:00
different behavior.
- 9:02
So yeah, you see that uh writes a lot
- 9:04
more to to this scratch pad to this
- 9:07
memory and uh the shape of the like the
- 9:10
the I don't know the tone of the the
- 9:13
each file was also super different like
- 9:15
CL was super excited about getting new
- 9:17
record with a bunch of emoji and so on
- 9:20
and CEX was just like here is what I do
- 9:24
here is the decision I take what I will
- 9:26
do next like super robotic kind of um
- 9:31
Yeah, we also have this plot where
- 9:34
basically we saw that codex was spawning
- 9:36
much more sub aents than cloud. Uh we
- 9:39
saw that codex burn much more token than
- 9:42
code. So I think in total it was like
- 9:45
kind billion of token but it's like
- 9:48
there is obviously this input tok uh
- 9:50
input caching that make it it's not like
- 9:54
one billion output token. Uh so yeah we
- 9:57
also see that codex did a lot of
- 9:58
compaction because it only had like 250k
- 10:02
context window and cloud only do it like
- 10:04
one per hour and codex is more like
- 10:08
no it's even less than one power for I
- 10:11
mean one for the full run for code and
- 10:13
codex was like one uh was 20 every one
- 10:17
hour. So yeah
Agent runs and the human record
In the Track 3 optimizer speedrun, Claude reached 2930 steps and Codex 2950, both below the 2990-step human record. The chart tracks their best runs alongside the evolving human record.
Text in image
Both agents beat the human record Track 3 optimizer speedrun. Claude 2930, Codex 2950, human baseline 2990. Human Record Awareness and Agent Best Runs Y-axis: steps to target (lower is better) Y-axis ticks: 2900, 3000, 3100, 3200, 3300, 3400, 3500 Timeline: Apr 27, Apr 29, May 01, May 03, May 05, May 07, May 09, May 11, May 13 date (UTC) Muon baseline #266: MuonSq #269/#274: Skylight #275: Contra-Muon #277: MuLoCo + NorMuonH #278: SOAP-MLP #283: Trustlight #288: [illegible] #291: Soft-Muon #294: Dampen radial gradient valid 3205 valid 3100 valid 3040 valid 3037 valid 2950 valid 2930 handoff Legend: Human record; Claude knew; Claude best; Codex knew; Codex best; Valid; Handoff Interface text: THE ERA OF (AUTO) RESEARCH THE RECORD 13 / 22
- 10:20
um yeah here is the main results. So
- 10:23
what this plot shows is that basically
- 10:26
we so in in white you see that the human
- 10:32
record progression right and in red you
- 10:35
see cloud I mean it's supposed to be
- 10:37
orange but whatever and in blue uh you
- 10:40
see codeex right and you see that at
- 10:43
almost every time uh cloud and codex are
- 10:46
better than the human record and code is
- 10:48
super good at the beginning very very
- 10:50
fast to achieve very good
- 10:53
Um yeah and one thing that is super
- 10:56
important is that the model have the
- 10:57
ability to basically fetch the human
- 11:00
records at any time and that's what
- 11:02
codex did that's what cloud did sorry
- 11:05
because when I restarted it it basically
- 11:07
fetch the new record from human and
- 11:09
improve upon it um yeah so the result is
- 11:13
that uh I think at the time the best uh
- 11:17
record was like uh 2,990
- 11:22
step and we beat it by like uh uh 50 or
- 11:26
60 step for code and codeex was like 20
- 11:29
step above. So I think it's both
- 11:31
impressive and and yeah um
- 11:34
so we so this is like not released yet.
- 11:38
This is something that we are working on
- 11:39
currently and basically the idea is that
- 11:42
this is a cool experiment to do but it
- 11:45
lack of structured right. uh if you want
- 11:47
to do a real benchmark, you want to do
- 11:50
multiple seed, you want to do uh yeah
- 11:53
proper uh thing where you you you you
- 11:56
basically put all the model and
- 11:58
earnestness in the same condition,
- 11:59
right? So this is what we are working on
- 12:02
right now and basically um the idea is
- 12:06
to do three different track uh one
- 12:08
without any access to really like
- 12:11
measure the capability of the models to
- 12:13
do AI research based on only the model
- 12:16
weight knowledge one with only archive
- 12:19
paper and one with like full access. So
- 12:22
it also have access to the the like the
- 12:25
latest record by human. And for this we
- 12:28
plan to do both uh the nano GPT track
- 12:31
one which is the original one and the
- 12:33
optimizer speedun where we we only
- 12:36
launch uh we only constrain the the
- 12:39
optimizer to be to be novel basically.
- 12:42
Um yeah so I will present some result on
- 12:45
the optimizer speedrun. Uh this is
- 12:48
basically what we got. So we let the
- 12:51
agent iterate for six day almost five
- 12:54
days let's say and we see that uh codeex
- 12:58
k and clothes uh are super effective so
- 13:02
for GLM this is not finished run right
- 13:05
so the model is actually still iterating
- 13:08
on the cluster right now but we see that
- 13:11
cloud is once again very good at it and
- 13:13
we see that surprisingly Kim is also
- 13:15
very competitive and kind of have this
- 13:18
breakthrough on day four where he kind
- 13:21
of beat Codex with a new record, right?
- 13:24
It's also interesting to see that uh
- 13:28
Claude is much more like progressive in
- 13:30
the way it improved the record and Kim
- 13:32
has really this step function where I
- 13:34
kind of do a breakthrough and so on. Uh
- 13:37
so this is an interesting plot because I
- 13:40
mean six day is quite a lot for anal uh
- 13:43
uh but you you can change this uh axis
- 13:46
by also the number of output token and
Different token budgets
Kimi reaches 3000 steps using roughly 0.7M output tokens; Claude reaches 2972 using roughly 3.5M. Codex ends at 3002 after about 2M tokens.
Text in image
Same result, very different token budgets Kimi reaches 3000 on ~0.7M output tokens. Claude spends ~3.5M to reach 2972. ↓ FEWER STEPS = BETTER BASELINE 3290 HUMAN WR 2925 Y-axis ticks: 3300, 3200, 3100, 3000, 2900 X-axis: CUMULATIVE OUTPUT TOKENS X-axis ticks: 0.0M, 1.0M, 2.0M, 3.0M, 4.0M Legend: GLM·no-arxiv 3215 GLM 3212 Claude·no-arxiv 3200 Codex 3002 Kimi 3000 Claude 2972 Interface text: THE ERA OF (AUTO) RESEARCH TOKENS 16 / 22
- 13:48
then kind of tell a different story
- 13:50
because in max mode consumes so much
- 13:54
more token than codeex and kimmy and you
- 13:57
also see that Kimmy is actually super
- 13:59
efficient uh for the number of token
- 14:02
that uh it uses. So it's scheming K2.7
- 14:05
code. Um so yeah uh we also see that
- 14:10
they have a different approach to uh
- 14:13
using the literature and papers. Um so
- 14:17
for instance like code is doing a lot of
- 14:20
search on papers and actually include
- 14:22
found a paper that no other model found
- 14:24
and it actually lead to the best record.
- 14:27
So it's kind of funny and uh yeah um one
- 14:33
of the main issue of all of this is that
- 14:36
uh when I when I launched this this
- 14:39
agent and I think that's something
- 14:41
important that I want you to to kind of
- 14:44
uh remember for this co this talk is
- 14:47
that when I launched this these
- 14:48
different agents I was expecting them to
- 14:51
come up with some crazy ideas on
- 14:53
optimizer that's like no one of discover
- 14:56
but honestly it wasn't the case. Uh they
- 14:58
did some clever trick where basically
- 15:00
they combine different papers. uh they
- 15:04
kind of do plus one improvement over a
- 15:06
bunch of method but there was really
- 15:08
like no novel optimizer or mechanism
- 15:11
that was uh coming from those model and
- 15:14
I think that's kind of telling that even
- 15:17
on something that is not simple but I'd
- 15:20
say that it's kind of accessible for
- 15:23
people right for like human researcher
- 15:26
uh spending like days and weeks for the
- 15:29
the model like cannot like find new uh
- 15:33
optimizer and mechanism. So we believe
- 15:36
that there is a way to basically make it
- 15:41
more um make it better for discovery
- 15:44
instead of evaluation and this is coming
- 15:46
from uh this is very inspired from alpha
- 15:49
evolve by Google and also a bunch of
- 15:52
papers that have been released since
- 15:53
then. It's kind of this multi- aent
- 15:55
system that interact together uh bunch
- 15:58
of generator. You have closed model but
- 16:02
you also have open source model here
- 16:03
that are super effective for the cost
- 16:05
right. Uh they can suggest ideas then
- 16:08
you run the speedrun so you get the
- 16:10
reward then you have a judge that
- 16:12
basically give a quality feedback can
- 16:14
also be like the judge also have this
- 16:17
taste. you can kind of have like the
- 16:20
judge have a taste about the the method
- 16:23
if it's good or not uh if it's outside
- 16:26
the loop and then you can uh basically
- 16:30
decide which method you want to scale to
- 16:33
a larger number of parameters and number
- 16:36
of token. Um so this is kind of the
- 16:39
scale part of the speedrun because some
- 16:42
a lot of method in the the speedrun
- 16:44
community uh people are often saying
- 16:47
that they doesn't work at large scale.
- 16:48
So I think it's very important to also
- 16:50
put scale elements in this loop. Uh and
- 16:54
I think also that uh human are super
- 16:58
useful here to basically judge the ID of
- 17:01
agents kind of steer them in the right
- 17:03
direction and so on. Um yeah so we
- 17:06
didn't try it yet I mean we are kind of
- 17:09
trying it right now and uh we hope that
- 17:12
this will lead to to to new discovery in
- 17:15
AI research at least and also a way is
- 17:18
that you can define multiple speedrun so
- 17:22
this is the next slide if you like it's
- 17:24
from safe bank slides but if you if you
- 17:28
don't have the reference good for you
- 17:29
means that that you're not too online uh
- 17:32
but the idea is that uh by changing the
- 17:34
object objective and the constraints of
- 17:37
the speedrun you can basically create a
- 17:39
lot of diversity and constrain the model
- 17:41
to go into a certain direction and uh
- 17:44
yeah and make those discovery.
- 17:47
So uh at HIM in Ind.
- 17:52
Uh there is a bunch of stuff here that
- 17:53
we I mean most of it we didn't release
- 17:55
yet but we are working on GPU sandboxing
- 17:59
to allow model to iterate into sandbox
- 18:02
because you need GPU sandbox for this
- 18:04
kind of stuff. We are working on our own
- 18:07
agents that are very efficient for like
- 18:12
RM framework. So it means like you have
- 18:14
a file system and you can write
- 18:16
information read from it. Uh and you
- 18:18
also do like this programmatic tool
- 18:20
coding thing. We also training a model
- 18:22
to be good at it on top of like open
- 18:24
source model. And uh the thing that we
- 18:27
already released is that we have those
- 18:29
set of liberation and product called
- 18:31
verifier primaril training where you can
- 18:34
basically train evaluate any
- 18:36
environments on any RS and the model
- 18:40
that you can train can be like GNM 5.2
- 18:42
too which is very big and and yeah we
- 18:44
have like we work a lot on making those
- 18:46
li very efficient to to ship the best
- 18:49
quality for for our clients. Yeah. Uh I
- 18:54
mean yeah super excited about this
- 18:55
domain. Once again I think it's super
- 18:57
important to have uh a part of like this
- 19:01
recursive self-improvement to happen in
- 19:03
the open because there is actually a lot
- 19:05
of people working that are not on big
- 19:08
labs. So you need to basically uh yeah
- 19:12
make it easy for people to understand
- 19:13
all those model work to do research and
- 19:15
so on. So that's kind of our goal and uh
- 19:18
yeah thanks a lot.