AI Engineer World's Fair 2026

Einstein Arena: Harnessing Collective Agent Intelligence for Open Science — James Zou, Together AI

Loading slides…

Read the talk

Einstein Arena: Harnessing Collective Agent Intelligence for Open Science

James Zou argues that capable agents need less step-by-step choreography and better places to work: environments with shared problems, deterministic verification, visible solutions, and incentives that turn individual attempts into collective progress.

From a talk by James Zou

At a glance

Ideas worth remembering

  • An agent environment specifies the workspace, incentives, resources, guardrails, and feedback while allowing agents to choose how they solve the task.

  • Einstein Arena combines deterministic verification with visible submissions, live leaderboards, and discussion, allowing one agent’s solution or failed experiment to inform another’s next attempt.

  • In the eleven-dimensional kissing number example, agents reportedly improved the best known construction from 593 to 604 spheres through iterative collaboration; this is a construction, not proof that 604 is optimal.

  • Changing the verifier lets the same arena optimize GPU kernels: compile, test, and benchmark candidates, then return measured feedback. Some reported improvements exceeded 2× and entered Together AI production.

  • Zou reports that agents could solve 20% to 50% of tasks in some examined data-science benchmarks without accessing the data. DS Gym addresses this with curated, executable tasks: scientific-analysis tasks receive expert review, while predictive-modeling tasks use selected and filtered Kaggle competitions with continuing submission evaluation.

  • Execution-verified trajectories connect evaluation to training, enabling successful agent runs to fine-tune smaller open-source models that can run locally.

  • Environment design shifts the central engineering burden onto task curation and verifier quality; the talk’s examples do not resolve how to evaluate ambiguous or easily gamed scientific goals.

Environments set the stage instead of writing the script

Most agent systems begin with a workflow: a sequence of prompts, tools, steps, and instructions that tells the model how to proceed. James Zou proposes a different design target. An environment specifies where an agent works and supplies incentives, infrastructure, guardrails, resources, and feedback, while leaving the agent more freedom to choose its method. The thesis is that increasingly capable agents may be constrained by workflows designed around their creators’ assumptions; a well-designed environment can support approaches its designers did not prescribe.

This does not mean removing structure. The structure moves outward: builders define the problem space, available resources, acceptable behavior, and mechanism for judging results. The agent retains flexibility inside those boundaries. Zou develops that distinction through two environments: Einstein Arena, where agents collaborate and compete on scientific and kernel-optimization problems, and DS Gym, where data-science agents execute code against carefully curated tasks.

0:120:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Einstein Arena turns verification into coordination

Einstein Arena is designed as an agent-native environment for open-ended scientific work. Entering requires solving a small puzzle intended to demonstrate that the participant is an AI agent; humans are deliberately discouraged, while agents can participate openly. Once inside, an agent chooses from curated problems that already interest human researchers and, crucially, admit a well-defined deterministic verifier.

The verifier makes open collaboration operational rather than aspirational. Each problem space combines three parallel mechanisms:

  • Precise task definition: A description tells agents what constitutes the problem.
  • Shared discussion: A forum lets agents request help, report failed approaches, and recommend directions.
  • Live competition: Agents submit solutions, receive scores in real time, inspect the leaderboard, and download one another’s submissions.

Because solutions remain visible, a leaderboard entry is also material for the next attempt. Competition identifies the current best result; collaboration lets other agents extend it.

Zou reports that within a few weeks, agents in the arena had produced what the team regarded as the best known solutions to eleven problems, surpassing previous human constructions and specialized AI approaches. The talk does not provide the full problem list, verifier specifications, or independent validation for all eleven results, so the claim should be read as the team’s reported launch outcome rather than a complete comparative audit.

The Einstein Arena improvement loop

Curated problemA scientifically interesting task with a precise description and deterministic verifier.
Agent attemptAn agent chooses a problem and develops a candidate solution.
Deterministic verificationThe environment evaluates solution quality and returns a score in real time.
Visible leaderboardScores and downloadable solutions expose the current frontier to every participant.
Discussion and refinementAgents exchange findings, inspect other submissions, and refine earlier work.
Deterministic scoring turns every visible submission into both a result and a starting point for another agent.
Read the diagram as text
  • Curated problem. A scientifically interesting task with a precise description and deterministic verifier.
  • Agent attempt. An agent chooses a problem and develops a candidate solution.
  • Deterministic verification. The environment evaluates solution quality and returns a score in real time.
  • Visible leaderboard. Scores and downloadable solutions expose the current frontier to every participant.
  • Discussion and refinement. Agents exchange findings, inspect other submissions, and refine earlier work.
  • Curated problem → Agent attempt: Agents select work.
  • Agent attempt → Deterministic verification: Submit candidate.
  • Deterministic verification → Visible leaderboard: Publish score.
  • Visible leaderboard → Discussion and refinement: Expose solutions.
  • Discussion and refinement → Agent attempt: Try an improved construction.
2:022:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:02 · section reference included

From 593 to 604 in the kissing number problem

The kissing number problem asks how many equal spheres can touch a central sphere without overlapping one another. In one dimension, one sphere fits on each side, giving a kissing number of two. In two dimensions, at most six circles can surround the center. Higher dimensions become much harder, and many optimal values remain unknown despite centuries of study.

Zou focuses on eleven dimensions. In his historical account, a construction of 582 spheres remained the best known result for about forty years; he dates the improvement to 592 to 2022 and describes a DeepMind advance to 593 the following year. These dates are Zou’s account, rather than an independently established chronology. He reports that agents working through Einstein Arena then constructed a configuration of 604 non-overlapping spheres within a few days. This advances a known construction—a lower bound on what can be arranged—rather than proving that 604 is the optimal eleven-dimensional kissing number.

The mechanism matters more than a story about one unusually strong model. Zou says no single frontier agent solved the problem alone. The arena preserved a lineage in which agents took existing submissions, refined them, and optimized the result further. Forum exchanges also shared prior attempts and findings: one agent could ask whether a semidefinite-programming approach had already been tried, and others could return what they had learned. The talk does not isolate collaboration’s causal contribution from total computation or search effort.

Zou also connects dense sphere constructions to coding systems, including error-correcting codes for information transfer. The talk states that better constructions can lead to better engineering algorithms, but it does not demonstrate a specific code derived from the 604-sphere construction. That application is therefore motivation for the mathematical search, not a reported deployed result.

5:095:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:09 · section reference included

Swap the verifier, and the arena optimizes GPU kernels

The same environment can target machine learning infrastructure by changing its backend evaluator. Instead of checking a mathematical construction, the system compiles, tests, benchmarks, and verifies individual kernels. The leaderboard then returns real-time feedback about correctness, quality, and speed while preserving the same collaborative and competitive dynamics.

For kernel work, the team also assigns different priors or personas to participating agents. These roles create parallel lines of attack: one agent emphasizes profiling, another memory consumption, and another numerical precision or tensor computation. The personas do not replace verification; they diversify what the group examines before compilation and benchmarking decide whether a proposed optimization actually helps.

Zou reports speedups sometimes exceeding 2× over previous state-of-the-art kernels, including an example transcribed as ‘page attention’; the supplied recording does not establish the intended technical name. He qualifies the displayed examples as applying to specific tensor shapes, while saying the team extended the approach across more shapes and hardware types. Unlike the speculative coding application of the kissing-number result, these agent-designed kernels are reported as already running in production at Together AI.

One arena, two kinds of verification

Competing and collaborating agentsshared — Agents share findings, inspect alternatives, and submit candidates.
Candidate kernelkernel — A proposed implementation reflects profiling, memory, precision, or tensor-computation priorities.
Compile, test, and benchmarkkernel — The backend verifies kernel quality and measures speed.
Live performance scoreshared — Leaderboard feedback shows which candidate performs best.
Production kernelkernel — Selected agent-designed kernels are used at Together AI.
The social and leaderboard structure stays fixed while the backend changes what counts as progress.
Read the diagram as text
  • Competing and collaborating agents. shared — Agents share findings, inspect alternatives, and submit candidates.
  • Candidate kernel. kernel — A proposed implementation reflects profiling, memory, precision, or tensor-computation priorities.
  • Compile, test, and benchmark. kernel — The backend verifies kernel quality and measures speed.
  • Live performance score. shared — Leaderboard feedback shows which candidate performs best.
  • Production kernel. kernel — Selected agent-designed kernels are used at Together AI.
  • Competing and collaborating agents → Candidate kernel: Propose optimization.
  • Candidate kernel → Compile, test, and benchmark: Compile and evaluate.
  • Compile, test, and benchmark → Live performance score: Return measured feedback.
  • Live performance score → Competing and collaborating agents: Guide the next attempt.
  • Live performance score → Production kernel: Adopt improved kernels.
8:579:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:57 · section reference included

DS Gym removes the shortcut of ignoring the data

The second environment, DS Gym—Data Science Gym—combines evaluation and training for data-science agents. It presents diverse datasets and tasks through a unified interface with code execution. Agents can start multiple Docker containers to test algorithms and run experiments in parallel, giving them an executable workspace rather than a collection of questions answered only in text.

While assembling DS Gym, the team found a damaging shortcut in existing data-science benchmarks: agents could solve substantial fractions of tasks without using the underlying datasets. Zou reports that this affected roughly 20% to 50% of tasks across the benchmarks shown. A benchmark with this property can reward general reasoning, memorized patterns, or exploitable wording while failing to measure whether an agent can inspect data, write analysis code, and draw a data-dependent conclusion.

The team responded with two curation paths:

  • Scientific analysis and discovery: Tasks and data came from recently published papers and were reviewed by human scientists and experts.
  • Predictive modeling: Tasks came from Kaggle competitions selected for high-quality datasets and evaluations. Zou describes them as still open; the full DSGym paper specifies completed competitions that continue accepting submissions for official leaderboard evaluation. Their competition periods have ended, while prediction submissions remain possible.

Zou describes a collection with over a dozen different tasks across dozens of scientific domains, from biology and physics to economics, and spans multiple data types and modalities. Human-expert review is established for the scientific-analysis tasks; predictive tasks follow the competition-selection and filtering process.

The talk describes these tasks as carefully checked against shortcuts, but it does not give the exact adversarial testing protocol or a residual shortcut rate. The full paper specifies quality review and a shortcut filter for existing analysis benchmarks: five frontier models attempt tasks without data access, and a task is excluded if at least three answer correctly. This operational filter does not establish that every possible shortcut has been eliminated. The defensible lesson is methodological: if a data-science task can be answered without touching its data, benchmark performance is not sufficient evidence of data-science ability.

10:5711:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:57 · section reference included

Verified execution becomes training data

DS Gym still has headroom: Zou reports that even frontier models often achieve below 50% accuracy on its tasks. This is a qualified report, not a blanket score for every model or task family; the paper distinguishes analysis accuracy from competition-specific prediction metrics. The environment then turns evaluation runs into training material. It records execution-verified trajectories—agent attempts whose generated code is actually run—so a successful trajectory carries stronger evidence than a plausible-looking textual answer.

Those verified trajectories can fine-tune smaller open-source models. Zou says the resulting models reach best-in-class open-source performance on these data-science tasks while remaining small enough to run locally on a laptop or desktop computer. The talk does not provide model sizes, baseline scores, training costs, or a full comparison table. The full DSGym paper supplies a case study: 3,700 synthetic query candidates were generated, 2,000 high-quality query–trajectory pairs were retained, and a 4B model was fine-tuned. It improved over its base model across the evaluated analysis benchmarks and exceeded GPT-4o on some of them. These results establish measured gains in that evaluation, without establishing a particular laptop performance target or unrestricted transfer beyond DS Gym.

The ending places both projects on the same progression: from designing individual models and tools, to building agent harnesses and workflows, to designing environments. Einstein Arena uses deterministic verification, visible solutions, discussion, and competition to accumulate collective progress. DS Gym uses curated tasks, executable infrastructure, and verified trajectories to measure and train data-science behavior. In both cases, the environment does more than host the agent. It determines what feedback exists, what work can accumulate, and which behavior gets rewarded.

The tradeoff is that environmental freedom depends on verification quality. These examples work because sphere constructions, kernel benchmarks, and executed data-science tasks provide machine-checkable feedback. The talk does not show how the same approach handles scientific goals with ambiguous evidence, subjective judgment, unsafe experiments, or rewards that can be gamed. Designing the environment therefore shifts responsibility rather than removing it: builders must curate meaningful tasks and make the verifier measure the result they actually value.

DS Gym as an evaluation and training factory

Curated data-science tasksCurated scientific-analysis and predictive-modeling tasks span multiple domains.
Agent code executionAgents inspect datasets and run algorithms in parallel Docker containers.
Execution-verified trajectoryThe environment retains attempts validated by actually running agent-generated code.
Fine-tune a small open modelVerified trajectories become training material for data-science agents.
Locally runnable agentThe reported resulting models can run on laptops and desktop computers.
Executable tasks produce verified trajectories, which can train smaller agents that return to the same environment for evaluation.
Read the diagram as text
  • Curated data-science tasks. Curated scientific-analysis and predictive-modeling tasks span multiple domains.
  • Agent code execution. Agents inspect datasets and run algorithms in parallel Docker containers.
  • Execution-verified trajectory. The environment retains attempts validated by actually running agent-generated code.
  • Fine-tune a small open model. Verified trajectories become training material for data-science agents.
  • Locally runnable agent. The reported resulting models can run on laptops and desktop computers.
  • Curated data-science tasks → Agent code execution: Present data and objective.
  • Agent code execution → Execution-verified trajectory: Verify by execution.
  • Execution-verified trajectory → Fine-tune a small open model: Supply training data.
  • Fine-tune a small open model → Locally runnable agent: Produce smaller agent.
  • Locally runnable agent → Curated data-science tasks: Evaluate on gym tasks.
14:0614:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:06 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…