Einstein Arena: Harnessing Collective Agent Intelligence for Open Science — James Zou, Together AI

All right. I think we'll go ahead and

get started with the with the

presentation. So my name is James So. I

am uh

going to explain some of the work we're

doing with Together AI and it's also in

collaboration with Stanford around

designing and optimizing environments

for AI agents to enable these agents to

make new kinds of scientific

discoveries.

All right.

So so that I guess the current paradigm

of how people often are using or

deploying AI agents is often involves

designing workflows that sort of tells

the agents you know what to do, right?

Or how the agent should work. And it's

typically done through a series of steps

or prompts, tools, and instructions.

In contrast, the way we imagine the

environment is that the environment

should really specify

not how the agent should work, but

really where the agent should work,

right? And the environment then should

provide a set of incentives and

infrastructure for the agents and

guardrails and resources so that agent

can then flexibly work within that

environment.

Right. And our thesis here is that as

agents become more and more powerful,

right? If we try to design workflows

that often can limit the capabilities

and creativity of the agents. Whereas if

we properly design the environment, this

can enables a lot more creativity and

capabilities and intelligence for the

agents to naturally emerge. This why I

think we're trying to shift away from

designing workflows and harnesses

towards designing environments.

So what I want to do today is to give a

few examples of the how we design

environments for agents.

And in particular also show how they're

able to then with with the right

environment able to actually solve some

really interesting and innovative

problems.

So, the first example I want to share is

the system that we environment that we

created called the Einstein Arena.

It's sort of like the one of the first

environments that enables AI agents to

be able to collaborate in the wild and

to compete to really solve open-ended

scientific problems.

So, we designed this Einstein Arena to

be really agent native. So, I So, that

means that

it's very easy for agents to just read

the skills talk on our on our arena and

be able to access the arena.

And it's actually also designed so that

it's intentionally very hard for humans

to enter the arena, right? So, you

actually have to solve a little puzzle

to prove that you're an AI agent in

order to participate in this arena. But,

any agent in the world can openly and

freely participate on the arena.

And once the agent actually enters into

the Einstein Arena, this is what they'll

see, right? They'll see actually see a

list of curated problems. Each of these

problems is actually a problem that we

curated, so it's a scientifically

interesting problem. And we curated

these problems so that first, there's

actually an existing community of human

researchers that are interested in these

problems. So, these are important

problems for human scientists. And

second is that for each of these

problems, we can actually create a

well-defined and deterministic

deterministic verifier to assess the

quality of the solutions to each of

these problems. And I'll give some

examples in a couple of slides.

So, So, the agents can actually decide

which of these problems they're

interested in once they log onto the

arena, right? So, if they enter into a

particular problem space, this is what

they'll see, right? They'll see some

description that precisely explains what

is the problem. We have a discussion

forum where the agents can communicate.

It's almost like a social network where

the agents can actually communicate and

talk to each other and ask for help or

give recommendations.

Um and we also have a leaderboard. This

is where the agent can actually see each

other's solutions. Right? So in any in

at any time they want, the agent can

actually submit a solution to one of

these problems. And because we have this

verifier, we can actually then determine

what is the quality of that solution and

provide a score in real time. So this

leaderboard is being constantly updated

in real time. And the agents can also

see how other agents are doing on this

problem. And they can also see other

agents' solutions and download those

solutions.

So there's both a collaboration dynamics

and also a competition dynamics in this

arena, right? They can collaborate and

ask each other questions and help in the

discussion forum. But agents are also

competing with each other. And that's

why I think this also sort of simulates

how human researchers can compete and

also collaborate to solve interesting

problems.

So we launched this AI instant arena

environment

earlier this year, I think in March. And

within a few weeks, it's already

actually we're very impressed and very

surprised that the agents were actually

able to already discover new solutions

to 11 problems that are of the best

solutions that have ever been found.

Right? So that means that the solutions

that they discovered by the agents on AI

instant arena were better than any

previous human solutions or any

solutions that we acquired using more

specialized AI tools.

So I'll just give you example of one

such solution or one such problem

which is called the kissing number

problem.

So this is actually a very famous

problem. It's been around for hundreds

of years. So for example, Isaac Newton

was already working on some version of

this kissing number problem. And it's

actually relatively easy to state.

Right? So the kissing number problem

basically asks that what is the maximum

number of spheres that you can place

around the central sphere so that these

additional spheres do not overlap each

other?

So for example, in one dimensions,

right? So around the central sphere I

can place one sphere to the left and one

sphere to the right without overlap. So

the kissing number in one dimension is

easy to compute. This is two.

In two dimensions, it's also easy to

show that you can at most place six

spheres. So, that's the kissing number

in two dimensions is six.

But, it turns out that in higher

dimensions, it actually becomes really

hard to compute what's the maximum

number of over non-overlapping spheres.

And the kissing number problem in higher

dimensions is actually open, right? It's

not been It's not clear what is the

optimal number.

And so, scientists have been trying to

work on this problem for the last

several centuries.

And in particular, right, so the kissing

number problem in 11 dimensions has

attracted a lot of interest for various

reasons.

So, this is actually sort of a

progression of the solutions in 11

dimensions.

So, in the 1980s, right, so it's best

known that there you can place 440

spheres, right, in 11 dimensions without

overlap.

And in

I think 19

uh

So, yeah, so so in in 1980, there was a

big advance that the first for the first

time showed that you can actually just

construct with 582 spheres in 11

dimensions without overlap.

Uh and then that sort of stuck there for

about 40 years, right, until 2022, where

a mathematician is able to publish a new

advance, right,

a breakthrough that's able to improve

that to 592 spheres.

And then there's another breakthrough

from DeepMind the following year that

advances that to 593 spheres.

But, with Alpha Zero, we know by having

these agents able to collaborate

actively, right, in the wild, within a

few days they were actually able to

construct a new solution that shows that

for the first time you can create 604

spheres in 11 dimensions that do not

overlap.

And this is not just a problem that's of

mathematical interest, because it turns

out that

the more of these sort of spheres you

can place in higher dimensions without

overlap that actually creates the better

coding systems including ways of like

doing error correction codes for

information transfer. Right, so this

actually is by creating this better

constructions that also leads to this

better engineering algorithms.

And in this case actually the

collaborations among these agents is

really critical for making these

advances, right? So this is a problem

where not a single agent is able to

solve by itself, right? Not you know,

GPT 5.5 or a cloud models that can't

really solve the problem by itself. So

the collaboration among multiple agents

is really critical.

And here we're actually able to show

that there's like this

sort of a lineage trace of how the

agents are able to collaborate and then

basically take each other's solutions

and refine that and further optimize it

to arrive at this breakthrough.

And you can also see some of these

interactions and discussions on Einstein

Arena, right? Where here's an example

where the one agent actually was asking

other agents, "Have you tried

you know, some of these approaches?" Um,

with uh, these STP approaches and then

the other agents showed that yes, we

have tried these approaches and here are

some of the things that we found. Right,

so the information sharing on the forums

on the arena is actually really

important to help the agents to arrive

at this solution together.

So in addition to solving these

interesting scientific problems, but

we've also been using platforms like the

right Einstein Arena uh, to help to

improve uh, you know, machine learning

and AI itself.

Right, so here's one example where we

actually use these agents to basically

help us to create better kernels

for and to speed up those kernels.

Right, and here we use the same

environment, right? Where the agents can

compete and they also can collaborate

and they see these leaderboards. And we

basically change the back end instead of

trying to verify the solutions to this

mathematics problem, here we're

basically trying to

you know, we will compile and benchmark

and test and verify the quality and the

speed of the individual kernels, right?

And then we'll provide a feedback to the

agents in real time in the form of these

leaderboards.

In these kernel settings, we also found

it to be quite useful to have different

agents with different personas,

right? And these different personas

actually corresponds to a different uh

roles and priors that agents can

actually have. So, for example, we have

one agent that looks at tends to look at

more of the profiling, another agent

that tends to look at more of the memory

consumptions, a third agent that looks

at, you know, the precisions, the tensor

computations. And these agents can and

then across different personas, they can

able to collaborate and a compete on the

arena to speed up the kernels.

And in this case, right here, the agents

were also able to collaborate and lead

to really quite substantial speed ups,

uh including sometimes over two two x

two-fold speed ups in some of these

production kernels. So, here I'm just

showing you a few examples where for

things like page attention, uh and these

are sort of for specific shapes, but we

also have generalized this to many

different shapes and different uh

hardware types, right? Where we're

actually seeing that we're getting up to

sometimes over two x speed up in these

kernels, and they uh compared to the

previous state-of-the-art kernels for

these problems.

And these improved kernels created

designed by the agents are actually

already used in in production at

Together AI.

So, in the last few minutes, I want to

show like a second example of a kind of

environment that we created as a way to

uh train and to create better data

scientist agents,

right? So, we call this DS Gym, which

stands for data science gym, which is

sort of like a unified environment that

we created for both for evaluating and

for training data science agents to

solve complex data science problems.

So, here in this DS Gym environment, we

also curated and created a unified list

of different data sets and tasks,

right? So, these data sets can combine

uh spans across many different settings.

And the agents are then able to interact

with these different data sets that we

have through a unified uh interface and

through code execution.

In the DSGM environment, we also provide

a unified infrastructure for the agents.

So, for example, the agents can actually

spin up many different Docker containers

to test their data science algorithms

and actually run them in parallel.

So, in the process of actually creating

the data sets and tasks for the data

DSGM environment, so we initially

actually wanted to incorporate some of

the existing data science benchmarks

that have been used to evaluate agents.

But we actually quickly realized that

many of the existing widely-used

benchmarks actually have many problems.

And one big problem is that they're

actually very vulnerable to shortcuts.

By shortcut, I mean here is that

uh down here what I'm showing are three

different common popular data science

benchmarks.

Right? And the in green here we see

shows like the performance of the agents

on these benchmarks.

Uh but the red bar also shows how well

they're able to the what fraction of the

benchmark the agents can actually solve

without actually using the data sets

themselves. Right? So, just by reasoning

or by, you know,

uh doing other shortcuts without

actually actually do working with the

underlying data sets.

And across many of these different

benchmarks, right, sometimes up to 20 to

50% of the tasks can be solved without

actually looking at any of the

underlying data.

Which I think is uh really a significant

problem with many of the existing

benchmarks.

So, to address that, we actually

carefully curated at our own our own

benchmarks, right, for both for

scientific analysis and also for

predictive modeling.

So, for scientific analysis and

discovery, the way we did this is that

we actually went through recently

published papers and then carefully

curated data and then also tasks from

those papers. And then we also had human

scientists and experts to review each of

those tasks.

And for predictive modeling, the way we

did this is go through all the different

Kaggle competitions to look for some of

the recent Kaggle competitions that are

still open and and where also you have

high quality data sets and also high

quality

evaluations. Then we curated those into

the DS Gym as a kind of task for

evaluating how well models agents can

actually build predictive models.

So all together in the DS Gym, we

actually have created over a dozen

different tasks. They span across

dozens of different scientific domains

ranging from biology to physics to

economics. It also involves many

different data types and data

modalities.

So this actually makes it very easy for

us to evaluate different models, both

open and closed source models. And one

thing we found is that the existing

models, even the frontier models, often

are only still achieves like less than

50% accuracy performance on the DS Gym

tasks. Right? So these are definitely

not saturated benchmarks.

We can also use a DS Gym as sort of like

a training factory to improve these open

source models.

Right? So one thing we did here is

actually generate in the DS Gym actually

the gym itself will actually create all

these execution verified trajectories,

which means that these are trajectories

generated by the agents that have been

verified through the through

through actually executing the code from

the agents.

Right? So by generating these execution

verified trajectories, then we are able

to like fine-tune sort of small open

source models

that actually now achieve sort of the

they're sort of the best in class open

source models in terms of solving these

kind of data science tasks. Right? And

these models are small enough that you

can actually run them locally on your

laptops and your computers.

So just to summarize the this part was

the data science gym. Right? So we with

DS Gym, we created this unified

execution layer so people can actually

run and all these different tasks across

dozens of different tasks across many

different domains. We have carefully

verified that there are no shortcuts in

these tasks, which has been sort of a

common challenge with existing data

science benchmarks.

And we also enable in the DSG and a way

to generate synthetic data, so you can

easily use that to improve and to train

your own data science agents.

So, just to summarize the presentation,

um I think the main takeaway here is

that I think we're in seeing this

interesting progression as in terms of

how we build different AI systems.

Right. So, then in the past, people have

been building these AI systems mostly by

designing individual models or

individual tools.

And currently, there's a lot of focus on

creating designing agents or harnesses

and workflows around agents.

But what our research shows is that I

think we're already moving towards the

next stage, where you're not then trying

to design workflows or specific or

specific agents, what we really want to

do is to design environments, which is a

set of infrastructure and incentives

that in that motivates the agents that

you solve more and more challenging

problems.

And with appropriate designs these

environments can actually unlock much

more creativity and collective

intelligence from the agents that's

that's limited by the existing

workflows.

And here are some of the references for

the papers that we published that

describes these in more detail. So,

thank you very much.

>> [applause]

[music]