# Measuring AGI: Interactive Reasoning Benchmarks

Greg Kamradt, ARC Prize Foundation

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/3XmFPwjG8pg-measuring-agi-interactive-reasoning-benchmarks) · [Video](https://www.youtube.com/watch?v=3XmFPwjG8pg)

## Measuring AGI: Interactive Reasoning Benchmarks

ARC-AGI-3 proposes a harder test than winning a familiar game: discover unfamiliar rules, learn through interaction, and compare that learning efficiency with humans.

### What does beating Pokémon prove?

[0:21](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=21s)

Watching **Claude** play Pokémon makes autonomous agents tangible: a small character moves through a familiar world while a model decides what to do next. OpenAI joined the game-playing demonstrations, and Kamradt points to a reported **Gemini** Pokémon completion. If an agent can finish the game, how much evidence of general intelligence does that provide?

The failures matter as much as the finish. Kamradt reports that Claude remained stuck in one place for three days, needed human interventions, and hallucinated actions. The accompanying post identifies the location as Cerulean City.

![Post with an orange outline around the claim that Claude is still stuck in Cerulean City, above a Pokémon livestream screenshot.](https://ai.engineer/api/talk-essay-media/193844e2facb418f948888e494e990ff0a5c33892d65749c9908d5f6b061ef79)

A post highlights Claude remaining stuck in Cerulean City for three days.

There is also the question of prior knowledge. Kamradt argues that Pokémon material was already present in the model’s training data. Exploring a game is impressive, but a familiar game cannot cleanly separate learning during play from recalling information acquired beforehand. **The benchmark needs to test how an agent acquires a new skill, not merely whether it can display an existing one.**

### Humans as the reference point

[1:15](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=75s)

Greg Kamradt leads [ARC Prize](https://arcprize.org/blog/arc-prize-2025), a nonprofit founded by François Chollet and Mike Knoop to guide progress toward open AGI. Its work includes participating in OpenAI’s December livestream announcing **o3** preview results on ARC-AGI. Its benchmark design starts from a specific reference point: humans are the only demonstrated example of general intelligence available to us.

That reference gives benchmark builders a practical search strategy: find problems people can solve but AI cannot. The remaining problems define a measurable capability gap. They also direct research toward particular failures, rather than making the benchmark merely a scoreboard. A useful task reveals something that a human can learn and a machine still struggles to acquire.

### Intelligence as skill acquisition efficiency

[2:38](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=158s)

Kamradt introduces unfamiliar, unprepared-for tasks through a definition he attributes to John McCarthy. [McCarthy’s own definition](https://www-formal.stanford.edu/jmc/whatisai/node1.html) concerns the science and engineering of making intelligent machines; the emphasis on unseen tasks is best understood here as Kamradt’s framing. For evaluation, the distinction is consequential: retrieving a familiar solution does not establish that the system learned a new skill during the test.

Chollet’s formulation is more compact: **“skill acquisition efficiency.”** It asks both whether a system can learn something new and how efficiently it does so. His 2019 paper, [On the Measure of Intelligence](https://arxiv.org/abs/1911.01547), connects that definition to a concrete benchmark. Human learning efficiency supplies the motivating comparison; ARC supplies tasks on which learning and subsequent application can be observed.

![Slide juxtaposing John McCarthy and François Chollet, with a paraphrased definition beneath McCarthy and “Skill-Acquisition Efficiency” beneath Chollet.](https://ai.engineer/api/talk-essay-media/feece1f9eecbef52a96cd21bf7b24baef999e1084bf936f515f4e3777609111f)

Definitions of intelligence emphasize unfamiliar tasks and skill-acquisition efficiency.

### Learn a transformation, then apply it

[3:47](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=227s)

An ARC-AGI-1 task separates learning from demonstration using colored grids:

1. Inspect example input grids and their corresponding output grids.
2. Infer the transformation that explains those pairs.
3. Apply that transformation to a new input, constructing the answer in a grid editor.

A correct answer demonstrates some generalization: the solver has carried a rule from the examples to a new case. The examples are the task’s training portion, not an instruction to retrain a model’s weights.

Kamradt describes ARC-AGI-2 as containing over 1,000 tasks. He emphasizes that each task targets a novel skill rather than asking the solver to repeat another task’s transformation. The intended unit of measurement is therefore a succession of small learning problems, not repeated execution of one mastered procedure.

Kamradt reports testing over 400 people in person in San Diego to establish human solvability. The team rented computers and conducted testing in person to protect task data. The important claim is task-level coverage: every included task could be solved by people, not that every participant solved every task. The [ARC-AGI-2 announcement](https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025) provides the corresponding human-validation account.

### What a single-turn benchmark leaves out

[5:11](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=311s)

The grid format still leaves substantial abilities untested. All necessary information is supplied at test time. The solver does not need to explore an environment to discover what information matters. An interactive test adds that missing responsibility: choose actions, inspect their consequences, discover goals, and interpret rewards.

This direction connects to David Silver and Richard Sutton’s [Welcome to the Era of Experience](https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf). Agents collect observations through engagement with the world, use those observations to construct world models, and infer rules for themselves. Learning exclusively from human-produced data imposes a ceiling that experience is intended to move beyond.

An **interactive reasoning benchmark** puts that process inside a controlled environment with defined rules and potentially sparse rewards. The evaluator knows the environment’s structure; the agent must discover enough of it to complete an objective. Sparse feedback makes exploration consequential because an action may yield useful information without immediately producing a reward.

### Games that are unfamiliar to the developer, too

[6:50](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=410s)

Games offer complex rules within a bounded scope, with enough design flexibility to create many different environments for both humans and artificial agents. But using games does not automatically produce a clean test of generalization. Kamradt identifies several weaknesses in earlier Atari evaluations:

- **Dense rewards:** frequent feedback can guide behavior without requiring the same discovery process as a sparse-reward environment.
- **Inconsistent reporting:** different performance scales make systems difficult to compare.
- **No hidden test set:** familiar evaluation games permit development against the test itself.
- **Developer knowledge:** a developer who knows the game can encode useful strategies into the system.

The last issue changes what a score means. Some apparent agent intelligence may have been supplied by the person building it.

Chess and Go establish that AI can master individual games. Systems such as [**Agent57**](https://deepmind.google/blog/agent57-outperforming-the-human-atari-benchmark/) and [**MuZero**](https://arxiv.org/abs/1911.08265) extend the discussion to suites of known games. Kamradt loosely describes this as beating 50 known games with unlimited compute and training data. The historical Atari suite contains 57 games, and that shorthand should not be read as a literal resource condition or as identical results from the two systems. The relevant distinction is that the games were known during development.

The proposed stronger test is to beat 100 games that neither the system nor its developer has seen before. Success would support a different capability claim: the system can acquire useful skills across unfamiliar environments without game-specific preparation by its creator. This is the ambition behind ARC-AGI-3, presented in the recording as a preview of ARC’s first interactive reasoning benchmark.

![Slide reading “Beat 100 games” and “That neither the system nor developer have seen before,” beside an illustration of a person and robots playing games.](https://ai.engineer/api/talk-essay-media/2328de09245e13081d0a84baf532fbd4e7634398e6129da9fa4b8d616a58400c)

Beat 100 games that neither the system nor its developer has seen before.

### Private scoring and discovery through play

[9:24](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=564s)

The preview proposes approximately 40 novel public training games so developers and agents can learn the interface. Kamradt also announces a public evaluation set, while assigning all performance reporting to a private evaluation set; he does not specify the public evaluation set’s separate role. Private evaluation would run without internet access, intended to prevent game data from escaping. The scored games must remain unfamiliar to both the model and its developer.

Novelty also applies to the instructions. A typical game benchmark might tell an agent that it is driving a racing car or playing a first-person shooter, then explain the controls. ARC-AGI-3 instead proposes dropping humans and AI into an environment without that framing. Understanding must emerge through exploration.

The example is **Locksmith**. Its initial screenshot does not explain the objective. As a player explores, they may discover things to pick up, walls, and possible goals. Those are hypotheses to establish through interaction, not instructions handed over at the start. Humans and AI face the same requirement: construct an account of how the world works, then use it to make progress. Kamradt presents this as a gap in the reasoning models available at the time.

### Testing the floor of reasoning ability

[11:06](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=666s)

The games are intended to require only **core knowledge priors**. Kamradt describes the tasks as lacking language, text, symbols, and trivia. The practical distinction is the absence of required external linguistic or specialist knowledge, rather than the exclusion of symbolic reasoning: visual patterns can themselves carry meaning. Instead of commissioning ever harder expert problems, ARC aims to test the reliability of basic capabilities, even when AI is already superhuman in particular domains.

Kamradt describes four priors as innate or acquired very early in human development:

| Prior | Intended foundation |
|---|---|
| Basic math | Small quantities, such as counting up to 10 |
| Basic geometry | Shapes and topology |
| Agentness | Recognizing other interacting agents; theory of mind |
| Objectness | Recognizing objects as units |

These constrain what a task may assume before it begins testing abstraction and reasoning.

Practice adds another dimension. Kamradt invokes a passage matching Dwarkesh Patel’s [Why I don’t think AGI is right around the corner](https://www.dwarkesh.com/p/timelines-june-2025): humans accumulate context, examine their failures, and find small improvements as they repeat a task. An interactive environment can expose that process over a sequence of decisions. ARC-AGI-3 is intended to provide the benchmark setting Kamradt sees as missing for those behaviors.

### Measure the learning, not just the win

[12:46](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=766s)

Finishing a game supplies an outcome, but skill acquisition efficiency also requires accounting for how the solver got there. The proposed evaluation would test hundreds of humans on the same games, recording completion duration and action count, then evaluate AI against that baseline. The comparison encompasses exploration, inference about the environment, creation of intermediate goals, and completion of the objective.

Kamradt sets a demanding interpretation: if AI cannot outperform humans in this process, he would not call it AGI. More broadly, he treats the continued existence of human-solvable problems that machines cannot solve as evidence against having reached AGI. That is the project’s proposed criterion, with **skill acquisition efficiency** as its main output metric.

### Building and pressure-testing the benchmark

[13:45](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=825s)

At the time of the preview, the plan was to release five sandbox games the following month in San Francisco, alongside a small agent competition with prize money. The purpose was to put the environments in front of strong agents early and learn from what happened. Kamradt’s larger release target was approximately 120 games by Q1 2026. These were forecasts in the recording. Each game would contain multiple levels, so the number of games alone understates both the available interactions and the operational work of creating them.

That work requires funding and attempts to break the evaluation. Kamradt invites direct donations, described as tax-deductible, and introductions to philanthropic donors, including individuals and LPs. He also seeks adversarial testers and participants in online or offline agent competitions to pressure-test ARC-AGI-3.

The implementation had already become simpler. The team initially tried Unity, then found it excessive for these small grid games and began building a lightweight Python engine. That choice also shaped recruitment: Python-capable game developers and game designers could help construct the environments.

### How long will it last, and what counts as efficient?

[15:48](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=948s)

Asked how long the new benchmark might take to saturate, Kamradt offers a tentative horizon of years rather than decades. He gives no exact deadline. The benchmark is intended to expose a present capability gap, not to remain permanently beyond AI.

An audience question about wattage and time makes the efficiency requirement more precise. Kamradt would ideally measure intelligence against two inputs:

- **Energy:** the human brain provides a physical reference for the resources consumed by a generally intelligent system.
- **Training data:** the amount of prior experience required matters when judging learning efficiency.

Both are difficult to obtain for closed models. Cost is a practical proxy, and interactive evaluation adds observable action counts and completion duration.

The games themselves, however, are turn-based and have no in-game wall clock. That distinction separates actions that change the environment from time spent deciding what to do. The preview leaves the exact duration measure unspecified; its clearest operational measure is how many interactions a solver needs.

### Recognizing objects—and the boundary of the test

[17:14](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1034s)

A second audience question asks what **objectness** means. Kamradt makes it concrete: a pixel surrounded by other pixels that all move together can be understood as part of one body. The useful representation groups their behavior instead of treating every pixel as independent.

A small Python example illustrates the shared-motion cue. Three labeled pixels retain their identities while each moves one column to the right. Their common displacement is evidence for treating them as one moving object:

```python
before = {"a": (1, 1), "b": (1, 2), "c": (2, 1)}
after = {"a": (1, 2), "b": (1, 3), "c": (2, 2)}

def shared_displacement(before, after):
    if not before or before.keys() != after.keys():
        return None
    shifts = {
        (after[key][0] - row, after[key][1] - column)
        for key, (row, column) in before.items()
    }
    return next(iter(shifts)) if len(shifts) == 1 else None

shift = shared_displacement(before, after)
assert shift == (0, 1)
```

This captures one cue for grouping, not a complete object-perception system.

Kamradt’s everyday analogy is a tree: its parts are understood as belonging to the same tree. The prior is a basic way of organizing perception into entities on which reasoning can operate.

The final question asks about tasks that neither humans nor AGI can perform. Those fall outside ARC’s stated aim. Its reference remains the set of tasks humans can solve, and the gap where machines cannot yet match them. Kamradt places human-impossible tasks in a separate discussion of superintelligence, leaving this benchmark focused on learning abilities for which human performance supplies a concrete standard.

## Resources

- [On the Measure of Intelligence](https://arxiv.org/abs/1911.01547) — François Chollet's framework for measuring intelligence through efficient skill acquisition and the original proposal for ARC.
- [ARC-AGI-2 launch and human validation](https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025) — Explains ARC-AGI-2's task design, human testing, two-attempt evaluation, and launch-era model results.
- [Welcome to the Era of Experience](https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf) — David Silver and Richard Sutton argue for agents that learn through interaction and experience beyond human-generated training data.
- [Why I don’t think AGI is right around the corner](https://www.dwarkesh.com/p/timelines-june-2025) — Dwarkesh Patel's essay on continual learning and the practical importance of improving through experience.
- [Agent57 and the Atari human benchmark](https://deepmind.google/blog/agent57-outperforming-the-human-atari-benchmark/) — DeepMind explains Agent57's performance across all 57 Atari benchmark games and why difficult games matter when assessing generality.
- [MuZero: Planning with a Learned Model](https://arxiv.org/abs/1911.08265) — The original MuZero paper describes learning a model for planning across Atari, Go, chess, and shogi.
- [ARC-AGI-3 technical report](https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf) — The later benchmark report explains environment construction, human calibration, dataset splits, and Relative Human Action Efficiency scoring.

## Slides

### Claude plays Pokémon

Claude Opus 4 searches a Pokémon town for a hinted-at house, updates its knowledge base, and sends a navigation target while the game view shows its character outside the Mart.

[0:23](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=23s) [Claude Opus 4 reasons about navigating a Pokémon town beside the game view.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d2a8ff8284c959c47c43f047d6d050bdd5cc6e368a21322a05071212799d7b2.webp)

Claude Opus 4 Plays Pokémon

STEP COUNT
28747

OPUS EDITION

Claude Opus 4

[illegible]
1. The Gym (at 30,19)
2. The Mart (in the south)
3. Some other building in the west

The hints specifically said "trashed house in the northeast corner." But I haven't seen any house that looks different or "trashed." However, maybe the hint is referring to a house that doesn't look obviously trashed from the outside.

Let me go back and check if there are any other buildings I can enter that I haven't tried yet. I should systematically check every building entrance.

Using tool: string_replace

Good, I've updated my knowledge base. Now let me go back up and look for buildings. I remember seeing a building entrance in the west that wasn't the gym. Let me go check that area.

Using tool: navigator - Target: (26, 26)

Thinking...

MART

Current Team
[illegible] Lv.17 — POISON — OK
SPLASH Lv.37 — BLASTOISE — WATER — HP: 119/119 — OK
luna Lv.12 — CLEFAIRY — NORMAL — HP: 40/40 — OK
DUX Lv.12 — FARFETCHD — NORMAL FLYING — HP: 37/37 — OK
wings Lv.15 — SPEAROW — NORMAL FLYING — HP: 41/41 — OK
gust Lv.16 — PIDGEY — NORMAL FLYING — HP: 39/39 — OK

Interface text:
AIE
Microsoft
smol ai

### o3 plays Pokémon

The o3 stream shows its reasoning beside a Pokémon battle: it plans another Tackle while watching Sprout’s HP, and the game screen shows Sprout using the move.

[0:34](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=34s) [A social media post introduces an o3 Pokémon stream, with a screenshot of a battle and reasoning interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c06a7a469dd380305501ed8c3f673a148de6b90a8de56effd5e2781fbed5dc4.webp)

Greg Brockman
@gdb

o3 plays Pokémon:

OpenAI Developers @OpenAIDevs · May 27

Watch o3 play Pokémon—live. See how it plans its next move, explains its reasoning, analyzes the map visually, and saves to memory.

Thank you community member @Clad3815 for putting this stream together! x.com/Clad3815/statu...

Interface text:
Total Runtime 00:36:10
Steps 58
Next Criticism T-22
Next Summary T-17
o3 Plays Pokémon
Connected

58
REASONING
Continuing the battle strategy
We didn't manage to knock out the opponent: Charmander is still standing strong. I realize we need another Tackle to keep the game moving along. However, the user might be expecting a summary of what's happening. So, I should provide a detailed reasoning for the next steps and the chat message about our actions. I'll proceed with Tackle but also keep an eye on HP, making sure it's safe at 10. Let's go ahead and craft that!

Token Usage
Total 41388
Input: 41239
Output: 149

REASONING
Charmander has literally 1 HP left, while SPROUT is at 10/19—perfectly safe. We just need one final Tackle. Main menu: > FIGHT highlighted, move cursor already on TACKLE. Input: A, then [illegible] to finish Charmander off and breeze through the post-battle text until we regain control in the lab.

[illegible]

KEY PRESS
Executing

CHARMANDER :L5
HP
SPROUT :L5
HP 10/19
SPROUT used TACKLE!

OBJECTIVES
1 Defeat NOVA in the lab rival battle
2 Head north to Viridian City for Oak's Parcel after victory
3 Heal at Viridian Pokémon Center
Others: None
OAK'S LAB
Inventory
Memory
Inventory is empty

0:32
AIE
aws

### Claude stuck in Cerulean City

Claude spent about three days stuck in Cerulean City. The post describes it overthinking and taking too few actions; Kamradt adds that it needed interventions and hallucinated actions.

[0:52](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=52s) [A social media post says Claude remained stuck in Cerulean City and shows the Pokémon stream.](https://assets.ai.engineer/sites/aiecode2025/sha256/aa71334409ccc3ba3f6007051d63109ac525ee823e3d2a15cdfb72c673389b49.webp)

QC
@QiaochuYuan

claude plays pokemon is still stuck in cerulean city after, i think, 3 days?
and the way it's stuck is kind of interesting: overthinks everything + incapable of getting bored. it would make more progress by taking way more actions even if they were random. makes ya think huh

10:32 AM · Mar 4, 2025 · 126.7K Views

Interface text:
Countdown to Restart
01:59:28
Claude Plays Pokémon

<thinking>
I tried pressing DOWN at (30, 27), but it didn't work as an exit to Route 5. I'm still in Cerulean City at the same coordinates.
Let me continue checking the remaining positions in the eastern area. I'll navigate to (31, 27) next.
</thinking>
[illegible]

<thinking>
I'm now at (31, 27), closer to the eastern edge of the city. Let me try pressing DOWN at this position to see if this is the exit to Route 5.
</thinking>
[illegible]

Current Team
[illegible]

ClaudePlaysPokemon
Claude Plays Pokemon - Debut Stream
Pokémon Red/Blue · English · AI · TwitchPlays
Follow

STREAM CHAT
[illegible]
Send a message

AIE
Microsoft
smol ai

### ARC Prize Foundation

Greg Kamradt introduces ARC Prize as a nonprofit founded by François Chollet and Mike Knoop. Its 2024 competition drew 1.4K teams and 17K submissions; the 2025 competition was live on Kaggle.

[1:14](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=74s) [ARC Prize Foundation overview with competition statistics, university tour, and leadership portraits.](https://assets.ai.engineer/sites/aiecode2025/sha256/281900f39194229a496a9eff75a626611be2ed14d7f63994104000a7633387d2.webp)

ARC Prize Foundation

A NORTH STAR FOR AGI

PRIZE

2024 COMPETITION
1.4K teams / 17K submissions / 40 papers published

2025 COMPETITION
Live now on Kaggle!

CONTENT

2024 University Tour, 1.5K students, 14 top AI programs

HARVARD  NYU  MIT  CMU  BOSTON UNIVERSITY  COLUMBIA

Francois Chollet
Co-Founder

Mike Knoop
Co-Founder

Greg Kamradt
President

@ARCprize  X  YouTube

Interface text:
AIE
Microsoft
smol ai

### The Target: Humans

The team uses humans as its target for general intelligence, creating problems that humans can solve but AI finds hard. The gap helps measure progress and guide research.

[1:43](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=103s) [Title slide: The Target: Humans, with the subtitle Proof of General Intelligence.](https://assets.ai.engineer/sites/aiecode2025/sha256/6590f5dbbb470109f1a7c44e6d35a15b24bd4c3a2da28aa0516409c82c96c6f4.webp)

THE TARGET: HUMANS
(Proof of General Intelligence)

AIE
Microsoft
smol ai

### The Definition of Intelligence

McCarthy’s paraphrased definition centers on tasks machines have not prepared for; Chollet defines intelligence by how efficiently new skills are learned.

[3:12](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=192s) [Portraits of John McCarthy and François Chollet accompany their respective descriptions of AI and intelligence.](https://assets.ai.engineer/sites/aiecode2025/sha256/49d8f2312f7dbf583a68bab476919cb3ddbd76dad875a64b20249cab42a42f13.webp)

THE DEFINITION OF INTELLIGENCE

John McCarthy (paraphrased):
“AI is the science & engineering of making machines do tasks they have never seen and have not prepared for beforehand.”

François Chollet:
“Skill-Acquisition Efficiency.”

Interface text:
AIE
aws
World’s Fair

### ARC-AGI: Learning and applying a skill

Two training examples extend a line in the marker’s color from the shape’s tip to the grid edge. The test asks the viewer to apply the rule to a yellow shape.

[3:45](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=225s) [Two training transformations appear beside an unanswered grid test.](https://assets.ai.engineer/sites/aiecode2025/sha256/1729a04ea8c6e9e06897c2ec06d422b7f525de916761b9b6ded27cf885b8dbf2.webp)

ARC-AGI: Measuring the ability to adapt to tasks you cannot prepare for

Ex.1 Input (15x10)
Ex.1 Output (15x10)

Ex.2 Input (12x12)
Ex.2 Output (12x12)

TEST
Input (11x16)
Output (11x16)
?

“Learn the skill”
Training

“Apply the skill”
Test

TASK ID: 25D487EB

AIE
Microsoft
smol ai

### ARC-AGI: Test solution

The test output applies the training examples’ pattern: a light-blue column extends from the yellow pyramid’s tip to the top of the grid.

[4:14](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=254s) [The test output extends a light-blue column above the yellow shape.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4198e3055f1de17e1c5fec69ef8d0e808328472100c5090109361dc528310a4.webp)

ARC-AGI: Measuring the ability to adapt to tasks you cannot prepare for

Ex.1 Input (15x10)
Ex.1 Output (15x10)
Ex.2 Input (12x12)
Ex.2 Output (12x12)

TEST
Input (11x16)
Output (11x16)

“Learn the skill”
Training

“Apply the skill”
Test

TASK ID: 25D487EB

AIE
Microsoft
smol ai

### ARC-AGI-2: 1,320 tasks

A collage of colorful grids illustrates ARC-AGI-2’s 1,320 tasks. The speaker says no two tasks require the same skill.

[4:22](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=262s) [A collage of colorful ARC-AGI-2 grid tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e36c74088f2a69410606a77c5a0c72ebbc8f3862e64924efd60b4d361a499ed.webp)

ARC-AGI-2 : 1,320 Tasks

[illegible]

AIE
Microsoft
smol ai

### Models can infer rules from a single snapshot

Three examples arrange colored blocks into a connected diagonal chain, leaving a test output to infer. The speaker notes that ARC-AGI presents all the information needed at test time in a single turn.

[5:16](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=316s) [Three input-output grid examples accompany a test input and an unanswered output.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3aadb7e1de70f44c8af944bf491273cbff344a4db998189bb5406b495c69b7a.webp)

Models can infer rules from a single snapshot

Ex.1 Input (10x10) → Ex.1 Output (10x10)
Ex.2 Input (10x10) → Ex.2 Output (10x10)
Ex.3 Input (10x10) → Ex.3 Output (10x10)
Test Input (10x10) → Test Output (3x3)

Interface text:
AIE
Microsoft
smol ai
World's Fair

[5:22](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=322s) [Three colored-grid examples show inputs and outputs alongside an unanswered test puzzle.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b27852ad52ecda7f90a942b4e3c5cbb3846f794d9a4a788787628deb788b47c.webp)

Models can infer rules from a single snapshot

Ex.1 Input (10x10) → Ex.1 Output (10x10)
Ex.2 Input (10x10) → Ex.2 Output (10x10)
Ex.3 Input (10x10) → Ex.3 Output (10x10)
Test Input (10x10) → Test Output (3x3)

Interface text:
AIE
aws
World's Fair

### Exploring an open world

A figure opens a door onto a colorful world as the speaker argues that tests of human-like intelligence should ask agents to explore, identify goals, and consider rewards.

[5:48](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=348s) [A person opens a door onto a colorful, candy-like landscape.](https://assets.ai.engineer/sites/aiecode2025/sha256/3893e3119e5b8099ffde7479ae370d8efa035ad0b4260f7ebf45c9412c679142.webp)

AIE
aws
World’s Fair

### Welcome to the Era of Experience

The speaker connects Silver and Sutton’s paper to agents that explore the open world, collect observations, and build world models to move beyond the human-data ceiling.

[5:58](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=358s) [The opening page of a paper by David Silver and Richard S. Sutton, including its abstract and the start of a section on human data.](https://assets.ai.engineer/sites/aiecode2025/sha256/0724febb7273f9f7b13c81c3e42512fddc8701a07004ea4ef2998b567716a209.webp)

Welcome to the Era of Experience
David Silver, Richard S. Sutton*

Abstract

We stand on the threshold of a new era in artificial intelligence that promises to achieve an unprecedented level of ability. A new generation of agents will acquire superhuman capabilities by learning predominantly from experience. This note explores the key characteristics that will define this upcoming era.

The Era of Human Data

Artificial intelligence (AI) has made remarkable strides over recent years by training on massive amounts of human-generated data and fine-tuning with expert human examples and preferences. This approach is exemplified by large language models (LLMs) that have achieved a sweeping level of generality. A single LLM can now perform tasks spanning from writing poetry and solving physics problems to diagnosing medical issues and summarising legal documents.

However, while imitating humans is enough to reproduce many human capabilities to a competent level, this approach in isolation has not and likely cannot achieve superhuman intelligence across many important

### Interactive Reasoning Benchmarks

Interactive reasoning benchmarks put agents in controlled environments with defined rules and sometimes sparse rewards. The speaker proposes games as a flexible medium for testing how agents explore and pursue objectives.

[6:36](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=396s) [Title slide reading Interactive Reasoning Benchmarks.](https://assets.ai.engineer/sites/aiecode2025/sha256/819f96511b08ec3fdebf9637a74d46c92a8c0be2afa82f94ee1db8c877d3b303.webp)

Interactive Reasoning Benchmarks

AIE
Microsoft
smol ai
World's Fair

### Beat 1 game

AI can beat individual games such as Go; the slide illustrates AlphaGo playing Lee Sedol.

[8:23](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=503s) [An illustration of AlphaGo playing Go against Lee Sedol accompanies the text “Beat 1 game.”](https://assets.ai.engineer/sites/aiecode2025/sha256/3c75ed65d53ecdba13a61e0ca394a4e8f40b308f64ced53c2503c103c361c654.webp)

Beat 1 game.

ALPHAGO
LEE SEDOL

Interface text:
AIE
Microsoft
smol ai
World's Fair

### Beat 50 games

The speaker cites Agent57 and MuZero as examples of AI beating 50 known games with unlimited compute and training data.

[8:40](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=520s) [Agent57 and MuZero are illustrated playing Atari games beside text about unlimited compute and training data.](https://assets.ai.engineer/sites/aiecode2025/sha256/561424caec9f6cb5e26a549281f60cc4ddc5def15a090cfea104bb79284ce06e.webp)

Beat 50 games.

With unlimited compute and training data.

ATARI
AGENT57
MUZERO

AIE
aws

### ARC-AGI-3

ARC previews version three, its planned first interactive reasoning benchmark.

[9:12](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=552s) [The title ARC-AGI-3 appears in outlined pixel-style lettering.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f66fcf3293c1b644d30c5830e6a94c25e9c328d0d6d78afb3d9994b0d546105.webp)

ARC-AGI-3

AIE
Microsoft
smol ai

### Public training and private evaluation

ARC plans about 40 public training games for learning the interface. Performance will be reported on private games neither developers nor AI have seen, without internet access. The slide labels 100 novel environments.

[9:26](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=566s) [Six public-training game previews sit beside six empty private-evaluation placeholders, with the caption “100 novel environments.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e538ee97f5241ab485e4d88618a2f74b0e551a5ee3e463253ddcb9350dcc107b.webp)

PUBLIC TRAINING
PRIVATE EVALUATION

FOCUS
<WE GO AGAIN

100 NOVEL ENVIRONMENTS

AIE
aws

[9:34](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=574s) [Six public training game views alongside six empty private evaluation placeholders, labeled 100 novel environments.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b7c50807959e2d45571ea0dc37af5f2f2e034961612f57a7f148cf375deedce.webp)

PUBLIC TRAINING
PRIVATE EVALUATION

100 NOVEL ENVIRONMENTS

AIE
Microsoft
smol ai

### Understanding through exploration

ARC-AGI-3 will place humans and AI in unfamiliar games and ask them to discover the rules, objects and goals through exploration. The pictured game is called Locksmith.

[10:05](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=605s) [Text describes adapting to tasks one cannot prepare for, beside a pixel-art game environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc2f27a8d2a4cae6249f9232ecff000d51c0b3ef6ffe23191caf88df02cdf26e.webp)

Understanding through exploration.

Measuring the ability to adapt to tasks you cannot prepare for.

AIE

### Core Knowledge Priors

The benchmark strips away language, symbols, trivia and cultural knowledge, relying instead on basic math, geometry, agentness and objectness to test abstract reasoning.

[11:06](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=666s) [Basic math, basic geometry, agentness and objectness appear opposite crossed-out language, symbols, trivia and cultural knowledge.](https://assets.ai.engineer/sites/aiecode2025/sha256/356da30bfe3432c93f53fac953ff1a3e309df2c9e061e2a88f609b9109579e6a.webp)

Core Knowledge Priors

Crossed out:
Language
Symbols
Trivia
Cultural Knowledge

Basic Math
Basic Geometry
Agentness
Objectness

Interface text:
AIE

### Continual learning

Dwarkesh Patel describes how humans build context and learn from failures. The speaker says ARC-AGI plans to build a benchmark environment to test this kind of continual learning in AI.

[12:20](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=740s) [A highlighted passage describes building context, examining failures and making small improvements through practice.](https://assets.ai.engineer/sites/aiecode2025/sha256/a30970d80dcda2c2757b4739ce8623a09c6122260db688bde46851a0b964f26b.webp)

Continual learning is a huge bottleneck
DWARKESH PATEL

model high level feedback. You’re stuck with the abilities you get out of the box. You can keep messing around with the system prompt. In practice this just doesn’t produce anything even close to the kind of learning and improvement that human employees experience.

The reason humans are so useful is not mainly their raw intelligence. It’s their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task.

How do you teach a kid to play a saxophone? You have her try to blow into one, listen to how it sounds, and adjust. Now imagine teaching saxophone this way instead: A student takes one attempt. The moment they make a mistake, you send them away and

Interface text:
AIE
Microsoft
smol ai

[12:29](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=749s) [A highlighted quotation describes humans building context, examining failures, and improving through practice.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d11ae76039dbabfc61a0651c33e9bb605a348b63be5915947cf6e8e4fc8389d.webp)

Continual learning is a huge bottleneck
DWARKESH PATEL

model high level feedback. You’re stuck with the abilities you get out of the box. You can keep messing around with the system prompt. In practice this just doesn’t produce anything even close to the kind of learning and improvement that human employees experience.

The reason humans are so useful is not mainly their raw intelligence. It’s their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task.

How do you teach a kid to play a saxophone? You have her try to blow into one, listen to how it sounds, and adjust. Now imagine teaching saxophone this way instead: A student takes one attempt. The moment they make a mistake, you send them away and

Interface text:
AIE
aws

### Skill-Acquisition Efficiency

The proposed benchmark will measure how long hundreds of humans take and how many actions they use on ARC tasks, then compare AI performance using the same measures.

[12:56](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=776s) [Skill-acquisition efficiency, with humans as the baseline for general intelligence.](https://assets.ai.engineer/sites/aiecode2025/sha256/a41866fc6e74f079065a9f95eba85cafae05b2a1e44ed3a8c271556199186dbb.webp)

Skill-Acquisition Efficiency
Proof of General Intelligence (Humans) as the baseline

Interface text:
AIE
World’s Fair
aws

### Preview and launch timeline

After the June 2025 sneak peek, the team planned a five-game sandbox preview and mini agent competition in July, with a goal of about 120 games for the Q1 2026 launch.

[13:43](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=823s) [Timeline shows a June 2025 sneak peek, July 2025 sandbox preview, and Q1 2026 general availability.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca670226212c85fdce93685d88dc6a6ecf773fe58dc185a319996e7250d16ed9.webp)

AI Engineer
World’s Fair
Sneak Peak
JUNE 2025

Preview
Sandbox Launch
JUL 2025

Launch
General Availability
Q1 2026

AIE
aws

### Get Involved with ARC Prize

ARC Prize seeks donations to fund games, adversarial testers to pressure-test ARC-AGI-3, and game designers and Python developers to help build games with its lightweight engine.

[14:27](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=867s) [ARC Prize invites sponsors, early testers, and game designers and developers to participate.](https://assets.ai.engineer/sites/aiecode2025/sha256/b970d452ae0ebc241c3812f7b5a3682d0d17a45dcabd58b33c862a583b0c4467.webp)

Get Involved with ARC Prize

Sponsor
Make a tax-deductible donation to fund the creation of games that challenge AI.

ARC-AGI-3 Early Tester
Build agents for adversarial testing and help shape the early architecture.

Game Designers & Developers
Make games that test agentic reasoning and guide progress towards AGI.

AIE
Microsoft
smol ai
World’s Fair

[14:31](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=871s) [Slide inviting sponsors, ARC-AGI-3 early testers, and game designers and developers to participate.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f828a1cad5f0488496b954cf3fee6296a12e7f27551e6a133b151f9167ec186.webp)

Get Involved with ARC Prize

Sponsor
Make a tax-deductible donation to fund the creation of games that challenge AI.

ARC-AGI-3 Early Tester
Build agents for adversarial testing and help shape the early architecture.

Game Designers & Developers
Make games that test agentic reasoning and guide progress towards AGI.

AIE
World’s Fair

## Full transcript

[0:00](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=0s) [on hold music] Today, we are gonna talk about why AI benchmarking is about to get a lot more fun.

[0:21](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=21s) But before we do, we need to go over some cool demos here. So I love the Claude Plays Pokémon demo. There's something s- about-- really special about seeing this little embodied agent make its own decisions, go and play Pokémon, uh, for us from our childhood here.

[0:35](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=35s) Now, OpenAI got in on the same game. I thought this was awesome. And then just the other day, Gemini beat Pokémon. I like seeing the little robots with AGI on top of their head.

[0:46](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=46s) That must mean that we're already there, right? Well, if we have these agents playing Pokémon, and it's already doing it and beating it, that means the game's over, right?

[0:52](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=52s) We're all done. Well, not quite because with Claude Plays Pokémon, we saw that it would get stuck in the same place for three days. It would need interventions. It would hallucinate different actions.

[1:04](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=64s) And not only that, there was a ton of Pokémon training data that was within the model itself. So although this is a really cool example about an agent exploring a world, there are a lot of things that we can go and improve on.

[1:15](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=75s) So as Kyle was saying, my name is Greg Kamradt, president un-- president of ARC Prize. We are a nonprofit with the mission to be a North Star guide towards open AGI.

[1:25](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=85s) We were founded last year by Mi-- uh, François Chollet and Mike Knoop. And just last December, we were invited by OpenAI to join them on their live stream to co-announce the o3 preview results on ARC-AGI.

[1:37](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=97s) Now, there's a lot of AI benchmarking companies out there, but we take a very opinionated approach as to how we should do this. And our opinion is that the best target we should be aiming for is actually humans.

[1:48](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=108s) And the reason why we think that is 'cause we see that humans are the one proof point of general intelligence that we know about. And if we use humans as the target, that does two things for us because what we do is we come up with problems that are feasible for humans, but hard for AI.

[2:04](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=124s) Now, while we do that, that does two things. Number one is it creates a gap, and when you have that gap, you can start to measure, well, how many problems can we come up with that humans can still do but AI can't?

[2:15](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=135s) And then number two is it guides research. So you can quantify that class of problems and then go tell researchers, "Hey, there's something really interesting going on on this side of the problem.

[2:25](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=145s) Uh, there's something that we need to go, uh, check out from there." All right? So if we're going to measure artificial general intelligence based off of humans, we need to actually define, well, what is general intelligence?

[2:38](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=158s) And there's two definitions that I love to, uh, quote. The first one was by John McCarthy, and he says that AI is the science and engineering of, uh, making machines do tasks, and this is the important part, that they have never seen beforehand and they have not prepared for beforehand.

[2:55](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=175s) This is very important because if you've seen a class of problem beforehand, if it's already in your training data, then you're simply just repeating memorization. You're not actually learning anything new on the fly, right?

[3:06](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=186s) The second, uh, person I like to quote on this is actually François himself, and he put it very eloquently within just three words, and he calls intelligence skill acquisition efficiency.

[3:18](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=198s) And this is, this is really beautiful here because skill acquisition, can you learn new things? And not only that, but how efficiently can you learn those new things? And humans are extremely...

[3:28](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=208s) Uh, spoiler, humans are extremely efficient at learning these new things. So François proposed this definition in his two thousand nineteen paper on the measure of intelligence, but he went further than that.

[3:37](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=217s) He didn't just define it. He actually proposed a benchmark to see can a human or s- or, uh, an AI, can it learn something new and then go repeat what it learned?

[3:47](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=227s) And this is where the ARC-AGI version one benchmark came out. So over here on the left-hand side, this is the learn the skill portion. This is what we call the training portion.

[3:54](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=234s) And what we show you is a transformation from an input to an output grid, and then the goal for the human or the AI is to look at it and say, "Hmm, what's going on here?"

[4:03](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=243s) And then on the left, we actually ask you to demonstrate that skill. So it's a little mini skill you learn on the left, and we ask you to demonstrate it on the right.

[4:09](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=249s) And if you can successfully do it, and this is what it looks like, it's just a grid editor here, then yes, you've learned what the transformation is, and you've actually applied this.

[4:17](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=257s) And so you're showing a non-zero level of generalization as you go through this. So our benchmarks, ARC-AGI 2, this is the most recent one, it has over a thousand tasks in it.

[4:27](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=267s) And the important part here is each one of these tasks is novel and unique. And what I mean by that is the skills required for one of them, we will never ask you to apply that same skill to another task.

[4:39](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=279s) Um, this is very important because we're not testing whether or not you can just repeat the skill you've already learned, but we wanna test all the little mini skills that you can do over time and so you can-- see if you can actually demonstrate those.

[4:50](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=290s) And if we're gonna back up that humans can actually do this, well, we need to go get first-party data. So our group as a nonprofit, we went down to, uh, San Diego, and we tested over four hundred people.

[4:59](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=299s) So rented a bunch of computers, and we did this in person to prever-- uh, have data privacy. And we made sure that every single task that was included in ARC-AGI was solvable by people.

[5:11](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=311s) So this isn't just an aim here. We're actually doing the work to, to, to go and do that. But if we think about it, there's actually quite a bit of human-like intelligence that's out of scope from what we call a single-turn type of benchmark.

[5:25](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=325s) With ARC-AGI, you have all the information presented that you need right at test time. You don't need to do any exploring or anything, and it's all through single turn.

[5:33](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=333s) So if we're gonna be measuring any human-like intelligence, and I would argue that if you are gonna measure human-like intelligence, it needs to be interactive by design.

[5:44](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=344s) And what you need to have is you need to be able to test the ability of an agent, whether that be biological or artificial, to explore an open world, understand what goals it needs to do, and ultimately look at the rewards and go from there.

[5:58](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=358s) So this is actually very in line with what Rich Sutton had just published within his paper, Welcome to the Era of Experience, and he argues that if we want agents that will, um, be readily adaptable to the human world, they need to engage with the open world They need to collect observational data, and they need to be

[6:16](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=376s) able to take that data to build a world model and make their own rules and really understand what it is, or else you're just gonna have the human ceiling, uh, the human data ceiling going forward from here.

[6:26](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=386s) If we're gonna be able to build this, we're gonna need a new type of benchmark that gets out of the single-turn, uh, realm, and this is where interactive reasoning benchmarks are gonna come in.

[6:36](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=396s) Now, an interactive reasoning benchmark is gonna be a benchmark where you have a controlled environment, you have defined rules, and you may have sparse rewards where an agent needs to navigate to understand what is going on in order to explore and, uh, complete the objective from here.

[6:50](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=410s) Now, there's an open question as to, all right, if our aim is interactive reasoning benchmarks, what is the medium in which we're gonna actually execute these benchmarks in? And it turns out that actually games are quite a v- uh, quite suitable for interactive reasoning benchmarks.

[7:07](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=427s) The reason for this is is games, they're a very unique set of, um, intersection of complex rules, defined scope, and you have large flexibility into creating these types of environments that you can then go put different artificial systems in or biological systems with it.

[7:23](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=443s) Now, I know what you may be asking here. "Wait, Greg, didn't we already do games? Didn't we do this 10 years ago? We already went through the Atari phase."

[7:33](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=453s) Well, yes, we did, but there's actually a huge amount of issues with what was going on during that realm there. Not, um, uh, even just starting with all the dense rewards that come with the Atari games.

[7:43](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=463s) There was a ton of irregular reporting, so everybody would report their own performance on these different scales, and it was tough to compare these models with it. There was no hidden test set that came, and then one of my, um, biggest gripes with the Atari phase was that all the developers, they already knew what the Atari games

[7:57](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=477s) were, so they were able to inject their own developer intelligence into their models themselves, and then all of a sudden, the intelligence of the performance, that, well, that's getting borrowed from the developer.

[8:07](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=487s) That's not actually getting done by the model itself from there. So if we were able to create a benchmark that overcame these shortcomings, well, then we'd be able to make a capabilities, uh, assertion about the model that beat it that we've never been able to make beforehand.

[8:24](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=504s) And so to put it another way that's a bit more visual, we know that AI can beat one game. This is proved. We-- AI can beat chess. AI can beat Go.

[8:30](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=510s) We've seen this many, many, many times here. And we know that AI can beat 50 games with 50 known games with unlimited compute and unlimited training data. We've seen this happen with Agent57 and MuZero.

[8:42](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=522s) But the assertion that we want to make is, well, what if AI beat 100 games that the system has never seen beforehand and the developer has never actually seen beforehand either?

[8:56](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=536s) If we were able to successfully p-put a test or put AI to this test, then we could make the capabilities assertion about that AI that we don't, we don't currently have in the market right now.

[9:08](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=548s) And I'm excited to say that that's exactly what ARC is gonna go build. So this is gonna be our version three benchmark. Today is a sneak pr-preview about what that's gonna look like, and this is gonna be our first interactive reasoning benchmark that is gonna come from ARC, and I wanna jump into three reasons why it's very

[9:24](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=564s) unique here. So the first one is, much like our current benchmark, we're gonna have a public training and a public evaluation set. So the reason why this is important with our public training, call it on the order of about 40 different novel games.

[9:37](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=577s) This will be where the developer and the AI can understand the interface and understand kinda what's going on here. But all performance reporting will happen on the private evaluation set, and this is very important because on this private evaluation set, there's no internet access allowed, so no data's getting out about this.

[9:52](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=592s) The scores that come out of private evaluation set will have been done by an AI that has never seen these games beforehand, and neither has the developer seen them.

[10:00](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=600s) So we can authoritatively say that this AI has generalized to these open domains here. Now, the second important point about what ARC-AGI 3 is gonna have is it's gonna force understanding through exploration.

[10:13](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=613s) One of my other gripes with, uh, current game benchmarks out there is you give a lot of instruction to the actual AI itself. "Hey, you're in a racing game," or, "Hey, you're in an FPS.

[10:22](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=622s) Go control the mouse and do all these things." We're gonna drop AI and humans i-into this world, and they won't know what's going on until they start exploring. So even as I look at this screenshot, this is actually one of our first games.

[10:34](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=634s) We call it Locksmith. We give all of our games a cool little name like that. As I look at this, it was-- I don't know what's going on, right?

[10:40](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=640s) But I start to explore, and I start to understand, oh, there's certain things I need to pick up. There may be walls. There may be goals and objectives. I'm not sure what those goals and objectives are right when I first start, but that's the point.

[10:53](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=653s) So not only are we gonna ask humans to explore and make up their own rules as to understand how to do the game, but we're gonna require the same thing for AI as well, and that's something that we're not currently seeing from the, from the reasoning models that we have from there.

[11:06](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=666s) Now, the third key point is that we're only gonna require core knowledge priors only. This is something that we carry from ARC-AGI 1 and 2 as well, but what this means is you'll notice the ARC tasks, there's no language.

[11:18](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=678s) There's no text that's being involved here. There's no symbols, and we're not asking you any trivia. So I call these, um, o-other benchmarks that rely on these, sometimes we try to make the hardest problems possible.

[11:28](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=688s) We go hire the best people in the world, and I call them PhD++ problems, right? And that's great, but AI's already superhuman. It's way smarter than me in a lot of different domains.

[11:38](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=698s) We take the alternative approach, which is let's look at more of the floor and look at the reliability side. Let's take all the core, um, anything outside of core knowledge and, and strip those away.

[11:46](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=706s) So core knowledge priors, the four of them that there are,

[11:50](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=710s) are basic math, and these are things that as humans that we're either born with or hardwired to gather right, immediately after birth. So basic math, meaning counting up to 10.

[11:59](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=719s) Basic geometry, so understanding different shapes and topology. And then agentness, which is understanding theory of mind, that there's other types of agents out there in the world that I know that they're interacting.

[12:08](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=728s) And then the fourth one is objectness. So as we create our benchmark, these are the four principles that we like to, um, go after when we try to test the abstract and reasoning piece.

[12:18](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=738s) Now, I was reading the recent, uh, Dwarkesh essay, and he actually put it really well in one of his paragraphs here. He was talking about one of the reasons why humans are, are great, and he says, "It's their ability to build up context, interogate- interrogate their own failures, and pick up small improvements and efficiencies as they practice

[12:35](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=755s) a task." We don't yet have this type of environment that can go and test this from a benchmark perspective for AI, and this is exactly what ARC-AGI is going to go build.

[12:46](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=766s) So before we wrap it up here, I want to talk about how we're going to evaluate AI because it's like, okay, cool, you go- they go play the game.

[12:51](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=771s) Well, what does it mean? How do you know if it's doing well or it's not? And we're going to bring it back to François's definition. So we're going to bring it back to s- uh, skill acquisition efficiency, and we're going to use humans, which again, is our only proof point of general intelligence.

[13:05](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=785s) We're going to use humans as the baseline. So we're going to go and test, um, at hundreds of humans on these exact ARC tasks, and we're going to measure how long does it take them, how many actions does it take them to complete the game, and then we're going to get a human baseline, and we're going to

[13:18](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=798s) be able to measure AI in the same exact way. So can the AI explore the environment, intuit about it, create its own goals, and complete the objectives faster than humans?

[13:27](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=807s) Well, if it cannot, I would go as far as to assert that we do not yet have AGI. And as long as we can come up with problems that humans can still do but machines cannot, I would again assert that we do not have AGI with it.

[13:38](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=818s) So we're going to be looking at skill acquisition efficiency as our main, uh, our main output metric here.

[13:45](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=825s) Today, we're giving a sneak peek about what this looks like. This is World's Fair. Actually, next month, uh, in San Francisco, we're going to give a sandbox preview. So we're going to release five games.

[13:52](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=832s) Um, we know better than to try to wait till the end. We're going to make contact with reality. We're going to put out these five. We're actually going to host a mini agent competition too.

[13:59](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=839s) So we want to see what is the best possible agent that people can do. We'll put up a little prize money. Um, and then we're going to look forward to launching about a hundred and twenty games, that's the goal, by Q1 of 2026.

[14:10](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=850s) Now, that sounds like it's not that many games, and you think it's not that many data points, but the richness of each one of these games goes really, really deep.

[14:17](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=857s) There's multiple levels. It goes deep with each one of them, and it's quite the operational challenge to make all of these, and that's a whole another side of the benchmarking process, which I'm happy to talk about later.

[14:27](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=867s) If this mission resonates with you, again, ARC Prize, we are a nonprofit. One of the best ways to get involved is through making a direct tax-deductible donation from that.

[14:35](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=875s) If anybody in the room knows any philanthropic donors, whether it be LPs or individuals, I'd love to absolutely talk to them. But then also, we're looking for adversarial testers.

[14:43](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=883s) We want to pressure test ARC-AGI-3 as best as we can. So if there's anybody who's interested in participating in the agent competition, whether it's offline or offline, uh, online or offline, let me know.

[14:53](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=893s) Happy to chat. And then also, kind of cool story, we originally started with Unity to try to make these games, and we quickly found out that Unity was way overkill for what we needed to do if you're just doing two-by-two, sixty-four-by-sixty-four, uh, games here.

[15:06](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=906s) So we're actually making a very lightweight Python engine ourselves. So if an- there's any game developers out there, anybody who wants to get involved with this and knows Python well, we're looking for game developers and game designers as well.

[15:20](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=920s) That is all we have today. Thank you very much. [audience applauding]

[15:29](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=929s) Kyle, do we have-

[15:31](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=931s) Yeah. I think we have time in this case for, for a couple of questions. If anyone wants to come up, there's microphones, one, two, three of them. Um, maybe a couple of questions.

[15:40](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=940s) Um, I'm going to kick that off-

[15:41](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=941s) Sure, yeah, yeah

[15:41](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=941s) ... if that's all right.

[15:42](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=942s) Yes, yes.

[15:42](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=942s) Um, question for you. So, um, and I don't know where, um-

[15:46](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=946s) I, I can repeat. Okay.

[15:48](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=948s) All right. All right. Um, I, uh, I, I, it's, it's very hard to make estimates about timelines famously, but, um, if you had to guess, how long do you think this new version of the benchmark you're making will take before it gets saturated?

[16:01](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=961s) Um, well, the wa- [laughs] the way I think about that is, uh, well,

[16:08](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=968s) I would say I'm counting in years, I'm not counting decades. We'll put it that way.

[16:12](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=972s) Okay.

[16:12](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=972s) Yes.

[16:13](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=973s) Interesting. All right. Uh, yeah, we'll, we'll take, uh, one at each mic. Looks like you've... it's well distributed, so starting over here.

[16:19](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=979s) Sure. Hi. Um, you mentioned efficiency as part of the requirements, and so I'm wondering for the, uh, the benchmarks if you're considering things like wattage or time or other ways of, of using that as one of the criteria.

[16:33](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=993s) Yeah. I, I love that question, and I would've put it in if I had more time, but m- I'm very opinionated about efficiency for measuring AI systems. If I could have two denominators for intelligence on the output, number one would be energy because you know how much energy the human br- uh, the human brain takes, and that's

[16:47](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1007s) our proof point of general intelligence, so you can take how much calories the human brain takes. So I would love to do energy. But the number two denominator is the amount of training data that you need for it, neither of which are very accessible for closed models in the current day, so we use proxies, and the proxy

[17:00](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1020s) is the cost. But then with interactive evals like this, you get another proxy, which is, um, action count and how long does it take you to actually do it.

[17:07](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1027s) We're not going to have a, uh, a wall clock within these games. It's going to be turn-based, um, so we won't have a wall clock to do it.

[17:14](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1034s) Awesome. All right. Question two and then we'll do three. Please keep them both very short. Yeah. Um, yeah, very quick question. Um, could you define more what do you mean by o- o- objectness?

[17:23](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1043s) Um-

[17:23](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1043s) Yes. It, it, that one's actually quite simple. It's just understanding that when you look out into the world, that there's, um, a mass of things that may act together.

[17:32](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1052s) So the crude way would be you have one pixel, but then it's surrounded by a whole bunch of other pixels, and they all move together. You understand all those pixels as one and it kind of acting as a one body rather than individual.

[17:42](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1062s) And, uh, really, evolutionary-wise, that's the same, that's a tree over there. All this is part of the same tree, that, that kind of thing.

[17:48](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1068s) Thanks.

[17:49](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1069s) Um, final question, I'll keep this one super short. Uh, how do you distinguish between tasks that, um, in the games that you gu- you guys are developing, how do you distinguish between tasks that humans cannot do and, and AGI also cannot do?

[18:03](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1083s) Like, what is the North Star there?

[18:06](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1086s) It, it's a good question. It, hu- tasks that humans cannot do are a bit out of scope for our thesis on how we want to drive towards AGI. So I would say that's not, that's not really the aim that we're looking, looking for on that.

[18:16](https://www.youtube.com/watch?v=3XmFPwjG8pg&t=1096s) Um, that's a whole different con- uh, conversation around super intelligence that maybe that's for another time. Thank you. [outro music]
