AI Engineer World's Fair 2025

Measuring AGI: Interactive Reasoning Benchmarks

Loading slides…

Read the talk

Measuring AGI: Interactive Reasoning Benchmarks

ARC-AGI-3 proposes a harder test than winning a familiar game: discover unfamiliar rules, learn through interaction, and compare that learning efficiency with humans.

From a talk by Greg Kamradt

What does beating Pokémon prove?

Watching Claude play Pokémon makes autonomous agents tangible: a small character moves through a familiar world while a model decides what to do next. OpenAI joined the game-playing demonstrations, and Kamradt points to a reported Gemini Pokémon completion. If an agent can finish the game, how much evidence of general intelligence does that provide?

The failures matter as much as the finish. Kamradt reports that Claude remained stuck in one place for three days, needed human interventions, and hallucinated actions. The accompanying post identifies the location as Cerulean City.

Post with an orange outline around the claim that Claude is still stuck in Cerulean City, above a Pokémon livestream screenshot.Open full source frame
A post highlights Claude remaining stuck in Cerulean City for three days.

There is also the question of prior knowledge. Kamradt argues that Pokémon material was already present in the model’s training data. Exploring a game is impressive, but a familiar game cannot cleanly separate learning during play from recalling information acquired beforehand. The benchmark needs to test how an agent acquires a new skill, not merely whether it can display an existing one.

0:210:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:21 · section reference included

Humans as the reference point

Greg Kamradt leads ARC Prize, a nonprofit founded by François Chollet and Mike Knoop to guide progress toward open AGI. Its work includes participating in OpenAI’s December livestream announcing o3 preview results on ARC-AGI. Its benchmark design starts from a specific reference point: humans are the only demonstrated example of general intelligence available to us.

That reference gives benchmark builders a practical search strategy: find problems people can solve but AI cannot. The remaining problems define a measurable capability gap. They also direct research toward particular failures, rather than making the benchmark merely a scoreboard. A useful task reveals something that a human can learn and a machine still struggles to acquire.

1:151:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:15 · section reference included

Intelligence as skill acquisition efficiency

Kamradt introduces unfamiliar, unprepared-for tasks through a definition he attributes to John McCarthy. McCarthy’s own definition concerns the science and engineering of making intelligent machines; the emphasis on unseen tasks is best understood here as Kamradt’s framing. For evaluation, the distinction is consequential: retrieving a familiar solution does not establish that the system learned a new skill during the test.

Chollet’s formulation is more compact: “skill acquisition efficiency.” It asks both whether a system can learn something new and how efficiently it does so. His 2019 paper, On the Measure of Intelligence, connects that definition to a concrete benchmark. Human learning efficiency supplies the motivating comparison; ARC supplies tasks on which learning and subsequent application can be observed.

Slide juxtaposing John McCarthy and François Chollet, with a paraphrased definition beneath McCarthy and “Skill-Acquisition Efficiency” beneath Chollet.Open full source frame
Definitions of intelligence emphasize unfamiliar tasks and skill-acquisition efficiency.
2:382:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:38 · section reference included

Learn a transformation, then apply it

An ARC-AGI-1 task separates learning from demonstration using colored grids:

  1. Inspect example input grids and their corresponding output grids.
  2. Infer the transformation that explains those pairs.
  3. Apply that transformation to a new input, constructing the answer in a grid editor.

A correct answer demonstrates some generalization: the solver has carried a rule from the examples to a new case. The examples are the task’s training portion, not an instruction to retrain a model’s weights.

Kamradt describes ARC-AGI-2 as containing over 1,000 tasks. He emphasizes that each task targets a novel skill rather than asking the solver to repeat another task’s transformation. The intended unit of measurement is therefore a succession of small learning problems, not repeated execution of one mastered procedure.

Kamradt reports testing over 400 people in person in San Diego to establish human solvability. The team rented computers and conducted testing in person to protect task data. The important claim is task-level coverage: every included task could be solved by people, not that every participant solved every task. The ARC-AGI-2 announcement provides the corresponding human-validation account.

3:473:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:47 · section reference included

What a single-turn benchmark leaves out

The grid format still leaves substantial abilities untested. All necessary information is supplied at test time. The solver does not need to explore an environment to discover what information matters. An interactive test adds that missing responsibility: choose actions, inspect their consequences, discover goals, and interpret rewards.

This direction connects to David Silver and Richard Sutton’s Welcome to the Era of Experience. Agents collect observations through engagement with the world, use those observations to construct world models, and infer rules for themselves. Learning exclusively from human-produced data imposes a ceiling that experience is intended to move beyond.

An interactive reasoning benchmark puts that process inside a controlled environment with defined rules and potentially sparse rewards. The evaluator knows the environment’s structure; the agent must discover enough of it to complete an objective. Sparse feedback makes exploration consequential because an action may yield useful information without immediately producing a reward.

5:115:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:11 · section reference included

Games that are unfamiliar to the developer, too

Games offer complex rules within a bounded scope, with enough design flexibility to create many different environments for both humans and artificial agents. But using games does not automatically produce a clean test of generalization. Kamradt identifies several weaknesses in earlier Atari evaluations:

  • Dense rewards: frequent feedback can guide behavior without requiring the same discovery process as a sparse-reward environment.
  • Inconsistent reporting: different performance scales make systems difficult to compare.
  • No hidden test set: familiar evaluation games permit development against the test itself.
  • Developer knowledge: a developer who knows the game can encode useful strategies into the system.

The last issue changes what a score means. Some apparent agent intelligence may have been supplied by the person building it.

Chess and Go establish that AI can master individual games. Systems such as Agent57 and MuZero extend the discussion to suites of known games. Kamradt loosely describes this as beating 50 known games with unlimited compute and training data. The historical Atari suite contains 57 games, and that shorthand should not be read as a literal resource condition or as identical results from the two systems. The relevant distinction is that the games were known during development.

The proposed stronger test is to beat 100 games that neither the system nor its developer has seen before. Success would support a different capability claim: the system can acquire useful skills across unfamiliar environments without game-specific preparation by its creator. This is the ambition behind ARC-AGI-3, presented in the recording as a preview of ARC’s first interactive reasoning benchmark.

Slide reading “Beat 100 games” and “That neither the system nor developer have seen before,” beside an illustration of a person and robots playing games.Open full source frame
Beat 100 games that neither the system nor its developer has seen before.
6:507:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:50 · section reference included

Private scoring and discovery through play

The preview proposes approximately 40 novel public training games so developers and agents can learn the interface. Kamradt also announces a public evaluation set, while assigning all performance reporting to a private evaluation set; he does not specify the public evaluation set’s separate role. Private evaluation would run without internet access, intended to prevent game data from escaping. The scored games must remain unfamiliar to both the model and its developer.

Novelty also applies to the instructions. A typical game benchmark might tell an agent that it is driving a racing car or playing a first-person shooter, then explain the controls. ARC-AGI-3 instead proposes dropping humans and AI into an environment without that framing. Understanding must emerge through exploration.

The example is Locksmith. Its initial screenshot does not explain the objective. As a player explores, they may discover things to pick up, walls, and possible goals. Those are hypotheses to establish through interaction, not instructions handed over at the start. Humans and AI face the same requirement: construct an account of how the world works, then use it to make progress. Kamradt presents this as a gap in the reasoning models available at the time.

9:249:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:24 · section reference included

Testing the floor of reasoning ability

The games are intended to require only core knowledge priors. Kamradt describes the tasks as lacking language, text, symbols, and trivia. The practical distinction is the absence of required external linguistic or specialist knowledge, rather than the exclusion of symbolic reasoning: visual patterns can themselves carry meaning. Instead of commissioning ever harder expert problems, ARC aims to test the reliability of basic capabilities, even when AI is already superhuman in particular domains.

Kamradt describes four priors as innate or acquired very early in human development:

PriorIntended foundation
Basic mathSmall quantities, such as counting up to 10
Basic geometryShapes and topology
AgentnessRecognizing other interacting agents; theory of mind
ObjectnessRecognizing objects as units

These constrain what a task may assume before it begins testing abstraction and reasoning.

Practice adds another dimension. Kamradt invokes a passage matching Dwarkesh Patel’s Why I don’t think AGI is right around the corner: humans accumulate context, examine their failures, and find small improvements as they repeat a task. An interactive environment can expose that process over a sequence of decisions. ARC-AGI-3 is intended to provide the benchmark setting Kamradt sees as missing for those behaviors.

11:0611:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:06 · section reference included

Measure the learning, not just the win

Finishing a game supplies an outcome, but skill acquisition efficiency also requires accounting for how the solver got there. The proposed evaluation would test hundreds of humans on the same games, recording completion duration and action count, then evaluate AI against that baseline. The comparison encompasses exploration, inference about the environment, creation of intermediate goals, and completion of the objective.

Kamradt sets a demanding interpretation: if AI cannot outperform humans in this process, he would not call it AGI. More broadly, he treats the continued existence of human-solvable problems that machines cannot solve as evidence against having reached AGI. That is the project’s proposed criterion, with skill acquisition efficiency as its main output metric.

12:4612:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:46 · section reference included

Building and pressure-testing the benchmark

At the time of the preview, the plan was to release five sandbox games the following month in San Francisco, alongside a small agent competition with prize money. The purpose was to put the environments in front of strong agents early and learn from what happened. Kamradt’s larger release target was approximately 120 games by Q1 2026. These were forecasts in the recording. Each game would contain multiple levels, so the number of games alone understates both the available interactions and the operational work of creating them.

That work requires funding and attempts to break the evaluation. Kamradt invites direct donations, described as tax-deductible, and introductions to philanthropic donors, including individuals and LPs. He also seeks adversarial testers and participants in online or offline agent competitions to pressure-test ARC-AGI-3.

The implementation had already become simpler. The team initially tried Unity, then found it excessive for these small grid games and began building a lightweight Python engine. That choice also shaped recruitment: Python-capable game developers and game designers could help construct the environments.

13:4513:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:45 · section reference included

How long will it last, and what counts as efficient?

Asked how long the new benchmark might take to saturate, Kamradt offers a tentative horizon of years rather than decades. He gives no exact deadline. The benchmark is intended to expose a present capability gap, not to remain permanently beyond AI.

An audience question about wattage and time makes the efficiency requirement more precise. Kamradt would ideally measure intelligence against two inputs:

  • Energy: the human brain provides a physical reference for the resources consumed by a generally intelligent system.
  • Training data: the amount of prior experience required matters when judging learning efficiency.

Both are difficult to obtain for closed models. Cost is a practical proxy, and interactive evaluation adds observable action counts and completion duration.

The games themselves, however, are turn-based and have no in-game wall clock. That distinction separates actions that change the environment from time spent deciding what to do. The preview leaves the exact duration measure unspecified; its clearest operational measure is how many interactions a solver needs.

15:4816:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:48 · section reference included

Recognizing objects—and the boundary of the test

A second audience question asks what objectness means. Kamradt makes it concrete: a pixel surrounded by other pixels that all move together can be understood as part of one body. The useful representation groups their behavior instead of treating every pixel as independent.

A small Python example illustrates the shared-motion cue. Three labeled pixels retain their identities while each moves one column to the right. Their common displacement is evidence for treating them as one moving object:

python

before = {"a": (1, 1), "b": (1, 2), "c": (2, 1)}
after = {"a": (1, 2), "b": (1, 3), "c": (2, 2)}

def shared_displacement(before, after):
    if not before or before.keys() != after.keys():
        return None
    shifts = {
        (after[key][0] - row, after[key][1] - column)
        for key, (row, column) in before.items()
    }
    return next(iter(shifts)) if len(shifts) == 1 else None

shift = shared_displacement(before, after)
assert shift == (0, 1)

This captures one cue for grouping, not a complete object-perception system.

Kamradt’s everyday analogy is a tree: its parts are understood as belonging to the same tree. The prior is a basic way of organizing perception into entities on which reasoning can operate.

The final question asks about tasks that neither humans nor AGI can perform. Those fall outside ARC’s stated aim. Its reference remains the set of tasks humans can solve, and the gap where machines cannot yet match them. Kamradt places human-impossible tasks in a separate discussion of superintelligence, leaving this benchmark focused on learning abilities for which human performance supplies a concrete standard.

17:1417:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:14 · section reference included

Resources

From the talk

Updates since the talk

  • The later benchmark report explains environment construction, human calibration, dataset splits, and Relative Human Action Efficiency scoring.

Transcript download
Full transcript

Loading transcript…