Robots Will Be Trained on Video Games - Pim de Witte, General Intuition
Loading slides…
Read the talk
Why General Intuition Trains Robot Operators on Video Games
Medal records what players see and which controls they use. Pim de Witte explains how those paired records help General Intuition train a model that can play a game or steer a robot, issuing several inputs at once as new frames arrive.
From a talk by Pim de Witte
At a glance
Ideas worth remembering
Preserve the operator’s inputs alongside the frames. A control such as a rudder can matter even when the camera gives little evidence that the operator touched it.
Expose the controls a human would use and label what each one does. That connects a text instruction such as waving back to an action the device can execute.
For tasks that need frequent corrections, examine actions per minute and simultaneous inputs as well as the number of recordings.
Assign long-term planning to an orchestrator and visually guided control to the action policy. For shared control, include the person’s inputs and give the policy only the controls it needs to supply.
What Olaf can see on the way down the mountain
Pim de Witte asks the audience to imagine training a model on information from across the internet and using it to solve scientific problems whose answers never appeared there. General Intuition is making a comparable bet on a different body of human experience. By learning which controls players use in response to what they see on screen, a model may learn to operate a physical machine from its camera feed.
To see what kind of intelligence that would take, de Witte asks the audience to picture Olaf, the snowman in Frozen, rolling down a mountain. A little stone pops up. Olaf spreads his pieces apart, clears the obstacle and comes back together. The important detail is when he gets to see the stone. He is already moving. He can take in new information while carrying out an action, then change what he does in time for it to matter.
An autoregressive language model can produce a sequence describing the whole maneuver. For an operator on the mountainside, the problem is deciding what to do with the next observation while the maneuver is underway. That timing requirement is de Witte’s reason for building an architecture specifically for action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Medal records the view and the input that went with it
The starting point was Medal, which de Witte describes as the world’s largest video game recorder. People use it to record themselves playing tens of thousands of games. From a spectator’s perspective, the product is a clip. From the perspective of someone training an operator, the valuable part is the association Medal can preserve between the frames on screen and the inputs the player supplied. A human has already made a decision about what to do in that situation, and the recorder has access to that side of the interaction.
De Witte sees a route from those records to robotics in the equipment people already use. The same person who plays a game can look at a camera feed and teleoperate a robot, in many cases with the same controller. He says robots below a certain number of degrees of freedom routinely ship with game controllers and that much of the supply chain already works this way. He gives no numerical cutoff. The overlap is in the actions available to the operator, even when the machines receiving those actions move quite differently.
General Intuition labels what each action means and gives a vision-based model a set of available controls. De Witte’s API example combines three pieces of information. A developer sends frames, supplies the instruction to wave back when somebody waves, and makes an action labeled greet available. The instruction tells the model which event to watch for. The frames tell it when the event occurs. Its output is the input bound to greet on that device. For the example to work, the device has to expose a control that performs the greeting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A screen or camera keeps supplying the next observation
De Witte uses coding to explain why a different interface makes sense here. A language model trained to predict the next text token can write software in turns. The developer asks for a change, the model works, and the developer can inspect the result before asking for something else. Much useful work fits that rhythm. A machine moving through space can be somewhere else by the time an operator finishes a turn.
General Intuition’s model continuously receives frames, either individually or in batches, from a screen or a robot camera. It predicts actions in the terms a game player would use, including keyboard and mouse inputs or game-controller commands. Several controls can be active together. A text prompt can specify the task, but the stream of observations keeps arriving as the model issues inputs and the scene changes.
De Witte compares the strategy to his reading of Anthropic’s focus on coding. Getting very good at code generation would give people a reason to express more problems in code. His team wants developers to bring more problems into pixel space. A screen already provides a visual representation, a camera provides one for a robot, and de Witte argues that other sensor information can often be represented visually too. A developer who can present the relevant observations that way and expose a set of controls has the two sides of the interface the model learned in games.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The controller already handles the motor math
A human teleoperator can steer a machine without knowing what torque to request from a motor. The manufacturer has already provided a device and a control system that turn an input the operator understands into the movement the machine needs. De Witte contrasts this with robotics efforts that train a model end to end to predict exact poses or motor values. His team places the learned policy at the point where the human would pick up the controller.
That choice determines what can transfer from Medal. A game clip provides evidence about choosing controls from visual observations. It would be a poor source for the exact motor values of a particular physical robot. Working through the controller lets the team train the decision for which it has demonstrations and leave machine-specific execution to the system built for that machine.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A billion uploads a year, measured again in action tokens
De Witte puts Medal’s upload volume at about a billion videos per year and says it was on track for three billion that year. He reports more than 1.5 trillion action tokens per month. The upload count measures how many clips reach Medal. The token count measures the volume of player input available alongside the frames. A player can hold one control while adjusting another, so a single maneuver can involve several inputs at once.
For a sense of scale, de Witte estimates that Medal collects in half a day as much data as frontier labs use to train robotics models. He supplies neither a named comparison dataset nor a common counting method, so the half-day figure is his estimate of relative volume. The bet behind it is that scaling data and compute on this particular prediction problem will produce an operator useful across computer interfaces and physical machines.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When to generate an environment and when to use the original view
General Intuition also builds world models, including Mirai. Their role here is to generate additional environments for the policy to learn in. The distinction is easiest to see from the outputs. The world model can supply a setting and the pixels the agent observes there. The policy takes an observation and chooses what to do. Producing another environment gives the policy another place to exercise that ability.
Making a simulation a prerequisite for learning a new activity would introduce two costs. First comes the compute needed to simulate the setting, which rises with the realism required. Then comes the work of establishing that the simulation preserves what matters about the original. De Witte traces a possible detour in which data passes through an LLM, someone creates a digital twin, the model is deployed into it, and verifiers check whether the two situations are equivalent. Only then does the robot get to learn the activity.
Consider a person trying to communicate something to a robot partner. De Witte wants the partner to be able to observe the very thing the person is looking at. If the team already has that view and the person’s actions, it has a demonstration in the operator’s own input modality. It can use that evidence directly and reserve generated worlds for the cases where additional places to practice are valuable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a drone operator benefits from a gamer’s actions per minute
De Witte calls attention to an example in which the model predicts multiple actions at the same time. For a drone operator making a succession of small adjustments, both the frequency and the overlap matter. One correction can already be underway when the camera view calls for another. Learning which controls people combine is part of learning the maneuver.
This is where he favors game recordings over starting with teleoperation footage. In his account, gamers supply more actions per minute, or APM, giving the policy more examples of fast and simultaneous inputs. The semantic labels also make a prediction easier to inspect. A developer can identify the key the policy selected and the behavior that key represents in the current environment. De Witte anticipates a speed advantage over LLMs on these workloads, though he gives no measured APM or latency comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One policy across games and more than ten robot types
De Witte reports that a single model can play video games at what he calls the highest level and control more than ten types of robots in real time. Across those embodiments it uses the same primitives: receive pixels, emit actions. He withholds the release videos from the livestream, saying the team wants to save them for the launch. The claim establishes the breadth the team reports, with individual games, robot tasks and performance scores left unspecified.
At the time of the recording, some customers already had private access and the New York team had begun giving live demonstrations. De Witte invites attendees to join a waitlist through a QR code. He says signup order will help determine access as the team rolls the models out over the following months.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An orchestrator sets the task and the policy explores the interface
One proposed application starts with an environment generated by an LLM. Someone still has to try it. An action model could explore the result through the controls available to a person. De Witte extends the idea to UX work. If a designer wants to know whether an interface is intuitive, watching an agent attempt to use it from the screen offers a concrete interaction to inspect. He presents this as a use to investigate, without reporting a usability study.
For longer jobs, the expected integration retains an orchestrator. De Witte names Gemini and Astra as possibilities for long-term planning. A planner can supply a text instruction while the action policy handles the succession of visually informed inputs needed to carry it out. His prospective science-lab example depends on both capabilities. Someone must organize what the robots should do, and the robots must reliably execute those instructions in space and time. He also points to a recursive opportunity for robotics businesses: making more robots is itself one of the problems that better robot operation could help solve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A player keeps the mouse while the model takes the keyboard
In an accessibility application the team is exploring, a player who can use one arm operates the mouse. The model supplies the keyboard inputs the game also requires. The player’s own actions are part of what the model observes alongside the screen. That gives the policy context for choosing the complementary keys while leaving the mouse under the player’s control.
This is a different allocation of work from handing an entire task to an agent. The person continues making decisions throughout play, and the model’s share of the controls is limited to what the person needs help operating. De Witte expects another early use to be in simulation, where someone generating environments with Astra or Gemini could put a policy to work exploring them instead of having to play through each one personally.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three teams and a first hands-on window
Medal and General Intuition are based in New York City. De Witte describes three teams. World modeling works on generating worlds. The policy team works on the agent that chooses actions. The third team works on data generation and environment generation, the entry point he suggests for engineers interested in joining. He values experience shipping an AI product end to end and an interest in learning how the models work. He mentions his own engineering path when encouraging candidates without a research background.
The release window he gives is late November or early December. He also announces plans for three hackathons and identifies locations for two, New York City and London. The first was planned for mid-November in New York, with details to go out through the partner waitlist. The attraction for participants was hands-on access of the kind the team already had internally.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The JEPA question turns on a rudder the camera cannot see
A question asks how General Intuition’s approach differs from JEPA. In de Witte’s comparison, a JEPA approach relies largely on inferring the actions behind observations. He allows that this may be the better approach in the limit and says his team borrows some of its ideas. His near-term objection is that it introduces a second scaling question. Along with learning a better policy, one has to count on getting better at recovering the actions that should supervise it.
Imagine a pilot moving the rudder at the back of a plane. From the available video frames, it can be extremely difficult to determine that the pilot used that control. Even if a viewer can see the scene change, the camera may reveal little about the specific input responsible. A synchronized control record gives the learner the pilot’s choice directly, including in sequences where attempting to reconstruct it from the image would be guesswork.
That example explains where General Intuition chose to concentrate its training effort. An archive of recorded inputs gives the team a particular advantage in predicting sequences of actions. De Witte sees less of a corresponding advantage in predicting the next pixels. The team therefore focuses its foundation models on action prediction while using a range of techniques for world modeling and welcoming environments built by others.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…