October 13, 2026Main stage4 min read

Frontier Models for Acting in Space and Time

Pim de Witte explains General Intuition’s bet on action-labeled game data: models that observe continuously and select actions while the world keeps moving, with language models available for longer-term planning.

Pim de WitteGeneral IntuitionCo-Founder & CEO

Watch "Frontier Models for Acting in Space and Time — conference livestream" on YouTube

From the conference livestream · 7:02:19–7:20:53 on YouTube. The player starts and stops at this session’s boundaries.

7:02:19 ↗

The world keeps moving while the model responds

Olaf rolls down a mountain in Frozen, separates his snowman body to avoid a stone, and comes back together. Pim de Witte uses the scene to make a point about acting: the character keeps observing while the action unfolds.

A turn-based interaction gives a model time to respond before the next instruction arrives. Robotics, games, and other real-time environments keep changing during the response. De Witte argues that this creates room for architectures designed around continuous observation and action, alongside language models that are strong at planning and coding.

General Intuition starts from game recordings collected through Medal. Human players already use visual input and controllers to operate within games. People can also use controllers to teleoperate robots. De Witte’s bet is that the overlap provides a useful route from large amounts of gameplay data to action in other environments.

The training data pairs frames with the actions taken at the time, and associates those actions with meaning. The model receives visual input and a set of available controls, with a text instruction when needed, then selects actions from that space.

7:05:10 ↗

Frames in, actions out

The first example is deliberately simple: someone waves, and the model should wave back. It receives frames and an instruction, then predicts an available action that carries the intended meaning.

In the wave-back example, the available action gives the model a way to respond to what it sees. New frames keep arriving during execution, letting the model update its behavior as the scene changes.

De Witte compares this specialization to the bet that coding models would attract more problems into code. General Intuition hopes more problems can be brought into pixel-and-action space. Keyboard, mouse, and controller actions become a common interface across different embodiments.

That interface also avoids having to predict every low-level physical value directly. A human using a controller does not calculate motor torque; an existing control system translates the command. De Witte proposes using that same layer rather than requiring one model to solve the entire physical-control problem from end to end.

Games contain frequent, simultaneous control changes, including the small adjustments needed to fly a drone. De Witte sees this action density in Medal's recordings as a training advantage. Knowing which actions occurred is more informative for this training objective than video alone.

7:09:37 ↗

Learn in the same visual world as the user

World models provide another part of the training environment. General Intuition works on generating environments in which its action models can learn, alongside using recorded gameplay.

De Witte contrasts this with a process that first describes an environment to a language model, builds a digital twin, checks its equivalence, and only then trains behavior inside it. Learning through the native visual input can shorten that chain. He expects the shorter path to improve speed on suitable real-time tasks. That comparison remains a forecast here, without a public benchmark in the talk.

The action labels also carry semantic meaning. The model is learning more than a key code detached from its effect. That matters when the same high-level instruction has to map onto the controls available in a different environment.

At the time of the talk, de Witte describes the system as privately deployed with some customers, with broader access still planned. He says one model can play games and control more than ten robot types using the same general primitives. Broader availability and the hackathon access he announces are still plans at this point.

7:13:44 ↗

Planning above the real-time action loop

One application de Witte proposes is exploring an interface to understand whether people can use it intuitively. Generated environments are another: a language model can create a world, and an action model can explore it.

He still expects an orchestrator model, such as Gemini or Astra, to handle long-term planning. The action model supplies the faster interaction with the current environment. This leaves room for different systems at different time scales rather than requiring the action model to replace every part of an agent.

The most concrete accessibility example is a person who can use a mouse but cannot operate a keyboard. De Witte describes the model observing the person’s input and supplying the missing keyboard actions. The point is shared control: the model need not duplicate the actions the user is already taking. Because observation continues during execution, it can adapt to that input.

He also anticipates scientific and robotic applications, while expecting much early use to occur in simulation. They remain proposed applications.

7:19:23 ↗

Why action labels matter when video is ambiguous

The closing audience question asks how the approach differs from JEPA. De Witte’s answer emphasizes the benefit of having observed action labels instead of relying entirely on inferring actions from frames. He expresses interest in JEPA-style ideas, while arguing that inference adds its own scaling problem in the near term.

His example is a plane’s rudder. A control change may be difficult to infer from the visible frames, even though it matters for the sequence of actions. Recorded controls remove some of that ambiguity. The same data advantage therefore helps action prediction more directly than it helps generating future pixels.

General Intuition can draw on other teams' world-modeling techniques while concentrating its paired visual-and-action data on choosing actions. The rudder example explains why: observing what the player did removes an ambiguity that more video alone may leave unresolved.

The world keeps moving while the model responds

Frames in, actions out

Learn in the same visual world as the user

Planning above the real-time action loop

Why action labels matter when video is ambiguous

Further reading