AI Engineer World's Fair 2025

Robotics: why now?

Loading slides…

Read the talk

Robotics: why now?

Vision-language-action models connect visual understanding to robot control, but dexterity and generalization depend just as much on how demonstrations are collected, annotated, and diversified.

From a talk by Quan Vuong and Jost Tobias Springenberg

Before you start: Familiarity with transformers and model training is helpful; no robotics background is required.

What changes when the environment stops being predictable?

What would it take for one model to control any robot and perform any task? That is the research goal Quan Vuong and Jost Tobias Springenberg introduce at Physical Intelligence. It is not a capability they consider ready: multiple scientific breakthroughs remain necessary. Publishing research, releasing models, and discussing the work openly are part of how they intend to get there.

Industrial robotics shows how much robots can accomplish when the environment is constrained. A factory provides structured surroundings and repeatable motions. Move into an everyday environment, however, and even an apparently simple task can become difficult. The opening comparison pairs an industrial robot with an older clip of a humanoid struggling beside a shopping cart.

Slide showing an industrial robot arm in a factory on the left and a humanoid robot beside a shopping cart on the right.Open full source frame
“Robotics Before”: an industrial robot arm and a humanoid beside a shopping cart.

The newer demonstrations change both sides of that comparison. A dancing humanoid displays complex physical motion. A laundry robot confronts a different difficulty: clothes have just come out of a running dryer, so nobody can precisely arrange each shirt before the task begins. The robot removes the shirts and puts them into a basket; Quan explains that the full video continues with transport to a table and folding. The challenge includes manipulating objects whose shapes and starting positions are difficult to control.

General progress in AI helps explain the change, but the specific bridge to physical behavior is the vision-language-action model, or VLA. It connects the ability to interpret a scene and an instruction with the ability to produce robot actions.

0:160:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

From answering questions to controlling a robot

A vision-language model, or VLM, takes text and images, embeds them, and passes those representations to a transformer that generates a textual answer autoregressively. A VLA adapts that arrangement for robotics. It also receives information about the robot itself, such as joint positions, and produces actions that directly control the robot.

ModelInputsOutput
VLMText and imagesTextual answer
VLAText, images, and robot stateRobot-control actions

The resemblance makes pretrained multimodal models useful starting points. It does not make training and deploying a robot policy equivalent to serving a chatbot.

For a downstream VLM application, the familiar pipeline starts with web data, perhaps supplemented by task-specific examples. An existing model can be fine-tuned on a cloud cluster and deployed using established inference libraries. Robotics introduces gaps at each of those stages.

  • Data: There is no established equivalent of the web as a source of training examples for frontier robot dexterity. Springenberg treats finding that source as a major open research question.
  • Architecture: A pretrained VLM backbone needs adaptation to generate actions at the high frequencies required for robot control.
  • Deployment: At the time of the talk, the team sees no standard solution for deploying large robot policies across multiple locations, on premises, and on devices.

The presentation develops the data and model problems; it leaves detailed deployment engineering aside.

2:242:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:24 · section reference included

A data engine built around human demonstrations

Physical Intelligence builds its collection pipeline from the ground up, aiming for useful capabilities immediately while leaving room to scale over subsequent years. Operating the data pipeline is a central part of the research effort. Springenberg estimates that getting the pipeline, data selection, and data quality right accounts for more than 50% of the work.

The collection process follows a concrete sequence:

  1. Choose tasks. An expanding set of tasks, including folding clothes and bagging groceries, tests what the robots can currently learn.
  2. Demonstrate with teleoperation. Human operators use a custom runtime and move leader arms attached to their own arms. Software transfers those motions to the robot’s end effectors, allowing people to demonstrate intricate manipulation.
  3. Schedule and track episodes. Collection runs around the clock. Each dash in the operations dashboard represents an operator performing an episode for a particular task.
  4. Annotate and filter. Demonstrations are stored in cloud buckets, annotated, and selected for training using those annotations.

This makes task selection, robot operation, collection throughput, and training-data quality parts of one connected system.

“From 0 to 1: Data Engine from the Ground Up” slide with task choices, a teleoperation image, an Operations Dashboard overlay, and dashed arrows leading to cloud storage and annotation.Open full source frame
The data engine connects tasks, teleoperation, operations tracking, and an annotation pipeline.

Training on the demonstrations produces policies that perform the tasks autonomously. The example here is π0, the model released late in the previous year, folding a shirt. The transition is from a human tracing dexterous movements through the teleoperation system to a learned policy controlling those movements itself.

5:095:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:09 · section reference included

More hours, then more kinds of environments

The presentation describes Open X-Embodiment as the largest public robot dataset available when the team started: about 3,800 hours, largely from static scenes in laboratories around the world. That historical estimate provides the baseline for the team’s own collection effort.

After six months, the team reports about 10,000 hours of successful episodes across tens of environments and hundreds of tasks. That collection supported capabilities such as the shirt-folding policy. These are the talk’s operational collection figures, whose scope includes the successful-episode accounting and collection interval.

Another six months brought more static-scene data and significant collection using mobile manipulation setups. The data now covered hundreds of scenes and environments, with a much wider range of tasks. No new total in hours is supplied. Mobility expands where a robot can act and the situations it encounters, so the collection effort grows in diversity as well as volume. The accompanying autonomous policy illustrates the resulting capability gains, while the presentation emphasizes that the larger, more varied pipeline also creates additional engineering challenges.

7:407:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:40 · section reference included

From early object selection to dexterous control

The model trajectory follows a pattern familiar from multimodal language models, with a delay. Over the preceding three years, VLMs progressed from conversational agents toward reinforcement-learning-trained multimodal reasoning models and coding assistants. VLAs began developing after language models had already acquired vision encoders.

RT-2 emerged in 2023 as an early example. The presentation credits colleagues now at Physical Intelligence, including Danny, with some of the early multimodal language-model work for robotics. These first VLAs demonstrated useful generalization: a robot could be asked to pick different objects in a scene resembling its training environments. Limited robot data still held them back, but the demonstrations helped generate wider interest in the field.

Dexterous models capable of controlling multiple robots began appearing in mid-to-late 2024. π0 was Physical Intelligence’s entry; the presentation describes it as open source and, in the team’s assessment, exceptionally dexterous. The broader category also includes Gemini Robotics and NVIDIA’s GR00T family. Those examples extend beyond the initial 2024 wave: Gemini Robotics and GR00T N1 were announced in March 2025, and the talk does not specify a GR00T version.

The architectural change is consequential: these models use diffusion-based action generation to support the fast output needed for physical control. A robot needs more than an appropriate description of what should happen next; it needs a stream of actions that can execute dexterous behavior. That leads to the next question: what happens when this control architecture is trained on substantially more diverse environments?

9:049:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:04 · section reference included

π0.5 couples task decomposition with continuous actions

π0.5 was developed to investigate how capabilities change as collection becomes more diverse, with the goal of generalizing to unfamiliar environments. Its training mixture combines static and mobile robot data with web data, object-detection data, and language annotations of robot demonstrations. A large annotation pipeline helps connect physical demonstrations to language and scene understanding.

The architecture starts with a pretrained transformer and adds an action-expert transformer. The large VLM backbone learns both to answer general questions about a scene and to break down a high-level request. For example, a request to clean a bedroom can become a subtask to pick up a pillow.

The action expert attends to the VLM’s internal representations, runs at a higher rate, and produces continuous robot actions using a flow-matching objective. The two components therefore connect different levels of the same task: the backbone selects a meaningful subtask, while the action expert generates the physical control needed to carry it out. The pillow subtask is a prediction to execute, not evidence that the pillow has already been moved.

Architecture diagram with multimodal data on the left, a pretrained VLA and action expert in the center, robot action data on the right, and subtask prediction and continuous actions above.Open full source frame
π0.5 architecture combines a pretrained VLA with an action expert and multiple data sources.
11:1811:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:18 · section reference included

Testing what transfers to an unfamiliar home

The presentation reports difficult tasks lasting up to ten minutes per episode in entirely unseen homes. In the first demonstration, colleague Chelsea prompts the model to perform cleaning tasks, including cleaning a kitchen surface, in a home absent from training. The result is presented as an emerging sign of broad generalization. The duration describes the demonstrated capability, rather than a measured ten-minute reliability rate.

To separate the effect of environmental diversity from simply providing more data, the team trains π0.5 with a fixed amount of data while varying how many homes contribute to it, then evaluates in a held-out location. The presentation reports that held-out performance generally improves as training locations increase, with the yellow curve matching and slightly surpassing the comparison trained with that scene included.

The π0.5 paper gives the more precise experimental conditions: the study varies training locations from 3 to 104 and uses 40,000 post-training steps to control the number of samples seen. Evaluation measures task progress on dishes, drawer packing, laundry, and bed-making in held-out mock homes. The paper describes the 104-location model as performing similarly to the control trained in the test home. The supported result is that broader environment coverage can close the gap with location-specific training; superiority over that control is not established.

The next demonstration makes the task hierarchy visible. In a new home, colleague Laura gives the high-level instruction to clean the bedroom. The model divides that request into activities such as putting trash in a bin and making the bed. The on-screen timer shows the robot operating autonomously for multiple minutes. Here, task decomposition supports a sustained sequence of physical actions from a single broad request.

12:4713:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:47 · section reference included

From unfamiliar homes to a partner’s robot

Quan returns with a different transfer problem: a robot located far from the team’s office makes a cup of coffee end to end. He says the Physical Intelligence team has never physically accessed this robot. He also reports that producing the video required few iterations, without giving a success rate.

Hardware and deployment remain important, but Quan identifies software and model intelligence as major bottlenecks. His scaling hypothesis is that a model should work across hardware platforms without requiring substantial effort from the model team on each one. The possibility of deploying thousands or millions of robots is a conditional illustration of why that matters, not a deployment count or forecast. The coffee-making robot offers one piece of evidence: the team says its model controls the task despite its lack of physical access or detailed knowledge of the robot’s internals.

The partnership is technically direct. Physical Intelligence sends the partner the actual model checkpoint so it can run inference, and the teams engage in low-level technical discussions. That demonstrates a practical collaboration model; it does not establish that the robot required no adaptation data or integration work. Quan invites introductions to other companies, in person or by email, because the larger problem is still unsolved.

The closing bottleneck is people. A model that can operate any robot on any task remains an open scientific, engineering, and operational problem. Quan identifies recruiting outstanding people as the organization’s biggest current constraint and invites candidates across the roles a research organization needs—including people who can make a case for a role not yet listed. The invitation extends to conversations in person, online applications, and Twitter messages: advancing the models also requires building the team that can collect the data, operate the systems, and solve the remaining research problems.

15:0315:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:03 · section reference included

Resources

From the talk

  • Physical Intelligence's original model introduction, including laundry demonstrations and its approach to continuous robot control.

  • Architecture, training mixture, post-training procedure and evaluations for π0.

  • An illustrated explanation of co-training and robot cleaning tasks in unfamiliar homes.

  • The original paper details the architecture, training recipe and controlled environment-diversity experiments.

  • RT-2Article9:40

    The original project page for transferring web-trained vision-language knowledge into robot actions.

  • A collection of robot demonstration datasets with standardized formats and accompanying RT-X research.

  • Google DeepMind's introduction to Gemini Robotics and Gemini Robotics-ER.

  • NVIDIA's announcement of the N1 model within the GR00T family, explaining its reasoning and action components.

Updates since the talk

Transcript download
Full transcript

Loading transcript…