AI Engineer Code 2025
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
Loading slides…
Read the talk
Agents are Robots Too: What Self-Driving Taught Me About Building Agents
A coding agent needs more than a capable model: it needs an observable environment, feedback from its actions, and a development loop that learns from failures.
From a talk by Jesse Hu
Before you start: Familiarity with tool-calling agents, shell commands, and basic model training will help; robotics concepts are introduced as they arise.
What surrounds the model?
What does it take to turn a capable model into an agent that works in the real world? Jesse Hu approaches that question through his experience in machine learning and robotics. He describes work at YouTube and Google on two-tower embeddings, early BERT, and mixture-of-experts models; at Waymo on data, reward modeling, and evaluation; and at Abundant on datasets for foundation-model labs training coding agents. The connection here is general robotics practice, rather than internal Waymo systems.
Hu frames the engineering burden as 1% model and 99% everything else. These are illustrative proportions, not a measured allocation. In robotics, everything else includes hardware, sensors, actuators, integration, deployment, simulation, and training. A strong model leaves most of the system still to build.
A digital agent has a body too. APIs and MCP tools give it ways to act; terminals, browsers, and virtual machines provide richer environments. An operating system and persistent filesystem expand that body further. The relevant comparison is not simply between a language model and a robot’s brain, but between two complete systems that sense and change their surroundings.
Both systems also need an offline stack: simulation and training, continuous retraining, monitoring, and human feedback loops. Hu recalls a recurring observation in self-driving: the strongest team may win through its offline tooling as much as through its model or runtime. Better development infrastructure lets engineers investigate behavior, iterate faster, and ship more reliably.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A command is not its outcome
Turn a steering wheel left. How far did the car actually turn? The command alone cannot answer that question: actuation is imperfect, so the controller must measure the result and adjust. That measurement closes the loop between intended motion and actual motion. Closed-loop control uses the consequences of an action to inform the next action.
The coding-agent equivalent is a long-running Bash command. If the agent cannot observe output while the process runs, determine whether it has completed, or stop it early, it has lost essential feedback and control. Starting the process is only the first step. The interface must also support observing and intervening in its execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Conversation turns choose when the agent can react
Robotics makes input design explicit. A system might observe through vision, lidar, radar, or a combination. It must also choose when to observe: periodically, when an event arrives, or at a faster fixed cadence. Hu gives 50 Hz—50 observations per second—as one example of periodic world-state sampling. Repeated observations let the controller update its state estimate and replan as the world changes.
Agents often inherit a different clock from conversation: take a turn, execute a tool, wait for the entire response, then take another turn. That is a design choice even when no one explicitly chose it.
| Interaction pattern | What it makes easier | What it constrains |
|---|---|---|
| Complete conversation turns | Reasoning about each input and output | Reaction while a tool is still running |
| Repeated world observations | Updating plans as conditions change | Requires handling ongoing changes |
Turns provide clear boundaries, but a boundary can become a delay. An agent waiting for a tool response may be unable to react immediately to a pop-up or intervene in a long-running process. The important question is when the environment can tell the agent that its plan needs to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose what an action means
Handcrafted tools and streams from tools or users are only some of the available interfaces. Terminal-Bench’s Terminus uses an interactive tmux session. Hu highlights character-level interaction, including Control-C and window commands: the agent can manipulate a terminal session instead of treating every command as an indivisible request and response. The interface provides those capabilities; observation frequency still depends on how the agent uses it.
For example, a Bash controller can start a process in a detached terminal, inspect its output, and later send an interrupt:
bash
# Start an illustrative long-running process.
tmux new-session -d -s agent-job \
'bash -c "while true; do date; sleep 1; done"'
# Observe the current terminal contents.
tmux capture-pane -p -t agent-job:0.0
# When an observation or user request calls for stopping it:
tmux send-keys -t agent-job:0.0 C-c
The key separation is between launching, observing, and interrupting. A controller can decide what to do between these operations instead of waiting for the original process to return.
Robotics offers the same freedom at different levels. An action might mean moving up one block and across two, or specifying a point in continuous space. It might operate in two or three dimensions, and control position, velocity, or acceleration. Each representation makes some behavior easy to express and other behavior awkward. For agents, the analogous choice includes structured MCP calls, terminal characters, or lower-level mouse and keyboard actions.
Hu describes a Dreamer example with mouse and keyboard interaction at 20 frames per second. The exact paper is not identified in the talk; the related Dreamer 4 paper uses that cadence in a Minecraft setting, not a general desktop-automation demonstration. The design question remains useful: which capabilities does an action interface expose, and which does it prevent the agent from expressing?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The world does not reset when a session ends
A driving game can spawn a car at the beginning of a session and discard it at the end. A real car occupies space before and after the task. How it arrived, where it will stop, its speed, and the motion of nearby vehicles all matter. The task exists inside a continuing world.
Coding agents are moving through a similar transition. A disposable session can produce an artifact and disappear. A persistent VM retains running processes and files. An agent joining a workplace also enters ongoing activity, such as existing Slack conversations. It needs to account for the environment it inherits, not just the instruction that started its session. That changes runtime behavior and the starting conditions needed for meaningful evaluation and simulation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small mistakes change the next input
Imitation learning trains on demonstrations, much as supervised fine-tuning trains a coding model on human examples. Reinforcement learning offers another approach, with simulation as one possible training environment. Hu introduces DAgger as relevant background to the distribution problem in imitation learning; it belongs to imitation learning rather than being another name for RL.
The difficulty is that the agent helps create its own future inputs. Once its behavior departs from the demonstrated path, it can encounter states unlike those in its training examples. A browser pop-up makes this concrete: a human handles it naturally, but an agent that has not learned that situation may become confused. Its next mistake can take it farther from familiar states, producing a cascade rather than an isolated prediction error.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Simulate the consequences, not just the prediction
An action changes the problem the agent must solve next. A classifier produces a prediction; an agent predicts, acts, encounters the consequences, and reevaluates its earlier decisions. Those consequences unfold in a messy world, where the initial plan may quickly stop matching reality.
Simulation provides a way to represent that messiness in a starting state and then explore what follows. Instead of evaluating only the path already observed, a simulator can play out alternative paths as the agent’s behavior changes. These are counterfactual rollouts: different possible continuations from a situation. Hu’s testing pyramid places simulation beneath test-track testing and production, making it part of the infrastructure for exploring consequential behavior.
A Markov decision process, or MDP, supplies vocabulary for this loop: an agent receives state and reward information, chooses an action, and affects an environment that produces the next situation. The value of the formalism is practical communication. It gives a team precise objects to discuss when deciding what the agent observes, what it controls, and how its behavior is evaluated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understanding the scene does not finish the task
Hu characterizes self-driving work from 2017 to 2020 as heavily focused on perception: identify objects, put boxes around them, and assume driving around those boxes will be relatively straightforward. That assumption concealed the complexity of choosing and executing actions. Recognizing the scene was not enough to drive through it.
Language agents face a comparable gap. A model may interpret a detailed description and generate a long, sophisticated reasoning trace, yet fail when its plan becomes a chain of real tool calls. A call fails, progress stalls, or the agent cannot correct its own mistake. Hu places much of the remaining engineering work in that execution-and-recovery loop.
Cars and code nevertheless offer unusually favorable starting points. Hu contrasts self-driving’s production use in limited settings with the demo-stage maturity of much other robotics. Cars already have refined human controls, electronic actuation, and built-in telemetry. They provide both an interface for taking actions and an interface for collecting evidence about what happened.
Coding has a similar advantage: predefined actions and observable results make software convenient to operate and learn from. Knowledge work that requires the full desktop is often harder to codify. When selecting a new domain, ask two connected questions: does an existing human interface make actions accessible, and does it expose enough information to observe their consequences?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use deployed failures to guide the next experiment
Improving an agent is a hill-climbing process. In a conventional application, implementing a feature usually gives a fairly direct route to delivering that feature. With an agent, an engineering change is often a hypothesis about improving a metric. Run the experiment and the score may rise—or fall. Progress comes from repeated experiments, not from assuming each implementation adds capability.
The self-driving development cycle connects those experiments to the world:
- Learn: improve the model or system using available evidence.
- Simulate: examine behavior before deployment, using simulation to support both learning and deployment confidence.
- Deploy: expose the system to real conditions.
- Collect logs: capture what happened in those conditions.
- Feed simulation: use those logs to ground the next round of simulated scenarios and experiments.
The return path matters. Without real-world evidence feeding back into it, simulation can drift away from the situations the deployed agent actually encounters.
Hu uses a hypothetical 70% benchmark score to show the limits of an aggregate number. A score gives a broad signal, but categories, cities, and failure modes reveal where behavior breaks down. Inspecting individual failures can tell the team what to change and where another experiment is likely to help. Hu describes that triage process as central to Abundant’s tooling and its work helping customers improve their systems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From impressive demonstrations to completed work
Hu’s closing slide displays a 2.5% automation rate on real work, labeled RLI. It invokes the Remote Labor Index and compares the state of agents with self-driving in 2015. Read the figure as a benchmark snapshot of end-to-end work completion, not a current estimate of economy-wide automation. Strong predictive models and impressive demonstrations still leave a substantial gap before reliable completion: actions have consequences, and real environments introduce complications that a plan must survive.
Closing that gap ties together feedback, observation timing, action interfaces, persistent state, simulation, and the infrastructure used to improve the system. Hu proposes agentics as a name for a practice built on these concepts—a field with reusable abstractions and scientific methods, rather than a succession of ad hoc agent implementations.
The reading path follows the problems an agent builder already encounters: open-loop and closed-loop control for feedback; MDPs and fully versus partially observable environments for describing the world; DAgger for imitation-learning distribution shift; and offline RL, introductory reinforcement learning, and recent robotics literature for learning from experience. These ideas become easier to grasp when attached to a stuck process, an unexpected pop-up, or a persistent environment. Agents act in the world, make mistakes, and have to recover. Building that recovery is part of building the agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains the tmux-based terminal interface, keystroke actions, external execution process and sandbox assumptions.
The original paper on collecting expert labels for states encountered by a learned policy and aggregating them across training iterations.
Further reading
The October 2025 launch report evaluates complete freelance deliverables across 240 projects and explains common failure categories.
Studies offline Minecraft learning from pixels and mouse/keyboard actions, with reinforcement learning performed inside a learned world model.
Shows Minecraft agent evaluations, imagined training trajectories and human interaction with the learned world model.
Full transcript
Loading transcript…