AI Engineer World's Fair 2025

Challenges in High Performance Robotics Systems

Rishabh Garg· Robotics Engineer, Tesla Optimus12:43

Loading slides…

Read the talk

Challenges in High Performance Robotics Systems

A simple CAN-connected robot reveals how bus delays, stale sensor inputs, logging and scheduling can turn correct policy outputs into erratic motion.

From a talk by Rishabh Garg

Before you start: Basic familiarity with control loops, threads and sensor-to-actuator data flow will help; no CAN experience is required.

Between the controller and the wire

A robot’s motor does not move. Did the policy fail to produce a command, or did the software fail to deliver it? A controller depends on two journeys: sensor data must reach the policy, and the resulting commands must reach the actuators. A failure anywhere along those paths can look like a bad policy.

Rishabh Garg builds a small toy robot around this diagnostic problem. The question is not simply whether the policy computes the right answer, but whether the surrounding system supplies the right input and delivers the answer when the motor needs it. That distinction becomes increasingly important as mechanical, electrical and software components interact.

Slide listing robot complexity, coordination of mechanical, electrical and software systems, and critical data flow, followed by “How to Root Cause unexplained behavior?”Open full source frame
Why this talk? Finding the root cause of unexplained robot behavior.
0:150:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

The missing millisecond

The toy architecture contains actuators, a CPU, an optional hardware accelerator and a sensor. Garg chooses CAN for communication because it is affordable, compatible with many components and fast enough for the example. His description of CAN as open source is best understood as enthusiasm for its accessible ecosystem; the protocol and particular open-source implementations are distinct.

The initial control loop is sequential: receive data, run the policy, transmit its output. With approximately 2 ms allocated to policy execution, it is tempting to expect a new output every 2 ms. Deployment introduces an extra gap, however. The receive and transmit operations at the edges of the loop also consume time.

In the toy budget, ten 100-bit messages at 1 Mbit/s require 1 ms of serialization. Five messages go out and five come back:

Totalbits=100bits/message×10messages=1,000bitsTimepermessage=100/1,000,000seconds=0.1msTotalbustime=1,000/1,000,000seconds=1msTotal bits = 100 bits/message × 10 messages = 1,000 bits Time per message = 100 / 1,000,000 seconds = 0.1 ms Total bus time = 1,000 / 1,000,000 seconds = 1 ms

Communication is now on the same order as computation, explaining the additional millisecond. This is a simplified budget: actual CAN frame structure, stuffing, arbitration, intermission and retries affect bus occupancy. The 100-bit assumption is not a universal frame size.

1:231:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:23 · section reference included

Overlap communication with computation

Accepting the communication delay gives an approximately 3 ms sequential loop; the proposed pipeline instead targets a 2 ms iteration cadence. The bus work does not disappear. The design changes when that work happens, separating transmission (TX), reception (RX) and policy execution, with communication running on a different thread from the policy.

The staggered schedule proceeds as follows:

  1. Receive initial sensor data to seed the first policy iteration.
  2. While that policy executes, begin receiving the data intended for the next iteration.
  3. At the next iteration, transmit the previous policy’s output while the next policy computation runs.

Pipelining recovers cadence by overlapping work. It does not establish a 2 ms end-to-end sensor-to-actuator latency, and the proposed schedule still depends on tasks completing at the intended boundaries.

New Design timeline with three consecutive Policy blocks, yellow RX blocks and blue TX blocks arranged around their boundaries, above a milliseconds axis.Open full source frame
The new design overlaps RX and TX with successive policy executions.
3:203:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:20 · section reference included

Watch what actually reaches the bus

After deployment, the actuators stutter, make catching-up sounds or move strangely. Again, the policy looks suspicious. Before changing it, inspect the commands that actually reach the wire. Attach an external CAN transceiver to the bus and connect it to a separate host, such as a laptop. This gives a view of traffic outside the robot’s control process.

On that host, candump from can-utils can display timestamped CAN traffic. For an already configured interface named can0, a basic capture is:

bash

candump can0

The measurement’s timestamp source matters. The current candump options include -H to request hardware rather than system timestamps; Garg does not specify the adapter or timestamp mode used in his example.

Plot the outgoing command timestamps. The expected spacing is 2 ms. In the illustrated failure, the gap from message two to message three grows to 4 ms, then message four follows message three with almost no gap. Message three is late; message four is on time. The actuator experiences a pause followed by closely spaced commands, producing the apparent catch-up motion. Messages seven and eight repeat the pattern. CAN frames still serialize: almost no gap here means small relative to the intended cadence, not simultaneous transmission.

A cycle-time plot makes this pattern easier to see. For each message, subtract the preceding message’s arrival time:

Cycletime[n]=arrivaltime[n]−arrivaltime[n−1]Cycle time[n] = arrival time[n] − arrival time[n − 1]

Healthy spacing produces a line around 2 ms. A delayed message produces a rise to 4 ms, followed by a near-zero interval when the next on-time message arrives. The pair reveals a missed interval followed by a burst, rather than a uniformly slower stream. With the symptom characterized, the next step is to find which task missed its timing boundary.

4:324:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:32 · section reference included

A late policy result becomes a burst

Policy execution time can vary. If an iteration runs long, its output misses the scheduled transmission opportunity. In Garg’s illustrated implementation, that result is queued. At the next opportunity, the system sends both the queued result and the current result, placing two commands on the bus in quick succession.

Desynchronization between communication threads can produce the same symptom. Synchronization is therefore part of the control system’s behavior, not merely an implementation detail. Fixing the transmit schedule improves the motion—but does not necessarily fix everything. Even a clean outgoing timing plot says nothing about whether each command was computed from fresh sensor data.

6:577:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:57 · section reference included

Regular commands can still use stale inputs

Now delay the receive thread instead. The next policy iteration starts before its new sensor data is available, so it reuses the previous input. In iteration two, the command is therefore based on older data. Iteration three then jumps ahead, skipping an intermediate data-processing step. The motor can again appear to catch up or jitter, even though outgoing messages arrive regularly.

The system must preserve the pairing between received data and the policy iteration that consumes it. Garg proposes two approaches:

  • Synchronization primitives: Use condition variables or semaphores to coordinate task handoffs.
  • Timing padding: Where the particular target lacks the needed primitives, leave a cushion so reception can finish before the intended policy iteration consumes its input.

The second option is conditional on the environment, not a general limitation of real-time operating systems. Garg specifies no padding duration; the purpose is to keep the intended RX data, computation and timely output together.

8:008:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:00 · section reference included

Logging becomes part of the timing problem

Logging received data and policy outputs seems harmless until accumulated logs must be written to storage. If the main control loop performs that write, it can stop controlling the robot while storage catches up. Garg reports a 30 ms robot freeze on a Raspberry Pi using an SD card when logging wrote to storage. The specific board, card and workload are not given.

His remedy is to assign asynchronous logging to another CPU core, described in the architecture as a third CPU. The design progressively separates work: policy execution, communication and now logging each have different timing needs. Keeping storage work away from the control path protects the deadline the robot is trying to meet.

A microcontroller can suffer a related failure without a disk or filesystem. Its logs may go directly to a peripheral such as UART, where transmission can take milliseconds depending on the amount logged. A diagnostic message can then create the very error it reports:

  1. A packet is dropped.
  2. The system logs the dropped packet.
  3. Sending that log takes enough time to miss the next packet.
  4. The next drop triggers another log, repeating the cycle.

The result can be a complete CAN data blackout while diagnostic logs continue. That combination is especially misleading: the system is visibly reporting errors, but reporting them is sustaining the failure. Observability work must fit within the system’s timing budget too.

Logging slide describing disk synchronization delays, up to 30 milliseconds on a Raspberry Pi with an SD card, and asynchronous logging on another CPU core. A second panel shows a Message Drop and Log feedback cycle.Open full source frame
Logging delays and the case for asynchronous logging on a different CPU core.
9:269:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:26 · section reference included

Let the entire receive path run

The final failure lies below the application. Received data passes through interrupt handling and kernel work before reaching a user process. Raising the robot application’s priority too aggressively can starve schedulable receive-path work that the application itself needs. The process urgently waiting for data prevents the work that would deliver it.

Garg calls this priority inversion and describes receive dropouts approaching seconds at a time. More precisely, the example concerns starvation of required receive-path work; it does not mean a user process can block every kind of kernel or interrupt execution. The remedy is to identify the participants in the complete pipeline and assign priorities that let all necessary work progress.

That brings the toy robot back to its original diagnostic question. Hardware, profiling and software scheduling must be considered together: overlap communication to reduce cycle time, synchronize handoffs to prevent jitter, keep logging from blocking control, and preserve the priorities needed to deliver data. These are foundational design concerns, but they determine whether a carefully constructed policy can actually control a physical robot.

Recap slide listing pipelining for lower cycle time, synchronization for jitter, logging strategies, and priority to avoid starvation.Open full source frame
Recap: pipelining, synchronization, logging strategies and priority.
11:1011:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:10 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…