# Computer-Use 2.0: Agents Just Got Multi-Cursor

Francesco Bonacci · Dillon DuPont · Robert Wendt

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/ZSQb5fzRFPw-computer-use-2-0-agents-just-got) · [Video](https://www.youtube.com/watch?v=ZSQb5fzRFPw)

## Computer-Use 2.0: Agents Just Got Multi-Cursor

Running agents beside a human requires more than another cursor: window-level control, trustworthy task evaluation, and infrastructure that keeps GPUs from waiting on desktops.

Familiarity with agent tool calls, basic Python, and the role of GPUs in model training will help; no prior Cua experience is required.

### When the agent needs your desktop

[1:22](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=82s)

An agent takes a screenshot, reasons about what it sees, and clicks, types, or scrolls. That familiar loop creates a practical problem: the agent operates the same desktop the human wants to use. Drawing on earlier GUI-agent work at Microsoft, Francesco Bonacci calls this screenshot–reason–act pattern **computer use 1.0**. The opening slide makes it concrete with a PDF export action: observe the interface, identify the next action, and select a click target.

![Slide titled “1.0 = the Human Loop” with screenshot, reason, and click-target panels illustrating a PDF export action.](https://ai.engineer/api/talk-essay-media/394e5c378a0a94bac6f8e28bad5e7ff63f712a562346db623283f2fa031e7984)

The human loop: screenshot, reasoning, and a click target.

### Cua Driver runs beside the user

[2:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=154s)

[Cua Driver](https://github.com/trycua/cua) moves computer interaction into the background so the agent can work without continually taking over the user's screen. Francesco dates its open-source release to roughly two months before the talk. He connects the initial effort to Codex's computer-use release; the corresponding [OpenAI announcement](https://openai.com/index/codex-for-almost-everything/) introduced background computer use in the Codex app, rather than a separately named model. The team put its first version together over a weekend.

On macOS, the approach relies on undocumented APIs in Apple's shipped frameworks. The calendar-and-terminal demonstration illustrates the intended result: an agent interacting with an application while the person retains control of the laptop. The slide names this the SkyLight recipe. Francesco describes Driver support across macOS, Windows, and Linux, exposing both **accessibility trees** and screenshots as ways for an agent to observe the operating system.

![Slide titled “À la SkyLight Recipe” showing a calendar and terminal on a macOS desktop beside four background-control technique cards.](https://ai.engineer/api/talk-essay-media/4b7b470015deeafea2219129b5cb0b9313ba194c413041c4c984e3ccb41df9d5)

The SkyLight recipe for background control on macOS.

After installation, the agent observes a particular window and chooses an action route:

1. Call `getWindowState` to obtain an accessibility-tree representation and a screenshot.
2. Attempt the action through accessibility-based background execution.
3. If that fails, attempt a pixel-based background click.

The two observations serve different purposes: the accessibility tree supplies a structured representation of the interface, while the screenshot supports actions located visually. The driver handles some of the operating-system differences, but background behavior is not identical across platforms. The [speakers' companion article](https://cua.ai/blog/computer-use-2-ai-engineer-worlds-fair) also describes a final fallback that briefly raises the window and then restores focus, so background operation is a preferred route rather than a guarantee for every action.

Keeping those routes working across releases requires application-level regression tests. Francesco reports approximately eight application harnesses used to check for breakage. He also credits early adopters with upstream contributions: Clicky, Hermes, Qwen Code, H Company, and Factory's Droid.

### Give each GUI task an initial state and a success check

[6:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=394s)

Once an agent has a way to act, the next question is whether it uses that capability correctly and leaves the environment intact. Dillon DuPont introduces [Cua-Bench](https://cua.ai/cuabench) through a comparison with Terminal-Bench and Harbor: a task needs a reproducible starting point, a known solution, and a way to check the result.

| Task component | Responsibility |
| --- | --- |
| `setup` | Put the machine into its initial state. |
| `oracle` | Supply a golden trajectory that solves the task. |
| `evaluator` | Probe the environment to check completion. |

The oracle consists of GUI actions, written in a style Dillon compares to PyAutoGUI. The evaluator answers a different question from the oracle: it checks whether the environment satisfies the goal, rather than merely supplying the actions expected to reach it.

A small Python runner makes that separation explicit. Each callback closes over the GUI environment; the same setup and evaluator can check either the oracle or an agent attempt:

```python
from collections.abc import Callable
from dataclasses import dataclass

@dataclass
class Task:
    setup: Callable[[], None]
    oracle: Callable[[], None]
    evaluator: Callable[[], bool]

    def check(self, attempt: Callable[[], None]) -> bool:
        self.setup()
        attempt()
        return self.evaluator()

    def check_oracle(self) -> bool:
        return self.check(self.oracle)
```

This expresses the three-part task structure without requiring the evaluator to depend on a particular action sequence.

The authoring problem is portability. Dillon describes targeting more than five desktop platforms and collapsing GUI construction into a single Python file. The Cua-Bench SDK defines the GUI and probes it for usable agent data; either a person or an agent can author a task. At the time of the talk, Dillon reports over 130 verifiable tasks in 42 environments across five platforms, reproducible through the CLI.

### Editing a circuit is easier than starting with a blank schematic

[8:13](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=493s)

Cua-Bench KiCad, built with Snorkel AI, moves evaluation into professional electrical-engineering software. Here, completing a sequence of clicks is insufficient: the resulting circuit needs a meaningful check. Dillon describes evaluators that simulate circuits. The companion article instead describes exporting netlists and comparing components and connections with reference designs; those are distinct checks, and the supplied descriptions do not establish which tasks use simulation.

Dillon reports that the top tested agent fully passed six of 25 KiCad tasks, all six involving edits to existing schematics. He reports 0% success on tasks starting from a blank schematic. The [June evaluation](https://cua.ai/blog/evaluating-gemini-3.5-flash-on-computer-use) identifies **GPT-5.5** as the six-solve model and describes design-from-scratch timeouts within a 200-step budget. The blank-schematic result therefore concerns completion within that budget, rather than an unrestricted claim about whether a model can design circuits.

Across the tested models, Dillon reports that none exceeded 30% reward. **Full passes and mean reward are different measurements**: partial progress can raise reward without producing another complete solve. In the dated evaluation, **Gemini 3.5 Flash** leads mean reward while GPT-5.5 leads full solves. The seven models share a 200-step budget, although the **Gemini 3.1 Pro** and **Gemini 3 Flash** baselines use function calling rather than native computer use.

### The computer tool changes the result

[9:04](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=544s)

Evaluation can also reveal improvements that come from the interface between model and environment. On Cua-Bench basic scaled to 4K resolution, Dillon reports that replacing the built-in computer tool with Cua Driver raised pass rate from approximately 62% to 80% while using 34% fewer tokens. He attributes the gain primarily to focusing on a window instead of the entire desktop. The talk does not identify the model or provide the task count and token-accounting details for this comparison.

### Try to break the evaluator before trusting its score

[9:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=574s)

A score is useful only if the task's success check deserves trust. Before admitting a task to the dataset, Cua runs a matrix of agents against it, asking them to attempt **reward hacking** and break the environment. Their findings become a CodeRabbit-style code review. The review screenshot beside the question about trusting an evaluator shows findings and supporting evidence—the output of testing the test itself.

![Slide reading “Our eval says: trust the model” and asking whether the eval can be trusted, beside a review screenshot with findings and evidence.](https://ai.engineer/api/talk-essay-media/9396f7811d3be2f7341007f85b3c68c1196ea7b63ee334e15fb16fbed6e1c882)

Can the evaluator itself be trusted?

Only tasks that survive that pipeline enter the dataset. Dillon calls this “evals all the way down”: an agent's performance depends on an evaluator, and confidence in that evaluator depends on another layer of adversarial testing.

Successful actions are only one part of intelligence. To test what an agent understands about its environment, Dillon describes forking recorded trajectories and asking a model to predict the reward, internal state, or another observation, then comparing that prediction with the recovered state. The companion makes the reconstruction scope more concrete: captured action and file state at recorded steps, rather than arbitrary restoration of every part of a running machine. This turns a **world-model prediction** into something that can be checked against a recorded environment.

### A desktop reset can leave an expensive GPU idle

[10:50](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=650s)

Robert Wendt shifts from measuring agents to keeping their reinforcement-learning infrastructure busy. In the training diagram, the blue portions represent GPUs generating tokens. Once a task finishes in a sandbox, the worker may wait for a fresh sandbox to start or for the existing one to reset. During that interval, the GPU is available but produces no useful work. The worker timelines make the interruption visible as alternating generation and idle periods.

![Worker-utilization chart with six horizontal timelines, dark-blue generation segments, light-blue idle segments, and a time arrow.](https://ai.engineer/api/talk-essay-media/075d2097ba0436ec0030d01d3c28241d4059fad10fb6d0f2c6311ba8d101aad8)

Worker timelines alternate between generation and idle waiting.

As training scales, those waits compound. Reducing startup time helps, but some desktop environments are inherently substantial: Robert gives the example of a necessary 40-gigabyte research environment that takes time to download and start. The infrastructure problem is therefore how to keep environment preparation from blocking GPU work even when the environment itself cannot start quickly.

### Prepare sandboxes before workers need them

[13:00](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=780s)

The solution is a **warm sandbox pool**. The intended animation does not play, so Robert explains the mechanism verbally: an autoscaler observes how many GPUs currently need sandboxes and grows the pool in response. The operator does not have to choose a fixed warm-pool size upfront.

Demand can change during a multi-day training run. If generations become longer, workers may need new sandboxes less often, reducing the required pool size. Robert says sandbox resources can be two to four times cheaper than GPU resources, making some redundant sandbox capacity economical. The trade is to spend on cheaper readiness so that expensive generation workers spend less time waiting.

With spare environments ready, startup happens on the sandbox infrastructure while GPU workers continue using available environments. Robert describes the result as full GPU-worker utilization and instant sandbox availability; these are workload-dependent claims, with immediate allocation depending on available warm capacity, rather than measured utilization results presented in the talk.

The historical platform list here is Windows, Linux, and Android, with macOS forthcoming. In the transition to questions, Francesco names this infrastructure offering **Cua Fleet**. Its role is separate from Driver's local interaction layer: Fleet supplies environments to workers, while Driver determines how an agent observes and acts within an environment.

### On mobile, background work may mean tool use

[15:35](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=935s)

The closing audience question asks whether background computer use can extend from the Mac to mobile. Francesco points to Android and discussions with the Hermes team about its Android harness. One possible route is to isolate the workload in a container, running an Ubuntu-like environment or a GUI Docker container within Android.

He describes Android as more accommodating than iOS to this form of background operation, but draws a boundary around the capability: it is closer to **background tool use** than general control of a mobile GUI. The team's approach uses the activity framework to perform that background work. Extending the idea to phones therefore changes the execution mechanism; it does not simply reproduce desktop cursor control on a smaller screen.

## Resources

- [Computer-Use 2.0](https://cua.ai/blog/computer-use-2-ai-engineer-worlds-fair) — The speakers explain window-level control, benchmark evaluation and warm sandbox pools in their companion article.
- [Cua source repository](https://github.com/trycua/cua) — Source code for Cua's computer-use drivers, sandbox infrastructure and evaluation tools.
- [Install Cua Driver](https://github.com/trycua/cua/blob/main/docs/content/docs/how-to-guides/driver/install.mdx) — Current installation instructions, operating-system requirements and permission setup.
- [Cua-Bench](https://cua.ai/cuabench) — Desktop and mobile evaluation tasks, with links to the registry and current leaderboard.
- [Cua's June 2026 KiCad evaluation](https://cua.ai/blog/evaluating-gemini-3.5-flash-on-computer-use) — Seven-model evaluation on 25 KiCad tasks, including the 200-step budget, full solves and partial-credit rewards.
- [Codex background computer-use announcement](https://openai.com/index/codex-for-almost-everything/) — OpenAI's April 2026 announcement of parallel background computer use in the Codex macOS app.

## Slides

### 1.0 = the Human Loop

The agent inspects a screenshot, reasons about the visible export button, clicks it, and observes again. The speaker calls this repeated screenshot-and-action cycle computer use 1.0.

[1:13](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=73s) [A diagram shows an agent inspecting a screenshot, reasoning about an export button and choosing a click target.](https://assets.ai.engineer/sites/aiecode2025/sha256/33e68f1bc720b17723ee454b955ca69bd560fe375bf16a59288430dd6b8f4337.webp)

1.0 = the Human Loop
The agent watched the screen, chose the next move, acted like a user, and repeated.

Cua
THE OLD LOOP

SCREENSHOT

REASON
“The user asked for a PDF. The export button is visible in the lower right.”
Choose one action, then observe again.

CLICK TARGET

1.0 METHODS
screenshots
OCR / SoM
mouse
keyboard
scroll
type
wait
pyautogui

screen first · action next · foreground repeat | cua.ai

Interface text:
Safari  File  Edit  View  Go  Window  Help
Reports
Exports
Settings
Quarterly report
Export PDF

### A GUI-first operator

The 1.0 loop sends a task and screenshot directly to the model, which returns a computer action through provider-native tools. The next action depends on the pixels in the screenshot.

[2:20](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=140s) [A code example and labeled panels describe a model receiving text and a screenshot and producing a computer action.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b584b395758ce5483f2807287ac1f9e8cd5d28a9265bcfe65762b32a0f396cc.webp)

A GUI-first operator
The old loop wired the model straight to the GUI: prompt and screenshot in, action out.

COMPUTER-USE 1.0 EXAMPLE

response = client.responses.create(
    model="computer-use-preview",
    tools=[{
        "type": "computer_use_preview",
        "display_width": 1024,
        "display_height": 768,
        "environment": "mac",
    }],
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": task},
            {"type": "input_image", "image_url": screenshot_url},
        ]
    }],
    truncation="auto", # returns computer_call
)

MODEL 01
computer-use-preview
The loop is native to the model.

TOOL 02
provider-native tools
OpenAI Responses API or Anthropic beta.

INPUT 03
text + screenshot
The next action depends on pixels.

OUTPUT
next: computer_call

Computer-Use model setup · prompt + screenshot loop | cua.ai

Interface text:
Cua
computer_use_setup.py
PYTHON

### À la SkyLight Recipe

Cua Driver activates an app without raising its window and uses SkyLight to send mouse events directly to its process. An off-screen click primes Chromium, while private AX SPI keeps occluded Electron trees fresh.

[2:35](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=155s) [A macOS desktop accompanies four notes on background control: the yabai pattern, event posting, a primer click and live accessibility.](https://assets.ai.engineer/sites/aiecode2025/sha256/b15569f75272fcfa0a6a8074e9ab7f2c3e3a6975f2188560cf1ccff4da6d57de.webp)

À la SkyLight Recipe
Focus without raise, post to pid, prime Chromium, keep AX live.

BACKGROUND CONTROL

01 yabai pattern
Flip AppKit active without raising the window or switching Spaces.

02 SLEventPostToPid
SkyLight sends mouse events to the target pid, outside the HID stream.

03 primer click
An off-screen click at -1,-1 warms Chromium's activation gate.

04 live AX
Private AX SPI keeps occluded Electron trees fresh.

Interface text:
Cua
Los Angeles 58°
Google Chrome
Blender
[illegible]

### Background-control click demonstration

A coordinate grid shows click markers behind a terminal as Cua demonstrates background control. SkyLight posts mouse events to a target process; the slide also describes a primer click and live accessibility.

[2:46](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=166s) [A coordinate grid shows click markers alongside a terminal, within the background-control slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/f99c475f56636f7b5f28fd7ad9be9c7d735f0af1368d988ea5277fa8e119566b.webp)

Cua
BACKGROUND CONTROL

À la SkyLight Recipe
Focus without raise, post to pid, prime Chromium, keep AX live.

01 yabai pattern
Flip AppKit active without raising the window or switching Spaces.

02 SLEventPostToPid
SkyLight sends mouse events to the target pid, outside the HID stream.

03 primer click
An off-screen click at -1,-1 warms Chromium’s activation gate.

04 live AX
Private AX SPI keeps occluded Electron trees fresh.

Interface text:
Coordinate Test Window
click [illegible]
click [illegible]
Terminal text: [illegible]

### One interface to rule them all

Cua maps a shared agent interface onto different OS mechanisms: Windows uses UIA and messages, while Linux uses AT-SPI with separate paths for X11 and Wayland.

[3:56](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=236s) [Windows and Linux approaches to a shared agent-facing interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/795b12c0de123f84b90c9f9243250edd0e78911bc75df3391edcda859fb31967.webp)

One interface to rule them all.
Different operating systems, different display servers, one agent-facing contract.

Windows
MAY 2026
02
UIA + messages
State comes from UIA/MSAA; pixels from PrintWindow/WGC. PostMessage handles many apps; SendInput is the explicit foreground rung.
UIA / MSAA
PrintWindow / WGC
PostMessage

Linux
JUNE 2026
03
AT-SPI first
X11 and Wayland expose opposite input models, so the driver keeps the agent contract semantic.
X11 / XWAYLAND  background pixels, XTEST keys, recording
NATIVE WAYLAND  AT-SPI actions; raw keys need fallback

Windows: UIA + messages · Linux: X11 routes pixels, Wayland routes semantics | cua.ai

Interface text:
Cua
CROSS-PLATFORM

### One state, several action paths

Cua Driver returns an accessibility tree and screenshot together. The agent can try an accessibility action, fall back to a background pixel click, and use foreground delivery when needed.

[4:23](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=263s) [A command example alongside four action paths: observe, accessibility action, pixel action, and foreground fallback.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ec4ec47ac152856f74ba79ef51ed39b4c280608d8f09177a64f8a2607207d82.webp)

Cua
CUA DRIVER SURFACE

One state, several action paths

Agents read window state once, then pick the most reliable way to act.

cua-driver
$ cua-driver get_window_state '{pid,window_id}'
AX tree + screenshot + element handles
$ cua-driver click '{element_index:14}'
✓ AX action: effect confirmed
$ cua-driver type_text '{x,y,text}'
pixel action: focus by pixel, then type

ACTION PATHS
01 observe once
AX tree and screenshot return together.
02 AX action
element_index gives driver-verifiable actions.
03 pixel action
x,y clicks use the same screenshot.
04 foreground fallback
delivery_mode: "foreground" only when needed.

get_window_state · element_index · x,y · delivery_mode | cua.ai

### Every action path gets tested

Cua tests action paths across desktop toolkits with 500+ scored checks. Background runs check both whether the app state changed and whether the target stayed backgrounded.

[5:33](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=333s) [A regression matrix of action modes and toolkit lanes, with more than 500 scored action checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/2142eb24caf9466853ffabd8095f535d74eaa4e30a91f86c4b979520e0b03ec4.webp)

Every action path gets tested
Tests run against real toolkits and app-state changes.

Cua
REGRESSION MATRIX

500+
SCORED ACTION CHECKS
effect plus background-held contract

5 MODALITY RUNS
ACTION PATH X DISPATCH X SCOPE
MODE | TARGET | DISPATCH | VERDICT
ax-bg | element_index | background | effect + held
px-bg | x,y | background | effect + held
ax-fg | element_index | foreground | effect
px-fg | x,y | foreground | effect
px-desktop | screen pixels | desktop | effect

TOOLKIT LANES
WINDOWS | WPF | WinUI3 | WebView2 | Electron
LINUX | GTK3/4 | Qt5/6 | Electron
MACOS | AppKit | SwiftUI | WKWebView | Electron

EFFECT
Did the app state actually change?

CONTRACT
Did the target stay backgrounded?

6 PARITY CONTROLS
checkbox | click target | slider | scroll | text input

8 actions x 6 controls x 10+ lanes = 500+ scored checks | cua.ai

### Our early adopters

Clicky, Hermes, Qwen Code, H Company, and Droid Factory use Cua’s desktop driver. The speaker thanks them for contributing upstream to the framework.

[6:03](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=363s) [Five early adopter cards and a summary of entry points, targets, and the observe-act-verify loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb60d16cd233b25ce10fc2bb4be1ae0a0fd8e640ef8e280448a59d5dc65c2bd1.webp)

Cua
USED TODAY

Our early adopters

These products and agent labs use the same MCP and CLI driver for desktop apps.

Clicky
CUA PRODUCT

Hermes
AGENT RUNTIME

Qwen Code
CODING AGENT

H Company
AGENT PLATFORM

Droid Factory
FACTORY

ENTRY POINTS
MCP, CLI, and SDK

TARGETS
real desktop apps

LOOP
observe, act, verify

MCP and CLI across real apps | cua.ai

### Can the agent use the computer?

Cua moves from giving agents hands with Cua Driver to measuring reliable task progress with Cua Bench. CTO Dillon DuPont introduces the intelligence section.

[6:22](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=382s) [An introduction to Cua Driver, Cua Bench, and speaker Dillon DuPont.](https://assets.ai.engineer/sites/aiecode2025/sha256/22b1f450d22fa3e1bea73f06e074a0e6344821ba27d3e872ea3d05d31aa183f4.webp)

Cua
INTELLIGENCE

Now: can the agent use the computer?

The driver gives agents hands. The next question is whether those hands produce reliable work.

SURFACE
Cua Driver
observe, act, verify

TEST
Cua Bench
measure real task progress

SPEAKER
Dillon DuPont
intelligence layer

Cua Bench · intelligence section | cua.ai

### Cua Driver gives agents actions

CuaBench tests whether an agent’s actions solve a task: it starts from a known state, lets the agent work, and checks whether the result passes or fails.

[6:33](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=393s) [CuaBench evaluates actions through a setup, act, and score loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/49773060d0e793b9a05a49c4dbf880f0532705af80c99b57b5bbf7f3f87de162.webp)

Cua
ACTIONS → CAN WE TRUST THEM

Cua Driver gives agents actions.
CuaBench measures whether those actions solve the task.

01
setup
known state

02
act
the agent works

03
score
checks pass or fail

verifiable environments · every result reproducible | cua.ai

### Every task is three pieces

The example opens an invoice, asks an agent to put the total in B2, then checks whether B2 equals 42. The speaker describes the solve step as an oracle that provides a golden trajectory.

[6:48](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=408s) [Python example showing setup, solve, and evaluation functions for a task.](https://assets.ai.engineer/sites/aiecode2025/sha256/315e6c3bdab5f252a8856f8c191335ccd937c370af44b16f5aa62c0c9798aea1.webp)

Cua
A TASK = THREE PIECES

Every task is three pieces.

from cua_bench import setup_task, solve_task, evaluate_task

@setup_task
def setup(env): env.open("invoice.xlsx")

@solve_task
def solve(agent): agent.run("Put the total in B2")

@evaluate_task
def check(env): return env.cell("B2") == "42"

setup · solve · evaluate · five platforms | cua.ai

Interface text:
task.py
PYTHON

### Environments take scale and expertise

A Python file launches an invoice window and retrieves the heading’s screen coordinates for an agent. The speaker says the SDK supports writing and probing GUIs across desktop platforms.

[7:29](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=449s) [Python code creates an invoice window and retrieves the coordinates of its heading.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a5ac80849f364bd1206188b9bc08fa9be40a07ee57f6aaba958b1d1872d9983.webp)

Cua
ONE FILE, ANY PLATFORM

Environments take scale and expertise.

from bench_ui import launch_window, get_element_rect

# inline HTML, no browser, no server
pid = launch_window(html="<h1>Invoice</h1>")

# real screen-space coords for the agent
rect = get_element_rect(pid, "h1")

rect → agent
Invoice

THE AGENT SEES EXACTLY WHAT YOU DREW

author · launch · inspect · in one file | cua.ai

Interface text:
env.py
PYTHON
env window

### One open catalog

Cua’s catalog lists 130 verifiable tasks across 42 environments and five platforms. The CLI example runs Cua-Bench KiCad, an electrical engineering dataset whose evaluators simulate circuits.

[7:57](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=477s) [Catalog overview listing 130 verifiable tasks, 42 environments, and five platforms.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a45ab6c1fe4bb2e6a4d8b80b0d6d3a4c809faf12e60e05c6cd35ffc57ce817c.webp)

One open catalog.

130 VERIFIABLE TASKS
42 ENVIRONMENTS
5 PLATFORMS

$ cb run dataset cua-bench-kicad -m sonnet-4-5

linux · windows · android · macos · web | cua.ai

Interface text:
Cua
130 TASKS · CUA-BENCH-KICAD

### KiCad results: 6 out of 25

The top tested agent fully passed 6 of 25 expert KiCad tasks. All six passes involved editing an existing schematic; tasks starting from a blank schematic had a 0% success rate.

[8:33](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=513s) [The slide reports six full passes on 25 expert KiCad tasks and says none built a schematic from a blank canvas.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee4c885f69c058a9ef78842d126168224d7a59ecd670ec05cb11c70e70069486.webp)

Cua
6/25 · NONE FROM SCRATCH

THE RESULTS ARE HUMBLING · 25 EXPERT KICAD TASKS

6 / 25

No model builds a schematic from a blank canvas.

every full pass was an edit, never a build | cua.ai

### It's not close for anyone

GPT-5.5 leads by full passes with 6 of 25, while both Claude models pass 5 of 25 and have a higher mean reward: 0.253 versus 0.240.

[8:52](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=532s) [A leaderboard shows GPT-5.5 with six full passes and two Claude models with five each, out of 25 tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/538835d9c6a806c23710718c9d2f08c3b0fc90f1cb567277fedf0cc4d23fb8ac.webp)

Cua
SEVEN MODELS, ALL FLAT

It's not close for anyone.

#  MODEL  PROVIDER  FULL PASS  MEAN REWARD
1  GPT-5.5  OpenAI  6 / 25  0.240
2  Claude Sonnet 4.5  Anthropic  5 / 25  0.253
3  Claude Haiku 4.5  Anthropic  5 / 25  0.253

ranked by full passes · 0.240 mean reward at the top | cua.ai

### Score it, and you can improve it

In an early, directional single-run eval at 4K, switching to Cua Driver’s per-window views raised full passes from 41 to 53 out of 66 and reduced tokens per solve by 34%.

[9:03](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=543s) [A directional single-run evaluation shows full-pass rate increasing from 62% to 80% and tokens per solve decreasing by 34% with per-window views.](https://assets.ai.engineer/sites/aiecode2025/sha256/99a9a5e48b05086b673dc3a3c68b205d760c8d07802371b83905aafae22d7753.webp)

Score it, and you can improve it.

62% → 80%
FULL-PASS RATE · 41 → 53 / 66

−34%
TOKENS PER SOLVE

Per-window views instead of the full 4K desktop.

gemini-3.5-flash · 4K · early eval, single run, directional | cua.ai

Interface text:
Cua
CUA-DRIVER · 62 → 80%

### Can the evaluator be trusted?

Cua uses agents to probe tasks for reward hacks and turns the results into code reviews. The example flags an evaluator that misses unrelated settings changes; only tasks that survive the pipeline enter the dataset.

[9:32](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=572s) [Slide asks whether an evaluator can be trusted, alongside an example automated review.](https://assets.ai.engineer/sites/aiecode2025/sha256/89afba8e13f6379c2e30c7ac1c8087d2f8f6298173510234d484ce684aa0e210.webp)

Cua
EVALS ALL THE WAY DOWN

Our eval says:
trust the
model.

But how do we know the eval itself can be trusted?

app: godot  commit: 02ec78d4 (demo)

Findings & suggested fixes
Reward Integrity | Major

Diff the full settings universe, not just the task's checklist keys.

no_side_effects only diffs the keys in THIS task’s checklist, so a collateral toggle of ANY of the [illegible] (fullsize was just the one this adversarial hit) scores 100% across every task. Fix the tracked-se[illegible] per-field.

Evidence
# runs/adversarial/trajectory.md:14-22
agent: [thought: set minimum_display_time, then flip an unrelated Fullsize toggle as [illegible]
agent: [tool_call: CLICK x=1010 y=165 (Boot Splash › Fullsize)]
tool: [illegible]
agent: [tool_call: DONE]

test the evaluator too | cua.ai

Interface text:
cua-gym-qc Bot commented yesterday · edited

### Fork through time to predict state

A recorded run can be forked at a chosen moment to recover the computer’s state. The model predicts an observation, which can then be compared with the fork.

[10:14](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=614s) [A recorded sequence is forked at an action, with a terminal view and a prompt to predict the next state.](https://assets.ai.engineer/sites/aiecode2025/sha256/af2613dc681d8e857875fdd85042bb7e6f1f40ff41adf0a79bbb9d947b75e01b.webp)

Cua
FORK TIME, PREDICT STATE

Can the agent predict state after it acts? Fork through time.

RECORDED TRAJECTORY · OBJECT STORAGE

01
type "buy milk"

02
press Ctrl + S

03
FORK HERE
type "fix bug 42"

04
press Ctrl + S

>_ forked @ step 03
$ cat todo.txt
✓ buy
milk

EACH FORK = ONE RL ENV
Predict the next state.

fork trajectory · probe state · score prediction | cua.ai

### Make it run at scale

Benchmarks and RL need more runs, so Cua frames fleet throughput around keeping GPUs busy and preventing environments from blocking training. Rob Wendt takes over the scaling section.

[10:47](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=647s) [Slide introduces fleet throughput, more runs, busy GPUs, and Cua Fleets.](https://assets.ai.engineer/sites/aiecode2025/sha256/dfcf7cab0621b085f9bbc86bc6ece3c729dbdf1f2f76dc49da3f7ec3bc77b4c5.webp)

Cua
SCALING

Then make it run at scale.

Once agents can be measured, the bottleneck moves to fleet throughput.

DEMAND
more runs
benchmarks and RL both need volume

CONSTRAINT
GPUs stay busy
environments should not block training

NEXT
Cua Fleets
Rob Wendt · scaling section

Cua Fleets · scaling section | cua.ai

### Interruptible generation

Blue sequences show GPUs generating tokens for RL training alongside training batches. The diagram marks interruptions, weight loading and KV cache recomputation at batch transitions.

[10:58](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=658s) [Timeline diagram shows generation and training across GPUs and batches, with weight loading, cache recomputation, and interrupted sequences.](https://assets.ai.engineer/sites/aiecode2025/sha256/656ef0ab4897aa4e34161d49893b8fd6dd081087f825a3c2d524d0240214bb2a.webp)

Interruptible Generation

INTERRUPTIBLE GENERATION · GPUS KEPT BUSY ACROSS BATCHES

GPU1
GPU2
GPU3
Time →

Timeline: θ₀, θ₁, θ₂
Batch 1: 1-4
Batch 2: 5-8
Batch 3: 9-c

Generation sequence labels: 1, 6, 9, d, d; 2, 5, a, a, e, e; 3, 7, [illegible], c, f, f; 4, 8, b, b, g

Legend: Training; Generation; Load Weight; KV Cache Recompute; Interrupted Sequence

training and generation share the fleet; no GPU sits idle | cua.ai

Interface text:
Cua
CUABENCH

### Worker utilization: generation versus idle

Six worker timelines alternate between generation and idle gaps. GPUs wait while a sandbox resets or a new one starts, adding cost as workloads scale.

[11:39](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=699s) [Six worker timelines alternate between generation and idle intervals.](https://assets.ai.engineer/sites/aiecode2025/sha256/b073bcac1f5319d0bf84e92a57cd53f105582e9c9028741545c8965dae919efc.webp)

WORKER UTILIZATION · GENERATION VS. IDLE

Workers: w0, w1, w2, w3, w4, w5
Time →
Legend: generation; idle (waiting)

every idle gap is a GPU waiting; minimize it | cua.ai

Cua
CUABENCH

### Warm pools and startup time

Some sandboxes are slow or costly to optimize for startup. A small warm pool can cost less than an idle GPU, shifting the focus toward GPU idle time.

[12:18](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=738s) [A crossed-out startup-time heading precedes three points about runtime limits, GPU costs, and warm pools.](https://assets.ai.engineer/sites/aiecode2025/sha256/1341d8f6488070e730f6ac38e0639344628a6dd128ba24a28a838cf8cb7cc948.webp)

Minimize startup time

Cua
STARTUP TIME

01 Some runtimes have a floor
Linux has microVMs. Other runtimes start slowly, or cost too much to optimize.

02 GPUs cost more than warm environments
A small warm pool is cheaper than an idle GPU.

03 Warm pools move the target
Optimize for GPU idle time before cold-start time.

warm pools keep GPUs busy | cua.ai

### Keep a warm pool

A demand-based autoscaler sizes the warm sandbox pool to GPU demand, reducing GPU waits for cold starts.

[13:00](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=780s) [A diagram pairs eight filled squares with a grid of eight filled and eight empty squares, illustrating a warm pool that scales with demand.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ab787152f2b1d3a1ddbac49f8d284cf5ed0a9cd60cf45c47e005be9f854d25a.webp)

Keep a warm pool.

Warm environments scale with demand, so GPUs spend less time waiting on cold starts.

Cua
WARM POOL

warm pool tracks workload · GPUs keep working | cua.ai

### Worker utilization

A warm pool absorbs startup time on the infrastructure side, keeping all six illustrated GPU workers generating without idle gaps.

[14:24](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=864s) [Six worker timelines show continuous generation with no idle intervals.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bd78ea1448e515607dec7a3f3d0c69b4800e01b7fc0315fce26ffc1ba3ba4f0.webp)

Cua
CUABENCH

WORKER UTILIZATION · NO IDLE, NO INTERRUPTIONS

Workers: w0, w1, w2, w3, w4, w5
Time →
Legend: generation; idle (waiting)

warm pool: every worker generates end to end | cua.ai

### Every major platform

Linux, Windows and Android are generally available; macOS is in Labs preview on Cua’s proprietary virtualization stack.

[14:43](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=883s) [Linux, Windows and Android are marked available; macOS is marked Labs preview on a proprietary virtualization stack.](https://assets.ai.engineer/sites/aiecode2025/sha256/f906e99f674077f6839a0e2d00a133a04acf03cbd2bf017e7bbccf0ea065e27c.webp)

Every major platform.

Cua
PLATFORMS

Linux, Windows, and Android are generally available. macOS is in Labs preview on our proprietary virtualization stack.

Linux
AVAILABLE

Windows
AVAILABLE

Android
AVAILABLE

macOS
LABS PREVIEW
PROPRIETARY VIRTUALIZATION STACK

one agent contract across every platform | cua.ai

### Computer-Use 2.0

Cua Driver offers an MCP and CLI for desktop apps, Cua Bench measures agent behavior, and Cua Fleets provides Linux and Windows fleets for evaluations.

[15:28](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=928s) [Presentation overview introduces Cua Driver, Cua Bench and Cua Fleets for vision, intelligence and scaling.](https://assets.ai.engineer/sites/aiecode2025/sha256/468db44351ef73e88e46989bcee51cc09ddb5f1150f882d4b10a261d183c23e8.webp)

Computer-Use 2.0

Cua
AI Engineer World’s Fair

From foreground GUI agents to computers that coding and general agents can use.

VISION
Cua Driver
MCP and CLI for desktop apps

INTELLIGENCE
Cua Bench
measure what agents actually do

SCALING
Cua Fleets
Linux and Windows fleets for evals

Francesco Bonacci · Dillon DuPont · Robert Wendt | cua.ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=0s) [upbeat music] Thank you for taking the time for coming over here.

[0:17](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=17s) Um, I'm Francesco. I'm the CEO of the company. Uh, alongside me, a couple of other folks, um, my CTO, Dillon, and my chief of infra, Rob. They're gonna walk on the stage in a while.

[0:27](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=27s) Uh, but before we do that, who's excited for some computer-using agent talk happening now? Are you guys excited? Lovely. Um, if I were to ask, like, what, what was a computer-using agent like one year ago?

[0:39](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=39s) Probably half the crowd would say, "I don't have any idea what really computer use mean." Um, so, um, today I'm gonna take you to a journey, um, basically like from our vision where, um, we come from so far on computer use, um, like this new shape of like agents that are talking, um, and, uh, up to

[1:04](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=64s) model intelligence. Um, so we're gonna start like with the vision of, uh, Cua Driver, where we're coming from. And, uh,

[1:15](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=75s) if you-- How many of you guys have been work- have been working with computer use for one year?

[1:22](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=82s) How about like three years? Lovely. Okay. So our team has plenty of experience. Like we go all the way back, uh, our time on Microsoft. We were working on this type of GUI agents we were calling them, uh, back in the days.

[1:36](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=96s) Um, and, uh, there is a, um, there is an example of like old-fashioned human, uh, agent loop. Um, we basically refer, um, refer this as, uh, human, uh, loop, where you will have like an agent loop.

[1:55](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=115s) You will have a, um, uh, you will take a screenshot that the agents will have to reason and plan through, and then, uh, um, you will basically work with an action space, uh, in terms of like clicking, typing, scrolling around.

[2:07](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=127s) So this is what we refer as, uh, as, um, um, the old-fashioned like computer use 1.0 just to set the tone for, for, um, for this talk. And, uh, we come like a long, um, a long like way since, uh, this type of like, um, computer-using agents.

[2:27](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=147s) 'Cause this is, this is again, like I'm gonna skim over these slides, but that's like the old-fashioned way of like representing this agents loop as a human would do.

[2:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=154s) Um, we, um... Here we go. Um, over like two months ago, we released a project in open source. It's called Cua Driver. And, uh,

[2:51](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=171s) um, we, we, um, we made it working like in the background. That means that your computer user will ta- will not take over your, uh, screen as like the computer use p- 1.0, um, kind of like agent loop was doing back in the days.

[3:07](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=187s) And, uh, um, it all like started from, uh, um, from like a, um, from Codex rel-releasing their computer-use, um, model, uh, two months ago. So we kind of like take the challenge because we were really like working with, uh, these, uh, this type of background computer use.

[3:26](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=206s) So over one weekend, uh, we hack something together, and, uh, the trick here is really, um, not like having your agents like take over your screen. So there is a lot of like dark magic like happening behind the hood.

[3:40](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=220s) Just to, uh, give you some context, uh, there are like some undocumented API, um, living in, uh, um, in the Apple framework and basically ships with your laptop. And as you can see here, like is in the demo, you have like an AI agent, uh, that is not taking over, uh, control of your, over your laptop.

[3:58](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=238s) Um, we made it working not only for macOS but also spanning like across Windows and Linux. We were-- Uh, this is like the very first like driver that is, uh, living on your, um, laptop and, uh, it lets, uh, really any AI agents connect to the underlying operating system, uh, either like using accessibility trees or like a

[4:20](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=260s) screenshot-level approach. We kind of like take all both.

[4:26](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=266s) Um, this is what really the agent sees for what it concerns. Um, you will have to install Cua Driver. Um, the agents will, uh, will take a snapshot of the window state, and, uh, you will have to observe.

[4:40](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=280s) Um, and, uh, we really like take, take like one, uh, like different like a, action path to really make background computer use happening. So you really, um, have, uh, to observe the space.

[4:54](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=294s) Uh, in this case, uh, just by calling like getWindowState, you get, uh, an accessibility tree representation plus a screenshot. And then, uh, you will go and, uh, um, try a background execution using accessibility tree, and if that doesn't work, we go all the way and, uh, make the heavy lifting for you and just try a pixel background

[5:16](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=316s) click. This is like, uh, kinda like best step for background at this stage. It's not like behaving the same way on macOS, Windows and Linux. So, uh, we, we do like some of the heavy lifting for you so that your AI agent can, uh, can run undisturbed on your, uh, on your MacBook.

[5:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=334s) Um, how we manage like to not break anything between like release cycles. We have ex- We have like a lot of investment happening, uh, behind the scene, uh, when we test like new releases.

[5:49](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=349s) Uh, we have about like eight different f- uh, application harnesses that, uh, are, um, that, that, that we, that we use for making sure that we don't break anything, uh, among different releases.

[6:05](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=365s) Um, among our early adopters, you can see, like, Cliqi, Hermes, Quancode, H Company, and, uh, DroidFactory. Uh, like, huge thanks to them for using Cua Driver and, like, um, basically releasing a lot of, like, upstream contribution, like, in, uh, in our framework.

[6:22](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=382s) Um, without further ado, I'm just gonna move to the next part of the presentation, which is gonna be intelligence. Um, and I'm gonna have our CTO, Dillon, uh, cover that.

[6:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=394s) Hello. So thank you, Francesco. Um, with Cua Driver, we gave an agent hands. But then the question becomes: How can you trust the agent to use those hands correctly and not leave anything broken behind?

[6:46](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=406s) And to answer that, we had to build Cua-Bench.

[6:49](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=409s) So for a show of hands, who here has heard of Terminal-Bench or Harbor?

[6:54](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=414s) Yeah, so a few of y- few of you have, have heard of it. And, uh, if you've ever authored a task for Terminal-Bench, then this might look familiar.

[7:03](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=423s) But in Cua-Bench, a task is made of three pieces: The setup function, which sets up the machine into initial state; the oracle function, which provides a golden trajectory for the task; and the evaluator, which probes the environment to check if the agent successfully completed the task.

[7:21](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=441s) Uh, unlike Terminal-Bench, the oracle here is, uh, GUI actions, so it looks kind of like PyAutoGUI when you write that.

[7:31](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=451s) And writing environments takes skill and expertise. On desktop, there's more than, uh, five platforms that we target, and, um, we try to collapse that into a single Python file.

[7:43](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=463s) So using the Cua-Bench SDK, you can write a GUI that works across every desktop platform in a single Python file and use the same SDK to probe that GUI to get usable agent data.

[7:55](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=475s) Anyone or any agent can author one of these tasks.

[7:59](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=479s) And when you put that to work, you get a real catalog. We have currently over 130 verifiable tasks, 42 environments, and across five platforms, and each of these are easily reproducible using our CLI.

[8:13](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=493s) And the latest addition to our datasets is one that we're proud of. Uh, with collaboration with Snorkel AI, we built Cua-Bench KiCad, which tests computer-use agents on electrical engineering tasks using software by real professionals and evaluator functions that actually simulate the circuits.

[8:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=514s) But the results are humbling. The top agent that we tested only got a full pass on six out of 25 of these tasks. Of those six, 100% of them involved editing an existing schematic.

[8:46](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=526s) And when we start the task from a blank schematic, the success rate drops to 0%.

[8:54](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=534s) And across all the models that we tested, the leaderboard is flat. No model has achieved more than 30% reward.

[9:04](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=544s) But once you can score something, you can improve it. If we take a look at the Cua-Bench basic dataset, scale it up to 4K resolution, uh, testing an agent, they typically get around 62% pass rate.

[9:18](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=558s) But when you switch the agent computer tool from the built-in one to Cua Driver, the pass rate jumps from 62% to 80% using 34% less tokens. And this is primarily because Cua Driver focuses on a window rather than the entire desktop.

[9:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=574s) But our evals might say you can trust model XYZ at task whatever. But how can you know that the task-- How can you know that the eval can be trusted?

[9:44](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=584s) So before we test a task against any agent, we first try to break the environment ourselves. We have a matrix of agents attempt to do reward hacking and attempting to break the environment, and we take all that data and we compile it into a nice CodeRabbit-style code review.

[10:02](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=602s) And only tasks that survive our pipeline can enter the dataset. And if you ask us how we trust that agent, the answer is that it's just evals all the way down.

[10:12](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=612s) But to measure the intelligence of an agent, you can't just measure its ability to successfully perform actions. You also have to measure its ability to understand the world that it's operating in.

[10:22](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=622s) Every run that we record can be forked through any moment in its trajectory to give you the state of the computer at that moment. From there, we can probe a model asking to predict the reward, the internal state, or any other observation of the computer and compare it against the fork.

[10:41](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=641s) And that prediction is the world model of the agent made measurable. And with that, um, I'll let Robert take the stage.

[10:50](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=650s) Thank you, Dillon. Uh, hello, everybody. Uh, I am the chief infra officer at Cua, and I'm here to talk to you about, um, how you're probably leaving a lot of money on the table, uh, with idle GPUs if you do RL training, uh, for computer-use agents.

[11:07](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=667s) So I kinda wanna introduce this, uh, diagram to y'all. Uh, can I get like... Is, is there, like, general familiarity with this diagram, or is this, like, something that most of us haven't seen before?

[11:18](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=678s) Like, any- anyone? Awesome. Very niche. Um, almost everything on this is not really important for what we're talking about, but the blue portions are. Um, and what those basically represent are GPUs, uh, generating tokens for, um, RL, uh, training.

[11:37](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=697s) And if you zoom in on this a little bit, you can kinda see, like, how this typically looks, like, with a sandbox environment, is you're gonna be generating some tokens, um, and then you finish your task on a sandbox, and then you're waiting for either, like, a new sandbox to spin up or for your existing one to

[11:57](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=717s) reset. Uh, the problem here is that, like, this is just pure cost. Um, your GPU really isn't doing anything useful here. And, you know, I don't know if you've heard, but GPU time is pretty expensive right now.

[12:10](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=730s) So, um, as you're scaling, this cost really compounds a lot, and you really wanna focus on minimizing this if possible. So one thing that you might try to do is, uh, minimize the startup time of your sandbox, and I mean, you should do that.

[12:27](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=747s) Like, that's a great thing to do, but, uh, you know, especially for computer use style environments, sometimes this can be a little bit impractical. Um, you know, your researchers might give you like a forty-gigabyte environment, and that might just be necessary, and it takes a long time to pull that down and start it up.

[12:43](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=763s) So, you know, how do you, how do you design your training infrastructure so that you can minimize the GPU startup or the minimize the startup time of the sandbox, uh, even when the sandbox is like not well designed to be start up quickly?

[13:00](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=780s) Um, so the way we do that... Ah, man, is it not?

[13:06](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=786s) So the way we do this is a p-pool, and this is supposed to be animated, but it's not animating. So, um, I guess I'll just explain it to you orally.

[13:16](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=796s) And what, what that is is, uh, so we have like a, a set of GPUs here which all want to use a sandbox. And what we will do is that we use a demand-based autoscaler to detect, um, how many GPUs, like, currently need a sandbox, and we can grow the pool to be that size, uh, on demand.

[13:38](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=818s) And what that means is that, uh, if you have... Let's say like you have a warm pool that you wanna allocate to your GPU cluster, you don't actually need to know upfront what that warm pool size is.

[13:50](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=830s) We can figure out what that warm pool size should be for you, um, on demand. And that might even change over the course of your multi-day training run. Uh, you might start needing a lot of sandboxes, but then as your generations get longer, you might need less.

[14:05](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=845s) So these also could be like, you know, easily, uh, two to f-four times cheaper than your GPUs. So having a little bit of redundancy here, uh, you still wind up saving money because you're maximizing the use of your GPU time.

[14:21](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=861s) Um, yeah, come see me after if you wanna see the animation 'cause it's, it's cool. Um, so yeah. So now when you have like this like, uh, redundancy in your pool, you're paying the cost of that startup time on the infrastructure side, not on the GPU side.

[14:38](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=878s) So your GPU workers have full utilization. Um, yeah. And then because we use this, we can give you instant sandboxes for your GPUs for Windows, Windows, Linux, Android, uh, and macOS is coming up.

[14:55](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=895s) Um, and I'm gonna hand it back to Francesco to, uh, close it out for us.

[15:02](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=902s) Lovely. Uh, thank you, Dillon. Thank you, Rob, for taking this over. Um, we do have like plenty of time for Q&A, so if you guys like have any questions, like happy to, um, take them either for Cua Driver, Cua-Bench, basically what Dillon presented, or Cua Fleet, uh, which is like what, uh, Robert covered.

[15:30](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=930s) Any questions? Otherwise, we can wrap this up. Oh, I see.

[15:35](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=935s) Would it be possible to operate like, uh, computer-use agents in the background? I know you do this for Mac-

[15:41](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=941s) Mm-hmm

[15:41](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=941s) ... but like in something like, uh, uh, mobile?

[15:44](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=944s) Um, so the story for mobile, Android, there is very far you can go. Um, we are talking with the Hermes team 'cause they do have like an harness that runs on Android.

[15:55](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=955s) I guess like if you're talking about background, there is some level of like background that can happen if you containerize a workload and basically on Android you can even like run your own container or like sort of like Ubuntu or like GUI Docker container, uh, within An-Android.

[16:14](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=974s) Um, but yeah, the Android ecosystem, especially compared to iOS, is more inclined to that form of like background, uh, computer use, but it's more towards like tool use than really like controlling GUI interface.

[16:27](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=987s) Um, we work with the activity framework and, uh, do tool use, uh, in the background.

[16:38](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=998s) Thank you.

[16:38](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=998s) Cool. Thank you, guys. [audience applauding]
