# Computer-use models will agentify the web, not APIs

Dhruv Batra

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/Ki980nV0__0-computer-use-models-will-agentify-web-not) · [Video](https://www.youtube.com/watch?v=Ki980nV0__0)

## Computer-use models will agentify the web, not APIs

Flight search already has APIs. Restaurant menus and school procurement records reveal why browser vision may supply the task interface the rest of the web lacks.

Basic familiarity with APIs, HTML and JavaScript will help you follow the browser examples.

### How will agents act on a web built for people?

[0:17](https://www.youtube.com/watch?v=Ki980nV0__0&t=17s)

If agents are going to read, click and buy on our behalf, how will they operate on a web that is already hostile to automated traffic? The usual answer is that websites will expose APIs through protocols: MCP servers, WebMCP and a growing collection of payment interfaces. Dhruv Batra accepts the prospect of agents driving web activity, but challenges that implementation assumption. Popular services may provide APIs; **the long tail cannot be assumed to follow.**

![Slide stating that computer-use agents will agentify the long tail of the web, not APIs.](https://ai.engineer/api/talk-essay-media/22b895b78d089dc2843277a63969f77fe8ee6f322e4149ad177323c0b9704097)

The opening thesis narrows the claim to the long tail of the web.

Consider a request for business-class flights from Miami to Palma de Mallorca in August. Sending a computer-use agent to click through `flights.google.com` is an unnecessarily indirect route when flight aggregators already accept parameters and return JSON. An LLM can call a tool against that structured interface. The buttons exist to produce the request behind the scenes; when the appropriate API already exists, use it.

### A gluten-free query becomes a visual navigation task

[2:34](https://www.youtube.com/watch?v=Ki980nV0__0&t=154s)

Now ask whether a restaurant has gluten-free menu items. The desired interface seems equally straightforward: supply a dietary filter and receive matching dishes. But that assumes the restaurant—or an aggregator—has already made its menu available as structured, queryable data. Batra walks through three restaurant sites to show what the agent actually encounters.

| Example | What the site provides | What the agent needs |
|---|---|---|
| Easy | German menu text and prices | Text extraction and language understanding |
| Medium | A Menu button leading to a selectable-text PDF | Navigation and PDF reading |
| Hard | A Spanish site leading to a gallery of pixelated menu JPEGs | Visual navigation and image reading |

The progression matters. On the first site, the language is not a fundamental obstacle for a multilingual model. On the second, the information remains text, but the agent must discover and open the document. On the third, a photograph-heavy page offers no obvious text menu until the visitor recognizes `Nuestra carta`, opens it and then opens individual gallery images. Batra reports that **ChatGPT** struggles with OCR on the downloaded menu JPEGs. A simple dietary question has become a sequence of navigation and perception problems before any filtering can begin.

![Enlarged restaurant-menu image with Spanish and English columns and illustrated food items.](https://ai.engineer/api/talk-essay-media/166ec0f5e2d89cfff58ed909f4ae9a045b00b67492caff2891041ce6acd89c2e)

The restaurant example reaches a menu displayed as an image in the browser.

The missing endpoint is therefore not just an omitted convenience. It presupposes a clean data representation that the restaurant may never have created. The public interface is the collection of pages, documents and images a person can work through.

### School procurement exposes the institutional bottleneck

[5:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=328s)

The enterprise version is a supplier asking whether a school district is currently procuring laptops. Given `mydistrict.gov`, the ideal interface would accept `status=open` and `topic=laptop`. Batra estimates a potential search space of 15,000–20,000 US school districts, though that range depends on whether regular districts or the broader category of education agencies is intended.

The demonstrations again become progressively harder. Ithaca Public Schools presents a portal that the agent must navigate. Another district places the relevant material under Finance → Purchasing → Current Solicitations, where the destination is a scanned PDF without extractable text. In the hardest example, obtaining information requires a Freedom of Information Act request. Staff scan the request email, upload it to Google Drive and attach the relevant PDFs. Expecting that institution to supply an MCP server overlooks the workflow it actually operates.

Batra uses roughly 200 million active websites and around a billion total websites as scale estimates. More consequential than the precise count is the ownership boundary. A coding agent might generate an API if given unrestricted access to an institution's software, but generating the code does not obtain that access, authorize deployment or change the institution's processes. Organizations still faxing documents will not all replace their infrastructure on an AI development schedule. **The bottleneck is institutional adoption as well as software construction.**

### The answer may not exist as page text

[8:35](https://www.youtube.com/watch?v=Ki980nV0__0&t=515s)

If the infrastructure will not change, why not point a coding agent at the HTML? A browser is software, so the information it displays must come from somewhere. The first complication is timing. In Batra's NBA.com example, a Minnesota Timberwolves–Brooklyn Nets page visibly shows a final score of 125–109. The initial HTML contains an empty placeholder. After a network-dependent delay, an asynchronous request returns JSON containing the score. Reading only the initial response misses the answer. Adding a wait seems like a reasonable repair—but the next example is not merely about waiting.

The next question is whether a 25mm osmium cube is in stock. A product dropdown shows three unavailable options and one available option. A human can interpret grayed-out, unclickable choices as unavailable, even when the interface does not explicitly label the remaining choice “in stock.” Yet the HTML option selector does not contain the availability text the visitor expects to find.

Batra describes product-count JSON containing a `quantity` value, followed by a separate rendering script that disables and grays out options whose quantity is zero. His account has the browser fetch that JSON; the [companion explanation](https://yutori.com/blog/the-bitter-lesson-for-web-agents) instead places it in the initial HTML and specifies a 25.4mm cube. Either placement leaves the central mechanism intact: **availability is computed from data and expressed through rendered state**, rather than supplied as a literal answer in the selector text.

A small JavaScript example captures that rendering rule. Here, a zero quantity makes the option unselectable without adding any “sold out” text:

```javascript
const stock = { label: "25mm osmium cube", quantity: 0 };
const select = document.createElement("select");
const option = new Option(stock.label, stock.label);

option.disabled = stock.quantity === 0;
option.style.color = option.disabled ? "#888" : "#111";
select.append(option);
document.body.append(select);
```

Searching `option.textContent` would find the product name, not the stock answer. Recovering the visible meaning requires understanding the data, the rendering rule and the resulting interface. Batra compares the browser to a game engine: predicting its pixels from source code requires reconstructing the computation that produces them.

### Screenshots provide a general interface

[12:08](https://www.youtube.com/watch?v=Ki980nV0__0&t=728s)

The web was built for human eyes. Its rendered interface is where text, images, scripts and interaction state come together for the intended consumer. Batra calls this a bitter lesson for web agents: accumulating website-specific extraction rules and scaffolding does not provide the same generality as interpreting the interface itself. Vision gives the agent access to the representation that website builders already maintain for people.

The first version of [**Navigator**](https://yutori.com/blog/introducing-navigator), released in November of the preceding year, used screenshots as inputs and button clicks and scrolls as outputs. Its discount-code demonstration turns a natural-language claim into a browser task. The code may apply only to a particular product, within certain dates or above a minimum cart value. Those conditions determine the interaction the agent must perform.

The validation procedure follows the same route a shopper would take:

1. Find the product described by the offer.
2. Add the eligible product to the cart, satisfying any minimum cart requirement.
3. Apply the discount code.
4. Inspect the resulting price to check whether the claimed 22% reduction was applied.

The percentage is the offer being tested, not a measured success rate. Batra describes these tasks as taking roughly 20–40 interaction steps, depending on complexity. The visual interface makes the task expressible even without a dedicated validation API; it does not eliminate practical accuracy gaps. His claim that the model can accomplish browser tasks is explicitly an in-principle claim.

### Use code for action and pixels for feedback

[14:23](https://www.youtube.com/watch?v=Ki980nV0__0&t=863s)

Needing vision does not require imitating every human action. In the next model demonstration, a Chrome extension displays an `Execute JS` action while multiple form fields are filled together. The model has generated a function to perform the updates instead of clicking and typing into each field separately. The browser can execute code, so an agent can use that capability whenever it is the efficient way to act.

**Choose the action method for the task, then inspect the rendered result.** The model can read code, write JavaScript or click a control, and use the next screenshot to check what happened. Batra describes this as a “sort of formal verification” system, but the mechanism shown is visual outcome checking, not mathematical proof. Successful execution of a function is not itself evidence that the intended form state appeared; the screenshot supplies that feedback.

The same browser capability can be replicated. An orchestrator launches multiple Navigators, each in its own cloud sandbox, to work on different websites simultaneously. The advantage over a human operator comes from parallel execution across browser instances, not from requiring one agent to finish every interaction faster than a person.

### When the benchmark becomes too easy

[15:49](https://www.youtube.com/watch?v=Ki980nV0__0&t=949s)

Are computer-use models reliable enough to make this useful? Batra answers with [Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web), showing model release dates against human-evaluated task success. Evaluators inspect the trajectories and judge whether the tasks were completed correctly. This measures outcomes across browser interactions, rather than whether an isolated click was plausible.

Batra reports approximately 97% human-evaluated success for [**Navigator n1.5**](https://yutori.com/blog/introducing-n1-5), with eight incorrect trajectories out of 300 on the evaluated Online-Mind2Web version. He describes the tasks as involving roughly 30–50 interaction steps. For him, that result is a reason to retire this version of the benchmark and build harder, longer tasks. The result belongs to that evaluated task set; the benchmark's tasks change over time, so its version matters when comparing releases.

The evaluated model also reflects the hybrid design: pixels come in, while both button clicks and code can come out. The progress claim therefore concerns a visual browser agent with multiple ways to act, not a system restricted to reproducing human mouse movements.

### The economic case depends on each step

[17:11](https://www.youtube.com/watch?v=Ki980nV0__0&t=1031s)

Reliability does not remove the cost of repeatedly observing a screen and taking an action. Batra acknowledges that computer-use workflows can be slow and expensive, especially when repeated across many tasks. In comparisons with **Opus 4.7** and **GPT-5.5** on browser-use benchmarks, he treats Navigator's small accuracy advantage as within statistical noise. The stronger practical argument is latency per step and cost per completed task.

Batra attributes the speed advantage to a smaller model footprint. For tasks involving roughly 20–30 interaction steps on the compared datasets, he cites about $0.80 versus $2.30 per task. The spoken comparison does not assign each price to a specific baseline or establish billing assumptions, so it supports the cost contrast without a precise model-by-model price ranking. Reducing the repeated cost of each interaction is what makes launching browser agents at scale more plausible.

### The browser layer becomes the API

[18:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=1108s)

Agents can become a primary way of acting on the web without every institution rebuilding its interface for them. Batra doubts that decades of infrastructure assembled for human use will be reinvented within a few years. Yet the caller's desired abstraction remains sensible: submit a task, expressed in natural language or programmatic parameters, and receive its result.

His proposed path is to build that abstraction above the existing web. Improving models make browser interactions more reliable and less expensive. Behind the endpoint, many browser instances navigate pages, click controls and collect results; the service returns structured output to its caller. The underlying restaurant, school district or merchant need not have created a new API. Another layer of software absorbs the complexity of its existing interface.

Batra projects a future in which this work costs less than a penny per task, runs in less than 100 milliseconds and sometimes executes in the user's browser. These are forecasts, not the current measurements presented earlier. If the caller can issue a task and obtain structured data cheaply and quickly, the implementation behind that boundary becomes less important. From the caller's perspective, the browser automation service is the API.

## Resources

- [Introducing Navigator](https://yutori.com/blog/introducing-navigator) — The original Navigator launch, explaining screenshot-driven browser actions, training and evaluations.
- [Introducing Navigator n1.5](https://yutori.com/blog/introducing-n1-5) — Hybrid visual and JavaScript browser interaction, benchmark comparisons, and a July 2026 evaluation addendum.
- [The bitter lesson for web agents](https://yutori.com/blog/the-bitter-lesson-for-web-agents) — Yutori's explanation of visual browser interaction, including the basketball-score and osmium-stock examples.
- [Online-Mind2Web benchmark](https://github.com/OSU-NLP-Group/Online-Mind2Web) — Official benchmark repository with tasks, evaluation instructions and task-update history.
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://arxiv.org/abs/2504.01382) — The paper introducing Online-Mind2Web and examining how automatic web-agent evaluation compares with human judgment.
- [Navigator API documentation](https://docs.yutori.com/reference/navigator) — Integration guidance for supplying screenshots and action history and handling Navigator's predicted browser actions.

## Slides

### A popular argument online

The familiar argument predicts agents will act across the web through APIs and protocols. The speaker agrees agents will drive web activity but rejects the proposed API mechanism.

[0:50](https://www.youtube.com/watch?v=Ki980nV0__0&t=50s) [The argument that agents will drive web actions, with APIs and protocols proposed as the mechanism.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4f7da189592e73f8998c5ae66acd024e48f94d4211e78c5587d3308beb287a8.webp)

A popular argument online

1. AI Agents will be the drivers of action on the web, not humans
2. How?
   a. The web will be “agentified”
3. How?
   a. With APIs that agents will call via
      i. (Web) Model Context Protocol (MCP/WebMCP)
      ii. {Agent, Universal} Commerce Protocol (ACP/UCP)
      iii. ...

Yutori

### Flight-search example

The speaker uses a business-class flight search to question why an agent would click through a site when flight aggregators already exist.

[1:50](https://www.youtube.com/watch?v=Ki980nV0__0&t=110s) [A phone displays a request for business-class flights from Miami to Palma de Mallorca in August 2026.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f0a51e6c3a81d641d6a916fd8c138387ae1bf7669eea8131dddeded04fd3a75.webp)

Find Business class flights from Miami to Palma de Mallorca for August 2026.

Interface text:
Yutori
23:59
Yutori
100

### Flight-search example

The speaker questions using a computer-use agent to click through Google Flights when flight aggregators can return JSON for a tool call. The slide contrasts the browser search with an illustrative API request.

[2:03](https://www.youtube.com/watch?v=Ki980nV0__0&t=123s) [The flight request appears beside a Google Flights search page.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6429cbe406c136fcf72a3eeed2171d7858f36bb66552c59d7c282a742c7453c.webp)

Yutori

Find Business class flights from Miami to Palma de Mallorca for August 2026.

Interface text:
23:59
Yutori
Google
Travel
Explore
Flights
Hotels
Vacation rentals
Sign in
Flights
Round trip
1
Business
Miami MIA
Palma PMI
Departure
Return
Search
Flexible? Discover the best flight deals with AI NEW
Describe your ideal trip, and let Google Flights find the best deals for you
Explore deals
Find cheap flights from United States to anywhere

[2:17](https://www.youtube.com/watch?v=Ki980nV0__0&t=137s) [A phone flight-search request alongside an illustrative API request for business-class flights.](https://assets.ai.engineer/sites/aiecode2025/sha256/4397001cb87a1f2bf8118732c11e0017387de0f7c84d0eae3081057b8cbfa931.webp)

Yutori

Find Business class flights from Miami to Palma de Mallorca for August 2026.

curl -X POST https://flights.google.com
     -H 'X-API-Key: YOUR_API_KEY'

  -d '{
       "origin": "MIA",
       "destination": "PMI",
       "departureDate": "2026-08-15",
       "passengers": [{ "type": "ADT" }],
       "cabin": "Business",
     }'

Interface text:
23:59
Yutori

### Restaurant menu as an API request

A hypothetical POST request turns a gluten-free menu question into a query. The speaker cautions that the restaurant may not offer such an endpoint.

[3:01](https://www.youtube.com/watch?v=Ki980nV0__0&t=181s) [A phone asks for gluten-free menu items beside an illustrative API request.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba666af78d18e7c0bd04193d7c2e90f9276cab3be42c4db7c726b7b73b32fcbd.webp)

Yutori

What are the gluten-free items on this restaurant’s menu?

curl -X POST https://my-favorite-restaurant.com/menu
     -H 'X-API-Key: YOUR_API_KEY'

  -d '{
       "query": "Gluten-free menu items",
     }'

Interface text:
23:59
Yutori
100

### Reality: restaurant websites

The speaker lines up three restaurant sites as easy, medium and hard examples before showing what their pages look like.

[3:18](https://www.youtube.com/watch?v=Ki980nV0__0&t=198s) [A slide lists easy, medium, and hard restaurant website examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/0927fdb835e62b4666f5996d603b08fca636a1bfdf480753b7f2690726269cb6.webp)

Yutori

Reality

Let me show you what restaurant websites look like

• Easy mode
  • https://xn--rainbowlnrnberg-7vb.com/
• Medium
  • https://www.eiscafe-baek.de/
• Hard
  • https://www.vegandveg.es/

### Restaurant website: opening view

The speaker calls this “easy mode”: the German ordering page displays menu text and prices that might be easy to scrape.

[3:29](https://www.youtube.com/watch?v=Ki980nV0__0&t=209s) [A restaurant ordering page with a menu, empty cart, and cookie banner.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7e66177aa7fb0fc882c3385ac36765a5dab4717c7f5bcfdde75975424c9abec.webp)

Speisekarte anzeigen
Restaurant Info
Anmeldung

Rainbowl Nürnberg
Marienstraße 15, 90402 Nürnberg
Geschlossen für Abholung
Vorbereitungsdauer 20 min
Speisekarte anzeigen

❤️ Die beliebtesten Produkte
Smoothie Bowls
Belgische Waffeln
Cake & More
Beloved Breads & Bagel
Summer Yummy’s
Coffee & More
Summer Specials on Ice

Produkt suchen...
ex. Nut-Ella Acai Bowl
Allergene

❤️ Die beliebtesten Produkte
Nut-Ella Acai Bowl
Nicecream aus gefrorenen Bananen, Acai Beere, Heidelbeere, Hafermilch. Getoppt mit Roasted Hazelnut Granola und frischen Früchten.
9,90 €

Abholung
Wähle ein Datum
Wähle eine Zeit
Deine Bestellung
Dein Warenkorb ist leer
Artikel hinzufügen

Diese Website verwendet Cookies
OrderYOYO möchte unsere eigenen und Drittanbieter-Cookies verwenden, um unsere Inhalte und Anzeigen gezielter zu gestalten und den Verkehr und das Nutzerverhalten auf unserer Website zu analysieren. Wir geben Informationen über deine Nutzung der Website an Dritte weiter, einschließlich sozialer Medien, Analysepartner und Werbepartner zum Zwecke von personalisierter und nicht personalisierter Werbung. Cookies tragen auch zu einer benutzerfreundlicheren Erfahrung auf unserer Website bei. Hier kannst du mehr über unsere Cookie Richtlinie lesen. Ausserdem findest du hier mehr Informationen darüber wie Google personenbezogene Daten verwendet.
Einstellungen verwalten
Zustimmen

Interface text:
rainbowlnürnberg.com
Feedback
Finish update

### Restaurant website: menu items

The speaker calls this German restaurant page “easy mode”: its menu exposes product descriptions and prices as text that may be easy to scrape.

[3:42](https://www.youtube.com/watch?v=Ki980nV0__0&t=222s) [A scrolled restaurant menu lists bowls, a bagel, bread, and a waffle with prices.](https://assets.ai.engineer/sites/aiecode2025/sha256/8334d075f688c5658661586371fe59557457d1df789d1bb0235c08b06f843997.webp)

❤️ Die beliebtesten Produkte

❤️ Die beliebtesten Produkte
Smoothie Bowls
Belgische Waffeln
Cake & More
Beloved Breads & Bagel
Summer Yummy’s
Coffee & More
Summer Specials on Ice
Homemade Lemonade
Homemade Smoothies
Bottles
Gutscheine

Nut-Ella Acai Bowl
Nicecream aus gefrorenen Bananen, Acai Beere, Heidelbeere, Hafermilch. Getoppt mit Roasted Hazelnut Granola und frischen Früchten.
9,90 €

Avocado Pink Hummus Bagel
Bagelbrot angeröstet, belegt mit hausgemachtem pinken Hummus, Avocado, Tomaten und Wildkräutersalat
8,90 €

Avocado Brot (vegan)
Frisches Sauerteigbrot mit Frischkäse, Avocado, frischen Tomaten, Sesam und Balsamicocreme
10,90 €

Dinkel-Vanille Protein Waffel
Waffelteig aus Dinkelmehl, Superfood Vanille Protein, Kokosreisdrink, Agavendicksaft, Banane, Apfelmus
5,00 €

Snickers Bowl
Nicecream aus gefrorenen Bananen, Peanut Butter, Erdnüsse, Datteln, Mandelmilch. Getoppt mit Peanut Crunch Granola, frischen Früchten und Karamell Sauce
9,90 €

Acai Bowl

Wähle eine Zeit
Deine Bestellung
Dein Warenkorb ist leer
Artikel hinzufügen

Diese Website verwendet Cookies
OrderYOYO möchte unsere eigenen und Drittanbieter-Cookies verwenden, um unsere Inhalte und Anzeigen gezielter zu gestalten und den Verkehr und das Nutzerverhalten auf unserer Website zu analysieren. Wir geben Informationen über deine Nutzung der Website an Dritte weiter, einschließlich sozialer Medien, Analysepartner und Werbepartner zum Zwecke von personalisierter und nicht personalisierter Werbung. Cookies tragen auch zu einer benutzerfreundlicheren Erfahrung auf unserer Website bei. Hier kannst du mehr über unsere Cookie Richtlinie lesen. Ausserdem findest du hier mehr Informationen darüber wie Google personenbezogene Daten verwendet.
Einstellungen verwalten
Zustimmen

Interface text:
rainbowlnürnberg.com
Feedback
Finish update

### EisCafé Bäk homepage

The Menu button leads from the café homepage to a PDF, so an agent needs a way to read the menu there.

[3:51](https://www.youtube.com/watch?v=Ki980nV0__0&t=231s) [Café homepage with an ice cream photograph and menu navigation.](https://assets.ai.engineer/sites/aiecode2025/sha256/f951c8ab3ed5e1b19e75b468edaa32a86f51593583e65bd5275fbc349d83c532.webp)

EisCafe Bäk

HOMEPAGE
MENU
GUEST ROOM
VOUCHERS
ARRIVAL
OPENING HOURS
GALLERY
ABOUT US

Ice cream
parlor Bäk

Bäk's Old Village School - A place worth lingering in.
We look forward to your visit!

OPENING HOURS

Interface text:
eiscafe-baek.de
Finish update

### Café menu PDF

The Menu button opens a PDF. Its highlighted drink listings are selectable text, which the speaker calls a medium-difficulty case for an agent.

[3:59](https://www.youtube.com/watch?v=Ki980nV0__0&t=239s) [A café menu PDF with drink listings highlighted and photographs of the café.](https://assets.ai.engineer/sites/aiecode2025/sha256/57d9f0dccc7ec46ba54f7821deb732e61e51c6e4113a2e74197811411d09cbdc.webp)

Heiße Getränke
Tasse Kaffee € 3,20
Pott Kaffee € 4,20
Kännchen Kaffee € 5,50
Café Crema € 3,50
Cappuccino mit geschäumter Milch € 3,90
Espresso € 2,90
Espresso Macchiato € 3,10
Espresso mit 1 Kugel Eis € 4,40
Doppelter Espresso € 4,20
Doppelter Espresso Macchiato € 4,40
Latte Macchiato € 4,30
Milchkaffee € 4,50
Kinder-Cappuccino: Heißer Milchschaum € 2,20
Glas Tee verschiedene Sorten € 3,60
Heiße Schokolade € 3,70
Heiße Schokolade mit Sahne € 4,10
Kurkuma Latti Ingwer € 4,20
Aufschlag Hafermilch € 0,30

Alkoholfreie kalte Getränke
- Softdrinks -
Mineralwasser 0,25l € 3,10
Stilles Wasser 0,25l € 3,10
Mineralwasser 0,75l € 6,90
Orangensaft 0,2l € 3,10
fritz-kola 0,2l € 3,20
fritz-kola ohne Zucker 0,2l € 3,20
fritz-kola Mischmasch 0,2l € 3,20
fritz-kola Bio Schorlen 0,2l € 3,30
(Apfel, Rhabarber)
fritz-kola Limo 0,2l € 3,30
(Zitrone, Orange, Apfel-Kirsch-Holunder)
Erdinger alkoholfrei 0,5l € 5,20
Flensburger Radler alkoholfrei 0,33l € 4,10

Eisige Getränke
Eiskaffee € 4,70

Unser Gastraum „Das alte Klassenzimmer“ steht auch für geschlossene Gesellschaften bis 30 Personen zur Verfügung.

Für festliche Anlässe außer Haus bieten wir Ihnen gerne unsere Eispyramiden an.

Liebe Gäste,
genießen Sie in entspannter Atmosphäre unsere Eiskreationen, leckere Kaffeespezialitäten, den selbstgebackenen Kuchen oder frische Waffeln.

Verweilen Sie in unserem Café-Garten oder im gemütlichen Gastraum mit Kamin.

Wir wünschen allen Gästen – ob groß oder klein – einen guten Appetit!

Inh.: Sven Unger
Schulstr. 25

EisCafé Bäk

Interface text:
eiscafe-baek.de/wp-content/uploads/2024/03/Eiscafe_Baek_Eiskarte.pdf
Eiscafe_Baek_Eiskarte.pdf
1 / 2
100%
Finish update

### Veg and Veg homepage

The speaker finds “Nuestra carta” on Veg and Veg’s photo-heavy homepage. The menu opens as a gallery of JPEGs, which he says ChatGPT struggles to read with OCR.

[4:35](https://www.youtube.com/watch?v=Ki980nV0__0&t=275s) [Restaurant homepage with a large Veg and Veg banner and a menu link.](https://assets.ai.engineer/sites/aiecode2025/sha256/d626c85717b536107d4603e4e93bbaa14d1bab9b7a1353416f1396296b1a3148.webp)

VEG and VEG
GASTROARTESANÍA

Inicio
Galería
Nuestra carta
Reseñas
Contacto

Interface text:
Inicio
vegandveg.es
Finish update
https://www.vegandveg.es/menu

### Reality: restaurant websites

The slide ranks restaurant websites from easy to hard. The speaker describes menu items embedded in gallery JPEGs that ChatGPT struggles to read, questioning whether a gluten-free API would be available.

[5:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=328s) [Slide listing restaurant websites as easy, medium and hard examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a36153db637b1bd5e589f5e86fb422378620c6e42d49fed68880a7f3090ab45.webp)

Yutori

Reality

Let me show you what restaurant websites look like

• Easy mode
  • https://xn--rainbowlnrnberg-7vb.com/
• Medium
  • https://www.eiscafe-baek.de/
• Hard
  • https://www.vegandveg.es/

Interface text:
2026-07-01-AIE
Lead presentation
Edit
Share
14 / 48
Finish update

### School district procurement API example

A proposed API request turns a buyer’s question about laptop procurement into filters for open RFPs and the topic “laptop.”

[5:36](https://www.youtube.com/watch?v=Ki980nV0__0&t=336s) [A phone asks whether a school district is procuring laptops, beside an example request filtering open solicitations for laptops.](https://assets.ai.engineer/sites/aiecode2025/sha256/409b8bee4925041502086d6eba9ade87bc3d6931004e777fedcc56cd66530a28.webp)

Yutori

Yutori
Is this school district procuring laptops right now?

curl -X POST https://my-district.gov/tasks
     -H 'X-API-Key: YOUR_API_KEY'

  -d '{
       "RFPstatus": "Open",
       "Topic": "laptop",
     }'

Interface text:
23:59
Finish update

### Finding a district purchasing page

The MCPS menu nests Purchasing & Warehouse under Finance. The speaker follows it to current solicitations and encounters a scanned PDF with no text.

[6:32](https://www.youtube.com/watch?v=Ki980nV0__0&t=392s) [A school district website menu shows Purchasing and Warehouse under Finance.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcc135e59a9e66910f77ba3032560947ca69b12ed9fd64480f5fc7ca0ace4ded.webp)

LIVE FEED   STAFF   NEWS   EVENTS   DINING   DOCUMENTS   ATHLETICS

Enroll at MCPS
Returning Student Registration
Transfer Requests

About Us
MCPS Leadership Cabinet
Advisory Committees
School Contact Information
MCPS Safety Initiatives
Budget
Calendar
Mission
School Board
Superintendent
Policies
Annual Notifications
Montgomery County Educational Foundation
Gun Safety
Tipline
Document

Departments
Athletics
Compliance
Communications
Curriculum
Equity
Facilities
Finance
  ESSER Funding
  Grants and Research Requests
  Payroll and Benefits
  Purchasing & Warehouse
Human Resources
Preschool
School Nutrition
Special Education
Student Services & Safety
Student Support and Intervention
Transportation
Technology

News
News Room
PowerSchool Data Breach
Emergency Communication

Interface text:
mcps.org/page/purchasing-warehouse
Finish update
https://www.mcps.org/page/calendar

### Berlin-Boylston district website

The district’s Our District menu reveals a Public Records Request link. The speaker says finding information at this district requires a freedom-of-information request.

[6:51](https://www.youtube.com/watch?v=Ki980nV0__0&t=411s) [The district homepage displays a school banner and navigation menu.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f6e24c3076f50361a424b627c334a5168b141a74abede1c13141b879bdaa3d9.webp)

Berlin-Boylston Regional School District

DISTRICT
SCHOOLS
On-Line Payments
School Committee Notices
Staff Links
Our District
Administration
School Committee
Community Organizations
Student Services
Directory

Congratulations!
U.S. Department of Education
National Blue Ribbon Schools
2023
Tahanto Regional High School
Lifetouch

Tahanto Blue Ribbon School Banner
Tahanto Blue Ribbon School Banner hangs in the main entry at Tahanto

Interface text:
bbrsd.org
Finish update
English

[6:55](https://www.youtube.com/watch?v=Ki980nV0__0&t=415s) [School district homepage with the public records request link visible in the district menu.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4188cd1298990a9cfb020d832ed6e56eb8601f7bbb6d374932aeb99ec0fdec9.webp)

Berlin-Boylston Regional School District

DISTRICT
SCHOOLS
On-Line Payments
School Committee Notices
Staff Links
Our District
Administration
School Committee
Community Organizations
Student Services
Directory

Our Goals
Contact Us
DESE District Report Cards
Calendar FY26
Employment Opportunities
Information for Current Employees
CORI Form and Fingerprinting Information
Public Records Request
School Choice
District Calendar

Congratulations!
U.S. Department of Education
National Blue Ribbon Schools
2023
Tahanto Regional High School
Lifetouch

Tahanto Blue Ribbon School Banner
Tahanto Blue Ribbon School Banner hangs in the main entry at Tahanto

Interface text:
bbrsd.org
Finish update
English

### Infrastructure changes slowly

Even if coding agents can generate API endpoints, the speaker argues that institutions change slowly and won’t grant unfettered access to their systems.

[7:58](https://www.youtube.com/watch?v=Ki980nV0__0&t=478s) [Slide stating that infrastructure changes slowly.](https://assets.ai.engineer/sites/aiecode2025/sha256/95d7507abfc124756ce3d1816f763537a508228545bbf256ffdd1a56bb91e04e.webp)

Infrastructure changes slowly

Yutori

### Timberwolves–Nets final score

The NBA page displays the answer: Minnesota beat Brooklyn 125–109 on November 3, 2025.

[9:10](https://www.youtube.com/watch?v=Ki980nV0__0&t=550s) [NBA page shows Minnesota defeating Brooklyn 125–109.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc2adfa0b6c56d3016309bb65aa7733aa54936f437f57c2dd4b4f76dd682789d.webp)

What was the final score of this game between the Minnesota Timberwolves and the Brooklyn Nets?

MIN 125 — FINAL — 109 BKN

Monday, November 3rd, 2025
WOLVES WIN IN BROOKLYN
Barclays Center, Brooklyn, NY

Interface text:
NBA
Games  Schedule  NBA Cup  Watch  News  Stats  Standings  Teams  Players  NBA Play  Fantasy
League Pass  Store  Tickets  Sign In

THU NOV 06, 6:00 PM PT: Clippers, Suns
FRI NOV 07, 4:00 PM PT, East Group B: Celtics, Magic
4:00 PM PT, East Group A: Cavaliers, Wizards
4:30 PM PT, East Group A: Raptors, Hawks
4:30 PM PT, East Group B: Pistons, Nets
4:30 PM PT, West Group C: Rockets, Spurs
5:00 PM PT, East Group C: Hornets, Heat
5:00 PM PT, West Group B: Mavericks, Grizzlies
League Pass  prime video

Watch Recap  Condensed Games

### Initial HTML placeholder

The initial HTML contains an empty score-strip placeholder; the browser fetches the game information asynchronously after the page loads.

[9:21](https://www.youtube.com/watch?v=Ki980nV0__0&t=561s) [Code for an empty score-strip loading placeholder overlays the faded game page.](https://assets.ai.engineer/sites/aiecode2025/sha256/53de44f6285b8d1b8784e7ea3ec1c4dda901aec6697afcda0040c930786160f1.webp)

What was the final score of this game between the Minnesota Timberwolves and the Brooklyn Nets?

<div class="ScoreStripGameSkeleton_gameContainer__bcST2"
     data-testid="score-strip-skeleton">
    <!-- Empty placeholder that shows loading UI -->
</div>

WOLVES WIN IN BROOKLYN

### Scoreboard request and response

The browser fetches the score asynchronously from a scoreboard endpoint. Its JSON response contains the answer missing from the initial HTML.

[9:35](https://www.youtube.com/watch?v=Ki980nV0__0&t=575s) [A network request, response headers, and the beginning of a JSON scoreboard response overlay the prior slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/978b6dc72935e6382d5e28eed5a2f45bb582dd798e06fa3181acf539ee60e57a.webp)

What wa[illegible] the
Minneso[illegible]ets?

<div clas[illegible]bcST2"
    da[illegible]
    <!-- [illegible] -->
</div>

URL: https://cdn.nba.com/static/json/liveData/scoreboard/todaysScoreboard_00.json

Method: GET

Headers:
- sec-ch-ua-platform: "your-operating-system"
- referer: https://www.nba.com/
- user-agent: Mozilla/5.0...

Response Headers:
- cache-control: max-age=10 (10-second cache)
- content-type: text/plain
- etag: "095a50e57d2dcd757c2c6ea63d5c71c5"
- last-modified: Tue, 04 Nov 2025 23:03:16 GMT

Response Body Structure:
{
  "meta": {
    "version": 1,
    "time": "2025-04-01 06:03:15.315",
    "code": 200
  },
  "scoreboard": {
    "gameDate": "2025-11-04",
    "leagueId": "00",
    "games": [
      {
        "gameId": "0022500156",
        "gameStatus": 3,
        "period": 4,

## Full transcript

[0:00](https://www.youtube.com/watch?v=Ki980nV0__0&t=0s) [upbeat music] Um, welcome. Let's get started. So my name is Dhruv Batra.

[0:17](https://www.youtube.com/watch?v=Ki980nV0__0&t=17s) I want to talk to you about an argument. There's an argument online that says, roughly speaking, AI agents will be the drivers of actions on the web, not humans.

[0:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=28s) That there will be more agents clicking buttons, reading things, buying things for us than human eyeballs on the web. Um, you drive one step deeper into the argument, and you ask, how will this happen?

[0:40](https://www.youtube.com/watch?v=Ki980nV0__0&t=40s) Because today, the web is extremely hostile to automated traffic. And the answer usually is, the web will be agentified. Um, you drive one step deeper into what does that mean and how will that happen?

[0:55](https://www.youtube.com/watch?v=Ki980nV0__0&t=55s) And the answer usually is with APIs that agents will call via some set of protocols, and the set of protocols grows over time. It was initially supposed to be MCP servers, then WebMCP, and for payments, there are twenty different competing protocols, and, you know, every single company wants to introduce their own protocol along the way.

[1:15](https://www.youtube.com/watch?v=Ki980nV0__0&t=75s) But okay, there will be APIs is the, is the argument. My claim today and argument, and what I hope to convince you today, is that this last bit is wrong.

[1:24](https://www.youtube.com/watch?v=Ki980nV0__0&t=84s) I think the first two I generally agree with. This last bit that suddenly the web will provide you APIs for accessing things is, is just delusional. And my claim is computer-use agents and computer-use models will agentify the web, not APIs, and more specifically, the long tail of the web.

[1:44](https://www.youtube.com/watch?v=Ki980nV0__0&t=104s) Um, the head of the distribution, the most popular website perhaps will give you the API, but the long tail will not. Um, and the story starts, uh, usually with these sorts of use cases.

[1:56](https://www.youtube.com/watch?v=Ki980nV0__0&t=116s) I, I agree that they also frustrate me. When people show a use case of, "Find me business class flights from Miami to Palma de Mallorca for August," and somehow they expect that behind the scenes there will be a computer-use agent clicking buttons on flights.google.com.

[2:11](https://www.youtube.com/watch?v=Ki980nV0__0&t=131s) I find this example... This is my own example. I, I'm not dunking on anybody else. I find this example bizarre. Why would you possibly do it this way? Aren't you aware that there are already aggregators?

[2:23](https://www.youtube.com/watch?v=Ki980nV0__0&t=143s) You send them a variable, they will send you a JSON. Your LLMs can do tool calling. Why would you click buttons? The purpose of those clicking buttons is to generate this behind the scenes.

[2:34](https://www.youtube.com/watch?v=Ki980nV0__0&t=154s) So we're on the same page there. The next step usually is, okay, let's roll this out to some other case. Um, I want to know, does my favorite restaurant or does this restaurant have gluten-free items on the restaurant's menu?

[2:50](https://www.youtube.com/watch?v=Ki980nV0__0&t=170s) And so I assume that there's going to be a similar endpoint somewhere for that restaurant website or via an aggregator... Oops.

[3:03](https://www.youtube.com/watch?v=Ki980nV0__0&t=183s) Um, that accepts a query that I can filter through, where I can ask for, "Give me your menu items that are gluten free." Um, I want to, for those of you who can already see, I want to rid you of the delusion that such an endpoint exists.

[3:20](https://www.youtube.com/watch?v=Ki980nV0__0&t=200s) Uh, and I want to show you just for the sake of, you know, being on the same page, what do restaurant pages look like. Um, there are three that are on the screen.

[3:29](https://www.youtube.com/watch?v=Ki980nV0__0&t=209s) This is the first one. This is what we imagine a prototypical... This is easy mode. You go to a webpage, there's no API, but at least it's text. It's in German, sure.

[3:40](https://www.youtube.com/watch?v=Ki980nV0__0&t=220s) A different part of the world. Models can speak languages. I see text and I see pricings. Maybe this is easily scrapable. Okay. This is easy mode. The medium mode is a, a, a, a page like this, where of course the thing that you're looking for is slightly hidden away.

[3:59](https://www.youtube.com/watch?v=Ki980nV0__0&t=239s) You press the Menu button, it actually takes you to a PDF. Okay, fine. If your agent has to do this, it needs a PDF reader mode. Fine. Like, you know, I can highlight these.

[4:10](https://www.youtube.com/watch?v=Ki980nV0__0&t=250s) Okay, so this is at least text. No problem. Like, you know, I will dump this into ChatGPT or whatever, and it'll do it. Here's the hard mode. This is what a hard mode website for a restaurant looks like, where it's just pictures of the people who created the restaurant, um, of where they're located, of menu items, and

[4:35](https://www.youtube.com/watch?v=Ki980nV0__0&t=275s) you're like, "Where is the menu?" Uh, does anybody speak Spanish?

[4:40](https://www.youtube.com/watch?v=Ki980nV0__0&t=280s) Nuestra carta. Si? Okay. Let's, let's check. That's our menu. Um, you click there. Oh my God, these are... What am I staring at? Okay, uh, you click on it.

[4:52](https://www.youtube.com/watch?v=Ki980nV0__0&t=292s) Oh, it's a gallery of, of individual... What is this? This is a, this is a pixelated... This is... There's no text here.

[5:02](https://www.youtube.com/watch?v=Ki980nV0__0&t=302s) I am looking at JPEGs embedded in a gallery which contain the menu items. You download this, and you put it into ChatGPT, and it's having str- it's struggling doing OCR on this thing.

[5:15](https://www.youtube.com/watch?v=Ki980nV0__0&t=315s) Okay, so this is what the web looks like. You're telling me this will give you an API endpoint that you can pass gluten-free items, and it'll tell you what that is.

[5:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=328s) That's an example number one. Let's to- take a look at another examples. Uh, this is much more, you know, enterprise focused. Um, um, I am a business. I want to sell to a school district.

[5:42](https://www.youtube.com/watch?v=Ki980nV0__0&t=342s) There's about fifteen to twenty thousand school districts in the US, and I want to ask the question, is this school district where I give you, you know, mydistrict.gov, is this school district procuring a laptop right now, right?

[5:56](https://www.youtube.com/watch?v=Ki980nV0__0&t=356s) It's a simple question, and I want a similar endpoint. I want to say RFP status open and topic laptop.

[6:04](https://www.youtube.com/watch?v=Ki980nV0__0&t=364s) And you think this will exist. Uh, let me tell you what these websites look like. Again, um, this is what easy mode looks like. This is, you know, Ithaca Public Schools.

[6:18](https://www.youtube.com/watch?v=Ki980nV0__0&t=378s) Um, okay, there's some portal. Um, maybe there's, you know, uh, some, you know, enrollment is maybe the wrong place to go. This is what slightly harder mode looks like.

[6:29](https://www.youtube.com/watch?v=Ki980nV0__0&t=389s) Um, this is a different public school. You look at the menu. Under finance, there's purchasing. Under purchasing, there's current solicitations. Um, scanned PDF again. Okay, very nice. Uh, so no text in here, but this is how they tell you about what they have, have purchased.

[6:49](https://www.youtube.com/watch?v=Ki980nV0__0&t=409s) Um, and for ultimate boss level, I want to show you a different school district where in order to find information, you have to file a request for access of infor- or information under Freedom of Information Act.

[7:06](https://www.youtube.com/watch?v=Ki980nV0__0&t=426s) And then what they do is they scan your email that you sent to them,

[7:11](https://www.youtube.com/watch?v=Ki980nV0__0&t=431s) and they will scan that email, put it on a Google Drive, and then attach PDFs associated with your request. These are the people you're telling me will give you an MCP server? [laughs]

[7:25](https://www.youtube.com/watch?v=Ki980nV0__0&t=445s) The, the amount of l- [laughs] delusion here is off the chart. Like,

[7:31](https://www.youtube.com/watch?v=Ki980nV0__0&t=451s) if there was ever a time to, to say go touch grass, like [laughs] I think this is it. Okay. Uh, so this is what...

[7:45](https://www.youtube.com/watch?v=Ki980nV0__0&t=465s) My claim here is that the web, we forget, is massive. It is extremely big. The, the number of websites are out there, uh, somewhat, you know, the active websites are somewhere two hundred million.

[7:57](https://www.youtube.com/watch?v=Ki980nV0__0&t=477s) The total number of websites are sitting in and around a billion. Infrastructure changes very slowly. Um, you can imagine as an engineer, you know, getting unfettered access to software systems and letting your favorite coding agent rip and generate an API endpoint.

[8:16](https://www.youtube.com/watch?v=Ki980nV0__0&t=496s) Even if that technological problem is solved, which I agree seems like on a horizon it should eventually be solved, you're not going to get unfettered access to these institutions, and these institutions change very slowly.

[8:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=508s) There are still places that are faxing each other. You're, you're not going to be able to suddenly change this overnight.

[8:35](https://www.youtube.com/watch?v=Ki980nV0__0&t=515s) Okay. So at this point of time, you might be thinking, "Fine. I'm stuck with the infra as it is. Um, so there's not going to be APIs available, but I have coding agents.

[8:45](https://www.youtube.com/watch?v=Ki980nV0__0&t=525s) Why don't I just throw them at the HTML? There is, after all, if the browser is doing it, it's a piece of code, my coding agent should be able to do it as well."

[8:55](https://www.youtube.com/watch?v=Ki980nV0__0&t=535s) Fair, fair point. I, you know, I also thought this way two years ago. Let me tell you what, what the web actually looks like. Um,

[9:04](https://www.youtube.com/watch?v=Ki980nV0__0&t=544s) if I ask you a question, what was the final score of this game between Minnesota Timberwolves and the Brooklyn Nets? Here's what the webpage looks like. This is a modern website.

[9:14](https://www.youtube.com/watch?v=Ki980nV0__0&t=554s) You know, it's NBA.com. You and I can see the score, 125 versus 109. Okay. Uh, behind the scenes, when you actually load the page and read it, this is what initially gets loaded.

[9:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=568s) There's an empty placeholder initially, and you wait a few hundred milliseconds to a few seconds, depending on your network connection, and your browser makes an asynchronous call later to fetch the information.

[9:41](https://www.youtube.com/watch?v=Ki980nV0__0&t=581s) So the browser makes a call to an endpoint, uh, that it's extracting that information from. That endpoint responds with a JSON that contains the answer. If you just read the HTML when you load it, the answer is not in the HTML, and so your, your chatbot doesn't have access to that either.

[10:00](https://www.youtube.com/watch?v=Ki980nV0__0&t=600s) So okay, you go, "Fine. These are just details. I just have to add some waits and sleeps and sure." Okay, fair enough. I will take you to another example.

[10:09](https://www.youtube.com/watch?v=Ki980nV0__0&t=609s) Um, I ask you the question on a product webpage: Is the 25mm osmium cube in stock or out of stock? You're on a, on this product website. Um, you scroll down.

[10:19](https://www.youtube.com/watch?v=Ki980nV0__0&t=619s) There is a dropdown. Uh, that dropdown is telling you and I as humans, you know, three things are sold out. One thing is in stock, even though it doesn't actually say in stock, like there's no text there that says in stock.

[10:31](https://www.youtube.com/watch?v=Ki980nV0__0&t=631s) But you understand that grayed out means sold out, and, you know, sometimes the sold out won't actually be there as a text. Sometimes it'll just be unclickable grayed out.

[10:40](https://www.youtube.com/watch?v=Ki980nV0__0&t=640s) Okay, surely this information must be in the code somewhere. You go and read the HTML, and it turns out there is an option selector. It actually doesn't say any of the things that I'm seeing on screen.

[10:51](https://www.youtube.com/watch?v=Ki980nV0__0&t=651s) It doesn't say sold out. It doesn't say available. So what's going on? Uh, turns out behind the scenes, uh, the browser makes a call, gets a JSON object, which is the variable, which is product count.

[11:05](https://www.youtube.com/watch?v=Ki980nV0__0&t=665s) That contains a variable called quantity. That quantity is, you know, 10 sometimes, zero sometimes. That's just, you know, how many things can the back end support right now. Some of those quantities are zero, and there's a different rendering script that anytime there's zero, grays it out and makes it unclickable.

[11:24](https://www.youtube.com/watch?v=Ki980nV0__0&t=684s) Fundamentally, what is happening here is this information that you are seeing on screen is not written somewhere as pure text. It is calculated. It is rendered. Um, and for people who work in the browser industry, they understand this.

[11:40](https://www.youtube.com/watch?v=Ki980nV0__0&t=700s) But, you know, often people who are coming from an AI background like me, uh, we didn't al-al-always understand this. The browser is a rendering engine. You are seeing pixels on screen.

[11:52](https://www.youtube.com/watch?v=Ki980nV0__0&t=712s) It's... Think of it as a game engine. Um, and you're asking, "Can I not read the source code of the game and predict exactly what the pixels are going to be?"

[12:00](https://www.youtube.com/watch?v=Ki980nV0__0&t=720s) Well, yes, eventually, but right now you're asking for an exact inversion of that process.

[12:08](https://www.youtube.com/watch?v=Ki980nV0__0&t=728s) Fundamentally, the web [chuckles] was built for human eyes. Pixels are the source of the truth because the consumers of the websites are humans. That is what it was built for.

[12:20](https://www.youtube.com/watch?v=Ki980nV0__0&t=740s) And so there is a implication here that we have to, we have to grapple with, which is machines will need vision to operate those things because the web world was built for, for human consumption.

[12:31](https://www.youtube.com/watch?v=Ki980nV0__0&t=751s) Um, in a way, this is the bitter lesson, uh, for web agents, that the more you end up writing scaffolds around existing websites, the-- it doesn't actually generalize to the long tail of the web.

[12:44](https://www.youtube.com/watch?v=Ki980nV0__0&t=764s) The thing that generalizes is the thing that it was designed for, which is the most general solution, just pixels in.

[12:53](https://www.youtube.com/watch?v=Ki980nV0__0&t=773s) This is what, um, you know, we and some others in the, in the area have been working on. Um, we have a model called Navigator. Um, the first version of the model went out in November last year.

[13:05](https://www.youtube.com/watch?v=Ki980nV0__0&t=785s) The first version acted purely like a human. Screenshot in, button clicks, and scrolls out. Um, I will tell you in a second, that is not where you should settle on, but it is a general solution.

[13:16](https://www.youtube.com/watch?v=Ki980nV0__0&t=796s) It lets you do things like this. Um, I give you an e-commerce website, and I tell you, "There is this discount code. Please go-- The discount code is applicable under certain constraints.

[13:27](https://www.youtube.com/watch?v=Ki980nV0__0&t=807s) Maybe it's only on a product, and it's only on a certain set of dates. Maybe it's only, um, with this minimum cart threshold." I describe that in natural language, and I tell you, "Tell me if this discount code is valid or not."

[13:38](https://www.youtube.com/watch?v=Ki980nV0__0&t=818s) There is no API for this. The reason-- The way to do it is just like a human can. You go to that website, you find the product that is described, you add it to cart, you apply the discount code, and you check whether the claim of twenty-two percent off was met or not.

[13:55](https://www.youtube.com/watch?v=Ki980nV0__0&t=835s) And that is, you know, screenshot in, button click out. This trajectory took, you know, twenty, thirty, forty steps, depending on the sophistication of the task. If you can do it on your browser, uh, this model can accomplish it in principle.

[14:10](https://www.youtube.com/watch?v=Ki980nV0__0&t=850s) In practice, of course, there are accuracy gaps and so on, but in principle, this task is solvable, whereas in a lot of earlier cases, even in principle, that task may not be solvable.

[14:23](https://www.youtube.com/watch?v=Ki980nV0__0&t=863s) So my claim is the web was built for human eyes. Machines will need vision, but of course, they do not need to be limited to human ways. Just because for the long tail you need to have a capability does not mean that is the only way you should do it.

[14:36](https://www.youtube.com/watch?v=Ki980nV0__0&t=876s) Um, here is an example showing that. The next version of the model that we trained can also write JavaScript on demand. So this is a Chrome extension. On the right, you see an action that says, "Execute JS, value default, text select."

[14:50](https://www.youtube.com/watch?v=Ki980nV0__0&t=890s) On the left, you saw that the model filled out multiple form fields simultaneously. The reason why it could do that is because it wrote a little bit of a function.

[14:59](https://www.youtube.com/watch?v=Ki980nV0__0&t=899s) It can read the code when necessary. It can write code when necessary, because the browser, after all, is an engine that can execute code. But it has a s-sort of formal verification system built in.

[15:11](https://www.youtube.com/watch?v=Ki980nV0__0&t=911s) It, it is seeing the screenshot. That is the source of the truth, so it knows whether it succeeded or not. So click buttons when you have to, write code when you have to, and look at the result, uh, through pixels, because that is the, that is the source of co-- of truth.

[15:26](https://www.youtube.com/watch?v=Ki980nV0__0&t=926s) And of course, because these things are, are machines, you can string them into multi-agent systems. So you can have an orchestrator that is launching multiple navigators in parallel, each with a cloud, cloud sandbox instance.

[15:38](https://www.youtube.com/watch?v=Ki980nV0__0&t=938s) They are clicking buttons on multiple websites. So you can accomplish things that would be s-superhuman, because no human would be able to parallelize over that many instances.

[15:49](https://www.youtube.com/watch?v=Ki980nV0__0&t=949s) Um, around here, usually, you know, in this flow of an argument is when people start asking, "Are computer-use models actually good enough, uh, for these tasks?" Um, there's actually a perception online, uh, that, um, it's not clear whether progress on computer use has been fast, and there are questions about why has progress been slow.

[16:08](https://www.youtube.com/watch?v=Ki980nV0__0&t=968s) That's not the reality I'm seeing, and that's not the reality that the numbers back up. Um, this is a popular benchmark, um, Online-Mind2Web. No benchmark is perfect. The point isn't, uh, that this is the, the right solution.

[16:24](https://www.youtube.com/watch?v=Ki980nV0__0&t=984s) But on the X-axis are re-- are release times of different models, on the Y-axis is performance, which is human evaled on, on this. So a human went in, looked at the trajectory, decided whether it was correct or not.

[16:36](https://www.youtube.com/watch?v=Ki980nV0__0&t=996s) And basically, this particular version of the benchmark is saturated. The, the last-- the w-- model that we just released, Navigator n1.5, is sitting at ninety-seven percent human eval. Eight trajectories out of three hundred are incorrect.

[16:48](https://www.youtube.com/watch?v=Ki980nV0__0&t=1008s) At this point of time, you should just retire the benchmark, build something harder. There's about, you know, thirty to fifty steps of interaction that are happening. The next step is to go for something larger.

[16:57](https://www.youtube.com/watch?v=Ki980nV0__0&t=1017s) So at least in numbers, what I'm seeing, we're seeing steady progress in computer-use agents becoming-- being able to do more and more things. And this is the model that I showed, pixels in, button clicks, and code out.

[17:11](https://www.youtube.com/watch?v=Ki980nV0__0&t=1031s) Around this time usually is when people start asking questions. "Okay, so they're getting better, but aren't these things slow? 'Cause after all, you're looking at a screen, you're gen-- you're clicking a button, there are lots of buttons to click, and, you know, these things are expensive if you're running them for hundreds of, of things."

[17:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=1048s) Um, that claim, I think, is largely true. There is some truth to it. Um, but I think people forget how, how much you can optimize these things out. Um, so this is our results compared to the frontier models.

[17:43](https://www.youtube.com/watch?v=Ki980nV0__0&t=1063s) Opus 4.7, GPT 5.5 on a couple of browser use benchmarks. Um, we're slightly better, but I think that's within statistical, uh, noise in terms of accuracy. That improvement I wouldn't beat the drum on.

[17:57](https://www.youtube.com/watch?v=Ki980nV0__0&t=1077s) What I would emphasize is latency per step and cost per task. In terms of latency, this is a smaller footprint model. That's why it's a lot faster than some of the trillion parameter plus models, and there are corresponding cost savings.

[18:11](https://www.youtube.com/watch?v=Ki980nV0__0&t=1091s) So if you have something like on these datasets, something like twenty, thirty steps of interaction, you're looking at eighty cents per task versus two dollar thirty, and that makes a big difference.

[18:21](https://www.youtube.com/watch?v=Ki980nV0__0&t=1101s) And so the models are getting cheaper, uh, in that sense, and you can launch them at, at scale.

[18:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=1108s) Um, so hopefully at a, at a high level, I've sort of convinced you that there is something off with this argument. You know, this is where, where what my goal was.

[18:38](https://www.youtube.com/watch?v=Ki980nV0__0&t=1118s) This is where I started. AI agents are going to be the dri- primary drivers of action. Um, that seems sort of an uncontestable statement because, you know, the underlying intelligence of the models is becoming larger and larger.

[18:53](https://www.youtube.com/watch?v=Ki980nV0__0&t=1133s) Um, there is a certain gain of productivity and efficiency and, you know, just ease of life that you get, so it makes sense. But I think this hypothesis that suddenly overnight, thirty years of infrastructure that was built layer upon layer for human consumption will in what, two, five, ten years be reinvented is, I think, a, a fantasy.

[19:17](https://www.youtube.com/watch?v=Ki980nV0__0&t=1157s) Um, but, but this is ultimately what you want, right? Ultimately, you want an endpoint that, um, you know, some higher level entity can go to and I say, "I want you to do X."

[19:30](https://www.youtube.com/watch?v=Ki980nV0__0&t=1170s) There is some task, maybe I give you, uh, that task description in natural language. Maybe I have some programmatic description with parameters, and there's, you know, some endpoint I should be able to do it.

[19:41](https://www.youtube.com/watch?v=Ki980nV0__0&t=1181s) So how will the web be agentified? My answer is just extrapolate this trend. This is where we stand today. Accuracies are getting higher, benchmarks are falling, latencies are getting smaller, costs are falling.

[19:55](https://www.youtube.com/watch?v=Ki980nV0__0&t=1195s) And that trend, I think, will continue. And the way we will agentify the web, [scoffs] and in, in, in some sense, this is depressing, but this is true, we will just pile on another layer of mess on top of the mess that the web is.

[20:10](https://www.youtube.com/watch?v=Ki980nV0__0&t=1210s) Which i- which is to say, you know, you will issue a task. Behind the scenes, there will be hundreds of browsers that are pretending, uh, and clicking buttons just like humans would.

[20:21](https://www.youtube.com/watch?v=Ki980nV0__0&t=1221s) Um, and they will give you a result in a structured format. It will cost less than a penny. It will run in your browser sometimes. It will, you know, run in less than a hundred milliseconds.

[20:33](https://www.youtube.com/watch?v=Ki980nV0__0&t=1233s) And at some point you will say, "Yeah, that's an API. Like, why do I care?" And that's how we'll, we'll agentify the web. So thank you. [clapping] [outro jingle]
