Browser Agents Don't Need Better Models. They Need Better Eyes. - Kushan Raj, ARK

Browser agents should be an obvious convenience, yet Raj opens by admitting that he rarely uses them himself. His first demonstration explains the hesitation: a browser challenge requires 30 sequential steps, but the reference agent spends roughly 10–20 seconds just clicking Start. At that pace, every observation, click, and recovery attempt compounds across the remaining workflow. 0:30

Raj then runs his prototype on the same challenge. It moves faster while using what he describes as a cheaper model. This motivates his central hypothesis: the model may already be capable enough, but the browser environment makes it spend too much time reconstructing what happened. In the slower run, the agent tries to click, fails to understand the result, and begins debugging through more browser observations. 0:48

A better runtime, in Raj’s framing, must let the model plan longer sequences, locate the step where execution failed, and choose its next interaction using a useful representation of the page. He briefly introduces a compressed website representation here, but the recording postpones its concrete token comparison and state-tracking mechanism until after the apparent closing. 1:01

Aadhaar download. Raj’s first practical example looks trivial: the desired download button is visible in a screenshot. Yet the demonstrated Claude run stalls after reaching it. From about 46 seconds into that demonstration, it takes another screenshot, scrolls without an evident reason, and continues observing the page. Raj says the full attempt lasts two minutes. Seeing the target once was not enough to establish whether the click worked or what state followed. 1:30

Raj’s prototype completes the corresponding workflow almost immediately after starting. The contrast suggests that browser speed depends on more than visual recognition: the runtime must make action outcomes legible enough that the model can continue instead of repeatedly inspecting the page. This remains a demonstration rather than a controlled evaluation; the talk provides no repeated trials, matched settings, success rates, or latency breakdown. 1:56

Kannada booking site. The next example adds unfamiliar language and form state. Raj wants to book a Sunday trek through a Kannada-language site, and says he is not fluent in Kannada. In the Claude run, the agent cannot select a date and eventually stalls. His prototype selects the control, enters the date, and completes the demonstrated step. 2:08

A date picker exposes the difference between locating an element and understanding an interaction. The agent must know whether a calendar or field became active, whether the selected date persisted, and whether the page accepted the input. Raj’s demonstrations suggest that this transition information is where conventional screenshot-driven behavior becomes brittle, although the talk does not test the approach against authentication challenges, CAPTCHAs, anti-bot defenses, or irreversible actions.

Raj is considering open-sourcing the project because he does not regard the implementation itself as highly defensible. The product could instead expose the execution command as an API: provide a URL and an intent, let the runtime perform the workflow, and return the result. He also considers offering the same capability through a website or plugin. 2:39

Those delivery options all wrap the same browser runtime, but the talk does not address the operational controls an execution service would need: credential handling, scoped authorization, confirmation before consequential actions, and evidence that the requested task actually completed. Raj closes this portion with the broader goal of making browser agents faster, cheaper, and more reliable enough for routine use. 3:09

After the apparent closing, the recording returns to the core technical mechanism. Raj compares three representations of one page: a full DOM of roughly 20,000 tokens, a screenshot of about 1,100 tokens, and his Markdown representation of about 1,800 tokens. The Markdown is intended to describe the entire page, while one screenshot shows only the current viewport. Token count and coverage both matter: cheaper whole-page context gives the model a navigational map without forcing it to rediscover off-screen content. 3:24

Raj proposes using the Markdown together with a screenshot. The two representations serve different purposes: Markdown supplies compact page-wide structure, while the screenshot retains rendered visual context. The recording does not explain the extraction algorithm or measure what the compression discards. Layout-dependent meaning, canvas content, nested controls, and other visual state could therefore remain difficult to represent faithfully. 3:34

The runtime also compares browser state across actions. It can report which elements appeared, which disappeared, and whether something blocking a target has been removed. If a click produces no expected change, the model receives that failed transition as feedback instead of having to infer it from another raw screenshot. 3:54

This turns browser operation into a closed loop: observe a compact page state, plan a sequence, execute an interaction, compare the new state with the old one, and either continue or revise the plan. “Better eyes” therefore means more than stronger image recognition. It means showing the model enough of the page and explaining what its actions changed. The recording ends before specifying the action protocol or presenting systematic evaluations, so the prototype supports a promising interface thesis rather than proving that model improvements no longer matter.