# Browser Agents Don't Need Better Models. They Need Better Eyes. - Kushan Raj, Sarvam AI

Kushan Raj, Sarvam AI

[Talk page](https://ai.engineer/talks/JnubYCYunk8-browser-agents-dont-need-better-models-they) · [Video](https://www.youtube.com/watch?v=JnubYCYunk8)

## Browser Agents Don’t Need Better Models. They Need Better Eyes.

Kushan Raj argues that browser agents often waste capable models on fragmented page views and ambiguous actions. His prototype instead gives the model compact whole-page context and explicit feedback about what changed after each interaction.

### The first click is already too slow

[0:00](https://www.youtube.com/watch?v=JnubYCYunk8&t=0s)

Browser agents should be an obvious convenience, yet Raj opens by admitting that he rarely uses them himself. His first demonstration explains the hesitation: a browser challenge requires **30 sequential steps**, but the reference agent spends roughly **10–20 seconds** just clicking Start. At that pace, every observation, click, and recovery attempt compounds across the remaining workflow. [0:30](https://www.youtube.com/watch?v=JnubYCYunk8&t=30s)

![Captures Raj’s faster prototype running on the same challenge for a direct visual contrast with the reference agent.](https://ai.engineer/api/talk-essay-media/c967e9912b7bf344341a7d04cb556aef741915a61bf9626efc718334e9fdcb64)

Captures Raj’s faster prototype running on the same challenge for a direct visual contrast with the reference agent.

Raj then runs his prototype on the same challenge. It moves faster while using what he describes as a cheaper model. This motivates his central hypothesis: the model may already be capable enough, but the browser environment makes it spend too much time reconstructing what happened. In the slower run, the agent tries to click, fails to understand the result, and begins debugging through more browser observations. [0:48](https://www.youtube.com/watch?v=JnubYCYunk8&t=48s)

A better runtime, in Raj’s framing, must let the model plan longer sequences, locate the step where execution failed, and choose its next interaction using a useful representation of the page. He briefly introduces a compressed website representation here, but the recording postpones its concrete token comparison and state-tracking mechanism until after the apparent closing. [1:01](https://www.youtube.com/watch?v=JnubYCYunk8&t=61s)

### Visible controls are not enough

[1:00](https://www.youtube.com/watch?v=JnubYCYunk8&t=60s)

**Aadhaar download.** Raj’s first practical example looks trivial: the desired download button is visible in a screenshot. Yet the demonstrated Claude run stalls after reaching it. From about 46 seconds into that demonstration, it takes another screenshot, scrolls without an evident reason, and continues observing the page. Raj says the full attempt lasts two minutes. Seeing the target once was not enough to establish whether the click worked or what state followed. [1:30](https://www.youtube.com/watch?v=JnubYCYunk8&t=90s)

![Shows the date entered and the demonstrated step completed, providing the outcome of the booking comparison.](https://ai.engineer/api/talk-essay-media/9f106cc62f64abff237de426d635db74cb37dfb787653e5d38f14437cfb3422b)

Shows the date entered and the demonstrated step completed, providing the outcome of the booking comparison.

Raj’s prototype completes the corresponding workflow almost immediately after starting. The contrast suggests that browser speed depends on more than visual recognition: the runtime must make action outcomes legible enough that the model can continue instead of repeatedly inspecting the page. This remains a demonstration rather than a controlled evaluation; the talk provides no repeated trials, matched settings, success rates, or latency breakdown. [1:56](https://www.youtube.com/watch?v=JnubYCYunk8&t=116s)

**Kannada booking site.** The next example adds unfamiliar language and form state. Raj wants to book a Sunday trek through a Kannada-language site, and says he is not fluent in Kannada. In the Claude run, the agent cannot select a date and eventually stalls. His prototype selects the control, enters the date, and completes the demonstrated step. [2:08](https://www.youtube.com/watch?v=JnubYCYunk8&t=128s) [2:35](https://www.youtube.com/watch?v=JnubYCYunk8&t=155s)

A date picker exposes the difference between locating an element and understanding an interaction. The agent must know whether a calendar or field became active, whether the selected date persisted, and whether the page accepted the input. Raj’s demonstrations suggest that this transition information is where conventional screenshot-driven behavior becomes brittle, although the talk does not test the approach against authentication challenges, CAPTCHAs, anti-bot defenses, or irreversible actions.

### Package the runtime as an API

[2:39](https://www.youtube.com/watch?v=JnubYCYunk8&t=159s)

Raj is considering open-sourcing the project because he does not regard the implementation itself as highly defensible. The product could instead expose the execution command as an API: provide a URL and an intent, let the runtime perform the workflow, and return the result. He also considers offering the same capability through a website or plugin. [2:39](https://www.youtube.com/watch?v=JnubYCYunk8&t=159s)

Those delivery options all wrap the same browser runtime, but the talk does not address the operational controls an execution service would need: credential handling, scoped authorization, confirmation before consequential actions, and evidence that the requested task actually completed. Raj closes this portion with the broader goal of making browser agents faster, cheaper, and more reliable enough for routine use. [3:09](https://www.youtube.com/watch?v=JnubYCYunk8&t=189s)

### Compress the page, then report the delta

[3:24](https://www.youtube.com/watch?v=JnubYCYunk8&t=204s)

After the apparent closing, the recording returns to the core technical mechanism. Raj compares three representations of one page: a full DOM of roughly **20,000 tokens**, a screenshot of about **1,100 tokens**, and his Markdown representation of about **1,800 tokens**. The Markdown is intended to describe the entire page, while one screenshot shows only the current viewport. Token count and coverage both matter: cheaper whole-page context gives the model a navigational map without forcing it to rediscover off-screen content. [3:24](https://www.youtube.com/watch?v=JnubYCYunk8&t=204s)

![Illustrates end-to-end page tracking and feedback when a click fails or a blocking element is removed.](https://ai.engineer/api/talk-essay-media/2c96190706bb00568651a8a25a17bfeb9560dbfd3b5632dd9fe8041d0f1af283)

Illustrates end-to-end page tracking and feedback when a click fails or a blocking element is removed.

Raj proposes using the Markdown together with a screenshot. The two representations serve different purposes: Markdown supplies compact page-wide structure, while the screenshot retains rendered visual context. The recording does not explain the extraction algorithm or measure what the compression discards. Layout-dependent meaning, canvas content, nested controls, and other visual state could therefore remain difficult to represent faithfully. [3:34](https://www.youtube.com/watch?v=JnubYCYunk8&t=214s)

The runtime also compares browser state across actions. It can report which elements appeared, which disappeared, and whether something blocking a target has been removed. If a click produces no expected change, the model receives that failed transition as feedback instead of having to infer it from another raw screenshot. [3:54](https://www.youtube.com/watch?v=JnubYCYunk8&t=234s)

This turns browser operation into a closed loop: observe a compact page state, plan a sequence, execute an interaction, compare the new state with the old one, and either continue or revise the plan. “Better eyes” therefore means more than stronger image recognition. It means showing the model enough of the page and explaining what its actions changed. The recording ends before specifying the action protocol or presenting systematic evaluations, so the prototype supports a promising interface thesis rather than proving that model improvements no longer matter.

The browser-state feedback loop
Whole-page structure and rendered context inform an action plan; state deltas then confirm whether execution worked or needs revision.
- Tracked browser page — The runtime maintains end-to-end page state across interactions.
- Markdown plus screenshot — Markdown supplies compact whole-page structure; the screenshot preserves rendered context.
- Multi-step plan — The model reasons over the compact state and constructs a longer sequence of tasks.
- Browser action — The runtime executes a click, selection, or form input.
- State-change feedback — The runtime reports appeared, disappeared, unblocked, or unchanged elements.
- Tracked browser page → Markdown plus screenshot (compress and render)
- Markdown plus screenshot → Multi-step plan (supply page context)
- Multi-step plan → Browser action (execute next step)
- Browser action → State-change feedback (compare outcome)
- State-change feedback → Multi-step plan (confirm or revise)
- State-change feedback → Tracked browser page (update tracked state)

## Key takeaways

- A capable model can still waste time when its browser interface supplies fragmented observations and weak confirmation that actions succeeded.
- For the demonstrated page, Raj compares a roughly 20,000-token DOM with an approximately 1,100-token screenshot and an approximately 1,800-token whole-page Markdown representation.
- Markdown and screenshots are complementary: one provides compact page-wide structure, while the other preserves rendered visual context.
- State deltas—what appeared, disappeared, became unblocked, or failed to change—give the model a clearer recovery signal than repeated screenshots alone.
- The examples support an interface-design hypothesis, not a general performance result: the talk provides no controlled success-rate, latency, cost, safety, or information-loss evaluation.

## Related talks

- [Bringing agents onto the world wide web](https://ai.engineer/talks/GqoNrUz8hEU-bringing-agents-onto-world-wide-web) — Extends the harness-first argument with browser infrastructure, network-request interception, reusable skills, and agent-oriented browser tools.
- [Harnesses in AI: A Deep Dive](https://ai.engineer/talks/C_GG5g38vLU-ai-agent-harnesses) — Examines how deterministic infrastructure, browser-state handling, and verification can improve a model-driven agent loop.
- [The Current State of Browser Agents](https://ai.engineer/talks/Djv8Sp11UjI-current-state-browser-agents) — Provides a broader comparison of screenshot- and DOM-based agents, including authentication, anti-bot defenses, latency, and browser-infrastructure constraints.

## Slides

### Browser challenge popups

Claude faces overlapping popups in step 1 of a 30-step browser challenge. Some warn that their X buttons are fake; Raj says the agent took 10–20 seconds just to click Start.

[0:24](https://www.youtube.com/watch?v=JnubYCYunk8&t=24s) [A browser agent faces several overlapping challenge popups.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0acbc6f8369748065046c789ca79ed47378f5a93e2e12c6b884051e0907bf5e.webp)

Step 1 of 30 - Browser Navigation Challenge

Challeng[illegible]
You are on st[illegible]
Wait for the [illegible]
UUB[illegible]
Code revealed!

Code revealed: UUB[illegible]

Scroll Down to Fi[illegible]
Section 4
This is filler content. Keep scrolling to find the navigation button.

Alert!
This is a popup message. The close button is fake! Look for another way to close.
Dismiss

[illegible]
Dismiss

Click here for amazing deals!
This is a popup message. Click X to close.
Close

Section 2: Importa[illegible]
Section 3
Section 4

Alert!
This is a popup message. The close button is fake! Look for another way to close.
Dismiss

War[illegible]
This i[illegible]
is fak[illegible]
Dismiss

Please select an option above
Submit & Continue

Interface text:
localhost:5173/step1?version=2
‘Claude’ started debugging this browser
Cancel
Finish update
Claude
Opus 4.6

You are a browser automation agent. Complete the task below by using computer tools to interact with the browser.

## Goal
“Complete all steps of the browser challenge. Quiz hint: the correct answer always matches 'Option [Letter] - Correct Choice'.”

## Guidelines
Closing misleading popup windows system...
Take screenshot
Click
Wallet empty · Add credits
Reply to Claude
Stop Claude

*This Claude with Chrom[illegible]
trying to navigate a browser cha[illegible]

### Challenge navigation and code entry

The challenge surrounds a code field with many similar navigation buttons. Raj argues that agents need an environment where they can plan clicks and diagnose failures.

[1:05](https://www.youtube.com/watch?v=JnubYCYunk8&t=65s) [A challenge page shows a code entry field and multiple navigation buttons.](https://assets.ai.engineer/sites/aiecode2025/sha256/26919bcddde0f52c45db40b2fec1f87e0e3a918612dac3266b46d07579d3a3df.webp)

Section 29
This is filler content. Keep scrolling to find the navigation button.

Enter Code to Proceed to Step 2:
Enter 6-character
Submit Code

Section 32
This is filler content. Keep scrolling to find the navigation button.

Move On
Continue Journey
Click Here
Next Section
Move On
Keep Going
Next Page
Continue
Keep Going
Next Page
Go Forward

Section [illegible]
This is filler content. Keep scrolling to find the navigation button.

Section 34
This is filler content. Keep scrolling to find the navigation button.

Section 35
This is filler content. Keep scrolling to find the navigation button.

Interface text:
~ aid-671 aria-checked: 'false' -> 'true' <button>
~ aid-673 disabled: '' -> None <button> "Submit & Continue"
localhost:5173/step3?version=1
[illegible] planned, 5 executed
[illegible] RSLT: [1] clicked aid=aid-660 ("Close") [2] clicked [illegible]
[illegible] clicked aid=aid-673 ("Submit & Continue") [4] slept 100[illegible]
Section 51

### Page representation

The page representation records which overlays block controls and content, including a disabled code form. Raj is considering open-sourcing the project and exposing the command through an API.

[2:45](https://www.youtube.com/watch?v=JnubYCYunk8&t=165s) [An editor displays a structured page representation with overlay and content entries.](https://assets.ai.engineer/sites/aiecode2025/sha256/94470bdc56bf8caeb382cccebc0bbc6ec28eeedf74ea0aaf3039f4efa69fe8c8.webp)

# Page Representation (V5)

## Overlays
radio[button] `aid-66` "Wrong option 2" [blocked by aid-75]
radio[button] `aid-67` "Not this one" [blocked by aid-75]
radio[button] `aid-68` "Incorrect choice" [blocked by aid-75]
radio[button] `aid-69` "The right choice" [blocked by aid-75]
radio[button] `aid-59` "Option A" [blocked by aid-75]
radio[button] `aid-60` "Option B - Correct Choice" [blocked by aid-75]
radio[button] `aid-61` "Option B" [blocked by aid-75]
radio[button] `aid-62` "Wrong option 3" [blocked by aid-75]
radio[button] `aid-63` "Correct answer" [blocked by aid-75]
button `aid-70` "Submit & Continue" [disabled, cursor:not-allowed] [blocked by aid-49, aid-43, aid-75]

## Content
`aid-81` z=100 "Challenge Step 1" [blocked by aid-75, aid-58]
  "You are on step 1. Complete the challenges and enter the code to proceed."
  buttons(15): "Next Section"*, "Next Section"*, "Proceed"*, "Keep Going"*, "Move On"*, "Proceed Forward"*, "Proceed Forward"*, "Advance"*, "Continue Reading"*, "Next Page"*, "Proceed Forward"*, "Continue Journey"*, "Keep Going"*, "Proceed"*, "Continue Reading"*
  `aid-82` z=10005 [blocked by aid-75, aid-58]
    "Click to Reveal: Click the button below to reveal the code"
    button `aid-5` "Reveal Code" [blocked by aid-75, aid-58]
  `aid-78` z=200 "Content Block 3 Loaded!" [animated] [blocked by aid-75, aid-58]
    "This content appeared 2500ms after page load."
  `aid-83` z=10002 "Enter Code to Proceed to Step 2:" blocks [aid-48, aid-71] [blocked by aid-75, aid-58]
    `aid-21` <form> [blocked by aid-75, aid-58]
      form `form-1` [blocked by aid-75]
        textbox[text] `aid-22` "Enter 6-character code"
        button[submit] `aid-23` "Submit Code" [disabled, cursor:not-allowed]
  `aid-71` z=200 "Content Block 2 Loaded!" [animated] [blocked by aid-75, aid-58, aid-83]
    "This content appeared 1500ms after page load."
  `aid-48` z=200 "Content Block 1 Loaded!" [animated] [blocked by aid-75, aid-58, aid-83]
    "This content appeared 500ms after page load."
  `aid-84` z=10003 blocks [aid-85] [blocked by aid-75, aid-58]
    buttons(14): "Continue"*, "Proceed"*, "Proceed Forward"*, "Click Here"*, "Advance"*, "Go Forward"*, "Keep Going"*, "Move On"*, "Continue Journey"*, "Next"*, "Next Section"*, "Next Page"*, "Continue Reading"*, "Next Step"*

Interface text:
browser-challenge
Upgrade to Pro
BROWSER-CHALLENGE
.pytest_cache
.ruff_cache
.venv
crawler
demo-video
claude-aadhar.mov
Claude-browser challenge.mov
claude-trekking.mov
feedback-action-log.md
feedback-blocker-recovery.md
feedback-mutations.md
our-agent.mov
ours-aadhar.mov
ours-trekking.mov
Screenshot 2026-04-03 at 12.32.23 A...
v5-metacritic-real-world.md
v5-step1-full-unedited.md
v5-step1-overlays-blocking.md
v5-step9-deep-blocking.md
v5-step9-full-unedited.md
Preview
Markdown
OUTLINE
TIMELINE
main
Launchpad
Git Graph
Pull Request #15
Ln 57, Col 26
Spaces: 2
UTF-8
LF
Prettier

### Running the trek agent

A terminal command gives the agent a website and a Savandurga trek-booking goal. Raj proposes exposing this URL-and-intent workflow as an API, website, or plugin.

[2:58](https://www.youtube.com/watch?v=JnubYCYunk8&t=178s) [A terminal shows a command configuring an agent for a trek-booking task.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b27657237b1a47c74a0661cc65690b07669b063a975617cc1cf05503081f135.webp)

~/browser-challenge (main) $ uv run -m crawler.main --url "https://aranyavihaara.karnataka.gov.in/lang/en" --goal "Book a trek: District Ramanagara, Trek Savandurga, Date 07-04-2026. Check availability and proceed with booking." --agent-mode raw --raw-model cerebras/gpt-oss-120b --headed --max-pages 5

Interface text:
Last login: Fri Apr 3 16:01:10 on ttys050
You have new mail.
ssh-agent is already running

### Implementation Overview

The talk moves into an implementation overview.

[3:20](https://www.youtube.com/watch?v=JnubYCYunk8&t=200s) [Implementation Overview title on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f9668f85412e5520c7f49db5ccb403f4b7653cced6fad7d36c77685a2a69204.webp)

Implementation Overview

### Page Representation

The demo condenses browser data into Markdown, including blocked elements and form state. Raj estimates roughly 1,800 tokens for this page’s Markdown versus 20,000 for its full DOM and 1,100 for one screenshot.

[3:27](https://www.youtube.com/watch?v=JnubYCYunk8&t=207s) [A markdown representation of a browser challenge, with an explanation of its browser data sources.](https://assets.ai.engineer/sites/aiecode2025/sha256/48deb8ec24f66b1826d36a6ba6ecaf60cac6bf7d9cbf5cb9deafc46b8baac975.webp)

The markdown essentially is constructed from DOM tree,
AX Trees and other native Browser tools/APIs
to condense a webpage into a simple representation

aid-81 z=100 "Challenge Step 1" [illegible]
"You are on step 1. Complete the challe[illegible]"
buttons(15): "Next Section", [illegible]
"Advance", "Continue Reading", [illegible]
Reading"*

aid-82 z=10005 [blocked by aid-75, [illegible]
"Click to Reveal: Click the button below to reveal the code"
button aid-5 "Reveal Code" [blocked by aid-75, aid-58]
aid-78 z=200 "Content Block 3 Loaded!" [animated] [blocked by aid-75, aid-58]
"This content appeared 2500ms after page load."
aid-83 z=10002 "Enter Code to Proceed to Step 2:" blocks [aid-48, aid-71] [blocked by aid-75, aid-58]
  aid-21

[blocked by aid-75, aid-58]
  form form-1 [blocked by aid-75]
    textbox[text] aid-22 "Enter 6-character code"
    button[submit] aid-23 "Submit Code" [disabled, cursor:not-allowed]
aid-71 z=200 "Content Block 2 Loaded!" [animated] [blocked by aid-75, aid-58, aid-83]
"This content appeared 1500ms after page load."
aid-48 z=200 "Content Block 1 Loaded!" [animated] [blocked by aid-75, aid-58, aid-83]
"This content appeared 500ms after page load."
aid-84 z=10003 blocks [aid-85] [blocked by aid-75, aid-58]
buttons(14): "Continue", "Proceed", "Proceed Forward", "Click Here", "Advance", "Go Forward", "Keep Going", "Move On", "Continue Journey", "Next", "Next Section", "Next Page", "Continue Reading", "Next Step"
aid-47 z=435 [animated] [blocked by aid-75, aid-58, aid-43, aid-49]
"Button!"
aid-53 z=439 [animated] [blocked by aid-75, aid-58]
"Try This!"
aid-85 [animated] [blocked by aid-75, aid-58, aid-84]
"Moving!"

Resources: extraction_json, full_dom

Interface text:
browser-challenge
Upgrade to Pro
demo-video > v5-step1-full-unedited.md
.pytest_cache
.ruff_cache
.venv
crawler
demo-video
claude-aadhar.mov
Claude-browser challenge.mov
claude-trekking.mov
feedback-action-log.md
feedback-blocker-recovery.md
feedback-mutations.md
our-agent.mov
ours-aadhar.mov
ours-trekking.mov
Screenshot 2026-04-03 at 12.32.23 A...
v5-metacritic-real-world.md
v5-step1-full-unedited.md
v5-step1-overlays-blocking.md
v5-step9-deep-blocking.md
v5-step9-full-unedited.md
OUTLINE
TIMELINE
main
Launchpad
Git Graph
Pull Request #15
Screen Reader Optimized

### DOM Mutation Tracking

After cookie consent is dismissed, the mutation feed records 200 changes, including new overlays, removed consent controls and shuffled button text. The agent can use these updates to reason from the current page state.

[3:49](https://www.youtube.com/watch?v=JnubYCYunk8&t=229s) [A mutation log shows elements added and removed and button text changed after cookie consent is dismissed.](https://assets.ai.engineer/sites/aiecode2025/sha256/1da98dd877ec4eda175e7bbe1e8028beb78e4d6bdeed4a088155a914181fec9a.webp)

System Feedback: Real-Time DOM Mutation Tracking

1. After Dismissing Cookie Consent — 200 Mutations in One Step

[MUTATIONS after step 2] (200 new)
  NEW (12):
    + aid-48 <div> "Content Block 1 Loaded! This content appeared 500ms after page load."
    + aid-49 <div> "Warning! This is a popup message. Click X to close."
    + aid-55 <div> "Click here for amazing deals! The close button is fake!"
    + aid-58 <div> "Please Select an Option — scrollable modal with radio buttons"
    + aid-75 <div> "Overlay Notice — close button below is fake! Look for the real one."
    ... 7 more new elements ...
  REMOVED (4):
    - aid-40 <div> "Cookie Consent — We use cookies..."
    - aid-41 <button> "Accept"
    - aid-42 <button> "Decline"
  TEXT_CHANGED (184):
    T aid-7  text: 'Continue' -> 'Continue Reading'    ← button text shuffled
    T aid-8  text: 'Continue' -> 'Proceed'
    T aid-9  text: 'Next Step' -> 'Move On'
    T aid-11 text: 'Go Forward' -> 'Click Here'
    ... and 174 more

The page deliberately shuffles button text on every mutation to confuse agents.
Our system captures every text change so the agent reasons from current state, not stale memory.

2. State Change Detection — Radio Button + Submit Enable

[MUTATIONS after step 9] (14 new)
  NEW (1):
    + obs-3 <span>                              ← checkmark indicator appeared
  CHANGED (2):
    ~ aid-63 aria-checked: 'false' -> 'true'    ← radio now selected

Interface text:
browser-challenge
Upgrade to Pro
BROWSER-CHALLENGE
.pytest_cache
.ruff_cache
.venv
crawler
demo-video
claude-aadhar.mov
Claude-browser challenge.mov
claude-trekking.mov
feedback-action-log.md
feedback-blocker-recovery.md
feedback-mutations.md
our-agent.mov
ours-aadhar.mov
ours-trekking.mov
Screenshot 2026-04-03 at 12.32.23 A...
v5-metacritic-real-world.md
v5-step1-full-unedited.md
v5-step1-overlays-blocking.md
v5-step9-deep-blocking.md
v5-step9-full-unedited.md
demo-video > feedback-mutations.md
Preview
Markdown
OUTLINE
TIMELINE
main
Launchpad
Git Graph
Pull Request #15
Screen Reader Optimized

### Mutation Feedback

The mutation feed reports a revealed code, a wrong-button error, and removal of a quiz overlay. After submission, 36 removed elements signal that the modal and its contents are gone.

[3:57](https://www.youtube.com/watch?v=JnubYCYunk8&t=237s) [Mutation examples show a challenge code being revealed, an error appearing, and an overlay being removed.](https://assets.ai.engineer/sites/aiecode2025/sha256/8340231c51582ac30f24a7fef67217f5073284b6f464bd30b605aafd64585713.webp)

[MUTATIONS after step 11] (3 new)
  NEW (1):
    + obs-4 <div> "PN5MRC"                         ← the 6-character code!
  CHANGED (1):
    ~ aid-5 disabled: None -> ''                    ← Reveal Code button now disabled
  TEXT_CHANGED (1):
    T aid-5 text: 'Reveal Code' -> 'Code Revealed'

Agent reads the code directly from the mutation feed. No re-extraction needed.

4. Wrong Button Feedback — Instant Error Signal

[MUTATIONS after step 13] (17 new)
  NEW (1):
    + aid-80 <div> "Wrong Button! Try Again!"         ← error toast appeared
  CHANGED (1):
    ~ aid-23 disabled: None -> ''                    ← Submit re-disabled

Agent sees the error toast in mutations, knows to try a different approach.

5. Overlay Removed — 36 Elements Vanish

[MUTATIONS after step 10] (51 new)
  REMOVED (36):
    - aid-58 <div> "Please Select an Option — scrollable modal"
    - aid-88 <div> "Please Select an Option..."
    - aid-91 <div> "Section 1: Introduction Lorem ipsum..."
    - aid-93 <div> "Important: Select Your Choice..."
    ... and 26 more

After quiz submission, the entire modal and all its children are removed.
Agent's next extraction will show a clean page.

Interface text:
browser-challenge
Upgrade to Pro
BROWSER-CHALLENGE
.pytest_cache
.ruff_cache
.venv
crawler
demo-video
claude-aadhar.mov
Claude-browser challenge.mov
claude-trekking.mov
feedback-action-log.md
feedback-blocker-recovery.md
feedback-mutations.md
our-agent.mov
ours-aadhar.mov
ours-trekking.mov
Screenshot 2026-04-03 at 12.32.23 A...
v5-metacritic-real-world.md
v5-step1-full-unedited.md
v5-step1-overlays-blocking.md
v5-step9-deep-blocking.md
v5-step9-full-unedited.md
Preview
Markdown
OUTLINE
TIMELINE
main*
Launchpad
Git Graph
Pull Request #15
Screen Reader Optimized

### Blocker Recovery

The feedback distinguishes fake close buttons from successful dismissals and reports newly reachable elements. In one example, the agent removes a blocker and retries the previously failed click.

[4:01](https://www.youtube.com/watch?v=JnubYCYunk8&t=241s) [A document compares failed and successful dismissals and shows elements becoming accessible after a blocker is removed.](https://assets.ai.engineer/sites/aiecode2025/sha256/61eca38e5118c19953b4e34a170c5f3f4db2244551c9ad33415ca6854f7ea735.webp)

`aid-479` z=9997 "Overlay Notice" blocks [aid-81, aid-353, aid-83, ...]
  button `aid-480` "Close" [blocked by aid-515, aid-473, aid-476, aid-461, aid-540, aid-506, aid-482]

The close button itself is blocked by 7 other overlays.
Agent must dismiss them in the right order — the representation tells it exactly what's in the way.

3. Dismiss Hit vs Miss — Fake Close Buttons

[FAST_CLICK aid-87] dispatch at (640, 32)
[DISMISS MISS] aid-87 -> still_visible        ← click didn't work (fake button)

vs.

[FAST_CLICK aid-77] dispatch at (697, 414)
[DISMISS OK] aid-77 -> removed                ← real dismiss button

Agent learns which buttons are decoys and adapts.

4. Unblocked Signal — What Becomes Accessible

After each overlay dismissal, the system reports exactly which elements are now reachable:

[UNBLOCKED] aid-13, aid-5, aid-9, aid-11, aid-12, aid-19, aid-7, aid-8, aid-18, aid-15, aid-16, aid-6, aid-14,
aid-20, aid-10, aid-17, aid-70 (blocker gone)

17 elements suddenly become clickable. Agent can now interact with them.

5. Failed Action → Diagnosis → Recovery

Full episode from a single page:

[step 6]  click aid-121 -> action failed aid=aid-121: blocked by aid-100
[step 7]  click aid-102 -> clicked aid=aid-102       ← fixes the blocker
[step 8]  click aid-121 -> clicked aid=aid-121       ← now succeeds

Interface text:
browser-challenge
Upgrade to Pro
feedback-mutations.md
feedback-action-log.md
feedback-blocker-recovery.md
demo-video > feedback-blocker-recovery.md
Preview  Markdown
BROWSER-CHALLENGE
.pytest_cache
.ruff_cache
.venv
crawler
demo-video
claude-aadhar.mov
Claude-browser challenge.mov
claude-trekking.mov
feedback-action-log.md
feedback-blocker-recovery.md
feedback-mutations.md
our-agent.mov
ours-aadhar.mov
ours-trekking.mov
Screenshot 2026-04-03 at 12.32.23 A...
v5-metacritic-real-world.md
v5-step1-full-unedited.md
v5-step1-overlays-blocking.md
v5-step9-deep-blocking.md
v5-step9-full-unedited.md
OUTLINE
TIMELINE
browser-challenge  main*  Launchpad  Git Graph  Pull Request #15  Screen Reader Optimized

## Full transcript

[0:00](https://www.youtube.com/watch?v=JnubYCYunk8&t=0s) Okay, everyone. I am Kushan.

[0:02](https://www.youtube.com/watch?v=JnubYCYunk8&t=2s) I worked at Seraphim as a founding

[0:03](https://www.youtube.com/watch?v=JnubYCYunk8&t=3s) engineer for 2 years. Let's talk about

[0:05](https://www.youtube.com/watch?v=JnubYCYunk8&t=5s) what I'm interested in right now, and

[0:06](https://www.youtube.com/watch?v=JnubYCYunk8&t=6s) that is browser agents.

[0:07](https://www.youtube.com/watch?v=JnubYCYunk8&t=7s) Browser agents as an idea are so cool,

[0:09](https://www.youtube.com/watch?v=JnubYCYunk8&t=9s) right? The browser agent should go

[0:11](https://www.youtube.com/watch?v=JnubYCYunk8&t=11s) crazy, right? I personally have not seen

[0:12](https://www.youtube.com/watch?v=JnubYCYunk8&t=12s) that adoption, and me myself, I don't

[0:14](https://www.youtube.com/watch?v=JnubYCYunk8&t=14s) use browser agents that much. I've been

[0:15](https://www.youtube.com/watch?v=JnubYCYunk8&t=15s) exploring that for some time. I've been

[0:17](https://www.youtube.com/watch?v=JnubYCYunk8&t=17s) trying to understand why that is. So, on

[0:19](https://www.youtube.com/watch?v=JnubYCYunk8&t=19s) my screen right now, we have the browser

[0:20](https://www.youtube.com/watch?v=JnubYCYunk8&t=20s) challenge.

[0:21](https://www.youtube.com/watch?v=JnubYCYunk8&t=21s) But, this is a very interesting

[0:23](https://www.youtube.com/watch?v=JnubYCYunk8&t=23s) benchmark for browser agents because

[0:25](https://www.youtube.com/watch?v=JnubYCYunk8&t=25s) there are so many things that you have

[0:26](https://www.youtube.com/watch?v=JnubYCYunk8&t=26s) to do, long rising sequencing of your

[0:28](https://www.youtube.com/watch?v=JnubYCYunk8&t=28s) tasks. And this actually reveals, you

[0:30](https://www.youtube.com/watch?v=JnubYCYunk8&t=30s) know, why browser agents suck. If you

[0:31](https://www.youtube.com/watch?v=JnubYCYunk8&t=31s) saw at the beginning of the video,

[0:33](https://www.youtube.com/watch?v=JnubYCYunk8&t=33s) the browser this agent took like maybe

[0:35](https://www.youtube.com/watch?v=JnubYCYunk8&t=35s) 10-20 seconds just to click the start

[0:37](https://www.youtube.com/watch?v=JnubYCYunk8&t=37s) button. And now, we're on step one.

[0:38](https://www.youtube.com/watch?v=JnubYCYunk8&t=38s) There are 30 steps, and it has taken so

[0:40](https://www.youtube.com/watch?v=JnubYCYunk8&t=40s) long just to click one button.

[0:42](https://www.youtube.com/watch?v=JnubYCYunk8&t=42s) Um so, enough of this. I want to show

[0:44](https://www.youtube.com/watch?v=JnubYCYunk8&t=44s) you what I've been building. So, same

[0:45](https://www.youtube.com/watch?v=JnubYCYunk8&t=45s) website.

[0:47](https://www.youtube.com/watch?v=JnubYCYunk8&t=47s) Um I've tried to sort of replicate the

[0:48](https://www.youtube.com/watch?v=JnubYCYunk8&t=48s) feeling of seeing what's happening, you

[0:50](https://www.youtube.com/watch?v=JnubYCYunk8&t=50s) know, you can see what the browser agent

[0:51](https://www.youtube.com/watch?v=JnubYCYunk8&t=51s) is thinking. But as you can see, it is

[0:53](https://www.youtube.com/watch?v=JnubYCYunk8&t=53s) so much faster and so much quicker, and

[0:54](https://www.youtube.com/watch?v=JnubYCYunk8&t=54s) I'm using a much cheaper model, right?

[0:56](https://www.youtube.com/watch?v=JnubYCYunk8&t=56s) The hypothesis here is models are pretty

[0:58](https://www.youtube.com/watch?v=JnubYCYunk8&t=58s) smart, but it's the infra around them

[0:59](https://www.youtube.com/watch?v=JnubYCYunk8&t=59s) that sucks. If you noticed in the video

[1:01](https://www.youtube.com/watch?v=JnubYCYunk8&t=61s) earlier, maybe I'll put a screenshot,

[1:03](https://www.youtube.com/watch?v=JnubYCYunk8&t=63s) the agent is trying to debug what's

[1:05](https://www.youtube.com/watch?v=JnubYCYunk8&t=65s) going on. It's trying to click

[1:06](https://www.youtube.com/watch?v=JnubYCYunk8&t=66s) something, but it doesn't understand

[1:07](https://www.youtube.com/watch?v=JnubYCYunk8&t=67s) what's going on. So, my core thesis here

[1:09](https://www.youtube.com/watch?v=JnubYCYunk8&t=69s) has been

[1:10](https://www.youtube.com/watch?v=JnubYCYunk8&t=70s) give a nice environment for the agent to

[1:12](https://www.youtube.com/watch?v=JnubYCYunk8&t=72s) use, right? So, where it can plan long

[1:13](https://www.youtube.com/watch?v=JnubYCYunk8&t=73s) sequences, it can figure out where it

[1:15](https://www.youtube.com/watch?v=JnubYCYunk8&t=75s) failed, what is going on, and it can

[1:17](https://www.youtube.com/watch?v=JnubYCYunk8&t=77s) plan the click correctly. I figured out

[1:18](https://www.youtube.com/watch?v=JnubYCYunk8&t=78s) is a cool representation which

[1:20](https://www.youtube.com/watch?v=JnubYCYunk8&t=80s) compresses the website and lets the

[1:22](https://www.youtube.com/watch?v=JnubYCYunk8&t=82s) agent see the entire page in very few

[1:24](https://www.youtube.com/watch?v=JnubYCYunk8&t=84s) few tokens. Now, I want to show you some

[1:26](https://www.youtube.com/watch?v=JnubYCYunk8&t=86s) actual examples. Let's say I want to

[1:28](https://www.youtube.com/watch?v=JnubYCYunk8&t=88s) download my Aadhaar. Um so, this is

[1:30](https://www.youtube.com/watch?v=JnubYCYunk8&t=90s) Claude trying to do it. So, I'd assume

[1:32](https://www.youtube.com/watch?v=JnubYCYunk8&t=92s) that this is very simple for a browser

[1:33](https://www.youtube.com/watch?v=JnubYCYunk8&t=93s) agent. I take a screenshot, you see the

[1:34](https://www.youtube.com/watch?v=JnubYCYunk8&t=94s) button right there, click it. Uh but

[1:36](https://www.youtube.com/watch?v=JnubYCYunk8&t=96s) then, what's interesting is that it got

[1:38](https://www.youtube.com/watch?v=JnubYCYunk8&t=98s) stuck after this point. So, from 46

[1:40](https://www.youtube.com/watch?v=JnubYCYunk8&t=100s) seconds until the end of this video, it

[1:42](https://www.youtube.com/watch?v=JnubYCYunk8&t=102s) took a screenshot, it scrolled for some

[1:43](https://www.youtube.com/watch?v=JnubYCYunk8&t=103s) reason, it took a screenshot.

[1:45](https://www.youtube.com/watch?v=JnubYCYunk8&t=105s) Basically, this entire process took 2

[1:47](https://www.youtube.com/watch?v=JnubYCYunk8&t=107s) minutes, whereas

[1:49](https://www.youtube.com/watch?v=JnubYCYunk8&t=109s) in my case, in our video,

[1:55](https://www.youtube.com/watch?v=JnubYCYunk8&t=115s) so, it just boots, and

[1:57](https://www.youtube.com/watch?v=JnubYCYunk8&t=117s) boom, done.

[1:58](https://www.youtube.com/watch?v=JnubYCYunk8&t=118s) And that's the beauty of a browser

[2:00](https://www.youtube.com/watch?v=JnubYCYunk8&t=120s) agent. Just how quick was that? And I'm

[2:02](https://www.youtube.com/watch?v=JnubYCYunk8&t=122s) using such a cheap model for this.

[2:03](https://www.youtube.com/watch?v=JnubYCYunk8&t=123s) Another interesting example is So, my

[2:06](https://www.youtube.com/watch?v=JnubYCYunk8&t=126s) friends and I are going trekking on

[2:08](https://www.youtube.com/watch?v=JnubYCYunk8&t=128s) on Sunday. I was wondering, you know

[2:10](https://www.youtube.com/watch?v=JnubYCYunk8&t=130s) what? Because this this site is in

[2:12](https://www.youtube.com/watch?v=JnubYCYunk8&t=132s) Canada and I am not very fluent in

[2:13](https://www.youtube.com/watch?v=JnubYCYunk8&t=133s) Canada.

[2:14](https://www.youtube.com/watch?v=JnubYCYunk8&t=134s) Like it took me some time to figure out

[2:16](https://www.youtube.com/watch?v=JnubYCYunk8&t=136s) this website. So, I asked Claude like,

[2:17](https://www.youtube.com/watch?v=JnubYCYunk8&t=137s) "Hey, can you book this for me?" And by

[2:19](https://www.youtube.com/watch?v=JnubYCYunk8&t=139s) the end of it, it's unable to pick a

[2:20](https://www.youtube.com/watch?v=JnubYCYunk8&t=140s) date and it just stuck. This is the

[2:22](https://www.youtube.com/watch?v=JnubYCYunk8&t=142s) video of my agent.

[2:34](https://www.youtube.com/watch?v=JnubYCYunk8&t=154s) You can see it selects it and

[2:39](https://www.youtube.com/watch?v=JnubYCYunk8&t=159s) puts in the date and boom, done. Right?

[2:41](https://www.youtube.com/watch?v=JnubYCYunk8&t=161s) It's so simple and convenient to use in

[2:43](https://www.youtube.com/watch?v=JnubYCYunk8&t=163s) theory. So, what's next, right? What am

[2:45](https://www.youtube.com/watch?v=JnubYCYunk8&t=165s) I planning on doing? I'm thinking of

[2:46](https://www.youtube.com/watch?v=JnubYCYunk8&t=166s) open sourcing this project because again

[2:48](https://www.youtube.com/watch?v=JnubYCYunk8&t=168s) my this code is not super defensible.

[2:50](https://www.youtube.com/watch?v=JnubYCYunk8&t=170s) The product that I want to give is again

[2:52](https://www.youtube.com/watch?v=JnubYCYunk8&t=172s) maybe an API that as you can see we were

[2:55](https://www.youtube.com/watch?v=JnubYCYunk8&t=175s) running these commands. Maybe I just

[2:57](https://www.youtube.com/watch?v=JnubYCYunk8&t=177s) want to expose this command as an API.

[2:59](https://www.youtube.com/watch?v=JnubYCYunk8&t=179s) Give me a URL, give me your intent and I

[3:01](https://www.youtube.com/watch?v=JnubYCYunk8&t=181s) will execute it for you and give it back

[3:02](https://www.youtube.com/watch?v=JnubYCYunk8&t=182s) to you and or maybe open this as a

[3:04](https://www.youtube.com/watch?v=JnubYCYunk8&t=184s) website

[3:05](https://www.youtube.com/watch?v=JnubYCYunk8&t=185s) or expose this as a plugin. But yeah, so

[3:08](https://www.youtube.com/watch?v=JnubYCYunk8&t=188s) bottom line is I want to make browser

[3:10](https://www.youtube.com/watch?v=JnubYCYunk8&t=190s) agents faster, cheaper and more reliable

[3:13](https://www.youtube.com/watch?v=JnubYCYunk8&t=193s) and just make sure everybody in the

[3:14](https://www.youtube.com/watch?v=JnubYCYunk8&t=194s) world is using them because they can

[3:15](https://www.youtube.com/watch?v=JnubYCYunk8&t=195s) just do so much for you.

[3:17](https://www.youtube.com/watch?v=JnubYCYunk8&t=197s) So yeah, that's the broad idea here.

[3:19](https://www.youtube.com/watch?v=JnubYCYunk8&t=199s) Thank you for watching.

[3:24](https://www.youtube.com/watch?v=JnubYCYunk8&t=204s) This

[3:25](https://www.youtube.com/watch?v=JnubYCYunk8&t=205s) entire

[3:26](https://www.youtube.com/watch?v=JnubYCYunk8&t=206s) markdown

[3:28](https://www.youtube.com/watch?v=JnubYCYunk8&t=208s) presents the website, that particular

[3:30](https://www.youtube.com/watch?v=JnubYCYunk8&t=210s) page. And let's actually do this

[3:31](https://www.youtube.com/watch?v=JnubYCYunk8&t=211s) interesting comparison, right? Let's go

[3:33](https://www.youtube.com/watch?v=JnubYCYunk8&t=213s) to AIS. The full DOM for this would be

[3:35](https://www.youtube.com/watch?v=JnubYCYunk8&t=215s) around 20,000 tokens. But so, let's say

[3:37](https://www.youtube.com/watch?v=JnubYCYunk8&t=217s) we have this screenshot.

[3:38](https://www.youtube.com/watch?v=JnubYCYunk8&t=218s) All right, this screenshot's about 1,100

[3:40](https://www.youtube.com/watch?v=JnubYCYunk8&t=220s) tokens.

[3:41](https://www.youtube.com/watch?v=JnubYCYunk8&t=221s) My markdown's about 1,800 tokens and

[3:43](https://www.youtube.com/watch?v=JnubYCYunk8&t=223s) instead in one screenshot where you

[3:44](https://www.youtube.com/watch?v=JnubYCYunk8&t=224s) could see only one particular snippet,

[3:46](https://www.youtube.com/watch?v=JnubYCYunk8&t=226s) you can see the entire

[3:48](https://www.youtube.com/watch?v=JnubYCYunk8&t=228s) website, right? A couple of other things

[3:50](https://www.youtube.com/watch?v=JnubYCYunk8&t=230s) that it's important to give feedback,

[3:51](https://www.youtube.com/watch?v=JnubYCYunk8&t=231s) right? So, we say that okay, hey, these

[3:53](https://www.youtube.com/watch?v=JnubYCYunk8&t=233s) are the new things that have popped up

[3:54](https://www.youtube.com/watch?v=JnubYCYunk8&t=234s) on the page. This is now gone, right?

[3:56](https://www.youtube.com/watch?v=JnubYCYunk8&t=236s) And similarly, we can say that you know,

[3:57](https://www.youtube.com/watch?v=JnubYCYunk8&t=237s) this thing that was blocking up things

[4:00](https://www.youtube.com/watch?v=JnubYCYunk8&t=240s) that you wanted to click has now been

[4:01](https://www.youtube.com/watch?v=JnubYCYunk8&t=241s) removed. You know, we give it feedback

[4:02](https://www.youtube.com/watch?v=JnubYCYunk8&t=242s) that you tried to click this, but that

[4:03](https://www.youtube.com/watch?v=JnubYCYunk8&t=243s) didn't happen because you know, we're

[4:04](https://www.youtube.com/watch?v=JnubYCYunk8&t=244s) keeping track of the entire end-to-end

[4:06](https://www.youtube.com/watch?v=JnubYCYunk8&t=246s) browser

[4:07](https://www.youtube.com/watch?v=JnubYCYunk8&t=247s) page.

[4:08](https://www.youtube.com/watch?v=JnubYCYunk8&t=248s) Right? So, all of this together, what I

[4:10](https://www.youtube.com/watch?v=JnubYCYunk8&t=250s) built is a very clean representation

[4:12](https://www.youtube.com/watch?v=JnubYCYunk8&t=252s) that that basically compresses the

[4:13](https://www.youtube.com/watch?v=JnubYCYunk8&t=253s) website, and you can give this along

[4:14](https://www.youtube.com/watch?v=JnubYCYunk8&t=254s) with the screenshot. It's pretty cheap

[4:16](https://www.youtube.com/watch?v=JnubYCYunk8&t=256s) token-wise. Um so, the model can reason

[4:18](https://www.youtube.com/watch?v=JnubYCYunk8&t=258s) really well, and then it can construct

[4:19](https://www.youtube.com/watch?v=JnubYCYunk8&t=259s) this long sequence of tasks to execute.
