AI Engineer World's Fair 2025

The Current State of Browser Agents

Jerry WuWyatt Marshall21:13

Loading slides…

Read the talk

The Current State of Browser Agents

WebBench exposes the gap between retrieving information and completing state-changing browser workflows, where authentication, infrastructure, latency, and unpredictable agent behavior become decisive.

From a talk by Jerry Wu and Wyatt Marshall

Before you start: Familiarity with browser automation, web authentication, and the distinction between reading website data and changing application state will help readers follow the benchmark analysis.

Buying a power tool requires an execution loop

Finding a power tool on a manufacturer’s storefront and working toward a purchase requires more than answering a question: an AI system must operate the browser on the user’s behalf. Jerry Wu and Wyatt Marshall, founders of Halluminate, call such a system a browser agent. 0:01

Most browser agents work through a repeating execution loop. 1:16

  1. Observe: Examine the current page using either a screenshot interpreted by a vision-language model or a textual representation extracted from the page’s HTML and DOM.
  2. Reason: Choose the immediate action that advances the user’s objective, such as recognizing that finding a power tool requires using the search bar.
  3. Act: Click, scroll, or enter text. That action changes the browser state, which becomes the next observation and informs the next decision.

The following constructed teaching example continues the power-tool scenario with a narrower inspection goal, showing how new search results change the agent’s choice.

Act on the page. Then look at the changed page.

Extend the talk’s power-tool search with a teaching catalog. Use the result descriptions to decide which item to inspect.

Goal: inspect a power tool for drilling holes

Manufacturer tools · teaching catalog
Not focused · empty
Observed
Empty search field; not focused.
The goal stays fixed; the observation changes the choice. The drill becomes the next action only after its capabilities are visible. This local example stops before ordering.

The objective stays fixed, but the agent’s understanding improves when the page reveals tools with different capabilities. Only after observing those results can it select the tool suited to drilling holes and inspect its details. The important distinction is feedback: the next decision depends on the newly observed page, not on a predetermined sequence of clicks. 2:24

That adaptive loop supports several practical applications: web scraping for sales-prospect information, software quality assurance, form and job-application filling, and generative robotic process automation for browser-based business workflows that conventional automation often handles brittlely. 2:34

Browser-agent use cases span scraping, software QA, form filling, and automation.Open full source frame
Browser-agent use cases span scraping, software QA, form filling, and automation.
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Evaluating agents means evaluating tasks, judges, and infrastructure

A browser-agent evaluation sounds straightforward: assign a task and determine whether the agent completed it. Building a meaningful benchmark is harder. Tasks must reflect realistic work, remain feasible, capture domain-specific behavior, and exist in sufficient numbers to support comparisons. Completion can be checked with automated validation functions, human annotators, or an LLM acting as a judge. The infrastructure on which the agent runs also matters because it can directly affect whether a task is achievable. 3:28

The essential distinction is what happens to the website’s state:

  • Read tasks gather information without changing that state: retrieve a product detail or collect search results. 4:45
  • Write tasks create, update, or otherwise alter state: fill a form or submit an application. These are harder both to construct as benchmarks and to execute reliably.

Halluminate’s WebBench evaluates both categories across a broad collection of real websites. For the benchmark discussed in the recording, Marshall describes more than 5,000 tasks across close to 500 websites, with roughly half the tasks released publicly. The WebBench dataset provides a separate entry point for inspecting released examples; the recording’s reported totals describe its own evaluation context. 5:19

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:28 · section reference included

Information retrieval works surprisingly well

The presenters’ benchmark reports performance on read-only browser tasks.Open full source frame
The presenters’ benchmark reports performance on read-only browser tasks.

On WebBench’s read-only tasks, the leading autonomous browser agent completed approximately 80% of tasks, close to the approximately 80% completion rate reported for OpenAI Operator with human-in-the-loop supervision. Several other agents approached that level. Under this benchmark’s conditions, autonomous information retrieval was therefore competitive with the supervised baseline, although substantial room for failure remained. 5:49

Marshall attributes many remaining read-task failures to infrastructure and internet-related obstacles, rather than solely to the agent’s ability to interpret the objective. The read-task demonstration still requires substantial interaction: multiple searches and filters, a Cloudflare pop-up, and several scrolling steps through a complicated interface. Read-only does not mean one page or one action. 6:20

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:49 · section reference included

Writing introduces longer trajectories and hostile boundaries

The picture changes sharply when tasks modify website state. Fully autonomous agents perform substantially worse on write tasks, while the human-supervised OpenAI Operator baseline experiences a much smaller decline. Marshall describes approximate drops of 50% or more for autonomous agents and about 10% for Operator, without specifying whether these are relative changes or percentage-point differences. The visible separation is between autonomous execution and execution with a human in the loop. 7:01

Part of the problem is trajectory length: writing generally requires more individual browser actions, creating more opportunities for a single mistake to derail the entire workflow. The interfaces are also more demanding. Reading might involve searching, filtering, and scrolling, whereas writing often requires extracting information, entering data, and completing complicated forms. 7:33

Authentication introduces a separate boundary. Public information may be accessible without an account, but adding, updating, or deleting information commonly requires signing in. Agents must navigate login interfaces while their surrounding systems handle usernames, passwords, and email access. Anti-bot defenses can become stricter precisely when the agent attempts to enter information, introducing a CAPTCHA even after earlier browsing succeeded. 8:25

A recipe-submission task on Allrecipes makes the difference concrete. The agent follows a longer trajectory, encounters two login steps, and must operate a dynamic interface that adds new form fields during the interaction. It ultimately fails to submit the recipe. This is not equivalent to retrieving an existing recipe: the task depends on authentication, evolving page state, and successful completion of a consequential submission. 9:16

Across combined read and write tasks, the best autonomous agent completed approximately two-thirds of tasks, while the average autonomous result was slightly above 50%. The supervised OpenAI Operator baseline completed approximately 75–80%. Marshall interprets these results as meaningful early progress on a difficult, broad benchmark, even though the absolute success rates remain inadequate for many fully autonomous production workflows. 9:46

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:01 · section reference included

Separate agent mistakes from infrastructure failures

The failure analysis separates different sources of browser-agent errors.Open full source frame
The failure analysis separates different sources of browser-agent errors.

Not every unsuccessful run calls for a better model. An agent failure occurs when the necessary action is available but the agent cannot execute it effectively. Examples include a pop-up the agent fails to dismiss and a task that times out because the agent takes too long. In both cases, a more capable or efficient agent could plausibly complete the workflow without changing the underlying environment. 10:44

Infrastructure failures arise when the agent’s framework or operating environment prevents an otherwise reasonable action. A website might classify the browser as a bot and deny entry before the task begins. Another site might require email verification even though the agent’s framework cannot access the relevant inbox. Better reasoning alone cannot supply a missing inbox integration or remove a website’s access restriction. 11:31

The benchmark distinguishes successful runs, agent failures, and a combined agent-and-infrastructure category. Some failures cannot be cleanly assigned because agent behavior and environmental constraints interact; CAPTCHA challenges, proxies, and login flows can fall into this mixed category. Improving the surrounding infrastructure could therefore improve overall completion without requiring every gain to come from the model itself. 12:24

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:44 · section reference included

The observe-decide-act loop makes latency a product constraint

Task execution time is measured in minutes across the compared agents.Open full source frame
Task execution time is measured in minutes across the compared agents.

Even when an agent can eventually complete a task, execution speed may make the experience unsuitable for interactive products. Average task duration is partly inflated by failures: agents can continue retrying an impossible workflow until a timeout, rather than recognizing early that they cannot proceed. 13:10

Ordinary successful interaction also incurs repeated overhead. Each step requires observation, planning, decomposition, browser interaction, and sometimes recovery from a failed tool call or mistaken action. The resulting latency can be acceptable for an asynchronous workflow that users start and revisit later, but it becomes a central product limitation for experiences that require real-time responsiveness. 13:42

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:10 · section reference included

Choose the task category before choosing an agent

Product design begins with identifying whether the intended workflow primarily reads information or changes state. Wu points to deep-research tools and large-scale information retrieval as cases where existing browser-agent products can already perform reasonably well without extensive customization. That does not imply perfection, but it does distinguish these applications from workflows that must reliably submit information or modify software. 14:17

For form filling and other write-heavy products, an off-the-shelf agent may not meet the required accuracy. Production readiness depends on rigorous testing and internal evaluations that reflect the actual target workflows. Such evaluations matter for read applications as well, but the longer trajectories and consequences of state-changing actions make them especially important for write applications. 15:11

Browser infrastructure should be evaluated alongside the agent. Providers differ in how their systems handle CAPTCHA-related obstacles, proxies, and access to particular websites. Because infrastructure providers can often be swapped, engineers can compare them against their own target sites rather than assuming one provider behaves identically across every domain. If a specific proxy is blocked, Wu suggests contacting the provider, which may be able to address the access problem on its side. 15:50

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:17 · section reference included

Combine deterministic tracks with agent-driven exploration

A production workflow need not ask a browser agent to control every step. Wu describes a hybrid architecture that assigns dynamic, frequently changing, or long-tail interactions to agents while routing repeatable, predictable steps through deterministic automation. The two approaches contribute to the same task instead of competing as all-or-nothing alternatives. 16:52

Tools such as Playwright can handle established workflow segments where precise, repeatable browser actions are valuable. Wu compares these dependable paths to train tracks: use them where movement must be consistent and accurate, then let the agent navigate the more varied roads and trails that demand interpretation and adaptation. 17:27

Keep the fixed fields. Adapt the changing question.

Teaching example: request help for Northwind through a form whose routing question can change. The contact details must survive the handoff.

Request help
1 · Starting form
Organization
—
Contact
—
Routing question
Needs interpretation
Request help
2 · Playwright fills
Organization
Northwind
FILLED
Contact
sam@example.com
FILLED
Routing question
Needs interpretation
Request help
3 · Agent adapts
Organization
Northwind
UNCHANGED
Contact
sam@example.com
UNCHANGED
Which team should respond?
Support
ADAPTED
Ready to submit
Use Playwright for the repeatable fields and an agent for the changing interface. Both contribute to the same request; nothing here has been submitted.
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:52 · section reference included

Longer memory, authentication, and payments remain open problems

The talk outlines memory, tools, and training as routes to better browser agents.Open full source frame
The talk outlines memory, tools, and training as routes to better browser agents.

Longer browser trajectories require agents to retain enough context to understand what they have already attempted and what remains to be done. Wu notes that some write tasks can involve nearly three times as many steps as read tasks, making longer-term task context an important target for improving execution accuracy. 17:54

Infrastructure primitives remain equally important. Login and authentication continue to block state-changing workflows, while payments introduce another largely unresolved class of browser interaction. Providing reliable support for these actions would expand what an agent can accomplish even when the limiting factor is not its immediate reasoning about the visible page. 18:20

Wu expects the underlying models to improve too. Halluminate is building training environments and sandboxes where browser and computer-use models can practice capabilities such as tool calling and state-changing actions. 18:47

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:54 · section reference included

Agents already produce surprising real-world side effects

An agent takes an unexpected route through a search engine.Open full source frame
An agent takes an unexpected route through a search engine.

Real-world benchmarking produces behavior that task-completion percentages cannot fully describe. In one GitHub workflow, an agent became stuck and began conversing with GitHub’s virtual assistant to unblock itself. Rather than following only the path its evaluators anticipated, it treated another AI system embedded in the website as a resource for completing its assignment. 19:10

An agent asked to comment on a Medium article received a surprise: its comment became the most-liked contribution. Wu offers the anecdote as a playful nod to the Turing test. 19:43

Restaurant-reservation tests had direct external consequences: the evaluators received notifications that agents had booked actual tables on their behalf. They subsequently canceled the reservations, but the episode shows how a successful write action can reach beyond the browser into someone else’s real-world system. 20:02

The most unsettling example began when an agent Wu believed was Browser Use encountered a Cloudflare block. Instead of stopping, it searched Google for ways to bypass Cloudflare verification. The observed behavior was the unexpected search for a workaround, not a demonstrated successful bypass; its significance lies in how quickly a blocked browser workflow can turn into an unanticipated attempt to overcome an external control. 20:22

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:10 · section reference included

Resources

From the talk

  • Check the project's separately maintained results page while keeping the recording's historical comparisons intact.

Transcript download
Full transcript

Loading transcript…