AI Engineer Code 2025

Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase

Loading slides…

Read the talk

Why 99% Accurate Browser Agents Still Fail

Derek Meegan of Browserbase explains how small risks compound across browser workflows, why completed transactions matter more than individual runs, and how an insurance-portal architecture reduces the decisions a model must make.

From a talk by Derek Meegan

At a glance

Ideas worth remembering

  • Under independent step success, 99% accuracy compounded across 100 steps yields only about 36.6% end-to-end success. Reducing model-directed steps is a reliability lever.

  • Define completion through a concrete artifact and measure success per customer transaction, including the attempts permitted by its retry budget.

  • In the insurance example, separate authentication, a packaged download operation, OCR-based verification, and a workflow skill narrow the model’s responsibility to the remaining ambiguity.

  • Production costs include browser infrastructure, integrations, observability, developer repairs, and reevaluation as websites and models change.

From a supervised demo to an unattended browser

A browser-agent demo gets favorable conditions: one site, one run, and a person watching. Production asks for thousands of unattended runs while websites change and models sometimes choose poorly. Derek Meegan, a software engineer at Browserbase, starts with that gap. The engineering problem is making a task complete repeatedly when nobody is there to rescue the browser.

The basic action loop separates choosing an action from executing it. Input tokens describe the page, the goal, and the steps already taken. The model proposes an action from a distribution of possibilities. A structured tool call then executes that action deterministically. Reliable execution of a click does not establish that clicking was the right decision; the uncertainty enters before the tool runs.

A browser offers several ways to observe and affect its state: network requests, the document object model and HTML, screenshots, the accessibility tree, and storage, cookies, URLs, tabs, and console information. Beneath these representations, a runtime can execute JavaScript or interact through CDP. Which parts an agent receives determines what it can reason about and how it can act.

Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 122 secondsOpen full source frame
A slide titled “The interface of the browser is complicated” lays out DOM, screenshot, accessibility tree, and browser state.

Three strategies connect those browser capabilities to the model:

  • Text representation. Combine HTML and the accessibility tree into a textual description of the page. The accessibility tree supplies semantic information about the interface.
  • Computer use. Represent the browser through successive screenshots, letting the model work from its visual appearance.
  • Dynamic code execution. Remove tailor-made tools and let the agent write code against the browser in an execution environment.

Meegan describes production systems as commonly using text, screenshots, or a combination. Dynamic execution offers a more open-ended interface, moving more of the interaction logic into code the agent generates.

0:160:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Every step costs something; the completed task earns the value

Browser workflows span a spectrum. A purpose-built automation knows its task in advance. At the other extreme, a just-in-time agent receives an arbitrary task from a user. Between them, the browser can serve a larger objective such as deep research or competitor analysis. This talk narrows to transactional workflows: completing a unit of work end to end, with enough flexibility to handle ambiguity along the way.

The difficult economic relationship is temporal. Each step incurs cost and adds another opportunity for failure, while the useful outcome arrives only after the whole task completes. Progress through a transaction can consume resources without delivering the requested result. A highly capable model improves individual decisions, but a long sequence still gives small risks many opportunities to accumulate.

How much does length matter? In Meegan’s thought experiment, every step succeeds independently with probability 99%. A 100-step workflow therefore succeeds with probability (0.99^{100} \approx 36.6%), roughly the 36% stated in the talk. The independence assumption makes this an illustration rather than a measured production success rate. Its useful lesson is that strong per-step accuracy can coexist with weak end-to-end reliability.

Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 336 secondsOpen full source frame
A slide displays “0.99¹⁰⁰ = 36%.”

That calculation creates a reason to change the system, rather than abandon browser agents. The model’s accuracy is one lever. The number of decisions it must make, the way completion is verified, and whether a failed attempt can be retried also shape the outcome.

3:183:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:18 · section reference included

Define success with an artifact, then measure the transaction

The business test is deliberately simple: (\text{profit} = \text{revenue} - \text{costs}). An agent is another line item and should return more value than it consumes. In Browserbase’s customer context, that value often comes from completing work on another customer’s behalf. The product being purchased is repeated task completion.

Meegan orders the engineering priorities as performance, cost, and maintainability. Here, performance means completing the requested task. Establish that capability first; then reducing its cost and keeping it working become engineering optimization problems. Cheap attempts have little value if they consistently fail to produce the result.

Completion needs an observable artifact:

  • Bill payment. A confirmation email can mark the completed payment.
  • Order placement. An order ID or receipt can identify the result.
  • Form submission. A new record can be queried deterministically in the destination system.

These artifacts give the system something concrete to inspect after the browser interaction. They turn success into a check on the outcome rather than a judgment about how plausible the agent’s actions looked.

A run and a transaction also need separate metrics. One run is one attempt; a transaction is the customer’s unit of work, which may allow several attempts. With a 50% success probability per independent attempt, four total attempts yield (1-(1-0.5)^4 = 93.75%) transaction success. The talk calls this roughly 94% while describing up to four retries; the arithmetic corresponds to four attempts in total. The distinction matters when setting a retry budget.

Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 550 secondsOpen full source frame
A slide compares 50% per-run success with 94% transaction success after retries.

The customer primarily wants the workflow completed reliably. Measuring only per-run success hides the benefit of retries; measuring transaction success captures it. Retries still consume resources, so their benefit belongs beside the accumulated cost of the attempts needed to finish the work.

6:176:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:17 · section reference included

Budget for browsers, repairs, and a changing web

Operating cost extends beyond model calls. Per-run expenses include model usage, the infrastructure and compute needed to run browser sessions, and additional integrations and tools. Meegan expects downward pressure on model prices to make intelligence a smaller part of the total over time. That is his forecast; browser resources and integration work remain part of the economic calculation regardless.

Maintenance has its own recurring work:

  • Observability. Capture both the agent’s decisions and what actually happened in the browser. A decision trace alone cannot explain every execution failure.
  • Developer time. Repair broken behavior, reconfigure the harness, or add capabilities when the existing system stops working well.
  • Reevaluation. Revisit the solution as sites change and models improve. A successful configuration at one point in time still needs ongoing evaluation.

Several kinds of change can undermine the system. Anti-bot measures can make the environment resist automation. The underlying task can shift, requiring the agent to interpret intent and recover from unfamiliar situations. A new API can offer a better implementation route than browser interaction. Even without an external change, a model can stray from the critical path between runs.

Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 672 secondsOpen full source frame
A slide titled “The risks that erode a working agent” lists environmental fightback, task changes, better methods, and models straying off task.

Model flexibility helps when the job contains ambiguity: better understanding of the user’s intention can support course correction. The same freedom also permits unnecessary detours. The architecture that follows tries to keep that flexibility where it contributes to the task while removing decisions whose answers are already known.

9:3410:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:34 · section reference included

Turn an insurance-document download into a verified workflow

The concrete example is a health insurance portal. A request asks the agent to log in and download an explanation of benefits. The smallest starting system sends the request to an agent, which performs the browser actions. It can do the job, but the model initially carries responsibility for the whole sequence. Each architectural change should improve performance, cost, or maintainability.

The first change packages downloading into one tool. Obtaining the document requires interacting programmatically with the page, retrieving the downloaded file from storage, and bringing it into the agent’s runtime. A dedicated tool performs that sequence as one operation. The browser and storage work still happens, but the model no longer has to decide how to coordinate every part of it on each run.

The next change makes the result inspectable. A verification tool uses OCR to extract entities from the downloaded document and compares them with a system of record. The observable outcome becomes a document plus a comparison result, allowing the agent to distinguish a completed task from merely having downloaded a file. Meegan presents this as deterministic verification; its conclusion depends on the extraction and comparison correctly capturing the document requirements, whose fields and error behavior are not specified here.

Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 825 secondsOpen full source frame
A flow diagram titled “Verify by OCR, deterministically” shows an agent, a system of record, and a highlighted OCR tool.

Authentication then moves out of the model’s responsibilities. In this example, the login workflow is treated as stable even though the business operation can contain ambiguity. A separate authentication function can log in once through a defined procedure and be reused across several operations on the same portal. Removing those model-directed steps aims to reduce unit cost and failure opportunities, while placing login maintenance in one reusable function.

12:0612:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:06 · section reference included

Give the remaining agent a critical path

After the predictable operations have moved into functions, the agent still needs guidance through the portal’s ambiguous business workflow. A skill supplies a standard operating procedure for that remaining work. If the agent knows it is at step two and the procedure identifies step three, the intended next action becomes more likely. This connects back to the opening probability model: instructions guide action selection, while deterministic tools execute the operations that have already been defined.

Where does the model still make decisions, and where does the document travel? The diagram follows the final explanation-of-benefits workflow. Authentication supplies an already authenticated browser to the runtime. The skill guides portal navigation, the download function delivers the file, and the verification tool compares extracted entities with the system of record.

Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 947 secondsOpen full source frame
A workflow diagram shows a request flowing through authentication, an agent runtime, download, and verification steps.

The resulting system is more complex than the initial request-to-agent demo, but the model’s job is smaller. Login has a reusable implementation. Downloading has a defined operation. Verification supplies an outcome check. The agent handles navigation with a skill and invokes those capabilities. Fewer decisions remain exposed to model variation, while the system retains flexibility for the part of the portal workflow that needs it.

That is the production tradeoff the ending leaves intact: reducing the agent’s responsibilities requires owning the surrounding software. Tools, authentication, observability, and reevaluation become ongoing engineering work. The last joke lands because the architecture has made reliable operation a concrete responsibility. Welcome to production; now someone is on call.

The explanation-of-benefits workflow

RequestLog in and obtain an explanation of benefits.
Authentication functionDefined operations — A serverless function authenticates the browser.
Agent runtimeModel-directed work — Navigate the portal with the authenticated browser.
Workflow skillModel-directed work — Describe the critical path and intended next steps.
Download functionDefined operations — Interact with the page and retrieve the file into the runtime.
Verification toolOutcome verification — Extract document entities with OCR and compare them.
System of recordOutcome verification — Provide the reference information for comparison.
Defined functions handle authentication and file retrieval. The skill guides the agent’s navigation, and document verification checks the result against a system of record.
Read the diagram as text
  • Request. Log in and obtain an explanation of benefits.
  • Authentication function. Defined operations — A serverless function authenticates the browser.
  • Agent runtime. Model-directed work — Navigate the portal with the authenticated browser.
  • Workflow skill. Model-directed work — Describe the critical path and intended next steps.
  • Download function. Defined operations — Interact with the page and retrieve the file into the runtime.
  • Verification tool. Outcome verification — Extract document entities with OCR and compare them.
  • System of record. Outcome verification — Provide the reference information for comparison.
  • Request → Authentication function: Start the workflow.
  • Authentication function → Agent runtime: Authenticated browser.
  • Workflow skill → Agent runtime: Navigation procedure.
  • Agent runtime → Download function: Invoke download.
  • Download function → Verification tool: Downloaded document.
  • System of record → Verification tool: Reference entities.
  • Verification tool → Agent runtime: Success or failure result.
1:161:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:35 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…