AI Engineer Code 2025
Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase
Loading slides…
Read the talk
Why 99% Accurate Browser Agents Still Fail
Derek Meegan of Browserbase explains how small risks compound across browser workflows, why completed transactions matter more than individual runs, and how an insurance-portal architecture reduces the decisions a model must make.
From a talk by Derek Meegan
At a glance
Ideas worth remembering
Under independent step success, 99% accuracy compounded across 100 steps yields only about 36.6% end-to-end success. Reducing model-directed steps is a reliability lever.
Define completion through a concrete artifact and measure success per customer transaction, including the attempts permitted by its retry budget.
In the insurance example, separate authentication, a packaged download operation, OCR-based verification, and a workflow skill narrow the model’s responsibility to the remaining ambiguity.
Production costs include browser infrastructure, integrations, observability, developer repairs, and reevaluation as websites and models change.
From a supervised demo to an unattended browser
A browser-agent demo gets favorable conditions: one site, one run, and a person watching. Production asks for thousands of unattended runs while websites change and models sometimes choose poorly. Derek Meegan, a software engineer at Browserbase, starts with that gap. The engineering problem is making a task complete repeatedly when nobody is there to rescue the browser.
The basic action loop separates choosing an action from executing it. Input tokens describe the page, the goal, and the steps already taken. The model proposes an action from a distribution of possibilities. A structured tool call then executes that action deterministically. Reliable execution of a click does not establish that clicking was the right decision; the uncertainty enters before the tool runs.
A browser offers several ways to observe and affect its state: network requests, the document object model and HTML, screenshots, the accessibility tree, and storage, cookies, URLs, tabs, and console information. Beneath these representations, a runtime can execute JavaScript or interact through CDP. Which parts an agent receives determines what it can reason about and how it can act.
Three strategies connect those browser capabilities to the model:
- Text representation. Combine HTML and the accessibility tree into a textual description of the page. The accessibility tree supplies semantic information about the interface.
- Computer use. Represent the browser through successive screenshots, letting the model work from its visual appearance.
- Dynamic code execution. Remove tailor-made tools and let the agent write code against the browser in an execution environment.
Meegan describes production systems as commonly using text, screenshots, or a combination. Dynamic execution offers a more open-ended interface, moving more of the interaction logic into code the agent generates.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Every step costs something; the completed task earns the value
Browser workflows span a spectrum. A purpose-built automation knows its task in advance. At the other extreme, a just-in-time agent receives an arbitrary task from a user. Between them, the browser can serve a larger objective such as deep research or competitor analysis. This talk narrows to transactional workflows: completing a unit of work end to end, with enough flexibility to handle ambiguity along the way.
The difficult economic relationship is temporal. Each step incurs cost and adds another opportunity for failure, while the useful outcome arrives only after the whole task completes. Progress through a transaction can consume resources without delivering the requested result. A highly capable model improves individual decisions, but a long sequence still gives small risks many opportunities to accumulate.
How much does length matter? In Meegan’s thought experiment, every step succeeds independently with probability 99%. A 100-step workflow therefore succeeds with probability (0.99^{100} \approx 36.6%), roughly the 36% stated in the talk. The independence assumption makes this an illustration rather than a measured production success rate. Its useful lesson is that strong per-step accuracy can coexist with weak end-to-end reliability.
That calculation creates a reason to change the system, rather than abandon browser agents. The model’s accuracy is one lever. The number of decisions it must make, the way completion is verified, and whether a failed attempt can be retried also shape the outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define success with an artifact, then measure the transaction
The business test is deliberately simple: (\text{profit} = \text{revenue} - \text{costs}). An agent is another line item and should return more value than it consumes. In Browserbase’s customer context, that value often comes from completing work on another customer’s behalf. The product being purchased is repeated task completion.
Meegan orders the engineering priorities as performance, cost, and maintainability. Here, performance means completing the requested task. Establish that capability first; then reducing its cost and keeping it working become engineering optimization problems. Cheap attempts have little value if they consistently fail to produce the result.
Completion needs an observable artifact:
- Bill payment. A confirmation email can mark the completed payment.
- Order placement. An order ID or receipt can identify the result.
- Form submission. A new record can be queried deterministically in the destination system.
These artifacts give the system something concrete to inspect after the browser interaction. They turn success into a check on the outcome rather than a judgment about how plausible the agent’s actions looked.
A run and a transaction also need separate metrics. One run is one attempt; a transaction is the customer’s unit of work, which may allow several attempts. With a 50% success probability per independent attempt, four total attempts yield (1-(1-0.5)^4 = 93.75%) transaction success. The talk calls this roughly 94% while describing up to four retries; the arithmetic corresponds to four attempts in total. The distinction matters when setting a retry budget.
The customer primarily wants the workflow completed reliably. Measuring only per-run success hides the benefit of retries; measuring transaction success captures it. Retries still consume resources, so their benefit belongs beside the accumulated cost of the attempts needed to finish the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Budget for browsers, repairs, and a changing web
Operating cost extends beyond model calls. Per-run expenses include model usage, the infrastructure and compute needed to run browser sessions, and additional integrations and tools. Meegan expects downward pressure on model prices to make intelligence a smaller part of the total over time. That is his forecast; browser resources and integration work remain part of the economic calculation regardless.
Maintenance has its own recurring work:
- Observability. Capture both the agent’s decisions and what actually happened in the browser. A decision trace alone cannot explain every execution failure.
- Developer time. Repair broken behavior, reconfigure the harness, or add capabilities when the existing system stops working well.
- Reevaluation. Revisit the solution as sites change and models improve. A successful configuration at one point in time still needs ongoing evaluation.
Several kinds of change can undermine the system. Anti-bot measures can make the environment resist automation. The underlying task can shift, requiring the agent to interpret intent and recover from unfamiliar situations. A new API can offer a better implementation route than browser interaction. Even without an external change, a model can stray from the critical path between runs.
Model flexibility helps when the job contains ambiguity: better understanding of the user’s intention can support course correction. The same freedom also permits unnecessary detours. The architecture that follows tries to keep that flexibility where it contributes to the task while removing decisions whose answers are already known.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn an insurance-document download into a verified workflow
The concrete example is a health insurance portal. A request asks the agent to log in and download an explanation of benefits. The smallest starting system sends the request to an agent, which performs the browser actions. It can do the job, but the model initially carries responsibility for the whole sequence. Each architectural change should improve performance, cost, or maintainability.
The first change packages downloading into one tool. Obtaining the document requires interacting programmatically with the page, retrieving the downloaded file from storage, and bringing it into the agent’s runtime. A dedicated tool performs that sequence as one operation. The browser and storage work still happens, but the model no longer has to decide how to coordinate every part of it on each run.
The next change makes the result inspectable. A verification tool uses OCR to extract entities from the downloaded document and compares them with a system of record. The observable outcome becomes a document plus a comparison result, allowing the agent to distinguish a completed task from merely having downloaded a file. Meegan presents this as deterministic verification; its conclusion depends on the extraction and comparison correctly capturing the document requirements, whose fields and error behavior are not specified here.
Authentication then moves out of the model’s responsibilities. In this example, the login workflow is treated as stable even though the business operation can contain ambiguity. A separate authentication function can log in once through a defined procedure and be reused across several operations on the same portal. Removing those model-directed steps aims to reduce unit cost and failure opportunities, while placing login maintenance in one reusable function.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the remaining agent a critical path
After the predictable operations have moved into functions, the agent still needs guidance through the portal’s ambiguous business workflow. A skill supplies a standard operating procedure for that remaining work. If the agent knows it is at step two and the procedure identifies step three, the intended next action becomes more likely. This connects back to the opening probability model: instructions guide action selection, while deterministic tools execute the operations that have already been defined.
Where does the model still make decisions, and where does the document travel? The diagram follows the final explanation-of-benefits workflow. Authentication supplies an already authenticated browser to the runtime. The skill guides portal navigation, the download function delivers the file, and the verification tool compares extracted entities with the system of record.
The resulting system is more complex than the initial request-to-agent demo, but the model’s job is smaller. Login has a reusable implementation. Downloading has a defined operation. Verification supplies an outcome check. The agent handles navigation with a skill and invokes those capabilities. Fewer decisions remain exposed to model variation, while the system retains flexibility for the part of the portal workflow that needs it.
That is the production tradeoff the ending leaves intact: reducing the agent’s responsibilities requires owning the surrounding software. Tools, authentication, observability, and reevaluation become ongoing engineering work. The last joke lands because the architecture has made reliable operation a concrete responsibility. Welcome to production; now someone is on call.
The explanation-of-benefits workflow
Read the diagram as text
- Request. Log in and obtain an explanation of benefits.
- Authentication function. Defined operations — A serverless function authenticates the browser.
- Agent runtime. Model-directed work — Navigate the portal with the authenticated browser.
- Workflow skill. Model-directed work — Describe the critical path and intended next steps.
- Download function. Defined operations — Interact with the page and retrieve the file into the runtime.
- Verification tool. Outcome verification — Extract document entities with OCR and compare them.
- System of record. Outcome verification — Provide the reference information for comparison.
- Request → Authentication function: Start the workflow.
- Authentication function → Agent runtime: Authenticated browser.
- Workflow skill → Agent runtime: Navigation procedure.
- Agent runtime → Download function: Invoke download.
- Download function → Verification tool: Downloaded document.
- System of record → Verification tool: Reference entities.
- Verification tool → Agent runtime: Success or failure result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Extends the discussion of skills and agent improvement by explaining how execution feedback can optimize prompts, programs, and reusable repository knowledge.
Full transcript
Loading transcript…