# Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase

Derek Meegan, Browserbase

AI Engineer Code 2025

[Talk page](https://ai.engineer/talks/5xi_S1f9sDU-why-99-accurate-browser-agents-still-fail) · [Video](https://www.youtube.com/watch?v=5xi_S1f9sDU)

## Why 99% Accurate Browser Agents Still Fail

Derek Meegan of Browserbase explains how small risks compound across browser workflows, why completed transactions matter more than individual runs, and how an insurance-portal architecture reduces the decisions a model must make.

### From a supervised demo to an unattended browser

[0:12](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=12s)

A browser-agent demo gets favorable conditions: one site, one run, and a person watching. Production asks for thousands of unattended runs while websites change and models sometimes choose poorly. Derek Meegan, a software engineer at Browserbase, starts with that gap. The engineering problem is making a task complete repeatedly when nobody is there to rescue the browser.

The basic action loop separates choosing an action from executing it. Input tokens describe the page, the goal, and the steps already taken. The model proposes an action from a distribution of possibilities. A structured tool call then executes that action deterministically. Reliable execution of a click does not establish that clicking was the right decision; the uncertainty enters before the tool runs.

A browser offers several ways to observe and affect its state: network requests, the document object model and HTML, screenshots, the accessibility tree, and storage, cookies, URLs, tabs, and console information. Beneath these representations, a runtime can execute JavaScript or interact through CDP. Which parts an agent receives determines what it can reason about and how it can act.

![Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 122 seconds](https://ai.engineer/api/talk-essay-media/6729615ce42b8e37c1ed93d7a9730bcde55e125e8fc11670e7f823eb0338cfcb)

A slide titled “The interface of the browser is complicated” lays out DOM, screenshot, accessibility tree, and browser state.

Three strategies connect those browser capabilities to the model:

- **Text representation.** Combine HTML and the accessibility tree into a textual description of the page. The accessibility tree supplies semantic information about the interface.
- **Computer use.** Represent the browser through successive screenshots, letting the model work from its visual appearance.
- **Dynamic code execution.** Remove tailor-made tools and let the agent write code against the browser in an execution environment.

Meegan describes production systems as commonly using text, screenshots, or a combination. Dynamic execution offers a more open-ended interface, moving more of the interaction logic into code the agent generates.

### Every step costs something; the completed task earns the value

[3:18](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=198s)

Browser workflows span a spectrum. A purpose-built automation knows its task in advance. At the other extreme, a just-in-time agent receives an arbitrary task from a user. Between them, the browser can serve a larger objective such as deep research or competitor analysis. This talk narrows to transactional workflows: completing a unit of work end to end, with enough flexibility to handle ambiguity along the way.

The difficult economic relationship is temporal. Each step incurs cost and adds another opportunity for failure, while the useful outcome arrives only after the whole task completes. Progress through a transaction can consume resources without delivering the requested result. A highly capable model improves individual decisions, but a long sequence still gives small risks many opportunities to accumulate.

How much does length matter? In Meegan’s thought experiment, every step succeeds independently with probability 99%. A 100-step workflow therefore succeeds with probability \(0.99^{100} \approx 36.6\%\), roughly the 36% stated in the talk. The independence assumption makes this an illustration rather than a measured production success rate. Its useful lesson is that strong per-step accuracy can coexist with weak end-to-end reliability.

![Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 336 seconds](https://ai.engineer/api/talk-essay-media/a0ec962f413b296dbfd0b08a2736c246f24af5a9e563d182025f53a103aa4cea)

A slide displays “0.99¹⁰⁰ = 36%.”

That calculation creates a reason to change the system, rather than abandon browser agents. The model’s accuracy is one lever. The number of decisions it must make, the way completion is verified, and whether a failed attempt can be retried also shape the outcome.

### Define success with an artifact, then measure the transaction

[6:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=377s)

The business test is deliberately simple: \(\text{profit} = \text{revenue} - \text{costs}\). An agent is another line item and should return more value than it consumes. In Browserbase’s customer context, that value often comes from completing work on another customer’s behalf. The product being purchased is repeated task completion.

Meegan orders the engineering priorities as performance, cost, and maintainability. Here, performance means completing the requested task. Establish that capability first; then reducing its cost and keeping it working become engineering optimization problems. Cheap attempts have little value if they consistently fail to produce the result.

Completion needs an observable artifact:

- **Bill payment.** A confirmation email can mark the completed payment.
- **Order placement.** An order ID or receipt can identify the result.
- **Form submission.** A new record can be queried deterministically in the destination system.

These artifacts give the system something concrete to inspect after the browser interaction. They turn success into a check on the outcome rather than a judgment about how plausible the agent’s actions looked.

A run and a transaction also need separate metrics. One run is one attempt; a transaction is the customer’s unit of work, which may allow several attempts. With a 50% success probability per independent attempt, four total attempts yield \(1-(1-0.5)^4 = 93.75\%\) transaction success. The talk calls this roughly 94% while describing up to four retries; the arithmetic corresponds to four attempts in total. The distinction matters when setting a retry budget.

![Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 550 seconds](https://ai.engineer/api/talk-essay-media/f55e461bde6dedcbe9462aaf086da86dfefaaf431cd77be69e769bb3c6f5d661)

A slide compares 50% per-run success with 94% transaction success after retries.

The customer primarily wants the workflow completed reliably. Measuring only per-run success hides the benefit of retries; measuring transaction success captures it. Retries still consume resources, so their benefit belongs beside the accumulated cost of the attempts needed to finish the work.

### Budget for browsers, repairs, and a changing web

[9:34](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=574s)

Operating cost extends beyond model calls. Per-run expenses include model usage, the infrastructure and compute needed to run browser sessions, and additional integrations and tools. Meegan expects downward pressure on model prices to make intelligence a smaller part of the total over time. That is his forecast; browser resources and integration work remain part of the economic calculation regardless.

Maintenance has its own recurring work:

- **Observability.** Capture both the agent’s decisions and what actually happened in the browser. A decision trace alone cannot explain every execution failure.
- **Developer time.** Repair broken behavior, reconfigure the harness, or add capabilities when the existing system stops working well.
- **Reevaluation.** Revisit the solution as sites change and models improve. A successful configuration at one point in time still needs ongoing evaluation.

Several kinds of change can undermine the system. Anti-bot measures can make the environment resist automation. The underlying task can shift, requiring the agent to interpret intent and recover from unfamiliar situations. A new API can offer a better implementation route than browser interaction. Even without an external change, a model can stray from the critical path between runs.

![Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 672 seconds](https://ai.engineer/api/talk-essay-media/19012efd30e09cf1fc181445e38fa463eb5508e91d75f088d23b055fd6bfe56e)

A slide titled “The risks that erode a working agent” lists environmental fightback, task changes, better methods, and models straying off task.

Model flexibility helps when the job contains ambiguity: better understanding of the user’s intention can support course correction. The same freedom also permits unnecessary detours. The architecture that follows tries to keep that flexibility where it contributes to the task while removing decisions whose answers are already known.

### Turn an insurance-document download into a verified workflow

[12:06](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=726s)

The concrete example is a health insurance portal. A request asks the agent to log in and download an explanation of benefits. The smallest starting system sends the request to an agent, which performs the browser actions. It can do the job, but the model initially carries responsibility for the whole sequence. Each architectural change should improve performance, cost, or maintainability.

The first change packages downloading into one tool. Obtaining the document requires interacting programmatically with the page, retrieving the downloaded file from storage, and bringing it into the agent’s runtime. A dedicated tool performs that sequence as one operation. The browser and storage work still happens, but the model no longer has to decide how to coordinate every part of it on each run.

The next change makes the result inspectable. A verification tool uses OCR to extract entities from the downloaded document and compares them with a system of record. The observable outcome becomes a document plus a comparison result, allowing the agent to distinguish a completed task from merely having downloaded a file. Meegan presents this as deterministic verification; its conclusion depends on the extraction and comparison correctly capturing the document requirements, whose fields and error behavior are not specified here.

![Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 825 seconds](https://ai.engineer/api/talk-essay-media/f2a2d7de364509763f045d56dfa0425b1891ce9b44ab1d41b10b7b500b5a3a33)

A flow diagram titled “Verify by OCR, deterministically” shows an agent, a system of record, and a highlighted OCR tool.

Authentication then moves out of the model’s responsibilities. In this example, the login workflow is treated as stable even though the business operation can contain ambiguity. A separate authentication function can log in once through a defined procedure and be reused across several operations on the same portal. Removing those model-directed steps aims to reduce unit cost and failure opportunities, while placing login maintenance in one reusable function.

### Give the remaining agent a critical path

[14:35](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=875s)

After the predictable operations have moved into functions, the agent still needs guidance through the portal’s ambiguous business workflow. A skill supplies a standard operating procedure for that remaining work. If the agent knows it is at step two and the procedure identifies step three, the intended next action becomes more likely. This connects back to the opening probability model: instructions guide action selection, while deterministic tools execute the operations that have already been defined.

Where does the model still make decisions, and where does the document travel? The diagram follows the final explanation-of-benefits workflow. Authentication supplies an already authenticated browser to the runtime. The skill guides portal navigation, the download function delivers the file, and the verification tool compares extracted entities with the system of record.

![Selected presentation frame from Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase at 947 seconds](https://ai.engineer/api/talk-essay-media/f034f993a9a6091ba40ba445c673c8078068ff8b023368500ff4a55422d092a8)

A workflow diagram shows a request flowing through authentication, an agent runtime, download, and verification steps.

The resulting system is more complex than the initial request-to-agent demo, but the model’s job is smaller. Login has a reusable implementation. Downloading has a defined operation. Verification supplies an outcome check. The agent handles navigation with a skill and invokes those capabilities. Fewer decisions remain exposed to model variation, while the system retains flexibility for the part of the portal workflow that needs it.

That is the production tradeoff the ending leaves intact: reducing the agent’s responsibilities requires owning the surrounding software. Tools, authentication, observability, and reevaluation become ongoing engineering work. The last joke lands because the architecture has made reliable operation a concrete responsibility. Welcome to production; now someone is on call.

The explanation-of-benefits workflow
Defined functions handle authentication and file retrieval. The skill guides the agent’s navigation, and document verification checks the result against a system of record.
- Request — Log in and obtain an explanation of benefits.
- Authentication function — A serverless function authenticates the browser.
- Agent runtime — Navigate the portal with the authenticated browser.
- Workflow skill — Describe the critical path and intended next steps.
- Download function — Interact with the page and retrieve the file into the runtime.
- Verification tool — Extract document entities with OCR and compare them.
- System of record — Provide the reference information for comparison.
- Request → Authentication function (Start the workflow)
- Authentication function → Agent runtime (Authenticated browser)
- Workflow skill → Agent runtime (Navigation procedure)
- Agent runtime → Download function (Invoke download)
- Download function → Verification tool (Downloaded document)
- System of record → Verification tool (Reference entities)
- Verification tool → Agent runtime (Success or failure result)

## Key takeaways

- Under independent step success, 99% accuracy compounded across 100 steps yields only about 36.6% end-to-end success. Reducing model-directed steps is a reliability lever.
- Define completion through a concrete artifact and measure success per customer transaction, including the attempts permitted by its retry budget.
- In the insurance example, separate authentication, a packaged download operation, OCR-based verification, and a workflow skill narrow the model’s responsibility to the remaining ambiguity.
- Production costs include browser infrastructure, integrations, observability, developer repairs, and reevaluation as websites and models change.

## Related talks

- [Beating RL With Reflection: GEPA and Optimize Anything](https://ai.engineer/talks/OA-Mc60Rboo-beating-rl-reflection-gepa-optimize-anything) — Extends the discussion of skills and agent improvement by explaining how execution feedback can optimize prompts, programs, and reusable repository knowledge.

## Slides

### What separates a demo from production?

Derek Meegan contrasts a watched, single-site demo with thousands of unattended production runs, then outlines four topics: web interaction, scaling challenges, agent value, and a real-world scenario.

[0:42](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=42s) [Agenda listing four topics: browser interaction, production difficulty, agent value, and a real-world scenario.](https://assets.ai.engineer/sites/aiecode2025/sha256/aefa4c017bd0eb48f9323ed6f43eee5484da8502dbddcde7dce10e348e4d192e.webp)

AGENDA

What separates a demo from production?

01 Bring order to chaos
How an agent interacts with the web.

02 Define the impossible
Why running them is genuinely hard.

03 Think about what's important
The value proposition of agents

04 Generate shareholder value
Exploring a real world scenario

### The output of a model is a probability distribution

The model assigns probabilities to candidate actions and selects the most likely one. Here, “click Submit” becomes a structured tool call that executes in the browser.

[1:25](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=85s) [A diagram connects input tokens to a distribution of actions and a structured tool call.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9a552d9360e43b060b8f5b60b15d1a8417e00adce38beccea3d558a3c0e2848.webp)

The output of a model is a probability distribution

BRING ORDER TO CHAOS

INPUT
Arbitrary tokens
Page state, the goal, and the history of the run so far.

THE MODEL
A distribution over actions
CANDIDATE ACTIONS →
argmax · click Submit · 0.71

OUTPUT
A structured tool call
{
  "tool": "click",
  "target": "Submit"
}

### The interface of the browser is complicated

The DOM exposes page HTML, screenshots show rendered pixels, and the accessibility tree describes roles and names. Network traffic and browser state add more layers, with JavaScript and CDP underneath.

[1:52](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=112s) [Browser layers include network traffic, the DOM, screenshots, accessibility tree, browser state, JavaScript runtime, and CDP.](https://assets.ai.engineer/sites/aiecode2025/sha256/99ff44a3e1f3b0b4b3080cf9e6b58784445f600a1c334030bd58cbe379635ccb.webp)

BRING ORDER TO CHAOS

The interface of the browser is complicated

NETWORK
Requests & responses flowing in and out

THE BROWSER

DOM
The full page HTML.

Screenshot
The rendered pixels.

Accessibility tree
Roles and names.

Browser state
Storage, cookies, console, URL, tabs, dialogs.

JAVASCRIPT RUNTIME
Execute arbitrary JavaScript

CDP
Chrome DevTools Protocol

### Wiring your agent to the browser

Agents can read a textual DOM and accessibility tree, click from screenshots, or drive the browser with JavaScript or CDP. Most production systems use one or both of the first two approaches.

[2:30](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=150s) [Three approaches use structured page representation, screenshots, or arbitrary JavaScript and CDP.](https://assets.ai.engineer/sites/aiecode2025/sha256/21b7486b932f1753f48cc81015e6abbc1c585271a40998486d95218234a6d250.webp)

BRING ORDER TO CHAOS

Wiring your agent to the browser

STRUCTURED REPRESENTATION
DOM + a11y tree
Read the page's structure and resolve to deterministic locators. Represents the browser as text.

COMPUTER USE
Screenshot-driven
The model looks at a screenshot and decides where to click. Represents the browser as a series of images.

DE-HARNESS THE AGENT
Arbitrary JS or CDP
Execute JavaScript or CDP to drive and observe the browser. Represents the browser through code.

In practice, most production systems lean on the first, second, or combination of the first and second.

### A spectrum of agenticness

Browser trajectories range from purpose-built, high-volume actions to ad hoc requests. In between, the browser serves a larger goal, such as deep research.

[3:16](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=196s) [Three columns compare planned, indirect and just-in-time browser trajectories from less to more agentic.](https://assets.ai.engineer/sites/aiecode2025/sha256/48e4d27bffe44b5781ae92d82d80fa4b1781603d6ee5c508a25c8c17c9683fea.webp)

BRING ORDER TO CHAOS

A spectrum of agenticness

PLANNED & DIRECT
A purpose-built flow to perform an action
Engineered for one job at high volume.
e.g. Pay vendor invoices

INDIRECT BROWSER TRAJECTORY
Extracts or judges the page
Browser is an implementation detail.
e.g. Performing deep research

JUST-IN-TIME
Spun up ad hoc for an arbitrary request
Reasons its way through on the spot.
e.g. Check in for my flight

LESS AGENTIC → MORE AGENTIC

### At scale, workflows are typically transactional

High-volume workflows repeat bounded transactions, but real workflows still contain ambiguity. Browser agents provide flexibility to handle it in production.

[3:57](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=237s) [Two panels contrast repeatable transactions with browser agents' ability to handle ambiguity.](https://assets.ai.engineer/sites/aiecode2025/sha256/ffbaf4d04f9318dfc7f48d500193c9dbf387cd4f804fb4fa7d409a18c1c95024.webp)

— BRING ORDER TO CHAOS

At scale, workflows are typically transactional

Planned, scoped trajectories comprise the majority of high-volume automation use cases.

01
Transactional by nature
Bounded, repeatable transactions, run the same way thousands of times.

02
A browser agent flexes
But no real workflow is fully deterministic. Agents offer the ability to absorb ambiguity.

Browser agents provide a tool to handle ambiguity in production.

### Define the impossible

The second section asks why transactional browser workflows are hard to run in production.

[4:23](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=263s) [Part 2 section divider: Define the impossible, the browser agent conundrum.](https://assets.ai.engineer/sites/aiecode2025/sha256/ddb061dd0c24d81f51ecfcbdd400cc3b58b5ec21bdd443e2cee415a8f937a412.webp)

PART 02 / 04

Define the impossible

The browser agent conundrum

02 / 04

### Cost accumulation is continuous, value realization is terminal

Each step in a browser trajectory adds cost and failure risk, but the vast majority of web automations pay off only when the task is complete.

[4:35](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=275s) [A diagram shows cost and risk accumulating with each step while value appears at completion; a callout says there is no partial credit.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f195db56632c5b1491d6abe2ae2d0d2a4a29532b38dd4ff6cbd0b24e6049c57.webp)

DEFINE THE IMPOSSIBLE

Cost accumulation is continuous,
Value realization is terminal

COST + RISK
accumulates every step →

VALUE
realizes only here ↑

No partial credit.
The vast majority of web automations only pay off on total completion.

### Reliability across 100 steps

In this thought experiment, 100 steps with independent 99% success rates yield an overall success rate of about 36%.

[5:09](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=309s) [A calculation shows 0.99 raised to 100 as 36%, illustrating cumulative failure risk over 100 steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/0083b209ec3e7013c0870d17102498ec123750527cba0e57acd7179243fbe41c.webp)

DEFINE THE IMPOSSIBLE

0.99 ^ 100 = 36%

100 steps. Each 99% reliable. A near-perfect agent completes barely a third of the time.

### So... never use a browser agent?

Meegan acknowledges that risk accumulates across browser-agent steps, but says the calculation alone does not tell the whole story of how agents deliver value.

[5:58](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=358s) [A callout says the reliability dynamic is real, but the math is not the whole story.](https://assets.ai.engineer/sites/aiecode2025/sha256/aa90dca936695b572bdbc11a7024345854784c58d82ca5e186c2c613635b40c7.webp)

— DEFINE THE IMPOSSIBLE

So... never use a browser agent?

THE REAL STORY

The dynamic is real. But the math isn't the whole story.

### Think about what's important

Meegan steps back from the reliability math to consider how agents deliver value.

[6:09](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=369s) [Section title: Think about what's important. How agents deliver value.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ff475335ad04ef95cc433f4dfbb0cc4fe3402d46f4e4c42c5dc0d88c7b4ab57.webp)

PART 03 / 04

Think about what's important

How agents deliver value

03 / 04

### Profits equal revenue minus costs

Meegan frames a browser agent as a line item: it should deliver more value than it costs.

[6:29](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=389s) [Profits equal revenue minus costs. Your agent should give you more than it takes.](https://assets.ai.engineer/sites/aiecode2025/sha256/a76e395e3ce2bdecd1972a247012134b1f6aa8b676a55880c5e25f092b0dc734.webp)

AN EQUATION NEVER BEFORE SEEN IN SILICON VALLEY

profits = revenue − costs

Your agent should give you more than it takes.

### What does a browser agent give you?

A browser agent ideally completes a customer’s task repeatedly. The challenge is to handle ambiguity while completing the task every time the environment stays the same.

[6:55](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=415s) [A browser agent ideally completes a customer's task the same way every time, creating dependable outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb145b80a46bd0cc5cea475ec2a55d2d76fe2c60150b59ef3221357a63554c10.webp)

THE FRAMEWORK

But what does a browser agent give you?

A task completed for your customer, the same way, every time.

An ideal browser agent creates dependable outcomes for your customers.

### Three dimensions, in order

Performance comes first: the agent must complete the task. Once it does, cost per run and maintainability as the world changes become optimization problems.

[7:36](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=456s) [Three ordered dimensions: performance, cost per run, and maintainability over time. Once performance is established, the other two become optimization problems.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fd7c17a6fb493277276a2f5e47cc36ceaba893757135ae212694c2c0ce27272.webp)

THE FRAMEWORK

Three dimensions, in order

01
Performance
The value it delivers. This is the most important.

02
Cost
Your COGS per run, what each completion costs.

03
Maintainability
COGS over time, what it costs as the world changes.

Once you create a performant browser agent, cost and maintainability become optimization problems.

### First, what does success look like?

A successful browser run leaves a concrete artifact: a bill confirmation, an order ID, or a new system record that can be queried.

[8:06](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=486s) [Examples of task success: a confirmation email for paying a bill, an order ID on a receipt, and a new system record for a submitted form.](https://assets.ai.engineer/sites/aiecode2025/sha256/f2a75f09220a7da6fbf0f742e1b2e897a387d7e09a69f4661e47f79bb7da0a20.webp)

DIMENSION 01 / PERFORMANCE

First, what does success look like?

Success is a concrete artifact the run leaves behind.

THE TASK
Pay a bill
→
THE PROOF
A confirmation email lands

THE TASK
Place an order
→
THE PROOF
An order ID on the receipt

THE TASK
Submit a form
→
THE PROOF
A new record in the system

### Then, how often is it right?

With a 50% success rate per run, allowing up to four tries raises the example’s per-transaction success rate to 94%. The customer experiences the completed transaction, even if it takes retries.

[8:38](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=518s) [A 50% individual-run success rate is shown alongside 94% per-transaction success with up to four tries. The slide recommends optimizing for the metric the customer feels.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dcee4f13c84645891c37398869084f2a0ff1f89de1250d1071948bfcf9dcc6c.webp)

DIMENSION 01 / PERFORMANCE

Then, how often is it right?

50%
Individual run success

94%
Per transaction · up to 4 tries

By permitting a few retries per transaction, the perceived success by the customer climbs.

Optimize for the metric the customer feels.

### Cost now, and cost over time

Per-run costs include models, compute, and tooling. Keeping an agent working also takes observability, developer time, and re-evaluation as models and sites change.

[9:32](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=572s) [Two panels compare per-run costs with the ongoing costs of maintaining an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/732c8417a1a1d973104e7d85bc5f726ba7ff3262579b1abab80b23fb7adf8555.webp)

DIMENSIONS 02 + 03 / COST & MAINTAINABILITY
Cost now, and cost over time

COST — PER RUN (COGS)
What each completion costs
• Model cost, usually the largest
• Infrastructure & compute
• Integrations & tooling

MAINTAINABILITY — OVER TIME
What it costs to keep working
• Observability to see what's happening
• Developer time to fix breakage
• Re-evaluation as models & sites move

Same question, two horizons. Land performance first, then optimize both.

### The risks that erode a working agent

An agent’s performance, cost, and maintainability can erode as sites resist automation, tasks shift, better methods emerge, or model runs stray off course. A site launching an API is one example of a better method.

[10:44](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=644s) [Four panels describe environmental resistance, changing tasks, better methods, and models straying off-task.](https://assets.ai.engineer/sites/aiecode2025/sha256/6aad50e6ee8e0018f17852dd0086ad29db4f170917dc33bb0c3c5efde48836d0.webp)

RISK — SPANS EVERY DIMENSION
The risks that erode a working agent

The environment fights back
Anti-bot defenses and layout churn at the web's edge.

The task changes
The underlying job shifts beneath the agent.

A better method appears
Cheaper or simpler approaches make yours legacy.

The model strays off-task
Run to run, it doesn't reliably follow the ideal trajectory.

### Generate shareholder value

Meegan turns to deployment at scale, treating cost and maintainability as optimization problems when a model performs well.

[12:04](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=724s) [Red section divider introduces concrete moves for deploying at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/45c54b9cafd2e14614e206ceb2aea4b4ebe67e72ac2d24a3dd085f107555dbd2.webp)

PART 04 / 04

Generate
shareholder value

The concrete moves for deploying at scale.

04 / 04

### Let's build a browser agent

As the health-insurance portal agent moves from demoable to production ready, each architecture step should lower cost, raise performance, or improve maintainability.

[12:23](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=743s) [A health-insurance portal agent example targets lower cost and higher performance and maintainability.](https://assets.ai.engineer/sites/aiecode2025/sha256/64ae69d3ef688e83bbbc48f0b09ec46df1268bc60f099d80641cad6df6b6f8d8.webp)

GENERATE SHAREHOLDER VALUE

Let's build a browser agent

We'll take a real task from demoable to production ready.

Our task: a health-insurance portal agent. →

EVERY LEVER SHOULD MOVE ONE
Cost ↓
Performance ↑
Maintainability ↑

### Start with just an agent

A request invokes an agent harness whose model and tooling layer drive and observe the browser to log in and download an Explanation of Benefits.

[12:42](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=762s) [A request invokes an agent harness containing a model and tooling layer, which drives and observes a browser.](https://assets.ai.engineer/sites/aiecode2025/sha256/db3e0d76fd20e822833044dd68620bca3efd26b4900c74045965161ec2da2468.webp)

GENERATE SHAREHOLDER VALUE / BUILD V1

Start with just an agent

The goal: "Log in and download my Explanation of Benefits."

Request → invokes → RUNTIME
AGENT HARNESS
Model
Tooling Layer

RUNTIME → drive/observe → Browser

### Wrap the brittle sequence in a custom tool

The agent calls download_eob to drive the browser, retrieve the saved file from storage, and return it to the sandbox in one tool call.

[13:01](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=781s) [An agent invokes a download tool that uses a browser and storage to retrieve a file.](https://assets.ai.engineer/sites/aiecode2025/sha256/d3861baba6833c53bcd29d6ed81aa5d50a508a38622cad26439e548f97d16219.webp)

GENERATE SHAREHOLDER VALUE / BUILD V2

Wrap the brittle sequence in a custom tool

Downloading requires programmatically interacting with the browser and retrieving the file from cloud storage. Collapse that sequence into a single tool call.

Agent
harness

invokes

download_eob
the tool

download

Browser

saved

Storage
downloads

retrieve file
file to sandbox

### Verify by OCR, deterministically

The agent passes the downloaded file to an OCR tool, which compares extracted fields with the system of record and returns a pass or fail.

[13:26](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=806s) [An agent passes a downloaded file to an OCR verifier, which checks it against a system of record and returns pass or fail.](https://assets.ai.engineer/sites/aiecode2025/sha256/d422a8a9907b52453939f622ae1e0ef58bb67d715f426768307d92908a578f7a.webp)

— GENERATE SHAREHOLDER VALUE / BUILD V3

Verify by OCR, deterministically

Permit the model to verify its work determinisitically with an OCR tool. This tool extracts key fields from the SOB and verifies them against a database.

Agent
runtime

invokes

download_eob
the tool

file to runtime
pass file

verify
OCR check

look up
result

System of record
database

match? pass / fail

### Take auth out of the model's hands

A serverless function logs in and performs MFA, then gives the agent an authenticated browser session. The same function can serve other workflows on the portal, reducing the steps the agent must handle.

[13:53](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=833s) [A serverless function authenticates a browser and supplies an authenticated session to an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/3be1fa00d9ffe2f851b776943a32775c1957064d8adbfa021840028c7ba11b61.webp)

— GENERATE SHAREHOLDER VALUE / BUILD V4

Take auth out of the model's hands

A serverless auth function logs in and performs MFA deterministically, then hands the authenticated browser to the agent. This auth function can be reused for other trajectories on the same portal.

Request
request

authenticate
serverless fn

authenticate on portal
authed session

Agent
harness

drive/observe

Browser

### Graduate the converged path into a skill

The agent harness gives the model a skill documenting the converged critical path. Like a procedure for an agent, it outlines the next step so the model is more likely to follow it.

[14:35](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=875s) [An agent harness contains a model, tooling layer, and critical-path skill, and interacts with a browser.](https://assets.ai.engineer/sites/aiecode2025/sha256/cebf417b7fb120da68b88f65e3f57e4b8479c13d9da5d1f17b27e8e1b6c69aef.webp)

GENERATE SHAREHOLDER VALUE / BUILD V5

Graduate the converged path into a skill

Run the hardened agent until it converges, then document the critical path into a skill. The agent invokes the skill at runtime and is therefore more likely to execute on the critical path.

RUNTIME
AGENT HARNESS
Model
Tooling Layer
Skill
critical path

Request
invokes
Browser
drive/observe

### The final result

A serverless function authenticates the browser. The agent follows a critical-path skill, calls a download tool, and passes the file to verification against the system of record.

[15:26](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=926s) [Architecture diagram connects serverless authentication, an agent harness, a browser, a download tool, storage, and OCR verification against a system of record.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc3433595441332b9d97a7f24a4d6123263b24a05a394b01f9edf467b6acf83b.webp)

GENERATE SHAREHOLDER VALUE / THE RESULT

The final result

Serverless auth logs in, the agent invokes the skill, downloads with the tool, and verifies its download, all constrained to the critical path. Same model but with a tailored harness around it.

Request
request
Serverless
auth fn
authed browser
authenticate on portal

RUNTIME
AGENT HARNESS
Model
Tooling Layer
Skill
critical path

drive/observe
Browser
invoke
download
download_eob
the tool
saved
Storage
downloads
file to runtime
pass file
verify
OCR check
match? pass / fail
look up
result
System of record
database

### Welcome to production

Meegan closes with the operational reality of putting browser agents into production: someone is now on call.

[16:24](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=984s) [Closing slide says welcome to production and congratulates the audience on being on call, alongside a pixelated illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/27516c6ca39ffe2f324bb1856362c776f4da82317010e2471664439c6661121f.webp)

Browserbase

HERE WE ARE

Welcome to production.

Congrats, now you're on-call.

Unlock the impossible.
browserbase.com

## Full transcript

[0:12](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=12s) All right.

[0:16](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=16s) Deploying browser agents. Uh, running a browser agent demo once is easy. Uh, it's on one site, it's one run, on a good day, with you watching and your fingers crossed. Uh, but we know production is vastly different. Thousands of runs, unattended, while the site can change or your model can just have an off day. So that's why we're here. What separates a demo from

[0:46](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=46s) production? My name is Derek Meegan. I'm a software engineer at Browserbase, and today we'll break up this talk into four parts. First, we'll bring order to chaos, and we'll understand concretely how agents interact with the web. Second, we'll define the impossible and explore why running them at scale is so hard. Third, we'll take a step back. We'll think about what's important and the value proposition that agents propose. And

[1:16](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=76s) then last, which is hopefully the reason why we're all here, we'll generate some shareholder value. So first, how agents interact with the web. I like to think about the output of a model as a probability distribution. You receive input tokens. They represent the page state, the overarching goal, or the steps that have already been taken. Then the model explores a distribution of possible actions, proposing the most likely one. Then that

[1:46](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=106s) action is transposed into a structured tool call and deterministically executed to interact with the browser.

[1:54](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=114s) But the interface of the browser is complicated. You have several distinct layers. You have requests flowing in and out of the browser. You have the document object model or the HTML representation of the page. You can screenshot the browser. The accessibility tree, which is a semantic textual representation of the page. And the browser state: storage, cookies, console URL, tabs, et cetera. Underneath all of this, you have a dynamic code

[2:24](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=144s) execution runtime where you can interact with the browser by writing arbitrary JavaScript or CDP.

[2:32](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=152s) How do you wire these together? Typically, in industry, there's three main strategies. Number one is creating a textual representation of the browser. This typically involves taking the HTML state of the page and the accessibility tree and creating a hybrid representation. Number two is computer use, where you represent the browser in a series of screenshots of the page. And finally, and becoming increasingly popular, is de-harnessing your agent, removing tailor-made tools in favor of

[3:02](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=182s) dynamic execution environments where the agent can write arbitrary code against the browser. In practice, most production systems lean on the first, the second, or a combination of the first and the second two strategies.

[3:18](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=198s) Now let's think about the type of agentic trajectories on the browser. And here I define three, spanning across a spectrum of agenticness. From least agentic, we have purpose-built browser trajectories that are specially made to complete a single task. To the most agentic, we have just-in-time browser automations, where we don't know what the task is beforehand, and the user provides us an arbitrary task to complete. In the middle, we have

[3:48](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=228s) something sort of between both. We're using the browser as an implementation detail to serve a larger goal, such as performing deep research or competitor analysis. At scale, I really want to hone in on these transactional workflows. Not because it makes things easy, but that's because we typically see at scale. These workflows are transactional by nature, where we want to complete a unit of work end to end. On the other side, browser agents provide us a flexible

[4:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=257s) execution environment to handle ambiguity in production. So we know how agents interact with the web and the different tripe-- types of trajectories we may see, and we're honing in specifically on transactional workflows. Why is it so hard in production? And I think it comes down to a key interaction. Cost accumulation is continuous, while value realization is terminal. Each step in your

[4:47](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=287s) trajectory, you're incurring some incremental cost. Additionally, you're incurring some incremental risk of failure. Yet I only receive value from this browser trajectory when all steps of the task have been completed. That's all to say that with browser agents, there is no partial credit, and the vast majority of web automations fall into this category. So you say, "Okay, well, each step, there's more risk and there's more cost. If I use a highly capable model,

[5:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=317s) I can sidestep this conundrum." Take this thought experiment for an example. You have a browser agent where each step has an independent success rate of ninety-nine percent. If your browser trajectory spans one hundred steps, the overall success rate at scale starts to look like thirty-six percent, only about a third of the time. That's not super great, and it's certainly not how we deliver reliable outcomes for customers in production. And so

[5:47](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=347s) you say, "Okay, well, we're continuously accumulating costs, we're tim-- continuously accumulating risk, and the percentage chance of value realization is very low." That would suggest to never use a browser agent. Uh, thankfully, that's not the subject of my talk today. I think this dynamic is real, but the math isn't the whole story. And so, it's time to take a step back. Let's think about what's important and how agents actually

[6:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=377s) deliver value. And to illustrate the value that I wish agents to deliver, uh, I'll propose a simple equation. Um, perhaps not used enough by many of us. And this equation is profit equals revenue minus costs. Yes, very simple. Um, but perhaps overlooked when thinking about browser agents. And this is all to say that your agent is just another line item, and that your agent should give you more than it takes.

[6:47](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=407s) But when we think about what your agent should give you, that begs the question of: what does a browser agent actually give you? And so, to answer this question, I like to think about the context of the customer. At Browserbase, we help our customers deploy browser automations for their users. Oftentimes, they're not automating the web for their own needs, but instead performing actions on the web on their customer's behalf. And so, a browser agent actually gives you a task completed for

[7:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=437s) your customer repeatedly. Well, uh, it gives you that ideally. And the question is, how you can create a dynamic system that affords the benefit of being able to wade through ambiguous problems, while also completing the task repeatedly every time when the environment stays the same? And so, we'll look across three dimensions, and in order of importance. First is performance, second is cost, and third is maintainability. And when we

[7:47](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=467s) think about performance, this is the primary measure of success that we want to focus on. Because once we know a browser agent is performance, it can complete the task we asked it to do, then cost and maintainability simply become optimization problems. And I like optimization problems because those can be addressed with engineering. So, the first thing we need to ask is what success actually looks like. And success in these transactional browser trajectories

[8:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=497s) look like some kind of concrete artifact that the run leaves behind. If you're paying a bill, that might look like a confirmation email. If you're placing an order, that mo- may look like an order ID or a receipt, or if you're submitting a form, that may look like a new record in a system that you can, uh, deterministically query. Second, how often is an agent correct? And there's two ways to measure that. Naively,

[8:47](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=527s) you may measure the success of an agent on a per run basis. Assume you have a browser agent. Its probability of success on any run is fifty percent. Yet, if you permit that agent to retry for a particular transaction, maybe up to four retries, your per transaction success rate quickly climbs to ninety-four percent. And so when you think about what metric matters, again, I'd implore that we ground ourselves in the customer. The customer does not

[9:17](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=557s) care necessarily how many retries you perform, but rather that the workflow was completed successfully and reliably. In short, permit retries and measure success against a per transaction basis.

[9:34](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=574s) Next, we'll explore cost, and we'll break cost down into two dimensions. The first is per run, which today is predominantly model cost. But as engineering evolves in the open source landscape for models, there is an incredible downward price pressure on the cost of intelligence. And I believe over time, model cost will become a smaller and smaller component of deploying browser agent trajectories. Next is infrastructure and compute. What are the real

[10:03](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=603s) resources you need to run your browser agent trajectories? And last is additional integration and tooling. Then we think about maintenance over time. We need to invest in observability, not only in how the agent made decisions, but what actually transpired in the browser. Second is developer time. When things break or things are not working optimally, people need to come in and reconfigure the harness or augment its capabilities. And last is

[10:32](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=632s) reevaluation as the problem changes. Sites change, models progress, and we need to reevaluate our point in time solution to last the test of time.

[10:46](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=646s) So, we talk about performance, we talked about unit costs, and we also talked about maintainability. What are the risk factors to the durability of these over time? Well, first and foremost, the environment can fight back. Anti-bots, and just generally, we understand the web as not the friendliest place for your agent. Second, the task changes. The underlying job shifts beneath the ag- shifts beneath the agent, and we ask the agent to wade through

[11:16](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=676s) ambiguity. Luckily, this also benefits from increase in model capability, as models are more able to intuitively understand the intention of the user, and therefore, in the face of something go wrong, course correct and ultimately complete the goal. Third is a better method appe- better methods appear. And this is sort of a canonical conundrum with automating the web. What happens if I'm automating the web today and that same website creates a API tomorrow? But for the vast

[11:45](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=705s) majority of browser agent use cases, uh, an API isn't coming anytime soon. And finally, the model strays off task. Models are inherently indeterministic. So from run to run, it may not id- follow the ideal path or the critical path to perform the action. So,

[12:06](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=726s) finally, we've come to the meat of the talk. How do we concretely generate shareholder value? This is the question. Uh, we understand that if a model performs well, then cost and maintainability are an optimization problem. So let's explore that in practice. Together, we're gonna go through a high-level architecture of a browser agent for real-world tasks. This task is automating a health insurance portal. Every step we take in creating this dynamic system should influence

[12:36](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=756s) one of these key levers: cost, performance, or maintainability. So first, we'll start with a lowly agent. The goal of this agent is to log in and download an explanation of benefits. A request comes in, the agent performs the action on the browser, and this is the minimum necessary system you need to just do something. Next, you realize that when you go to download that explanation of benefits, that downloading involves

[13:06](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=786s) programmatically interacting with the page. And then once you perform that download, you need to retrieve that file from storage and bring it into the agent's runtime. So you create a tool that does this all in one transaction. Instead of the model needing to make decisions on the fly, you encapsulate a complex operation into a single tool call. Next, you wanna give the agent an opportunity to verify its work real time. This can be done by giving it another tool.

[13:36](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=816s) This tool is an OCR tool that deterministically extracts entities from the document and compares that against a system of record. Great. Now the agent knows if it's did it-- if it really did its job right or not because it has a deterministic mechanism to verify that. Next, you realize that while this portal may have ambiguity in your business logic, that the authentication workflow is never changing, and you want to reuse the same

[14:05](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=845s) authentication mechanism for many business operations on the same portal. So you pull that authentication logic out from the responsibility of the model. Again, we're reducing the number of steps that the agent has to take to complete the task successfully. This means lower unit cogs, higher performance, and easier maintainability. As well as we can reuse this authentication function across multiple business logic workflows on the same portal.

[14:35](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=875s) Last, we've contained the responsibility of the agent to only the area of ambiguity necessary, so we draft a skill. These skills look very similar to a standard operating procedure that we would have humans use in the past, except this standard operating procedure is for an agent, and it allows the agent to follow the critical path more closely. At the beginning of the talk, we discussed how the output of an

[15:05](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=905s) agent can be thought of like a probability distribution. So if we introduce the critical path to the agent, it knows it's at step two and knows that it needs to take step three, because we have outlined it in this skill, then it's more likely to take that step three. It knows exactly where it needs to go, and we remove ambiguity from the agent's decision process. In the end, this is the final result. We have actually a pretty complex system. A request comes in, a

[15:34](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=934s) serverless authentication function authenticates the browser. That browser is provided to the agent runtime. The agent navigates the portal using the skill provided, calls a deterministic download function, provides that downloaded file to a verify tool, and can know authoritatively whether its trajectory succeeded or failed. In the end, this system takes steps out of the model's responsibility that simply do not need to

[16:04](https://www.youtube.com/watch?v=5xi_S1f9sDU&t=964s) be in the responsibility of the agent. We reduce the number of steps it needs to take while still being it-- able to achieve the adap-- desired outcome. Thus, the system's more maintainable and it's more performant, and we can overcome the browser-agent conundrum. So here we are. Welcome to production, and congrats. Now you're on call. Thank you.
