How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs

Loading slides…

Read the talk

How Brainbase took Domino’s competitor-pricing agent from 41% to 97% accuracy

Milind Sunil Shah follows a competitor’s pizza price from a local listing to a cell an analyst can sign off on, showing how Brainbase used failed runs to improve accuracy, halve cost per source and cut runtime.

From a talk by Milind Sunil Shah

At a glance

Ideas worth remembering

  • Give each price a target ID, fulfillment method, source and interpretation rule so an analyst can trace a disputed cell back to the offer.

  • Save what the browser showed during collection. Historical spreadsheets supply expected answers, but replaying a pricing case also requires the offers available at the time.

  • Use code for arithmetic and completeness, a trace-reading LLM for item and promotion selection, and a tool-using judge for flags that need investigation.

  • Inspect the tool calls around a failure before adding more verification. Brand B needed a wider search, while Brand D needed a cheaper way to select images and a stopping rule.

What does a pizza cost down the street?

Milind Sunil Shah, a member of the technical staff at Brainbase Labs, builds and manages internal agents for enterprises. His example begins in the CMO’s office for one Domino’s region. Domino’s had more than 20,000 stores across more than 90 markets, but the question this office wanted answered was local. What was the lowest price a customer could pay for a particular item at a nearby competitor, given the store, ordering channel, fulfillment method and available promotions?

An analyst would open the competitor’s website or delivery app, select the local store, compare delivery and takeaway prices, look for promotions and enter the cheapest qualifying price in a spreadsheet. The pricing team used the finished sheet for analysis. An error early in that sequence could survive every subsequent step. A price from the wrong store looked just as plausible in a completed cell as the right one.

Six cards trace the manual task from opening a competitor’s site to entering prices in a spreadsheet. A warning identifies two ways a cell can be wrong, and a bottom strip shows where the agent flags uncertainty for a human.Open full source frame
Six steps for collecting competitor prices

Brainbase’s first agent achieved 41% accuracy. By version eleven, accuracy had reached 97%, cost per source had fallen from $2 to $1, and elapsed time per run had fallen from 129 to 71 minutes. A source was an item–competitor–website or app combination, so checking a competitor’s own site and checking a delivery app were separate work. Shah attributed the gains to changes in instructions, skills and playbooks. The team used the same model throughout.

0:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

1. Agree on what the analyst signs off on

Brainbase began its six-step process by asking the people doing the job to describe a successful day. What did they deliver, who judged it, how much work did it take, and which decisions required a person? An analyst spent about two hours of active work each day preparing the spreadsheet, and every cell was checked. That baseline measured human attention. The 129 and 71 minutes measured elapsed time for an agent run.

The regional office wanted a faster, repeatable collection process that retained the analyst’s judgment on edge cases. The agent would read the list of items, find and interpret prices, populate the existing report and flag anything it could not verify. The analyst would sign off before the pricing team used it. Accuracy, cost per source and runtime all mattered. Even the feasibility of discount search was an open question at kickoff, so the team needed a way to tell when the agent had found a price but missed the offer a customer could actually use.

A four-row table applies the kickoff questions to the pricing team’s spreadsheet. It gives the analyst’s time per run, the cell-by-cell accuracy check, and the human sign-off requirement. A footer lists the measures of success.Open full source frame
Domino’s brief for collecting local competitor prices
2:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:23 · section reference included

2. Connect each target to a source, a pricing rule and a cell

Shah uses ontology for the definitions that make the analyst’s task executable. The team wrote down what to collect, where to find it, where to put it and what to record when the evidence ran out. Brainbase gave the agent a memory primitive resembling a Postgres database. A targets table held the requested items, a rules file described how to interpret an offer, and a price observations table held what the agent found.

Consider Shah’s hypothetical target 14. The request was for a large Brand B pizza, for delivery, on app B. Suppose the agent opened the listing and saw 24 struck through beside a sale price of 19. Both numbers belonged to the correct product. A previous failure had prompted an explicit rule to enter what the customer would pay at checkout. The agent would therefore record 19 as the observed price, retain 24 as the full price and save the source alongside the target ID.

The report already contained placeholder item IDs. To fill the cell marked 14, the agent could look up the corresponding observation and substitute its price. That arrangement also worked in reverse during review. An analyst questioning 19 could trace the cell to the page the agent read and the rule that made 19 the answer.

Four linked panels follow Brand B’s large delivery pizza from target ID 14 to a spreadsheet cell. The agent records the checkout price of 19, the full price of 24, and a source link, then replaces ID 14 with 19.Open full source frame
The agent looks it up instead of guessing.

Several definitions came from live traces. A large pizza could qualify as an individual item or as part of a bundle. A takeaway listing could not satisfy a delivery target, a distinction the agent had confused. An offer of 30% off required the agent to calculate the discount before entering the price. When it could not verify an answer, the prescribed output was a gap.

3:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:35 · section reference included

3. Reconstruct the input, then choose how to judge the answer

Shah separated two evaluation questions. Could the agent reproduce an analyst’s answer on a past case, and what could the team check during a live run? For the offline benchmark, past spreadsheets supplied the expected prices. To replay a case, Brainbase also needed the offers an analyst could see that day. The team looked for dated snapshots of competitor websites on archive.org, but the same approach failed for delivery apps, which served their data differently. Starting with version one, Brainbase saved browser screenshots during each run so its internal benchmark included the offers the agent had actually seen.

Shah distinguished three ways to score the work. First, deterministic checks in code could inspect the spreadsheet and its arithmetic. Did the agent replace each placeholder ID with a price? Did it calculate the discount correctly? If a price moved 20% up or down relative to the previous run, did it raise a flag? The last check tested whether the agent had called attention to a change. A substantial price change could still be genuine.

Second, an LLM as a judge could read the trace to decide whether the selected item matched the target and whether the agent applied the right promotion. Third, an agent as a judge could revisit each flagged price. It could take multiple turns and make tool calls to determine whether the evidence justified the flag. The human analyst then received the final flags and investigated the reason when something had gone wrong.

Two columns show how Domino’s scores replayed pages against corrected spreadsheets and checks live runs. The live checks progress from code to an LLM, a page-checking agent, and an analyst. The LLM’s questions are highlighted.Open full source frame
A benchmark before shipping, evals on every run.
5:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:43 · section reference included

4. Measure one item on one app in one city

Brainbase chose a slice small enough to score from day one. One item on one app in one city still required the agent to select the store, interpret an offer and deliver a completed report. At 41% accuracy, more than two hours a run and about $2 per source, version one gave the engineers specific misses to investigate and a baseline for what each attempt cost.

The team made fixes before widening the assignment. It added burgers and takeaway options, then another app, then other websites. That order matters when reading the version-one and version-eleven figures. The later agent covered a broader task, and the engineers had a record of failures from the smaller task to test as they expanded.

Two bars show 100% completion and 41% of cells correct without an analyst’s fix. Five mistake types point to specific fixes. A July timeline tracks expansion from two medium pizzas on one app in one city to competitors’ own sites.Open full source frame
Version one: done, and 41% right.
7:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:42 · section reference included

5. Score a failure, change one thing and measure the cost

Iteration meant returning to the definitions, benchmark and running agent. Shah advised prioritizing failures by frequency and expense. A recurring miss might warrant a new pricing rule, a benchmark case that preserved the offer, or a change in how the agent used the browser. He recommended changing one thing at a time where possible and fixing instructions before considering a model change. Additional verification had a price of its own, so the team checked what each fix did to runtime and cost.

8:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:49 · section reference included

The $21 deal below the linked section

For one Brand B target, the agent received a link to a large pizza for delivery on the competitor’s own website. The link highlighted the relevant section. The agent clicked through, found the large pizza, read its price and finished. During review, an analyst spotted a cheaper $21 deal farther down the same page.

The item selection was correct. The mistake was treating the first matching offer as sufficient evidence of the lowest price. Brainbase changed the instruction to require the agent to examine the entire page, including offers outside the section the link opened.

A delivery menu contrasts the agent’s pick, a large pizza priced at 26, with the analyst’s find in Deals, a large pizza and drink priced at 21. Beside it are the resulting instruction and the check applied on every run.Open full source frame
One correction became a rule and a check.
9:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:44 · section reference included

The promotion card the agent found at minute 13

On Brand D’s promotions page, prices appeared inside images. The agent inspected all 26 promotional images, then examined them again at four, five and six times magnification. It found the right card at minute 13 and continued zooming to double-check it. At minute 15, the team stopped the run manually. Here the question was how to narrow the search and give verification an endpoint.

An engineer reviewed the sequence of tool calls alongside the website. The HTML DOM offered information the agent could inspect before reading pixels, including image titles and alt text. Brainbase changed the browser instructions to examine those first, then inspect the relevant images. It also set a limit on zooming.

For this case, the revised agent used one-third as many image reads, cost over 50% less and produced a finished report. The trace showed something the runtime figure alone could not reveal. By minute 13 the agent had already found the card, and its remaining calls were spent rechecking it.

Two panels compare runs on Brand D’s promotions page. The first used 61 image reads, cost $6.93, and stopped at minute 15. The second used HTML titles to narrow the search and finished the report with 21 image reads for $3.17.Open full source frame
Same answer, half the cost.
10:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:28 · section reference included

6. Give the pricing team ownership of the next correction

At handoff, Shah asks teams to write down who runs the agent, who reviews the output, who converts a correction into a rule or benchmark case, and who decides when a version is ready to ship. Fixing a cell gives the pricing team the right number today. Preserving the case lets the owner of the next version check whether it can find that number unaided.

In the Domino’s deployment, the agent could run daily on the team’s schedule or on demand through an email trigger. Analysts who had checked every cell now reviewed exceptions and could add corrections to the benchmark themselves, with Brainbase available to help. At the time Shah described the project, the teams were working on expanding it to additional markets and areas.

11:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:56 · section reference included

Transcript download
Full transcript

Loading transcript…