How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs
Loading slides…
Read the talk
How Brainbase took Domino’s competitor-pricing agent from 41% to 97% accuracy
Milind Sunil Shah follows a competitor’s pizza price from a local listing to a cell an analyst can sign off on, showing how Brainbase used failed runs to improve accuracy, halve cost per source and cut runtime.
From a talk by Milind Sunil Shah
At a glance
Ideas worth remembering
Give each price a target ID, fulfillment method, source and interpretation rule so an analyst can trace a disputed cell back to the offer.
Save what the browser showed during collection. Historical spreadsheets supply expected answers, but replaying a pricing case also requires the offers available at the time.
Use code for arithmetic and completeness, a trace-reading LLM for item and promotion selection, and a tool-using judge for flags that need investigation.
Inspect the tool calls around a failure before adding more verification. Brand B needed a wider search, while Brand D needed a cheaper way to select images and a stopping rule.
What does a pizza cost down the street?
Milind Sunil Shah, a member of the technical staff at Brainbase Labs, builds and manages internal agents for enterprises. His example begins in the CMO’s office for one Domino’s region. Domino’s had more than 20,000 stores across more than 90 markets, but the question this office wanted answered was local. What was the lowest price a customer could pay for a particular item at a nearby competitor, given the store, ordering channel, fulfillment method and available promotions?
An analyst would open the competitor’s website or delivery app, select the local store, compare delivery and takeaway prices, look for promotions and enter the cheapest qualifying price in a spreadsheet. The pricing team used the finished sheet for analysis. An error early in that sequence could survive every subsequent step. A price from the wrong store looked just as plausible in a completed cell as the right one.
Brainbase’s first agent achieved 41% accuracy. By version eleven, accuracy had reached 97%, cost per source had fallen from $2 to $1, and elapsed time per run had fallen from 129 to 71 minutes. A source was an item–competitor–website or app combination, so checking a competitor’s own site and checking a delivery app were separate work. Shah attributed the gains to changes in instructions, skills and playbooks. The team used the same model throughout.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
1. Agree on what the analyst signs off on
Brainbase began its six-step process by asking the people doing the job to describe a successful day. What did they deliver, who judged it, how much work did it take, and which decisions required a person? An analyst spent about two hours of active work each day preparing the spreadsheet, and every cell was checked. That baseline measured human attention. The 129 and 71 minutes measured elapsed time for an agent run.
The regional office wanted a faster, repeatable collection process that retained the analyst’s judgment on edge cases. The agent would read the list of items, find and interpret prices, populate the existing report and flag anything it could not verify. The analyst would sign off before the pricing team used it. Accuracy, cost per source and runtime all mattered. Even the feasibility of discount search was an open question at kickoff, so the team needed a way to tell when the agent had found a price but missed the offer a customer could actually use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
2. Connect each target to a source, a pricing rule and a cell
Shah uses ontology for the definitions that make the analyst’s task executable. The team wrote down what to collect, where to find it, where to put it and what to record when the evidence ran out. Brainbase gave the agent a memory primitive resembling a Postgres database. A targets table held the requested items, a rules file described how to interpret an offer, and a price observations table held what the agent found.
Consider Shah’s hypothetical target 14. The request was for a large Brand B pizza, for delivery, on app B. Suppose the agent opened the listing and saw 24 struck through beside a sale price of 19. Both numbers belonged to the correct product. A previous failure had prompted an explicit rule to enter what the customer would pay at checkout. The agent would therefore record 19 as the observed price, retain 24 as the full price and save the source alongside the target ID.
The report already contained placeholder item IDs. To fill the cell marked 14, the agent could look up the corresponding observation and substitute its price. That arrangement also worked in reverse during review. An analyst questioning 19 could trace the cell to the page the agent read and the rule that made 19 the answer.
Several definitions came from live traces. A large pizza could qualify as an individual item or as part of a bundle. A takeaway listing could not satisfy a delivery target, a distinction the agent had confused. An offer of 30% off required the agent to calculate the discount before entering the price. When it could not verify an answer, the prescribed output was a gap.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
3. Reconstruct the input, then choose how to judge the answer
Shah separated two evaluation questions. Could the agent reproduce an analyst’s answer on a past case, and what could the team check during a live run? For the offline benchmark, past spreadsheets supplied the expected prices. To replay a case, Brainbase also needed the offers an analyst could see that day. The team looked for dated snapshots of competitor websites on archive.org, but the same approach failed for delivery apps, which served their data differently. Starting with version one, Brainbase saved browser screenshots during each run so its internal benchmark included the offers the agent had actually seen.
Shah distinguished three ways to score the work. First, deterministic checks in code could inspect the spreadsheet and its arithmetic. Did the agent replace each placeholder ID with a price? Did it calculate the discount correctly? If a price moved 20% up or down relative to the previous run, did it raise a flag? The last check tested whether the agent had called attention to a change. A substantial price change could still be genuine.
Second, an LLM as a judge could read the trace to decide whether the selected item matched the target and whether the agent applied the right promotion. Third, an agent as a judge could revisit each flagged price. It could take multiple turns and make tool calls to determine whether the evidence justified the flag. The human analyst then received the final flags and investigated the reason when something had gone wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
4. Measure one item on one app in one city
Brainbase chose a slice small enough to score from day one. One item on one app in one city still required the agent to select the store, interpret an offer and deliver a completed report. At 41% accuracy, more than two hours a run and about $2 per source, version one gave the engineers specific misses to investigate and a baseline for what each attempt cost.
The team made fixes before widening the assignment. It added burgers and takeaway options, then another app, then other websites. That order matters when reading the version-one and version-eleven figures. The later agent covered a broader task, and the engineers had a record of failures from the smaller task to test as they expanded.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
5. Score a failure, change one thing and measure the cost
Iteration meant returning to the definitions, benchmark and running agent. Shah advised prioritizing failures by frequency and expense. A recurring miss might warrant a new pricing rule, a benchmark case that preserved the offer, or a change in how the agent used the browser. He recommended changing one thing at a time where possible and fixing instructions before considering a model change. Additional verification had a price of its own, so the team checked what each fix did to runtime and cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The $21 deal below the linked section
For one Brand B target, the agent received a link to a large pizza for delivery on the competitor’s own website. The link highlighted the relevant section. The agent clicked through, found the large pizza, read its price and finished. During review, an analyst spotted a cheaper $21 deal farther down the same page.
The item selection was correct. The mistake was treating the first matching offer as sufficient evidence of the lowest price. Brainbase changed the instruction to require the agent to examine the entire page, including offers outside the section the link opened.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The promotion card the agent found at minute 13
On Brand D’s promotions page, prices appeared inside images. The agent inspected all 26 promotional images, then examined them again at four, five and six times magnification. It found the right card at minute 13 and continued zooming to double-check it. At minute 15, the team stopped the run manually. Here the question was how to narrow the search and give verification an endpoint.
An engineer reviewed the sequence of tool calls alongside the website. The HTML DOM offered information the agent could inspect before reading pixels, including image titles and alt text. Brainbase changed the browser instructions to examine those first, then inspect the relevant images. It also set a limit on zooming.
For this case, the revised agent used one-third as many image reads, cost over 50% less and produced a finished report. The trace showed something the runtime figure alone could not reveal. By minute 13 the agent had already found the card, and its remaining calls were spent rechecking it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
6. Give the pricing team ownership of the next correction
At handoff, Shah asks teams to write down who runs the agent, who reviews the output, who converts a correction into a rule or benchmark case, and who decides when a version is ready to ship. Fixing a cell gives the pricing team the right number today. Preserving the case lets the owner of the next version check whether it can find that number unaided.
In the Domino’s deployment, the agent could run daily on the team’s schedule or on demand through an email trigger. Analysts who had checked every cell now reviewed exceptions and could add corrections to the benchmark themselves, with Brainbase available to help. At the time Shah described the project, the teams were working on expanding it to additional markets and areas.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…