[
  {
    "startMs": 12720,
    "text": "Hey, so I'm Milind, MTS at Brainbase. We help enterprises deploy internal agents at scale and manage them. Today I'm gonna be talking about hill climbing back office agents and how we did that successfully with a customer."
  },
  {
    "startMs": 26380,
    "text": "So yeah, our customer in this case that I'm go-gonna be talking about throughout, and like the framework we used repeatedly is, is, is Domino's. As you all know, they have twenty-plus thousand stores, ninety-plus markets. We started off with a single region with their CMO office. We wanted to answer this question that they kept asking: What does a pizza cost at the competitor down the street? What, what does it cost in their delivery apps? What does it cost on their actual website, with promos, without promos? Things like that. They essentially wanted a spreadsheet at the end with the lowest price"
  },
  {
    "startMs": 55700,
    "text": "po-possible for a bunch of different items. Before we built the agent for them, they had an analyst manually doing these steps. They all look easy in isolation. You open a competitor's website, pick the right store, compare prices with delivery, takeaway, find the right promos, find the cheapest price, and write it all down into a spreadsheet. It, it all looks easy enough, but if you get one of these steps wrong, you look like you, you, you get a cell at the end that looks finished, but it's just wrong. We had to build an agent that does the same."
  },
  {
    "startMs": 85900,
    "text": "Reads a bunch of items, finds the website, interprets prices, fill the spreadsheets, hand it off to an agent, and flag anything that it thinks it's not sure of."
  },
  {
    "startMs": 96400,
    "text": "W-When we created version one of this agent for them, we had forty-one percent accuracy. By the end, at version eleven, we had ninety-seven percent accuracy. Our cost per source, per source means, um, per item, per, per br- competitor, per website, uh, went from two dollars per source to one dollar per source, and the amount of minutes it took per run went from one twenty-nine to seventy-one. All of this when just was-- just, just through instructions and, and skills and playbooks changing. Here is like the six steps that we use,"
  },
  {
    "startMs": 126540,
    "text": "not just with Domino's, but with other customers as well. We start off with kickoff, followed by ontology, benchmarks, first agent, iterating, and handing off. I'll talk about what these steps mean in detail and what we did for Domino's in each of these, and hopefully it's a repeatable framework that you all can use internally as well. So step one is kickoff. You should ideally talk to your customer or end user and agree on what the win looks like. As of today, what, what is the job? What does a human sign off on at the end of it? How long does it take? How much does"
  },
  {
    "startMs": 156580,
    "text": "it cost? How accurate does your agent need to be? Who judges the output? And what must the agent never do alone? What assumptions should it never make? For Domino's, we ended up creating this brief that worked really well for us. Our outcome at the end had to be a faster, repeatable way to collect local competitor pricing without losing the judgment edge cases need. So we asked them a- all of these questions. As of today, the analysts create a spreadsheet and the pricing team, uh, fin-finally analyzes it. An analyst"
  },
  {
    "startMs": 186540,
    "text": "spent two hours daily manually, actively working on creating the spreadsheet. Every cell was then checked and, and just confirmed, and the agent at the end just had to ask for the analyst to sign off and flag anything else it had to. Our win in numbers looked like just having the highest accuracy, the lowest cost per source as possible, runtime to be as low as possible, and before we even started, they weren't even sure if we can do dis-discount search at all. Step two is ontology. What does that exactly mean?"
  },
  {
    "startMs": 216640,
    "text": "We, we try to write down all the terms that, that the agent needs to know to never make assumptions, which means we need to tell the agent exactly what it's supposed to collect, where each answer comes from, where does it go, what does each term mean to the analysts that, that actually do the term, uh, do the work manually, and what should the agent do when it can't figure something out. If, if there's one rule here, it is that every term should ideally have a definition. If you don't write something down, the agent's gonna guess and probably get something wrong."
  },
  {
    "startMs": 247500,
    "text": "For Domino's, here's what it looked like. We at Brainbase have a memory primitive of our own that has memory that looks like a Postgres database that the agent can do whatever it wants with. We gave it a targets table where we gave it all the items. In this case, let's look at ID number fourteen. ID fourteen is brand B, it's a large pizza, uh, takeaway method-- sorry, it's delivery, and on app B. The agent hypothetically does browser use, finds the price on the website, finds that it's twenty-four and nineteen. Twenty-four is"
  },
  {
    "startMs": 277460,
    "text": "the striked out price, nineteen is the sale price. We also now have a rules file where we actually tell the agent to specifically mark down the price that it, that a user pays at checkout, not twenty-four. This came from like a valid failure. What it saves at the end, we created a table again called price observations, where it, with the corresponding ID, stores what price it saw, what the full price was, and the actual source. And at the end, we give it a spreadsheet that has all placeholder item IDs, and all it has to"
  },
  {
    "startMs": 307480,
    "text": "do is look up the ID and replace the price. When a cell is wrong, it points to the exact page and the rule behind why it chose the selected price. Other rules that we mentioned that came through actual live traces were definition of what a large pizza looks like. It's anything that's actually the item itself or a bundle. Delivery is not equal to takeaway. It, it got confused with that. It's supposed to do promotion math. If it's thirty percent off the price, it should do that before entering it in the spreadsheet."
  },
  {
    "startMs": 337880,
    "text": "And if it can't verify it, write it down as a gap and do not guess. Step three is figuring out benchmarks and evals. Now, now these steps can be interchangeable slightly, but essentially you're trying to figure out what you can run offline evals on and what you can run online evals on. For offline evals, you should ideally start off by asking them if they have past cases that they already have answers for. In this case, we had analysts from Domino's who had spreadsheets that they worked on. Can we actually rerun those Um, input"
  },
  {
    "startMs": 367820,
    "text": "cases to get the same final output. And then at the end, how do you figure out how to score each thing that the agent's doing? For evals, you try to figure out what you can check deterministically, what you can check via an LLM as a judge, and what you can check as an agent as a judge. That's a concept that we think is slightly more unique, where if there's an output artifact, you probably want an agent to take multiple turns, multiple tool calls to figure out how correct that, that artifact is. And at the end, what happens when something goes wrong and what happens when it's flagged?"
  },
  {
    "startMs": 397920,
    "text": "You should try to score as many things as you can with these three things. So yeah, at Domino's, our benchmark essentially started off with old spreadsheets that they had. We tried to create the input environment by going to archive.org and having snapshots of comparative websites at those given dates. What we saw that-- was that it didn't work with delivery apps because they didn't sl-serve data the same way. So we ended up storing browser screenshots for everything the agent was doing, starting from version one of the agent, and that's how we created our, our environment for, for the internal"
  },
  {
    "startMs": 427720,
    "text": "benchmark. For evals, we used code for things like checking if every ID cell was replaced with the price, if the discount map was computed correctly, if the price was twenty percent plus or minus compared to the previous run, was it flagged or not? LLM as a judge just checks the trace to see if the right item was selected, if the right promo was applied. Agent as a judge actually goes to every flagged price and actually confirms if the flag was valid or not. And the analyst at the end will go look at the final flags and"
  },
  {
    "startMs": 458040,
    "text": "try to figure out what the reason for it was if something went wrong. Step four is actually shipping some version of a first agent. You should ideally ship a narrow first agent in your stack that you can score from ver- from, from day one. You should probably figure out what the smallest slice worth scoring is. You should figure out how accurate things should be, how long... And, and you should use this agent to figure out how long does the run take, what, what does it cost, what does it get wrong, and things like that. Version one's job is to be measured, it's not to be"
  },
  {
    "startMs": 487980,
    "text": "good. You should widen the score-- uh, widen the scope after you get some baseline score that you're trying to meet. For Domino's, uh, like I mentioned earlier, version one for us was just forty-one percent correct. It took more than two hours. It was two dollars per source, and, and it just wasn't working well. We found a bunch of changes, made fixes, and then improved and, and increased the scope. So we started off with just one app, one item in one city, moved on to adding burgers and takeaway"
  },
  {
    "startMs": 517560,
    "text": "options, moved on to adding another app, and finally other websites as well. So starting off small, narrow scope worked out really well for us to, to figure out how we can fix things. Step five is to just iterate. You should score, fix, and ship. Here is where you'll do steps two, three, and four. Keep iterating, improving your ontology, improving your evals, your benchmarks to get things that work. You should figure out which failures are most expensive, most frequent, fix them, what changes should be used to fix"
  },
  {
    "startMs": 547580,
    "text": "them. If that change results in increased runtime or cost, if yes, are you okay with that increased runtime and cost? And how can you use those failures to create and add to your benchmark cases or eval criteria? Ideally, change one thing at a time and fix the instructions before you want to change the model entirely. At Domino's, we-- I, I highlighted two examples. Before I get to that, here is a graph of how our progression went. V1 started off at forty-one percent, um, more than two dollars a source."
  },
  {
    "startMs": 577900,
    "text": "V11 after a bunch of changes is one dollar a source and, and ninety-seven percent accurate. So yeah. Here's an example of a very simple rule change that we made. We gave the agent a link of, of a large pizza to find for brand B via delivery on its own website. It went to the link which had the section already highlighted. It clicked on it, found the large pizza, found the price, and ended there. The analyst caught, that is somebody manually caught at the end, that there"
  },
  {
    "startMs": 607940,
    "text": "was actually a deal below that website, below on the same page that was just twenty-one dollars instead and, and it missed that, right? So all we had to do there was add a simple rule to not stop at the section that it gets linked to, but actually look at the entire page. And a simple change like that helped us just, like, catch this and, and make sure it doesn't happen again. Another more intricate change that we had to make was actually browser use behavior of said agent. So in one of the runs, we saw that for brand D,"
  },
  {
    "startMs": 637980,
    "text": "this, this is a brand called brand D promotions page, where all the promotion prices were inside images, so not actually captured in text. What the agent did is it looked at all twenty-six promo images to find the item it, it was looking for. It looked at, looked at them again by zooming in f- four x, five x, six x. It found the right card at minute thirteen, but it still kept zooming to double-check and verify, and it, it was just in a loop, and we had to manually stop it at minute fifteen. We realized after"
  },
  {
    "startMs": 667740,
    "text": "manually looking at the trace and the website, that the HTML DOM is actually super important and could have saved the agent a lot of work here. So we, we tell the agent in, in our latest rules to go look at the HTML DOM first, look at things like image titles, alt text, before looking at the actual image reads and just stopping past a certain zoom level. What that resulted in is, is one-third the number of image reads, reducing cost by over fifty percent, and just us getting an actual answer and a finished report. So while we"
  },
  {
    "startMs": 697640,
    "text": "have automated checks and eval criteria to help us catch a lot of things, some things require our engineers to actually look at traces and, and browse around used tool calls and, and just see what went wrong. So yeah, we made a bunch of these changes, got to V11, got to ninety-seven percent accuracy, and now it's just figuring out, \"Hey, like, like, what next?\" So, so our, our way of this is trying to figure out how best to hand off to the customer at the end. We wanna figure out who's running these agents, who's reviewing the outputs, who's trying to correct, who's trying to take the"
  },
  {
    "startMs": 727380,
    "text": "corrections into and converting it into rules and benchmark cases, and who decides when a new version is ready to ship. All these questions are things you should probably have written down. Um, you're trying to hand off an entire loop and not just the agent. Ideally, the user should be empowered enough to make these changes if they're smart enough themselves. So for, for Domino's, that's exactly what we did. The agent now runs on their schedule, either multi-- um, daily or even via email trigger manually. Analysts review only the exceptions at this point. Each"
  },
  {
    "startMs": 756600,
    "text": "correction that they catch, they're automatically able to add as a benchmark case. If they can't, then they obviously-- we're always available for them. And we are now at a place where we are expanding with them, um, trying to get into multiple markets and multiple other, um, areas."
  },
  {
    "startMs": 774140,
    "text": "So yeah, this is basically how we, we used, um, our framework to work with Domino's to go from an agent that barely works to an agent that works almost perfectly while reducing the cost. We found this framework to be very effective for various other use cases, especially where you have messy sources of data. We're trying to figure out the right answer and, um, thanks."
  }
]
