[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12639,
    "text": "All right. Hello everybody. Thank you so"
  },
  {
    "startMs": 14240,
    "text": "much for coming. My name is Hassan. Uh"
  },
  {
    "startMs": 16240,
    "text": "this is my dad Omar. And uh we're very"
  },
  {
    "startMs": 18560,
    "text": "excited to be here today uh to to talk."
  },
  {
    "startMs": 21119,
    "text": "Uh I think the only father-son talk at"
  },
  {
    "startMs": 23600,
    "text": "this conference. So um thank you."
  },
  {
    "startMs": 26200,
    "text": "[cheering] Thank you. [applause]"
  },
  {
    "startMs": 26880,
    "text": "Thanks."
  },
  {
    "startMs": 29920,
    "text": "So, I'm here to talk about, you know,"
  },
  {
    "startMs": 31760,
    "text": "can LLMs actually uh make financial"
  },
  {
    "startMs": 35200,
    "text": "forecasts of a company? And so, we used"
  },
  {
    "startMs": 36960,
    "text": "Costco. We gave an open model kind of"
  },
  {
    "startMs": 39360,
    "text": "everything uh that was public from"
  },
  {
    "startMs": 41360,
    "text": "Costco's filings up until a certain"
  },
  {
    "startMs": 43040,
    "text": "point, and we had it try to predict"
  },
  {
    "startMs": 44960,
    "text": "Costco's profits over the next eight"
  },
  {
    "startMs": 47360,
    "text": "quarters. So, we'll start off with some"
  },
  {
    "startMs": 49360,
    "text": "intros. My name is Hassan. Uh I lead"
  },
  {
    "startMs": 51440,
    "text": "developer experience at Together AI. The"
  },
  {
    "startMs": 53120,
    "text": "main thing to know about me is I really"
  },
  {
    "startMs": 54480,
    "text": "like building AI apps. Everything I do"
  },
  {
    "startMs": 56239,
    "text": "is open source on GitHub. And uh this is"
  },
  {
    "startMs": 58320,
    "text": "another one of the apps that I built uh"
  },
  {
    "startMs": 60000,
    "text": "but this one was in collaboration uh"
  },
  {
    "startMs": 61760,
    "text": "with my dad and I work over at Together"
  },
  {
    "startMs": 64000,
    "text": "AAI. We used it for the open source"
  },
  {
    "startMs": 65760,
    "text": "models in in this talk. We're a platform"
  },
  {
    "startMs": 67680,
    "text": "for uh running and deploying and"
  },
  {
    "startMs": 70479,
    "text": "training open source models."
  },
  {
    "startMs": 72960,
    "text": "Hey guys, my name is Omargari. So um"
  },
  {
    "startMs": 76000,
    "text": "usually parents they teach children but"
  },
  {
    "startMs": 78159,
    "text": "Hessen he was teaching me a lot during"
  },
  {
    "startMs": 80560,
    "text": "the last six years when I uh I got to"
  },
  {
    "startMs": 82720,
    "text": "coding after 25 years of experience with"
  },
  {
    "startMs": 85280,
    "text": "the big four firms auditing firms uh"
  },
  {
    "startMs": 88000,
    "text": "couple of positions as a CFO and audit"
  },
  {
    "startMs": 90479,
    "text": "uh head of internal audit of"
  },
  {
    "startMs": 92400,
    "text": "multi-billion dollar company. So today"
  },
  {
    "startMs": 95200,
    "text": "I'm in the human in the loop for"
  },
  {
    "startMs": 96960,
    "text": "Hassan's um engineering work."
  },
  {
    "startMs": 101360,
    "text": "Okay. So my coding experience didn't"
  },
  {
    "startMs": 104320,
    "text": "start just last six years. Um I had a"
  },
  {
    "startMs": 106799,
    "text": "lot of passion for coding at at age of"
  },
  {
    "startMs": 108640,
    "text": "16. I was coding in assembly and the"
  },
  {
    "startMs": 111520,
    "text": "reason I did that I started with basic a"
  },
  {
    "startMs": 113759,
    "text": "tennis game small tennis game on an"
  },
  {
    "startMs": 115600,
    "text": "epson with four lines on the screen and"
  },
  {
    "startMs": 118560,
    "text": "just this four um you know tennis uh"
  },
  {
    "startMs": 120719,
    "text": "court but the ball was too slow. It took"
  },
  {
    "startMs": 123280,
    "text": "like almost a minute to get to the other"
  },
  {
    "startMs": 125040,
    "text": "side. And then I learned assembly"
  },
  {
    "startMs": 127040,
    "text": "because of my brother's uh IT"
  },
  {
    "startMs": 128640,
    "text": "background. And then I rewrote the the"
  },
  {
    "startMs": 131280,
    "text": "same game and it took the ball. I can't"
  },
  {
    "startMs": 133360,
    "text": "see it anymore. It was too fast to so I"
  },
  {
    "startMs": 136319,
    "text": "had to slow it down to really see"
  },
  {
    "startMs": 138080,
    "text": "things. So just wanted to share that"
  },
  {
    "startMs": 139520,
    "text": "with you."
  },
  {
    "startMs": 141280,
    "text": "All right. So spoiler alert. Um the LLM"
  },
  {
    "startMs": 145040,
    "text": "was accurate up until about 4.4%. So the"
  },
  {
    "startMs": 148560,
    "text": "our our forecast of Costco's actual"
  },
  {
    "startMs": 150560,
    "text": "membership was off by 4.4%. And so I I"
  },
  {
    "startMs": 152879,
    "text": "thought this was pretty good, right? We"
  },
  {
    "startMs": 154239,
    "text": "had this LLM build a financial model. We"
  },
  {
    "startMs": 156160,
    "text": "evaled it. Um ended up giving us this"
  },
  {
    "startMs": 158239,
    "text": "result. Um, and then I showed it to my"
  },
  {
    "startMs": 160640,
    "text": "dad and it turned out it got the right"
  },
  {
    "startMs": 162720,
    "text": "answer or got close to the right answer,"
  },
  {
    "startMs": 164640,
    "text": "uh, but for the wrong reasons. And so"
  },
  {
    "startMs": 166000,
    "text": "the rest of this talk is going to be"
  },
  {
    "startMs": 167280,
    "text": "going over, you know, how we built this"
  },
  {
    "startMs": 168879,
    "text": "financial model, how we evalued it, and"
  },
  {
    "startMs": 170959,
    "text": "actually what it ended up getting wrong"
  },
  {
    "startMs": 172800,
    "text": "and and kind of lessons learned along"
  },
  {
    "startMs": 174560,
    "text": "the way, including kind of the"
  },
  {
    "startMs": 175760,
    "text": "overarching lesson, which is having like"
  },
  {
    "startMs": 177920,
    "text": "an expert human in the loop, is is very,"
  },
  {
    "startMs": 179840,
    "text": "very important when you're building, you"
  },
  {
    "startMs": 181440,
    "text": "know, AI agents in in these different"
  },
  {
    "startMs": 183040,
    "text": "spaces. So, really quick background"
  },
  {
    "startMs": 185360,
    "text": "about Costco. Costco sells a lot of"
  },
  {
    "startMs": 187760,
    "text": "stuff. They sell $300 billion worth of"
  },
  {
    "startMs": 189599,
    "text": "stuff every year at close to cost. So"
  },
  {
    "startMs": 191920,
    "text": "they make very little money from that"
  },
  {
    "startMs": 193440,
    "text": "$300 billion. Um actually more than half"
  },
  {
    "startMs": 195840,
    "text": "of their revenue is from their"
  },
  {
    "startMs": 197120,
    "text": "membership fees, right? So they charge"
  },
  {
    "startMs": 198480,
    "text": "you to to come at the door. Um their"
  },
  {
    "startMs": 201440,
    "text": "total operating profit was about $12"
  },
  {
    "startMs": 203280,
    "text": "billion and and a little bit more than"
  },
  {
    "startMs": 204879,
    "text": "half of that was from membership fees."
  },
  {
    "startMs": 206560,
    "text": "So membership fees are are, you know,"
  },
  {
    "startMs": 208080,
    "text": "extremely important when we're trying to"
  },
  {
    "startMs": 209519,
    "text": "forecast uh Costco's uh revenue and"
  },
  {
    "startMs": 212080,
    "text": "profits. And thankfully Costco gave us"
  },
  {
    "startMs": 214480,
    "text": "the perfect test on July 2024 because"
  },
  {
    "startMs": 217200,
    "text": "they raised their membership prices for"
  },
  {
    "startMs": 219680,
    "text": "the first time since 2017. So Costco has"
  },
  {
    "startMs": 222480,
    "text": "two tiers. They have a standard tier"
  },
  {
    "startMs": 223680,
    "text": "that they raised from 60 to $65 and they"
  },
  {
    "startMs": 226400,
    "text": "have an executive tier that they raised"
  },
  {
    "startMs": 228000,
    "text": "from 120 to $130 uh. So what we did is"
  },
  {
    "startMs": 231519,
    "text": "we came up with this cutoff which was a"
  },
  {
    "startMs": 233360,
    "text": "couple weeks before the announcement and"
  },
  {
    "startMs": 235760,
    "text": "we took an open source model and we gave"
  },
  {
    "startMs": 238239,
    "text": "it access to all of these documents from"
  },
  {
    "startMs": 240480,
    "text": "Costco uh up until that point and asked"
  },
  {
    "startMs": 243519,
    "text": "it like hey if Costco you know increases"
  },
  {
    "startMs": 245920,
    "text": "its membership fee um what are the the"
  },
  {
    "startMs": 249200,
    "text": "the revenue and the profit that you"
  },
  {
    "startMs": 251120,
    "text": "would uh forecast. So we actually"
  },
  {
    "startMs": 252879,
    "text": "started doing this with a couple of"
  },
  {
    "startMs": 254640,
    "text": "recent open source models like GLM 5.3"
  },
  {
    "startMs": 256720,
    "text": "and Kimmy K3. But it turns out they"
  },
  {
    "startMs": 258560,
    "text": "already knew that Costco actually"
  },
  {
    "startMs": 260239,
    "text": "increased their prices um because it was"
  },
  {
    "startMs": 262240,
    "text": "already in their training data, right?"
  },
  {
    "startMs": 264000,
    "text": "So we had to go and find, you know,"
  },
  {
    "startMs": 266160,
    "text": "older open models who were trained, you"
  },
  {
    "startMs": 268639,
    "text": "know, up until before July 2024. So we"
  },
  {
    "startMs": 271360,
    "text": "found GBT OSS120B and Llama 3.3 uh7B. Uh"
  },
  {
    "startMs": 275759,
    "text": "and we this is an example of a question"
  },
  {
    "startMs": 277360,
    "text": "we asked it. Hey, did Costco change its"
  },
  {
    "startMs": 279040,
    "text": "membership fees in 2024? The newer"
  },
  {
    "startMs": 281280,
    "text": "models uh knew the answer. The older"
  },
  {
    "startMs": 283199,
    "text": "models didn't. And so we wanted to give"
  },
  {
    "startMs": 285520,
    "text": "we wanted to use these older models"
  },
  {
    "startMs": 286880,
    "text": "because you know you want a completely"
  },
  {
    "startMs": 288080,
    "text": "unbiased uh uh point of view here. So"
  },
  {
    "startMs": 290240,
    "text": "that's kind of the first lesson. If"
  },
  {
    "startMs": 291360,
    "text": "you're trying to do any sort of"
  },
  {
    "startMs": 292240,
    "text": "forecasting, you need to try to find a"
  },
  {
    "startMs": 293840,
    "text": "model whose training data ends before"
  },
  {
    "startMs": 296080,
    "text": "the event that you're actually trying to"
  },
  {
    "startMs": 297520,
    "text": "forecast. Um and uh for you know open"
  },
  {
    "startMs": 300639,
    "text": "weight models are actually really good"
  },
  {
    "startMs": 301759,
    "text": "for for this kind of use case. Um so"
  },
  {
    "startMs": 304160,
    "text": "this is kind of how everything works. We"
  },
  {
    "startMs": 306080,
    "text": "went and scraped about 214 uh public"
  },
  {
    "startMs": 308880,
    "text": "filings from Costco. uh you know,"
  },
  {
    "startMs": 311039,
    "text": "quarterly and yearly earning reports,"
  },
  {
    "startMs": 312800,
    "text": "sales reports, um all of that. We uh"
  },
  {
    "startMs": 316320,
    "text": "ended up making it into a list of about"
  },
  {
    "startMs": 318479,
    "text": "3,000 facts that were all timestamped"
  },
  {
    "startMs": 320639,
    "text": "and and then we filtered these facts to"
  },
  {
    "startMs": 323039,
    "text": "only the facts available before our"
  },
  {
    "startMs": 324720,
    "text": "cutoff date. We gave it to a model and"
  },
  {
    "startMs": 326960,
    "text": "we actually spun up um 16 sessions of of"
  },
  {
    "startMs": 329840,
    "text": "GPTOSs 12B. We had the model actually"
  },
  {
    "startMs": 333039,
    "text": "write a bunch of formulas and write this"
  },
  {
    "startMs": 334800,
    "text": "financial model. we ran some checks and"
  },
  {
    "startMs": 336880,
    "text": "then we picked uh the from those 16 runs"
  },
  {
    "startMs": 339520,
    "text": "we picked the ones that actually passed"
  },
  {
    "startMs": 341199,
    "text": "uh our checks. So in in this case like"
  },
  {
    "startMs": 344000,
    "text": "what does the model actually do right in"
  },
  {
    "startMs": 345600,
    "text": "this previous slide where we we gave it"
  },
  {
    "startMs": 347680,
    "text": "all these facts. What does the model"
  },
  {
    "startMs": 349360,
    "text": "output here? So we actually had the"
  },
  {
    "startMs": 350960,
    "text": "model output formulas here um in and and"
  },
  {
    "startMs": 354080,
    "text": "so an example of this is all the way to"
  },
  {
    "startMs": 356320,
    "text": "the left we say like we account for"
  },
  {
    "startMs": 358080,
    "text": "membership fee revenue on a deferred"
  },
  {
    "startMs": 359600,
    "text": "basis. This is straight from Costco's"
  },
  {
    "startMs": 361039,
    "text": "quarterly report. We give that to the"
  },
  {
    "startMs": 362479,
    "text": "model and it constructs this uh uh"
  },
  {
    "startMs": 364880,
    "text": "structure JSON where it says you know"
  },
  {
    "startMs": 366800,
    "text": "the operand is ratable the term is 52"
  },
  {
    "startMs": 368880,
    "text": "weeks. And so what this means is"
  },
  {
    "startMs": 370240,
    "text": "basically, you know, even if a user"
  },
  {
    "startMs": 372000,
    "text": "comes in and pays a $65 membership fee,"
  },
  {
    "startMs": 374479,
    "text": "it's actually spread across 52 weeks."
  },
  {
    "startMs": 376880,
    "text": "Um, and so this is really really"
  },
  {
    "startMs": 378080,
    "text": "important when you're doing uh uh uh"
  },
  {
    "startMs": 379680,
    "text": "forecasting to understand how to to to"
  },
  {
    "startMs": 382960,
    "text": "forecast. And so we had the model"
  },
  {
    "startMs": 384479,
    "text": "construct a list of all of these"
  },
  {
    "startMs": 386400,
    "text": "formulas. And so the main takeaway here"
  },
  {
    "startMs": 388000,
    "text": "is like have models write structured"
  },
  {
    "startMs": 389840,
    "text": "data instead of um having them, you"
  },
  {
    "startMs": 392080,
    "text": "know, do the math and do and do"
  },
  {
    "startMs": 393440,
    "text": "everything all at once. Um the model"
  },
  {
    "startMs": 395600,
    "text": "also came up with a bunch of"
  },
  {
    "startMs": 396720,
    "text": "assumptions. Uh and so it had to guess"
  },
  {
    "startMs": 398240,
    "text": "on a few things. So, for example, what"
  },
  {
    "startMs": 400000,
    "text": "percentage of people are in the standard"
  },
  {
    "startMs": 401759,
    "text": "tier versus the executive tier? Um, how"
  },
  {
    "startMs": 404720,
    "text": "many members were in Canada can US and"
  },
  {
    "startMs": 406319,
    "text": "Canada and impacted by this change. U,"
  },
  {
    "startMs": 408240,
    "text": "so it made a few assumptions. My dad's"
  },
  {
    "startMs": 410080,
    "text": "going to go over some of these"
  },
  {
    "startMs": 410880,
    "text": "assumptions later and why some of them"
  },
  {
    "startMs": 412319,
    "text": "may or may not be wrong. Um, and and"
  },
  {
    "startMs": 414400,
    "text": "here's kind of the full picture over"
  },
  {
    "startMs": 415759,
    "text": "here for the full financial model. On"
  },
  {
    "startMs": 417280,
    "text": "the left, you have the assumptions,"
  },
  {
    "startMs": 418639,
    "text": "which we went over some on the last"
  },
  {
    "startMs": 420000,
    "text": "slide, um, that the model came up with."
  },
  {
    "startMs": 422880,
    "text": "And then we have 14 formulas that the"
  },
  {
    "startMs": 424560,
    "text": "model also came up with. Um, and and"
  },
  {
    "startMs": 426479,
    "text": "then we have a few numbers that are kind"
  },
  {
    "startMs": 427919,
    "text": "of directly from uh Costco's filings."
  },
  {
    "startMs": 429919,
    "text": "And this is actually an Excel"
  },
  {
    "startMs": 431039,
    "text": "spreadsheet that that I created. I'm"
  },
  {
    "startMs": 432560,
    "text": "kind of just trying to visualize it in a"
  },
  {
    "startMs": 434000,
    "text": "way uh where where it's uh it's a little"
  },
  {
    "startMs": 436400,
    "text": "bit nicer to to look at. Um, and so"
  },
  {
    "startMs": 438800,
    "text": "after we had the model create this"
  },
  {
    "startMs": 440400,
    "text": "financial model, we uh actually had it"
  },
  {
    "startMs": 444319,
    "text": "go through a bunch of these checks. Uh,"
  },
  {
    "startMs": 446240,
    "text": "and so there's just basic checks. For"
  },
  {
    "startMs": 447599,
    "text": "example, a formula can't depend on"
  },
  {
    "startMs": 449039,
    "text": "itself. For example, raising the"
  },
  {
    "startMs": 450400,
    "text": "membership fee needs to raise revenue."
  },
  {
    "startMs": 452400,
    "text": "you know, just making sure that like"
  },
  {
    "startMs": 453759,
    "text": "some a few basic things were were"
  },
  {
    "startMs": 455520,
    "text": "correct. And whenever a model whenever"
  },
  {
    "startMs": 458000,
    "text": "one of those runs, because remember we"
  },
  {
    "startMs": 459440,
    "text": "had these 16 runs, whenever one of those"
  },
  {
    "startMs": 461199,
    "text": "runs got one of these things wrong, we"
  },
  {
    "startMs": 462800,
    "text": "ended up passing the error back to the"
  },
  {
    "startMs": 464720,
    "text": "model uh up to two different times uh so"
  },
  {
    "startMs": 467520,
    "text": "that it kind of tried to self-correct"
  },
  {
    "startMs": 469759,
    "text": "itself. And what was interesting here,"
  },
  {
    "startMs": 471440,
    "text": "so these are actually the the 16"
  },
  {
    "startMs": 473120,
    "text": "different runs that we have. They're all"
  },
  {
    "startMs": 474639,
    "text": "running. They're all creating their"
  },
  {
    "startMs": 475680,
    "text": "financial model. They're writing the the"
  },
  {
    "startMs": 477520,
    "text": "formulas. We're passing the errors back."
  },
  {
    "startMs": 479840,
    "text": "And what ended up happening is something"
  },
  {
    "startMs": 481199,
    "text": "interesting where we had a bunch of them"
  },
  {
    "startMs": 482800,
    "text": "fail anyway. Um because they just failed"
  },
  {
    "startMs": 485120,
    "text": "more than the two times we we we gave"
  },
  {
    "startMs": 487039,
    "text": "them back their their error. Um and so"
  },
  {
    "startMs": 489520,
    "text": "we ended up uh using these five that"
  },
  {
    "startMs": 491599,
    "text": "ended up passing uh for for our uh"
  },
  {
    "startMs": 493919,
    "text": "financial forecast in general. And"
  },
  {
    "startMs": 495360,
    "text": "that's kind of another lesson. We ended"
  },
  {
    "startMs": 497120,
    "text": "up starting with just like one prompt"
  },
  {
    "startMs": 498560,
    "text": "that we kept tuning and tuning and"
  },
  {
    "startMs": 499840,
    "text": "tuning. We're like, \"Oh, like surely"
  },
  {
    "startMs": 501440,
    "text": "like we're going to give this a GPT"
  },
  {
    "startMs": 502800,
    "text": "OSS120B and it's going to pass, you"
  },
  {
    "startMs": 504560,
    "text": "know, all the checks.\" Um, but that"
  },
  {
    "startMs": 506160,
    "text": "ended up like just taking so much time"
  },
  {
    "startMs": 507759,
    "text": "that we were like, what if we just spun"
  },
  {
    "startMs": 509199,
    "text": "up, you know, 20 16 different runs. Uh,"
  },
  {
    "startMs": 512159,
    "text": "and in this case, it works really well"
  },
  {
    "startMs": 513360,
    "text": "because the model is really cheap. So,"
  },
  {
    "startMs": 514719,
    "text": "it only cost us 19 cents to spin up"
  },
  {
    "startMs": 516560,
    "text": "these like 16 different runs and we had"
  },
  {
    "startMs": 518479,
    "text": "more of them that that passed this way."
  },
  {
    "startMs": 521599,
    "text": "So, this is the million-dollar question"
  },
  {
    "startMs": 523680,
    "text": "that we asked, right? If cost what if"
  },
  {
    "startMs": 525760,
    "text": "Costco raises its fee uh this September,"
  },
  {
    "startMs": 528320,
    "text": "what happens to its actual revenue? So"
  },
  {
    "startMs": 530959,
    "text": "we had GLM 5.3 translated into just like"
  },
  {
    "startMs": 533440,
    "text": "some structured output that we can feed"
  },
  {
    "startMs": 534880,
    "text": "into the financial model. And then the"
  },
  {
    "startMs": 536560,
    "text": "model gave us this number of like it"
  },
  {
    "startMs": 538720,
    "text": "predicted that uh profit would increase"
  },
  {
    "startMs": 541760,
    "text": "by $560 million over two years. Uh and"
  },
  {
    "startMs": 545120,
    "text": "so this is it just in the app. We send"
  },
  {
    "startMs": 547440,
    "text": "this over uh GLM 5.3 interprets it. It"
  },
  {
    "startMs": 550080,
    "text": "sends it as structured data to the"
  },
  {
    "startMs": 551760,
    "text": "model. And remember the model is just a"
  },
  {
    "startMs": 553519,
    "text": "financial model. It takes 10"
  },
  {
    "startMs": 554640,
    "text": "milliseconds to run uh because the mo"
  },
  {
    "startMs": 556480,
    "text": "because the uh our GPOSS model already"
  },
  {
    "startMs": 558800,
    "text": "made it. And here's that 560 million uh"
  },
  {
    "startMs": 561120,
    "text": "number right there. So, um this is it"
  },
  {
    "startMs": 563920,
    "text": "just like visualized on a scale. The"
  },
  {
    "startMs": 565760,
    "text": "blue line is the predicted uh uh uh"
  },
  {
    "startMs": 568480,
    "text": "profit and orange is actually the real"
  },
  {
    "startMs": 571279,
    "text": "profit extracted from Costco's uh"
  },
  {
    "startMs": 573120,
    "text": "earning reports. And so you can see we"
  },
  {
    "startMs": 574800,
    "text": "were accurate to about 4.4% uh which is"
  },
  {
    "startMs": 578080,
    "text": "the average of the five builds. The"
  },
  {
    "startMs": 579360,
    "text": "builds gave us between 0.7 and 5.2%. So"
  },
  {
    "startMs": 583040,
    "text": "again, I thought this was pretty good,"
  },
  {
    "startMs": 584320,
    "text": "right? We had the model build uh we had"
  },
  {
    "startMs": 586560,
    "text": "our AI model build this financial model."
  },
  {
    "startMs": 588880,
    "text": "We ran it. It was accurate within 4.4%."
  },
  {
    "startMs": 591600,
    "text": "Uh and then I showed it to my dad. Yeah,"
  },
  {
    "startMs": 594160,
    "text": "definitely. It's amazing to see that AI"
  },
  {
    "startMs": 596000,
    "text": "can can get to financial models which is"
  },
  {
    "startMs": 598160,
    "text": "the you know the area of accountants and"
  },
  {
    "startMs": 599920,
    "text": "and financial people but with with very"
  },
  {
    "startMs": 602560,
    "text": "small guidance but the first thing you"
  },
  {
    "startMs": 604560,
    "text": "look at when you try to look at a"
  },
  {
    "startMs": 606240,
    "text": "financial model is to audit it is to"
  },
  {
    "startMs": 608240,
    "text": "look at assumptions. So these are some"
  },
  {
    "startMs": 610320,
    "text": "of the assumptions that were used in"
  },
  {
    "startMs": 611760,
    "text": "Hassan's model and uh I just want to"
  },
  {
    "startMs": 614399,
    "text": "talk your attention to two things. The"
  },
  {
    "startMs": 615760,
    "text": "executive share like Hassan mentioned"
  },
  {
    "startMs": 617440,
    "text": "before. Uh it was um you know big swings"
  },
  {
    "startMs": 620320,
    "text": "there. Um US Canada the last um you know"
  },
  {
    "startMs": 623680,
    "text": "um assumption did not move um sales and"
  },
  {
    "startMs": 627920,
    "text": "um and fees with much with zero. So"
  },
  {
    "startMs": 631040,
    "text": "basically that's a lever that is not"
  },
  {
    "startMs": 632880,
    "text": "working and it should be and something"
  },
  {
    "startMs": 634640,
    "text": "was wrong. Something went wrong there"
  },
  {
    "startMs": 636560,
    "text": "and it should be fixed."
  },
  {
    "startMs": 639440,
    "text": "Um so these are why the executive share"
  },
  {
    "startMs": 642880,
    "text": "for example the executive share the 15%"
  },
  {
    "startMs": 644959,
    "text": "the mother took 15%. Um and the problem"
  },
  {
    "startMs": 647839,
    "text": "is that the mother had just one citation"
  },
  {
    "startMs": 650240,
    "text": "increased penetration of executive"
  },
  {
    "startMs": 651920,
    "text": "membership but there was no number to"
  },
  {
    "startMs": 654160,
    "text": "back it up. So that's what the auditors"
  },
  {
    "startMs": 656480,
    "text": "looking at at how are all the facts"
  },
  {
    "startMs": 658800,
    "text": "backed up with you know with with real"
  },
  {
    "startMs": 660800,
    "text": "numbers. So the citation is real but the"
  },
  {
    "startMs": 663200,
    "text": "problem and the the the number was ma"
  },
  {
    "startMs": 665200,
    "text": "was made up by AI and not only that AI"
  },
  {
    "startMs": 668480,
    "text": "also draw the boundaries of that"
  },
  {
    "startMs": 670399,
    "text": "assumption which is uh zero to um 0 to 5"
  },
  {
    "startMs": 674399,
    "text": "to 30%. But we will see that it's not"
  },
  {
    "startMs": 677200,
    "text": "correct actually um the that's what the"
  },
  {
    "startMs": 680640,
    "text": "AI did 5 to 30% but the problem is the"
  },
  {
    "startMs": 684720,
    "text": "real number that was already disclosed"
  },
  {
    "startMs": 686720,
    "text": "in the in in the filings um was 45%. So"
  },
  {
    "startMs": 690880,
    "text": "in this case AI failed to um you know um"
  },
  {
    "startMs": 694959,
    "text": "identify that number from the from the"
  },
  {
    "startMs": 696720,
    "text": "filing. So that's one uh one problem and"
  },
  {
    "startMs": 699680,
    "text": "um the surprise why did it land correct"
  },
  {
    "startMs": 702720,
    "text": "like hasn't mentioned it's 4.4% 4% which"
  },
  {
    "startMs": 705440,
    "text": "is a okay uh performance because of two"
  },
  {
    "startMs": 710399,
    "text": "errors that canceled out two assumptions"
  },
  {
    "startMs": 712320,
    "text": "that canceled out and the first one was"
  },
  {
    "startMs": 714800,
    "text": "an overstatement of the base how many"
  },
  {
    "startMs": 717040,
    "text": "people instead of applying the um"
  },
  {
    "startMs": 719040,
    "text": "increase on just the US Canada portion"
  },
  {
    "startMs": 722560,
    "text": "of the members the model applied it to"
  },
  {
    "startMs": 725040,
    "text": "international to to the whole population"
  },
  {
    "startMs": 727040,
    "text": "actually and the second thing that was"
  },
  {
    "startMs": 729200,
    "text": "understated it was too low it's the"
  },
  {
    "startMs": 731839,
    "text": "percentage uh instead of 45% which is"
  },
  {
    "startMs": 734480,
    "text": "the real number. Um AI made up this 15%"
  },
  {
    "startMs": 737920,
    "text": "and that was too low. It landed $69"
  },
  {
    "startMs": 741120,
    "text": "which is close to 66 but um again the"
  },
  {
    "startMs": 744000,
    "text": "model behind it was wrong. And this is"
  },
  {
    "startMs": 747360,
    "text": "um this is what Hassan was mentioning"
  },
  {
    "startMs": 749279,
    "text": "before um that um like that led to um"
  },
  {
    "startMs": 754399,
    "text": "like the 16 the 16 models they failed"
  },
  {
    "startMs": 758000,
    "text": "and they failed um all with the same"
  },
  {
    "startMs": 760399,
    "text": "thing with big ranges from half billion"
  },
  {
    "startMs": 762720,
    "text": "to $6 billion in in terms of fee"
  },
  {
    "startMs": 765760,
    "text": "increase while the actual number that we"
  },
  {
    "startMs": 768079,
    "text": "recalculated was just $450 million. So"
  },
  {
    "startMs": 771440,
    "text": "that that was one of the other issues"
  },
  {
    "startMs": 773680,
    "text": "here. And then um just a trick here the"
  },
  {
    "startMs": 776800,
    "text": "way to know whether a model or um to to"
  },
  {
    "startMs": 780320,
    "text": "something is wrong if you have two"
  },
  {
    "startMs": 782240,
    "text": "things cancelling out like an auditing"
  },
  {
    "startMs": 784160,
    "text": "trick you just fix one of the two and"
  },
  {
    "startMs": 786079,
    "text": "the model will look worse. So before the"
  },
  {
    "startMs": 789440,
    "text": "performance was 1.7 because things were"
  },
  {
    "startMs": 791839,
    "text": "washing out canceling out and then once"
  },
  {
    "startMs": 794560,
    "text": "we correct just one we just we put like"
  },
  {
    "startMs": 796880,
    "text": "the 45% instead of 15% that was made up"
  },
  {
    "startMs": 800320,
    "text": "by AI suddenly the problem the error was"
  },
  {
    "startMs": 803920,
    "text": "28%. So that's what we say luck is not a"
  },
  {
    "startMs": 807519,
    "text": "control in the auditing world. Usually"
  },
  {
    "startMs": 810079,
    "text": "in auditing we use three lines of"
  },
  {
    "startMs": 811839,
    "text": "defense. We have management controls"
  },
  {
    "startMs": 813760,
    "text": "which is the engineering checks that"
  },
  {
    "startMs": 815279,
    "text": "hasn't um coded and then we have the uh"
  },
  {
    "startMs": 818720,
    "text": "internal audit or some some internal um"
  },
  {
    "startMs": 821279,
    "text": "accounting checks that should be"
  },
  {
    "startMs": 822560,
    "text": "performed by internal um uh audit"
  },
  {
    "startMs": 824560,
    "text": "function and then we have a third layer"
  },
  {
    "startMs": 827120,
    "text": "which is the independent check that's"
  },
  {
    "startMs": 829200,
    "text": "external auditors. So these layers they"
  },
  {
    "startMs": 832000,
    "text": "will benefit like the model will benefit"
  },
  {
    "startMs": 833760,
    "text": "from adding layer two and three and"
  },
  {
    "startMs": 835680,
    "text": "that's the role I was playing like this"
  },
  {
    "startMs": 838079,
    "text": "CPA in the loop or what I call uh"
  },
  {
    "startMs": 840959,
    "text": "actually the judgment professional"
  },
  {
    "startMs": 843199,
    "text": "judgment in the loop um to because we"
  },
  {
    "startMs": 845760,
    "text": "can we can hardcode most of those checks"
  },
  {
    "startMs": 848240,
    "text": "but still we need uh we need a human um"
  },
  {
    "startMs": 851199,
    "text": "looking at that."
  },
  {
    "startMs": 852160,
    "text": "Great. So, you know, as my dad said, we"
  },
  {
    "startMs": 854880,
    "text": "we had the model here. We passed it"
  },
  {
    "startMs": 856560,
    "text": "through engineering checks. It did"
  },
  {
    "startMs": 858880,
    "text": "pretty well. Uh, you know, we got a"
  },
  {
    "startMs": 860399,
    "text": "little bit lucky with the result, but"
  },
  {
    "startMs": 862000,
    "text": "was really missing is these like"
  },
  {
    "startMs": 863120,
    "text": "accounting checks or this like kind of"
  },
  {
    "startMs": 864480,
    "text": "human in the loop uh uh activity. So, uh"
  },
  {
    "startMs": 867519,
    "text": "and a lot of these accounting checks"
  },
  {
    "startMs": 868560,
    "text": "that my dad came up with, you know,"
  },
  {
    "startMs": 869920,
    "text": "we're were able to to codify. So, one"
  },
  {
    "startMs": 872160,
    "text": "was the the executive 45% like my dad"
  },
  {
    "startMs": 874560,
    "text": "said. Um so, it's really important to"
  },
  {
    "startMs": 876480,
    "text": "have these like domain specific checks"
  },
  {
    "startMs": 878240,
    "text": "when you're building uh these these AI"
  },
  {
    "startMs": 880399,
    "text": "agents. You know, I don't come from the"
  },
  {
    "startMs": 881519,
    "text": "from the accounting world. when I built"
  },
  {
    "startMs": 882800,
    "text": "this and I saw 4% I was like oh amazing."
  },
  {
    "startMs": 885279,
    "text": "Um but but really it's um it's yeah this"
  },
  {
    "startMs": 888959,
    "text": "is really the number one thing to"
  },
  {
    "startMs": 890000,
    "text": "remember from this talk. If you take one"
  },
  {
    "startMs": 891120,
    "text": "thing away um having a human in the loop"
  },
  {
    "startMs": 893440,
    "text": "or ideally a subject matter expert uh in"
  },
  {
    "startMs": 895839,
    "text": "the loop is really really really"
  },
  {
    "startMs": 897600,
    "text": "important. Um they can take a look at"
  },
  {
    "startMs": 899600,
    "text": "your agent or your model or or whatever"
  },
  {
    "startMs": 901440,
    "text": "you've actually built. They can check"
  },
  {
    "startMs": 902880,
    "text": "the assumptions, check the inputs, check"
  },
  {
    "startMs": 904399,
    "text": "the outputs, make the right corrections."
  },
  {
    "startMs": 906480,
    "text": "you can codify their checks in the model"
  },
  {
    "startMs": 908639,
    "text": "and it it just makes it into something"
  },
  {
    "startMs": 910480,
    "text": "that's a lot more autonomous and uh uh"
  },
  {
    "startMs": 913040,
    "text": "uh just a lot a lot better overall. And"
  },
  {
    "startMs": 916480,
    "text": "that is it. Thank you all so much for"
  },
  {
    "startMs": 918079,
    "text": "coming. Thank you."
  }
]
