[
  {
    "startMs": 1964,
    "text": "[music]"
  },
  {
    "startMs": 14000,
    "text": "Uh afternoon everybody. Uh I'm going to"
  },
  {
    "startMs": 16720,
    "text": "talk today about engineering agency out"
  },
  {
    "startMs": 19439,
    "text": "of the happy path. Um this is going to"
  },
  {
    "startMs": 21840,
    "text": "be focused on AI leadership, engineering"
  },
  {
    "startMs": 25199,
    "text": "leadership at growth stage companies."
  },
  {
    "startMs": 28640,
    "text": "Um, and I'd like to start by asking"
  },
  {
    "startMs": 30560,
    "text": "everybody to think about the strongest"
  },
  {
    "startMs": 32800,
    "text": "domain experts on your team. So these"
  },
  {
    "startMs": 35120,
    "text": "are people who uh, you know, if sales"
  },
  {
    "startMs": 37360,
    "text": "wants to build a new feature or deliver"
  },
  {
    "startMs": 39200,
    "text": "something new in order to close an"
  },
  {
    "startMs": 40559,
    "text": "account or an urgent customer ticket"
  },
  {
    "startMs": 42800,
    "text": "appears, uh, these are the people who"
  },
  {
    "startMs": 44480,
    "text": "get at paint. Uh, they're the people"
  },
  {
    "startMs": 46800,
    "text": "maybe who are responsible for a small"
  },
  {
    "startMs": 48800,
    "text": "team or for a critical business process."
  },
  {
    "startMs": 51440,
    "text": "And I want to think about how these"
  },
  {
    "startMs": 52719,
    "text": "people actually spend most of their work"
  },
  {
    "startMs": 54480,
    "text": "weeks."
  },
  {
    "startMs": 56239,
    "text": "Uh my experience is that most experts"
  },
  {
    "startMs": 58399,
    "text": "I've worked with have their attention"
  },
  {
    "startMs": 60160,
    "text": "sort of smeared over multitasking,"
  },
  {
    "startMs": 62800,
    "text": "loading up on complicated situations,"
  },
  {
    "startMs": 65280,
    "text": "making decisions that could have been"
  },
  {
    "startMs": 66880,
    "text": "routine or obvious in retrospect and"
  },
  {
    "startMs": 69760,
    "text": "only a small fraction of their attention"
  },
  {
    "startMs": 71840,
    "text": "is left over to sort of move the"
  },
  {
    "startMs": 73280,
    "text": "business forward."
  },
  {
    "startMs": 76000,
    "text": "Um now like the cost of like raw white"
  },
  {
    "startMs": 78640,
    "text": "collar output is dropping. It's"
  },
  {
    "startMs": 81360,
    "text": "cratering. I've talked to a bunch of"
  },
  {
    "startMs": 82720,
    "text": "people over the conference and"
  },
  {
    "startMs": 84159,
    "text": "regardless of what you think about"
  },
  {
    "startMs": 85360,
    "text": "frontier models and where the future's"
  },
  {
    "startMs": 86960,
    "text": "going to go in terms of new capacity."
  },
  {
    "startMs": 88720,
    "text": "Everybody I've talked to has agreed that"
  },
  {
    "startMs": 90400,
    "text": "existing capacity is going to get very"
  },
  {
    "startMs": 92320,
    "text": "very cheap."
  },
  {
    "startMs": 94159,
    "text": "Cheap execution increases the load and"
  },
  {
    "startMs": 96799,
    "text": "the on on judgment on deciding what to"
  },
  {
    "startMs": 99119,
    "text": "do. Uh organizational judgment was"
  },
  {
    "startMs": 102320,
    "text": "already in short supply. Uh and this is"
  },
  {
    "startMs": 104320,
    "text": "going to make it even worse. I think"
  },
  {
    "startMs": 105680,
    "text": "this is currently underpriced."
  },
  {
    "startMs": 108399,
    "text": "So with that in mind, I'm going to talk"
  },
  {
    "startMs": 109520,
    "text": "today about using AI for full"
  },
  {
    "startMs": 110880,
    "text": "automations. By full automations, I mean"
  },
  {
    "startMs": 113520,
    "text": "taking humans out of the happy path"
  },
  {
    "startMs": 115439,
    "text": "completely. Using humans maybe for"
  },
  {
    "startMs": 117119,
    "text": "exceptions, but not not human in the"
  },
  {
    "startMs": 118880,
    "text": "loop um style automations. I think that"
  },
  {
    "startMs": 121840,
    "text": "these types of processes can improve"
  },
  {
    "startMs": 123920,
    "text": "margins kind of obvious. They can reduce"
  },
  {
    "startMs": 126320,
    "text": "complexity of a business by"
  },
  {
    "startMs": 127600,
    "text": "encapsulating whole chunks of work. Uh"
  },
  {
    "startMs": 129759,
    "text": "and then you can recover judgment from"
  },
  {
    "startMs": 131200,
    "text": "your experts and hopefully route it"
  },
  {
    "startMs": 132959,
    "text": "towards more important um tasks or"
  },
  {
    "startMs": 135120,
    "text": "questions. I'm going to talk about that"
  },
  {
    "startMs": 136480,
    "text": "specifically a bit later."
  },
  {
    "startMs": 138800,
    "text": "Um, now the key to full automations and"
  },
  {
    "startMs": 141520,
    "text": "my experience has been engineering"
  },
  {
    "startMs": 142879,
    "text": "agency out of the happy path out of"
  },
  {
    "startMs": 145280,
    "text": "these automations. This is a bit of a"
  },
  {
    "startMs": 147120,
    "text": "reversal. You know, agents have"
  },
  {
    "startMs": 148800,
    "text": "permanently changed and expanded my idea"
  },
  {
    "startMs": 150879,
    "text": "of what I think is possible with"
  },
  {
    "startMs": 152239,
    "text": "software. But in enterprise, agents have"
  },
  {
    "startMs": 154560,
    "text": "problems. They're a pain to maintain."
  },
  {
    "startMs": 157120,
    "text": "Uh, especially if skills and harnesses"
  },
  {
    "startMs": 158800,
    "text": "are changing. Uh, they fail in unusually"
  },
  {
    "startMs": 162080,
    "text": "shaped ways that enterprises find hard"
  },
  {
    "startMs": 163760,
    "text": "to work with. And then they increase the"
  },
  {
    "startMs": 166080,
    "text": "burden of decisions. When I see an"
  },
  {
    "startMs": 168080,
    "text": "expert or a leader responding to agent"
  },
  {
    "startMs": 170480,
    "text": "notifications, you know, maybe on their"
  },
  {
    "startMs": 171840,
    "text": "phone or or on their laptop, maybe they"
  },
  {
    "startMs": 173440,
    "text": "have an app for it. I'm often"
  },
  {
    "startMs": 174959,
    "text": "suspicious. Is that actually the best"
  },
  {
    "startMs": 176560,
    "text": "use of their, you know, valuable time"
  },
  {
    "startMs": 178400,
    "text": "and attention?"
  },
  {
    "startMs": 180319,
    "text": "Full automations, on the other hand,"
  },
  {
    "startMs": 181599,
    "text": "they free up attention. They're legible."
  },
  {
    "startMs": 184319,
    "text": "So this means that, you know, they take"
  },
  {
    "startMs": 185840,
    "text": "an input, they produce an output, they"
  },
  {
    "startMs": 187440,
    "text": "encapsulate the whole thing, and they"
  },
  {
    "startMs": 189040,
    "text": "hide the complexity. And they're"
  },
  {
    "startMs": 191440,
    "text": "organizationally familiar. Like large"
  },
  {
    "startMs": 193280,
    "text": "human organizations have been building"
  },
  {
    "startMs": 194800,
    "text": "automations in software for decades."
  },
  {
    "startMs": 197040,
    "text": "Think marketing, think DevOps. And"
  },
  {
    "startMs": 199200,
    "text": "they've been building uh automations in"
  },
  {
    "startMs": 201840,
    "text": "industry for centuries. Full automations"
  },
  {
    "startMs": 203840,
    "text": "are one of the key tools by which human"
  },
  {
    "startMs": 206159,
    "text": "organizations can do bigger and bigger"
  },
  {
    "startMs": 207840,
    "text": "things."
  },
  {
    "startMs": 214000,
    "text": "And that is giving the whole system an"
  },
  {
    "startMs": 215760,
    "text": "escape hatch. So, one of the problems"
  },
  {
    "startMs": 217680,
    "text": "with full automations is, and I'm sure"
  },
  {
    "startMs": 219519,
    "text": "people have seen this, when you get uh"
  },
  {
    "startMs": 221200,
    "text": "you sign up for something and maybe you"
  },
  {
    "startMs": 223120,
    "text": "accidentally get a whole marketing drip"
  },
  {
    "startMs": 224959,
    "text": "email when you were signing up as a"
  },
  {
    "startMs": 226239,
    "text": "vendor. Um, one of the interesting"
  },
  {
    "startMs": 228400,
    "text": "things you can do with AI and agency"
  },
  {
    "startMs": 230159,
    "text": "with automations is you can pre-filter"
  },
  {
    "startMs": 231760,
    "text": "inputs. You can check does this task"
  },
  {
    "startMs": 234959,
    "text": "actually match the happy path of the"
  },
  {
    "startMs": 236640,
    "text": "automation. You can reject adversarial"
  },
  {
    "startMs": 238959,
    "text": "situations. If something looks like it's"
  },
  {
    "startMs": 240879,
    "text": "business critical, sort of not a normal"
  },
  {
    "startMs": 242560,
    "text": "course of business operation, you can"
  },
  {
    "startMs": 244319,
    "text": "escalate that to a human. I think this"
  },
  {
    "startMs": 246480,
    "text": "is kind of neat. This is the use of AI,"
  },
  {
    "startMs": 248799,
    "text": "AI agency to reduce the variance of an"
  },
  {
    "startMs": 251680,
    "text": "automation to make things more smooth,"
  },
  {
    "startMs": 253920,
    "text": "which thereby lets us expand the scope"
  },
  {
    "startMs": 255599,
    "text": "of what we can automate. So I'm going to"
  },
  {
    "startMs": 258000,
    "text": "give a case study today. My background"
  },
  {
    "startMs": 259840,
    "text": "is working in bioarma data at scale. I"
  },
  {
    "startMs": 262000,
    "text": "had a company called deal enablement for"
  },
  {
    "startMs": 264720,
    "text": "bioharmaceutical mergers, licensing and"
  },
  {
    "startMs": 266800,
    "text": "acquisitions. In 23 we sold to clarvate."
  },
  {
    "startMs": 269600,
    "text": "Clarvate is a really interesting"
  },
  {
    "startMs": 270639,
    "text": "company. They have an annotated history"
  },
  {
    "startMs": 272320,
    "text": "of the entire you know drug development"
  },
  {
    "startMs": 274320,
    "text": "industry. hundreds of millions of"
  },
  {
    "startMs": 275919,
    "text": "document or tens of millions of"
  },
  {
    "startMs": 277120,
    "text": "documents, hundreds of thousands of"
  },
  {
    "startMs": 278560,
    "text": "edits every year and a rich ontology for"
  },
  {
    "startMs": 280639,
    "text": "working of those sort of tens of"
  },
  {
    "startMs": 282000,
    "text": "thousands of ways of indexing it."
  },
  {
    "startMs": 284928,
    "text": "[snorts] Um the automation we built was"
  },
  {
    "startMs": 286880,
    "text": "automating the indexing and the"
  },
  {
    "startMs": 288720,
    "text": "relationships between those documents."
  },
  {
    "startMs": 290639,
    "text": "So this is processing tens of thousands"
  },
  {
    "startMs": 292400,
    "text": "of new tasks a month when a document's"
  },
  {
    "startMs": 294160,
    "text": "edited or inserted or something changes."
  },
  {
    "startMs": 296639,
    "text": "We were able to achieve three and a half"
  },
  {
    "startMs": 298160,
    "text": "n of accuracy on the tasks that the"
  },
  {
    "startMs": 300560,
    "text": "business most valued. And I'll talk"
  },
  {
    "startMs": 301840,
    "text": "about what that means in a few minutes."
  },
  {
    "startMs": 303840,
    "text": "Um, this was expensive to build. This"
  },
  {
    "startMs": 305759,
    "text": "was a capex cost. AI engineers aren't"
  },
  {
    "startMs": 307759,
    "text": "cheap. But once it was built, it was"
  },
  {
    "startMs": 309680,
    "text": "cheap to operate. It mostly stayed"
  },
  {
    "startMs": 311759,
    "text": "within its encapsulation. Um, and it was"
  },
  {
    "startMs": 315120,
    "text": "stable."
  },
  {
    "startMs": 317919,
    "text": "So building this out, our initial"
  },
  {
    "startMs": 319840,
    "text": "approach, I think in 2024, we were"
  },
  {
    "startMs": 322240,
    "text": "excited by 40, we were excited by 03. We"
  },
  {
    "startMs": 325440,
    "text": "did something, maybe we were early for"
  },
  {
    "startMs": 326720,
    "text": "agents, but we built something that's"
  },
  {
    "startMs": 328320,
    "text": "sort of similar to what people would"
  },
  {
    "startMs": 329600,
    "text": "today think of as a research and"
  },
  {
    "startMs": 330960,
    "text": "assignment agent. After a few sprints,"
  },
  {
    "startMs": 333520,
    "text": "we hit, you know, 80 83% performance. We"
  },
  {
    "startMs": 335759,
    "text": "were really excited. We had been working"
  },
  {
    "startMs": 336960,
    "text": "in NLP for a decade. We were used to"
  },
  {
    "startMs": 338960,
    "text": "spending months fine-tuning a BERT model"
  },
  {
    "startMs": 340800,
    "text": "to just do one sort of component of one"
  },
  {
    "startMs": 342639,
    "text": "of these pipelines. We compiled a"
  },
  {
    "startMs": 345120,
    "text": "thousand devalves. We went we sat down"
  },
  {
    "startMs": 346960,
    "text": "with the domain experts."
  },
  {
    "startMs": 349199,
    "text": "Um, I will say one good thing that we"
  },
  {
    "startMs": 351280,
    "text": "did is that we brought the domain"
  },
  {
    "startMs": 352720,
    "text": "experts a whole bunch of examples. And"
  },
  {
    "startMs": 355840,
    "text": "um I've worked on other projects where"
  },
  {
    "startMs": 358400,
    "text": "you go to a domain expert, you give them"
  },
  {
    "startMs": 359840,
    "text": "a spreadsheet and you say, \"Hey, fill"
  },
  {
    "startMs": 361440,
    "text": "out a 100 evals.\" That doesn't go well."
  },
  {
    "startMs": 364160,
    "text": "Uh bringing a blank spreadsheet to a"
  },
  {
    "startMs": 366160,
    "text": "busy expert uh doesn't create momentum."
  },
  {
    "startMs": 369280,
    "text": "Uh by bringing them all these these"
  },
  {
    "startMs": 371039,
    "text": "these evals, these places we'"
  },
  {
    "startMs": 372720,
    "text": "experienced, we very quickly were able"
  },
  {
    "startMs": 374080,
    "text": "to get a the domain experts on board"
  },
  {
    "startMs": 376240,
    "text": "with what's possible and b get them"
  },
  {
    "startMs": 378240,
    "text": "fixated and interested in the places in"
  },
  {
    "startMs": 380240,
    "text": "which it failed. Um I do think that most"
  },
  {
    "startMs": 383360,
    "text": "automations are going to require input"
  },
  {
    "startMs": 384960,
    "text": "with domain experts working on these"
  },
  {
    "startMs": 386560,
    "text": "cases. If you can help them get started"
  },
  {
    "startMs": 389520,
    "text": "with evaluation,"
  },
  {
    "startMs": 394720,
    "text": "but writing evals is an art and it can"
  },
  {
    "startMs": 396319,
    "text": "help get them started more quickly. But"
  },
  {
    "startMs": 399039,
    "text": "experts, we sat down with these eval."
  },
  {
    "startMs": 401039,
    "text": "What did we find? Well, there was a"
  },
  {
    "startMs": 402880,
    "text": "large bucket of undecided or contentious"
  },
  {
    "startMs": 406639,
    "text": "cases. These were tasks where actually"
  },
  {
    "startMs": 409360,
    "text": "two experts might confidently disagree"
  },
  {
    "startMs": 411280,
    "text": "about what the answer is or where the"
  },
  {
    "startMs": 413840,
    "text": "organization itself hadn't quite"
  },
  {
    "startMs": 415360,
    "text": "decided. and everybody was not sure."
  },
  {
    "startMs": 417360,
    "text": "Mixed in with that sort of"
  },
  {
    "startMs": 418560,
    "text": "heterogeneously was a small bucket of"
  },
  {
    "startMs": 420400,
    "text": "critical errors. These were errors where"
  },
  {
    "startMs": 422319,
    "text": "everybody agreed a misstep would, you"
  },
  {
    "startMs": 424160,
    "text": "know, threaten material accounts. We we"
  },
  {
    "startMs": 426400,
    "text": "just couldn't have them. And then mixed"
  },
  {
    "startMs": 428080,
    "text": "in with that was a whole sort of maybe 3"
  },
  {
    "startMs": 430720,
    "text": "or 5% of exceptions or escalations,"
  },
  {
    "startMs": 433680,
    "text": "places that needed more attention."
  },
  {
    "startMs": 436960,
    "text": "Uh do a quick example of these three. So"
  },
  {
    "startMs": 439199,
    "text": "the errors pretty straightforward. Even"
  },
  {
    "startMs": 441120,
    "text": "top models, if you have multi-step"
  },
  {
    "startMs": 442639,
    "text": "workflows fail 2 3% of the time. It's"
  },
  {
    "startMs": 445440,
    "text": "annoying when you see it at scale."
  },
  {
    "startMs": 446800,
    "text": "Sometimes you can't even reproduce it,"
  },
  {
    "startMs": 448319,
    "text": "but it happens. For undecided or"
  },
  {
    "startMs": 450400,
    "text": "ambiguous cases, quick example, um, in"
  },
  {
    "startMs": 453199,
    "text": "drug development, uh, drug developers"
  },
  {
    "startMs": 455199,
    "text": "care a lot about targets. These are the,"
  },
  {
    "startMs": 457360,
    "text": "you know, key pathway or molecule that"
  },
  {
    "startMs": 458800,
    "text": "your drug actually impacts. Gout is a"
  },
  {
    "startMs": 461280,
    "text": "disease that's caused by overprouction"
  },
  {
    "startMs": 463039,
    "text": "of interlucan one beta. Culticine is a"
  },
  {
    "startMs": 465759,
    "text": "drug that reduces interlucan one beta by"
  },
  {
    "startMs": 468160,
    "text": "preventing a chain reaction basically."
  },
  {
    "startMs": 470720,
    "text": "uh so colon itself binds to a protein"
  },
  {
    "startMs": 474240,
    "text": "called tubulin. Our system wanted to"
  },
  {
    "startMs": 476400,
    "text": "know does cultureine target interlucan"
  },
  {
    "startMs": 478479,
    "text": "one beta. And if you ask GPT55 sometimes"
  },
  {
    "startMs": 481599,
    "text": "it'll say no and if you ask you know"
  },
  {
    "startMs": 483440,
    "text": "opus 48 sometimes it'll say yes. They"
  },
  {
    "startMs": 485759,
    "text": "actually sort of both go back and forth"
  },
  {
    "startMs": 487120,
    "text": "on this. This is a toy case. It's"
  },
  {
    "startMs": 489039,
    "text": "actually the real situation is a little"
  },
  {
    "startMs": 490400,
    "text": "more complicated but it's"
  },
  {
    "startMs": 491360,
    "text": "representative. It gives a sort of"
  },
  {
    "startMs": 492560,
    "text": "flavor of um what undecided cases look"
  },
  {
    "startMs": 495120,
    "text": "like. Uh in terms of unusual or"
  },
  {
    "startMs": 497759,
    "text": "exception, I've got another bio example"
  },
  {
    "startMs": 499199,
    "text": "that I really like. Uh, this is a really"
  },
  {
    "startMs": 501039,
    "text": "exciting new potential drug called Verve"
  },
  {
    "startMs": 502960,
    "text": "102, possibly cure heart disease, which"
  },
  {
    "startMs": 505360,
    "text": "is like it's pretty cool. I hope to get"
  },
  {
    "startMs": 507199,
    "text": "this or something like it for my little"
  },
  {
    "startMs": 508720,
    "text": "boy when he turns 18. Um, this does have"
  },
  {
    "startMs": 512560,
    "text": "a decidable output. We do want to index"
  },
  {
    "startMs": 514800,
    "text": "the sort of six mechanisms of this drug,"
  },
  {
    "startMs": 517039,
    "text": "but the four of them that I've circled"
  },
  {
    "startMs": 518479,
    "text": "here are all actually just new to"
  },
  {
    "startMs": 519760,
    "text": "science. So, like these didn't exist a"
  },
  {
    "startMs": 521680,
    "text": "few years ago. So, in order to properly"
  },
  {
    "startMs": 524159,
    "text": "solve this, we're going to require some"
  },
  {
    "startMs": 525519,
    "text": "human attention. The humans are going to"
  },
  {
    "startMs": 526800,
    "text": "have to think about how to index"
  },
  {
    "startMs": 528399,
    "text": "ontologies. This is an escalation"
  },
  {
    "startMs": 532480,
    "text": "path forward. How are we going to handle"
  },
  {
    "startMs": 533760,
    "text": "these three buckets? We need to make"
  },
  {
    "startMs": 535120,
    "text": "sure that errors both get routed into a"
  },
  {
    "startMs": 538320,
    "text": "place where we can grind out high"
  },
  {
    "startMs": 539680,
    "text": "performance and then we need to grind"
  },
  {
    "startMs": 541360,
    "text": "out the performance for exceptions and"
  },
  {
    "startMs": 543360,
    "text": "for undecided cases. We want to route"
  },
  {
    "startMs": 544959,
    "text": "those into places to keep them out of"
  },
  {
    "startMs": 546720,
    "text": "the normal system because they're going"
  },
  {
    "startMs": 548320,
    "text": "to require different types of handling."
  },
  {
    "startMs": 551040,
    "text": "And I think probably in retrospect we"
  },
  {
    "startMs": 553040,
    "text": "spent at least as much time building"
  },
  {
    "startMs": 555120,
    "text": "case routing as we actually did on the"
  },
  {
    "startMs": 557120,
    "text": "final error climbing for the sort of"
  },
  {
    "startMs": 559040,
    "text": "critical set. Is the AI ever confused"
  },
  {
    "startMs": 561519,
    "text": "about what's going on? Is it unusual a"
  },
  {
    "startMs": 563360,
    "text": "case? Abort. Use the AI to describe"
  },
  {
    "startMs": 565440,
    "text": "what's going on. Categorize it away."
  },
  {
    "startMs": 567360,
    "text": "Skip it from the pipeline. Is it"
  },
  {
    "startMs": 569440,
    "text": "plausibly business critical? Okay, then"
  },
  {
    "startMs": 571600,
    "text": "send that to the extremely robustly"
  },
  {
    "startMs": 573680,
    "text": "tested, you know, thousands of evals"
  },
  {
    "startMs": 575360,
    "text": "pipeline. Hopefully you can ship an"
  },
  {
    "startMs": 577200,
    "text": "answer. If through that pipeline it's"
  },
  {
    "startMs": 579360,
    "text": "still not sure, fine. and ship it back"
  },
  {
    "startMs": 580800,
    "text": "to a human as an exception. But vast"
  },
  {
    "startMs": 582640,
    "text": "majority of the time you can ship an"
  },
  {
    "startMs": 583920,
    "text": "answer"
  },
  {
    "startMs": 586080,
    "text": "is it undecided pipeline that out as"
  },
  {
    "startMs": 587920,
    "text": "well. Um for the majority of cases"
  },
  {
    "startMs": 589920,
    "text": "though they didn't fall into any of"
  },
  {
    "startMs": 591040,
    "text": "these cases and we could just ship as"
  },
  {
    "startMs": 592800,
    "text": "usual."
  },
  {
    "startMs": 598080,
    "text": "um in terms of the critical path. So you"
  },
  {
    "startMs": 600399,
    "text": "know we had this this small set of tasks"
  },
  {
    "startMs": 602160,
    "text": "that were absolutely business critical"
  },
  {
    "startMs": 604399,
    "text": "once we took out the ambiguous cases and"
  },
  {
    "startMs": 606399,
    "text": "once we took out the sort of um unusual"
  },
  {
    "startMs": 608959,
    "text": "escalation cases we're able to get to"
  },
  {
    "startMs": 610720,
    "text": "about 95% accuracy with just sort of"
  },
  {
    "startMs": 612720,
    "text": "standard approaches going from 95 to 3"
  },
  {
    "startMs": 615279,
    "text": "and a half 9s. This is a case of I've"
  },
  {
    "startMs": 618079,
    "text": "got a whole bag of tricks on it. It's a"
  },
  {
    "startMs": 619519,
    "text": "case of hill climbing. It's a case of"
  },
  {
    "startMs": 620880,
    "text": "looking at a lot of evals using all of"
  },
  {
    "startMs": 622720,
    "text": "the normal tricks to juice performance"
  },
  {
    "startMs": 624160,
    "text": "out of LLMs. I basically do believe"
  },
  {
    "startMs": 626320,
    "text": "though that if your inputs are well"
  },
  {
    "startMs": 628000,
    "text": "structured and you're looking at a"
  },
  {
    "startMs": 629360,
    "text": "specific set of cases that you've"
  },
  {
    "startMs": 630720,
    "text": "articulated from your business, getting"
  },
  {
    "startMs": 632480,
    "text": "to three and a half nines is possible"
  },
  {
    "startMs": 634079,
    "text": "for most automations."
  },
  {
    "startMs": 636640,
    "text": "Um, [snorts] we deployed this only once"
  },
  {
    "startMs": 638399,
    "text": "a random sample showed that this"
  },
  {
    "startMs": 640240,
    "text": "performance was better than the historic"
  },
  {
    "startMs": 641839,
    "text": "work. This is cool. So, um, I do think"
  },
  {
    "startMs": 645279,
    "text": "specifically um, I'm going to claim that"
  },
  {
    "startMs": 647200,
    "text": "this system performed better than PhD"
  },
  {
    "startMs": 649120,
    "text": "humans at scale. And I'll make that"
  },
  {
    "startMs": 651200,
    "text": "qualifier because I know that a PhD"
  },
  {
    "startMs": 653120,
    "text": "human if they're sitting down on a"
  },
  {
    "startMs": 654720,
    "text": "singular case and they're looking at all"
  },
  {
    "startMs": 656160,
    "text": "the details, they will outperform this"
  },
  {
    "startMs": 658079,
    "text": "system, but not at scale. Not when they"
  },
  {
    "startMs": 660240,
    "text": "have to look at hundreds of cases a"
  },
  {
    "startMs": 661519,
    "text": "week, not when they have fatigue, not"
  },
  {
    "startMs": 663279,
    "text": "when if you know one in a thousand times"
  },
  {
    "startMs": 664720,
    "text": "they misclick and not uh you know given"
  },
  {
    "startMs": 667360,
    "text": "time pressure of their other work."
  },
  {
    "startMs": 671440,
    "text": "Um so full automation is the happy path."
  },
  {
    "startMs": 673600,
    "text": "Saves you time. That's great. You get"
  },
  {
    "startMs": 675519,
    "text": "better critical business critical task"
  },
  {
    "startMs": 677120,
    "text": "outcomes. That's also great. You know,"
  },
  {
    "startMs": 678640,
    "text": "this is a if you can take the places"
  },
  {
    "startMs": 680320,
    "text": "that are most valuable to the business"
  },
  {
    "startMs": 682000,
    "text": "and grind them out until you're"
  },
  {
    "startMs": 683120,
    "text": "performing better than humans at scale,"
  },
  {
    "startMs": 684560,
    "text": "that's great. There's a third benefit"
  },
  {
    "startMs": 686320,
    "text": "here, which is that it organizes the"
  },
  {
    "startMs": 687839,
    "text": "exhaust from the automation in a way"
  },
  {
    "startMs": 689360,
    "text": "that your experts can pay attention to."
  },
  {
    "startMs": 690800,
    "text": "And I think this is pretty neat. So,"
  },
  {
    "startMs": 693360,
    "text": "back to that expert from the beginning."
  },
  {
    "startMs": 695200,
    "text": "Hopefully, we've cleared their plate"
  },
  {
    "startMs": 696800,
    "text": "somewhat. We've taken some of the"
  },
  {
    "startMs": 697920,
    "text": "routine work out of their job. In"
  },
  {
    "startMs": 700000,
    "text": "addition, because we've invested so"
  },
  {
    "startMs": 701440,
    "text": "heavily in routing, we've now organized"
  },
  {
    "startMs": 703440,
    "text": "that we can organize their attention in"
  },
  {
    "startMs": 704880,
    "text": "a structured way among the around the"
  },
  {
    "startMs": 707279,
    "text": "remaining cases. So they're going to get"
  },
  {
    "startMs": 709839,
    "text": "a pile organized undecided and ambiguous"
  },
  {
    "startMs": 712720,
    "text": "cases, exceptions and unusual cases, and"
  },
  {
    "startMs": 715440,
    "text": "then a small number of critical cases"
  },
  {
    "startMs": 716959,
    "text": "the automation wasn't sure about. For"
  },
  {
    "startMs": 719440,
    "text": "those last ones, the sort of ones that"
  },
  {
    "startMs": 720800,
    "text": "are critical, they can quickly review"
  },
  {
    "startMs": 722399,
    "text": "them, very very few in number, and ship"
  },
  {
    "startMs": 724320,
    "text": "them. [snorts] For the ones that are"
  },
  {
    "startMs": 726720,
    "text": "organizationally ambiguous, this is"
  },
  {
    "startMs": 728480,
    "text": "actually kind of interesting. You know,"
  },
  {
    "startMs": 730000,
    "text": "the first thought is maybe this is a way"
  },
  {
    "startMs": 732000,
    "text": "to incentivize the organization to"
  },
  {
    "startMs": 733920,
    "text": "specify. You know, you've got 20% maybe"
  },
  {
    "startMs": 736160,
    "text": "of the total tasks that the organization"
  },
  {
    "startMs": 738399,
    "text": "wasn't sure with"
  },
  {
    "startMs": 742240,
    "text": "these are often one of the reasons that"
  },
  {
    "startMs": 744800,
    "text": "they're not yet fully specified is that"
  },
  {
    "startMs": 746639,
    "text": "they're only marginally valuable. So the"
  },
  {
    "startMs": 749040,
    "text": "alternative here is that you can"
  },
  {
    "startMs": 750399,
    "text": "actually if the expert has a big pile of"
  },
  {
    "startMs": 753279,
    "text": "them in front of them they can actually"
  },
  {
    "startMs": 754959,
    "text": "advocate that hey let's permanently"
  },
  {
    "startMs": 756480,
    "text": "freeze this work let's prioritize it"
  },
  {
    "startMs": 758720,
    "text": "let's not think about it again until"
  },
  {
    "startMs": 760079,
    "text": "there's a business case too and let's"
  },
  {
    "startMs": 761680,
    "text": "get it off our plate that is a form of"
  },
  {
    "startMs": 764079,
    "text": "again reducing complexity in an"
  },
  {
    "startMs": 765680,
    "text": "organization having fewer things to"
  },
  {
    "startMs": 767600,
    "text": "think about so the focus and attention"
  },
  {
    "startMs": 769040,
    "text": "can go in the right place"
  },
  {
    "startMs": 771598,
    "text": "[snorts] I think this is even more"
  },
  {
    "startMs": 772800,
    "text": "interesting so groundbreaking work like"
  },
  {
    "startMs": 774800,
    "text": "that verve 102 drug for for heart"
  },
  {
    "startMs": 776800,
    "text": "disease it's only like 3% % of new drug"
  },
  {
    "startMs": 779040,
    "text": "development work. The vast majority of"
  },
  {
    "startMs": 780480,
    "text": "drug development work sort of pattern"
  },
  {
    "startMs": 781680,
    "text": "matches. Everybody's familiar with the"
  },
  {
    "startMs": 783040,
    "text": "rough shape of what you're looking at."
  },
  {
    "startMs": 784720,
    "text": "So an expert might see this once a week,"
  },
  {
    "startMs": 786880,
    "text": "once every other week, one of these"
  },
  {
    "startMs": 788399,
    "text": "cases, they're scattered throughout. Uh"
  },
  {
    "startMs": 790880,
    "text": "the the expert might think, hey, I could"
  },
  {
    "startMs": 792800,
    "text": "build something to handle these cases."
  },
  {
    "startMs": 794880,
    "text": "But if you're only seeing it"
  },
  {
    "startMs": 795920,
    "text": "sporadically, it's hard to build an"
  },
  {
    "startMs": 797360,
    "text": "intuition for how important it is, how"
  },
  {
    "startMs": 799040,
    "text": "expensive it would be, who would pay for"
  },
  {
    "startMs": 800480,
    "text": "it. You start to bucket those. And now"
  },
  {
    "startMs": 803120,
    "text": "the expert can think maybe when they sit"
  },
  {
    "startMs": 804880,
    "text": "down with two or three hours and look at"
  },
  {
    "startMs": 806240,
    "text": "them in aggregate, hey there's patterns"
  },
  {
    "startMs": 808320,
    "text": "here or hey maybe this is the specific"
  },
  {
    "startMs": 810720,
    "text": "investment I propose to start solving"
  },
  {
    "startMs": 812720,
    "text": "these for the organization and then if"
  },
  {
    "startMs": 815279,
    "text": "an an account you know a real customer"
  },
  {
    "startMs": 817200,
    "text": "comes and says hey I have a problem this"
  },
  {
    "startMs": 819200,
    "text": "expert now is ready to figure out how to"
  },
  {
    "startMs": 821519,
    "text": "how to deal with that how to move the"
  },
  {
    "startMs": 822959,
    "text": "business forward on it. So we saw this"
  },
  {
    "startMs": 824720,
    "text": "happen organically on a 8 figure account"
  },
  {
    "startMs": 827680,
    "text": "um and the expert had basically by"
  },
  {
    "startMs": 830639,
    "text": "having the data structured had already"
  },
  {
    "startMs": 832079,
    "text": "pre-formed an opinion about where to"
  },
  {
    "startMs": 833680,
    "text": "invest and where to build forward. I"
  },
  {
    "startMs": 836160,
    "text": "think this is actually the case for most"
  },
  {
    "startMs": 838560,
    "text": "scaled business processes. If you're"
  },
  {
    "startMs": 841440,
    "text": "working with the with with the market"
  },
  {
    "startMs": 843120,
    "text": "with data in aggregate you're collecting"
  },
  {
    "startMs": 845360,
    "text": "leading indicators of market demand,"
  },
  {
    "startMs": 847199,
    "text": "right? you know, the places that are"
  },
  {
    "startMs": 848880,
    "text": "slightly unusual or the places that um"
  },
  {
    "startMs": 851279,
    "text": "that the market's moving now that they"
  },
  {
    "startMs": 852639,
    "text": "weren't a few years ago, but they look"
  },
  {
    "startMs": 854399,
    "text": "like exceptions and if they're"
  },
  {
    "startMs": 855519,
    "text": "encountered infrequently, it's not"
  },
  {
    "startMs": 856880,
    "text": "enough signal to have somebody think"
  },
  {
    "startMs": 858320,
    "text": "about what to do with them. They're"
  },
  {
    "startMs": 859839,
    "text": "difficult to prioritize if they occur"
  },
  {
    "startMs": 861519,
    "text": "randomly. If you can organize this"
  },
  {
    "startMs": 863680,
    "text": "signal, this exhaust these exceptional"
  },
  {
    "startMs": 865279,
    "text": "cases that don't land in the happy path,"
  },
  {
    "startMs": 867199,
    "text": "your best talent can actually start to"
  },
  {
    "startMs": 868959,
    "text": "think about what to do with it. So,"
  },
  {
    "startMs": 871440,
    "text": "quick, we built a full automation. We"
  },
  {
    "startMs": 874480,
    "text": "can save hours. We can improve task"
  },
  {
    "startMs": 876720,
    "text": "level outcomes. We can reduce"
  },
  {
    "startMs": 878639,
    "text": "organizational complexity by taking a"
  },
  {
    "startMs": 880399,
    "text": "big bucket of tasks, encapsulating them,"
  },
  {
    "startMs": 882320,
    "text": "and having nobody have to think about"
  },
  {
    "startMs": 883519,
    "text": "them anymore. And then we can route"
  },
  {
    "startMs": 885279,
    "text": "talent towards higher value problems."
  },
  {
    "startMs": 887440,
    "text": "This is a form of organizational"
  },
  {
    "startMs": 889199,
    "text": "refactoring."
  },
  {
    "startMs": 891519,
    "text": "I will say that full automations,"
  },
  {
    "startMs": 894079,
    "text": "especially if you're trying to very high"
  },
  {
    "startMs": 896079,
    "text": "quality output, something like doing a"
  },
  {
    "startMs": 898000,
    "text": "PhD level expert's job, are expensive."
  },
  {
    "startMs": 901040,
    "text": "AI engineers are expensive. Having an AI"
  },
  {
    "startMs": 903680,
    "text": "engineer spend hours reading a"
  },
  {
    "startMs": 904959,
    "text": "spreadsheet of eval is expensive. Once"
  },
  {
    "startMs": 907839,
    "text": "you get to that 95% going from 95 to"
  },
  {
    "startMs": 910160,
    "text": "99.95, that's an expensive process,"
  },
  {
    "startMs": 912560,
    "text": "etc., etc., etc. Beyond that, you"
  },
  {
    "startMs": 914880,
    "text": "actually have to think if you're going"
  },
  {
    "startMs": 915920,
    "text": "to be refactoring a um a non-engineering"
  },
  {
    "startMs": 919199,
    "text": "process, you're going to have"
  },
  {
    "startMs": 920320,
    "text": "organizational change, trust building,"
  },
  {
    "startMs": 922959,
    "text": "ownership change. If you're going to"
  },
  {
    "startMs": 924800,
    "text": "make an investment of this size,"
  },
  {
    "startMs": 926000,
    "text": "hopefully you're also working with your"
  },
  {
    "startMs": 927199,
    "text": "sales and marketing team to figure out"
  },
  {
    "startMs": 928880,
    "text": "how to juice that for revenue, how to"
  },
  {
    "startMs": 930480,
    "text": "increase customer satisfaction, how to"
  },
  {
    "startMs": 931920,
    "text": "expand into a neighboring market. So,"
  },
  {
    "startMs": 933519,
    "text": "this is a major project. Um, thinking"
  },
  {
    "startMs": 936880,
    "text": "about where to do them takes a little"
  },
  {
    "startMs": 938560,
    "text": "bit of of um of judgment. You want to"
  },
  {
    "startMs": 940959,
    "text": "pick a domain that is scaled and that"
  },
  {
    "startMs": 942959,
    "text": "already has material ROI. I would"
  },
  {
    "startMs": 944880,
    "text": "actually not propose building full"
  },
  {
    "startMs": 946560,
    "text": "automations if you're a startup, if"
  },
  {
    "startMs": 948240,
    "text": "you're in discovery, if you're looking"
  },
  {
    "startMs": 949440,
    "text": "at new and new businesses. Keep the"
  },
  {
    "startMs": 950959,
    "text": "flexibility of humans. But if you have a"
  },
  {
    "startMs": 953040,
    "text": "bucket of tasks that's associated with"
  },
  {
    "startMs": 954800,
    "text": "seven figures of revenue and that maybe"
  },
  {
    "startMs": 956800,
    "text": "there's a customer ticket, hopefully"
  },
  {
    "startMs": 958320,
    "text": "there's 10 customer tickets, the"
  },
  {
    "startMs": 959920,
    "text": "organization knows that there's pain"
  },
  {
    "startMs": 961279,
    "text": "here, that can justify an investment and"
  },
  {
    "startMs": 963920,
    "text": "that is usually something that a CFO can"
  },
  {
    "startMs": 965759,
    "text": "understand. You know, if we say, hey,"
  },
  {
    "startMs": 967519,
    "text": "this is associated with seven figures of"
  },
  {
    "startMs": 969600,
    "text": "revenue and there's five tickets."
  },
  {
    "startMs": 970959,
    "text": "Customers don't like it or they want"
  },
  {
    "startMs": 972240,
    "text": "something else here. That's a pretty"
  },
  {
    "startMs": 973600,
    "text": "good leading signal."
  },
  {
    "startMs": 976399,
    "text": "Sort of like in terms of the tasks"
  },
  {
    "startMs": 978079,
    "text": "themselves, look for business processes"
  },
  {
    "startMs": 980320,
    "text": "that are running. ideal if it has"
  },
  {
    "startMs": 982320,
    "text": "written specifications. I wouldn't worry"
  },
  {
    "startMs": 983839,
    "text": "too much about if the written"
  },
  {
    "startMs": 984880,
    "text": "specifications are any good."
  },
  {
    "startMs": 986720,
    "text": "Specifications are really hard and I"
  },
  {
    "startMs": 988639,
    "text": "think that even if there has been"
  },
  {
    "startMs": 990399,
    "text": "momentum to build specifications that"
  },
  {
    "startMs": 993279,
    "text": "reflects that on the inside we the team"
  },
  {
    "startMs": 996079,
    "text": "believes that this is specifiable the"
  },
  {
    "startMs": 998320,
    "text": "team believes that this is not just raw"
  },
  {
    "startMs": 1000000,
    "text": "judgment complicated detail orient"
  },
  {
    "startMs": 1005920,
    "text": "work now which is actually"
  },
  {
    "startMs": 1008320,
    "text": "shockingly shockingly new and very very"
  },
  {
    "startMs": 1010560,
    "text": "interesting. It's ideal if this is"
  },
  {
    "startMs": 1012079,
    "text": "detail- oriented work because you can"
  },
  {
    "startMs": 1014079,
    "text": "take this sort of expensive thought,"
  },
  {
    "startMs": 1016880,
    "text": "memory, um, wrote work off of of humans"
  },
  {
    "startMs": 1020480,
    "text": "who tend to fail at it at scale."
  },
  {
    "startMs": 1022880,
    "text": "Subjective judgment work is not. So"
  },
  {
    "startMs": 1024720,
    "text": "that's where you have to draw the line."
  },
  {
    "startMs": 1026079,
    "text": "Uh, AI for like if your business is"
  },
  {
    "startMs": 1028400,
    "text": "built around a process, AI to automate"
  },
  {
    "startMs": 1030880,
    "text": "that specific differentiated process is"
  },
  {
    "startMs": 1032880,
    "text": "still in my opinion um, it's not where"
  },
  {
    "startMs": 1035120,
    "text": "the action is."
  },
  {
    "startMs": 1037039,
    "text": "So a bag of tricks here. I'm not going"
  },
  {
    "startMs": 1038640,
    "text": "to get into this, but if you are working"
  },
  {
    "startMs": 1040160,
    "text": "on something where you're going from 95"
  },
  {
    "startMs": 1041839,
    "text": "to 99.95%,"
  },
  {
    "startMs": 1043600,
    "text": "here's a bag of tricks. This talk will"
  },
  {
    "startMs": 1044959,
    "text": "be online. You can take a photo of this,"
  },
  {
    "startMs": 1046400,
    "text": "contact me. I'd love to chat. I'm very"
  },
  {
    "startMs": 1048079,
    "text": "interested in the ways that models fail."
  },
  {
    "startMs": 1050000,
    "text": "I feel like knowing where models fail is"
  },
  {
    "startMs": 1051679,
    "text": "really interesting for what to build"
  },
  {
    "startMs": 1053280,
    "text": "next. Um, but I'd be I'd love to chat"
  },
  {
    "startMs": 1055280,
    "text": "about any places where you have clean"
  },
  {
    "startMs": 1057039,
    "text": "input, uh, a sort of specified process,"
  },
  {
    "startMs": 1059679,
    "text": "and you can't get to three and a half"
  },
  {
    "startMs": 1061120,
    "text": "nights of accuracy. I think the two that"
  },
  {
    "startMs": 1063200,
    "text": "I'll flag out of this is the top one"
  },
  {
    "startMs": 1064720,
    "text": "which is read and write evals and the"
  },
  {
    "startMs": 1066480,
    "text": "bottom one which is build a thousand"
  },
  {
    "startMs": 1067840,
    "text": "evals and expect to spend time reading"
  },
  {
    "startMs": 1069520,
    "text": "them. Um that was really one of the big"
  },
  {
    "startMs": 1072240,
    "text": "learnings from my team is is everybody"
  },
  {
    "startMs": 1074320,
    "text": "had to invest in reading exactly why"
  },
  {
    "startMs": 1076640,
    "text": "this time the AI did not do the thing we"
  },
  {
    "startMs": 1079200,
    "text": "expected on the emails and it's"
  },
  {
    "startMs": 1080480,
    "text": "exhausting work because you're reading"
  },
  {
    "startMs": 1082000,
    "text": "psychopantic text and you're trying to"
  },
  {
    "startMs": 1083760,
    "text": "figure out what specifically went wrong"
  },
  {
    "startMs": 1086480,
    "text": "or against my expectations."
  },
  {
    "startMs": 1088720,
    "text": "I'm close with this quote. This is one"
  },
  {
    "startMs": 1090400,
    "text": "of my favorite quotes is Rockefeller uh"
  },
  {
    "startMs": 1092799,
    "text": "paraphrase as soon as you can hire"
  },
  {
    "startMs": 1094640,
    "text": "someone to do your job. Sit down, kick"
  },
  {
    "startMs": 1096799,
    "text": "up your feet, think about how to make"
  },
  {
    "startMs": 1098320,
    "text": "the company more money. What is he"
  },
  {
    "startMs": 1100400,
    "text": "saying here? He's saying if your job is"
  },
  {
    "startMs": 1102960,
    "text": "specifiable, get someone else to do it."
  },
  {
    "startMs": 1105520,
    "text": "What we want you to be doing is"
  },
  {
    "startMs": 1106799,
    "text": "thinking, applying your judgment,"
  },
  {
    "startMs": 1108480,
    "text": "looking for new opportunity. I think"
  },
  {
    "startMs": 1110559,
    "text": "that's perennial advice. It's great"
  },
  {
    "startMs": 1111919,
    "text": "advice. My hope is that AI might be able"
  },
  {
    "startMs": 1114320,
    "text": "to help us do this even better. If AI"
  },
  {
    "startMs": 1116720,
    "text": "can help route our attention towards the"
  },
  {
    "startMs": 1119120,
    "text": "most valuable places, the most valuable"
  },
  {
    "startMs": 1120880,
    "text": "opportunities, then maybe we can sit"
  },
  {
    "startMs": 1122799,
    "text": "down and actually think about the most"
  },
  {
    "startMs": 1124799,
    "text": "valuable things first. Um, my bet is"
  },
  {
    "startMs": 1128080,
    "text": "that orchestration layer and and um the"
  },
  {
    "startMs": 1131600,
    "text": "right harnesses, the market's going to"
  },
  {
    "startMs": 1133360,
    "text": "sort that out and there's not going to"
  },
  {
    "startMs": 1134880,
    "text": "be a huge amount of differentiated"
  },
  {
    "startMs": 1136320,
    "text": "advantage in terms of picking the right"
  },
  {
    "startMs": 1138480,
    "text": "AI orchestration software in 2026."
  },
  {
    "startMs": 1141600,
    "text": "My bet is that the companies that"
  },
  {
    "startMs": 1143840,
    "text": "succeed the most with AI are going to be"
  },
  {
    "startMs": 1146000,
    "text": "the ones who figure out how to best"
  },
  {
    "startMs": 1147600,
    "text": "manage their collective attention and"
  },
  {
    "startMs": 1149200,
    "text": "judgment because the premium on that is"
  },
  {
    "startMs": 1150960,
    "text": "going to only go up at least for so long"
  },
  {
    "startMs": 1153600,
    "text": "as human judgment holds a premium. Um"
  },
  {
    "startMs": 1156480,
    "text": "that's it for my talk. It's exciting"
  },
  {
    "startMs": 1158240,
    "text": "times everyone. Uh I hope everybody"
  },
  {
    "startMs": 1160080,
    "text": "enjoys the conference and uh really"
  },
  {
    "startMs": 1162160,
    "text": "excited about all the stuff that we're"
  },
  {
    "startMs": 1163280,
    "text": "going to be building."
  },
  {
    "startMs": 1177703,
    "text": "[music]"
  }
]
