[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13519,
    "text": "Hello everyone. Um, this is a"
  },
  {
    "startMs": 15360,
    "text": "practitioner report uh from real"
  },
  {
    "startMs": 17359,
    "text": "production work. So, let's get into it."
  },
  {
    "startMs": 21520,
    "text": "Um, I'll skip the generic uh yet another"
  },
  {
    "startMs": 24160,
    "text": "loop agent intro. This is about the hard"
  },
  {
    "startMs": 26880,
    "text": "part most agent demos skip."
  },
  {
    "startMs": 30080,
    "text": "and about turning messy operational"
  },
  {
    "startMs": 32079,
    "text": "knowledge into something an agent can"
  },
  {
    "startMs": 35200,
    "text": "execute safely."
  },
  {
    "startMs": 37280,
    "text": "This comes from real work uh in my"
  },
  {
    "startMs": 39680,
    "text": "company I'm working for supporting"
  },
  {
    "startMs": 42559,
    "text": "global shipping operations and grounded"
  },
  {
    "startMs": 44960,
    "text": "in production."
  },
  {
    "startMs": 48399,
    "text": "On paper it's uh one workflow usually"
  },
  {
    "startMs": 50719,
    "text": "but uh in reality every shipment is an"
  },
  {
    "startMs": 53600,
    "text": "orchestration of many parallel state"
  },
  {
    "startMs": 55840,
    "text": "machines. While they agree the happy"
  },
  {
    "startMs": 59280,
    "text": "paths work the moment one drifts you get"
  },
  {
    "startMs": 62480,
    "text": "exception work."
  },
  {
    "startMs": 66240,
    "text": "The easy majority is already automated"
  },
  {
    "startMs": 69200,
    "text": "in many companies. What's left is the"
  },
  {
    "startMs": 72080,
    "text": "long tail and more exceptions than"
  },
  {
    "startMs": 75680,
    "text": "system built uh to handle them. That"
  },
  {
    "startMs": 78720,
    "text": "tail is uh the expensive part."
  },
  {
    "startMs": 83200,
    "text": "And then there's my favorite category."
  },
  {
    "startMs": 85280,
    "text": "And it comes with a special uh plate"
  },
  {
    "startMs": 89040,
    "text": "here. See for EI builder dreams and"
  },
  {
    "startMs": 93439,
    "text": "their laptops. This what you can find"
  },
  {
    "startMs": 96320,
    "text": "outside of AI bubble in San Francisco."
  },
  {
    "startMs": 102240,
    "text": "The signal process uh depends on many"
  },
  {
    "startMs": 104799,
    "text": "systems being coherent at once. If any"
  },
  {
    "startMs": 107759,
    "text": "step uh can't complete the happy path"
  },
  {
    "startMs": 110720,
    "text": "breaks and then it takes expert uh"
  },
  {
    "startMs": 113920,
    "text": "archist expert orchestration across uh"
  },
  {
    "startMs": 116880,
    "text": "multiple incomplete systems."
  },
  {
    "startMs": 120399,
    "text": "All these uh variations um path pathways"
  },
  {
    "startMs": 124159,
    "text": "should be captured in SOPs. SOPs is a"
  },
  {
    "startMs": 127119,
    "text": "standard operating procedure common and"
  },
  {
    "startMs": 129039,
    "text": "regulated industries. So an expert and"
  },
  {
    "startMs": 131920,
    "text": "the model read them uh the same way."
  },
  {
    "startMs": 136959,
    "text": "That gap is the hard part. Stable intent"
  },
  {
    "startMs": 140400,
    "text": "detection tool calls you can guarantee"
  },
  {
    "startMs": 143280,
    "text": "are safe integrating with legacy back"
  },
  {
    "startMs": 146400,
    "text": "ends and results evaluated with experts."
  },
  {
    "startMs": 152319,
    "text": "Uh I call this uh tribal dungeons. Uh"
  },
  {
    "startMs": 155360,
    "text": "the knowledge exists but not in a form"
  },
  {
    "startMs": 158640,
    "text": "uh agent can execute and you can safely"
  },
  {
    "startMs": 161599,
    "text": "run a process. You can't safely run a"
  },
  {
    "startMs": 164239,
    "text": "process. The organization cannot"
  },
  {
    "startMs": 165599,
    "text": "represent"
  },
  {
    "startMs": 169360,
    "text": "standard legacy SOPs [clears throat]"
  },
  {
    "startMs": 171680,
    "text": "bunch of bunch of screenshots organized"
  },
  {
    "startMs": 175040,
    "text": "in sequence and but screenshots not uh a"
  },
  {
    "startMs": 178480,
    "text": "process. A legacy SOPs explain what a"
  },
  {
    "startMs": 181760,
    "text": "person sees and clicks. And an agent SOP"
  },
  {
    "startMs": 185680,
    "text": "needs a more complex uh setup,"
  },
  {
    "startMs": 188319,
    "text": "preconditions, uh decisions,"
  },
  {
    "startMs": 190400,
    "text": "identifiers, back end calls, validation,"
  },
  {
    "startMs": 193360,
    "text": "recovery, and evidence of uh successful"
  },
  {
    "startMs": 196560,
    "text": "execution."
  },
  {
    "startMs": 201040,
    "text": "Experts own the what, agents own the"
  },
  {
    "startMs": 204640,
    "text": "how. And exception becomes a guardrail."
  },
  {
    "startMs": 208159,
    "text": "Most of the effort is the translation"
  },
  {
    "startMs": 210799,
    "text": "and negotiation between them to align on"
  },
  {
    "startMs": 214560,
    "text": "common sense."
  },
  {
    "startMs": 218319,
    "text": "Three parts here um in this architecture"
  },
  {
    "startMs": 221519,
    "text": "it's SOP memory uh organized as SOP"
  },
  {
    "startMs": 224799,
    "text": "corpus"
  },
  {
    "startMs": 226319,
    "text": "execution runtime and theme feedback"
  },
  {
    "startMs": 228720,
    "text": "capture. The agent loop is not the"
  },
  {
    "startMs": 231440,
    "text": "system. The refining loop around the"
  },
  {
    "startMs": 233840,
    "text": "agent is the system"
  },
  {
    "startMs": 236400,
    "text": "and it's the most complex part. Oh,"
  },
  {
    "startMs": 238959,
    "text": "sorry SAP is okay. It's this slide for"
  },
  {
    "startMs": 242159,
    "text": "UK. This is correct one. So and it's"
  },
  {
    "startMs": 246080,
    "text": "good illustration why the the same thing"
  },
  {
    "startMs": 249439,
    "text": "is means different and uh describing"
  },
  {
    "startMs": 252640,
    "text": "differently in different countries and"
  },
  {
    "startMs": 255360,
    "text": "it's creating a lot of variations"
  },
  {
    "startMs": 256560,
    "text": "between each country"
  },
  {
    "startMs": 259680,
    "text": "and that corpus is a asset the company"
  },
  {
    "startMs": 262880,
    "text": "company's process memory uh modified and"
  },
  {
    "startMs": 265840,
    "text": "aligned with every country um conditions"
  },
  {
    "startMs": 270160,
    "text": "and far bigger than than than runtime"
  },
  {
    "startMs": 272720,
    "text": "you could see the proportion 20 to1"
  },
  {
    "startMs": 276240,
    "text": "So and this is concurrently operating"
  },
  {
    "startMs": 278880,
    "text": "system"
  },
  {
    "startMs": 281840,
    "text": "and this is the scale we run in"
  },
  {
    "startMs": 283759,
    "text": "production today"
  },
  {
    "startMs": 286320,
    "text": "over 200 instances and spikes and"
  },
  {
    "startMs": 290000,
    "text": "latencies deviates from few minutes to"
  },
  {
    "startMs": 293199,
    "text": "up to 10 minutes."
  },
  {
    "startMs": 296960,
    "text": "Um and mainly yeah the mainly main"
  },
  {
    "startMs": 300080,
    "text": "reason for it that u we depending on"
  },
  {
    "startMs": 302639,
    "text": "many legacy system which is uh so cannot"
  },
  {
    "startMs": 305840,
    "text": "be faster than agent loop itself."
  },
  {
    "startMs": 311440,
    "text": "Expert time is the bottleneck. So the"
  },
  {
    "startMs": 315520,
    "text": "theme bench uh does the triage for us."
  },
  {
    "startMs": 319199,
    "text": "It clusters the failures and hands back"
  },
  {
    "startMs": 322320,
    "text": "something you can act on. Not just look"
  },
  {
    "startMs": 324400,
    "text": "at look at it."
  },
  {
    "startMs": 328880,
    "text": "The trace is the shared evidence that"
  },
  {
    "startMs": 331840,
    "text": "lets an expert and an engineer review"
  },
  {
    "startMs": 334080,
    "text": "the same case and agree on what"
  },
  {
    "startMs": 336479,
    "text": "happened."
  },
  {
    "startMs": 339280,
    "text": "A correction only counts when it becomes"
  },
  {
    "startMs": 341440,
    "text": "an executable change. And that's the"
  },
  {
    "startMs": 345440,
    "text": "line between an opinion and a production"
  },
  {
    "startMs": 347919,
    "text": "fix."
  },
  {
    "startMs": 352880,
    "text": "And and this is where quality comes"
  },
  {
    "startMs": 355280,
    "text": "from. not from vibes uh not from a"
  },
  {
    "startMs": 359759,
    "text": "bigger model from replaying real"
  },
  {
    "startMs": 361840,
    "text": "examples with u disabled rights to uh"
  },
  {
    "startMs": 366000,
    "text": "protect the production systems and"
  },
  {
    "startMs": 368720,
    "text": "checking whether behavior improved."
  },
  {
    "startMs": 373440,
    "text": "You can see here on the"
  },
  {
    "startMs": 376400,
    "text": "uh cognitive proportion u or this effort"
  },
  {
    "startMs": 379280,
    "text": "ratio uh between each activity in our"
  },
  {
    "startMs": 381600,
    "text": "project. So usually uh pipe coding ends"
  },
  {
    "startMs": 385840,
    "text": "here."
  },
  {
    "startMs": 387600,
    "text": "Here there ends um"
  },
  {
    "startMs": 390560,
    "text": "specdriven development because it cannot"
  },
  {
    "startMs": 393680,
    "text": "uh grow improve accuracy more than this"
  },
  {
    "startMs": 397360,
    "text": "stage on this scale. And this is uh"
  },
  {
    "startMs": 400639,
    "text": "where the real work starts. Nothing"
  },
  {
    "startMs": 404400,
    "text": "exotic. It's engineering common"
  },
  {
    "startMs": 406400,
    "text": "engineering sense applied at scale."
  },
  {
    "startMs": 410160,
    "text": "So if uh you don't know all this uh"
  },
  {
    "startMs": 413680,
    "text": "terminology which developed over lastuh"
  },
  {
    "startMs": 417199,
    "text": "30 years in software development"
  },
  {
    "startMs": 418800,
    "text": "argument to check because this is what"
  },
  {
    "startMs": 421360,
    "text": "every AI agent uh AI coding agent should"
  },
  {
    "startMs": 423840,
    "text": "know uh to help you develop reliable"
  },
  {
    "startMs": 427120,
    "text": "production systems"
  },
  {
    "startMs": 432080,
    "text": "and accuracy it's uh wasn't designed uh"
  },
  {
    "startMs": 435120,
    "text": "in one diagram up front it was earned"
  },
  {
    "startMs": 439039,
    "text": "one small correction at the time at the"
  },
  {
    "startMs": 441039,
    "text": "scale you see here. So we have over"
  },
  {
    "startMs": 444960,
    "text": "100,000"
  },
  {
    "startMs": 446720,
    "text": "corrections over last 9 months in the"
  },
  {
    "startMs": 449599,
    "text": "system when we developing it"
  },
  {
    "startMs": 453265,
    "text": "[clears throat]"
  },
  {
    "startMs": 455440,
    "text": "and this um heat maps uh turned"
  },
  {
    "startMs": 459199,
    "text": "thousands of traces into priorities. is"
  },
  {
    "startMs": 461680,
    "text": "how we keep experts and engineers uh"
  },
  {
    "startMs": 464560,
    "text": "looking at the same problems and"
  },
  {
    "startMs": 466160,
    "text": "prioritize where the the most beneficial"
  },
  {
    "startMs": 469680,
    "text": "work for them. Every cell is a group of"
  },
  {
    "startMs": 473039,
    "text": "tracked scenarios we have and uh usually"
  },
  {
    "startMs": 477440,
    "text": "to turn one block in red it's around one"
  },
  {
    "startMs": 482000,
    "text": "two months of force for the whole team"
  },
  {
    "startMs": 486960,
    "text": "whole team of engineers and also AI"
  },
  {
    "startMs": 489360,
    "text": "agents"
  },
  {
    "startMs": 491280,
    "text": "um the agent failed is uh where the"
  },
  {
    "startMs": 493599,
    "text": "investigation starts not where it ends"
  },
  {
    "startMs": 496000,
    "text": "each failure maps to a specific uh fix"
  },
  {
    "startMs": 502319,
    "text": "discovery needs agent freedom and"
  },
  {
    "startMs": 505520,
    "text": "production needs a cage. Uh the harness"
  },
  {
    "startMs": 508160,
    "text": "isn't there to give the agent more room."
  },
  {
    "startMs": 510560,
    "text": "It's there to make the dumb mistakes"
  },
  {
    "startMs": 512640,
    "text": "impossible."
  },
  {
    "startMs": 516640,
    "text": "So on this scale please be careful is"
  },
  {
    "startMs": 518800,
    "text": "not a guard guard. Uh if we have wrong"
  },
  {
    "startMs": 521599,
    "text": "workflow then classifier eval. If it's"
  },
  {
    "startMs": 524399,
    "text": "wrong right then right gate. If it's"
  },
  {
    "startMs": 527120,
    "text": "wrong assumption then it's a mere view."
  },
  {
    "startMs": 529760,
    "text": "A preventive measure eliminates the"
  },
  {
    "startMs": 531600,
    "text": "unsafe path"
  },
  {
    "startMs": 535600,
    "text": "on critical paths. U review and approval"
  },
  {
    "startMs": 538399,
    "text": "stay in the loop. The engine engineering"
  },
  {
    "startMs": 540480,
    "text": "focus is uh to build safe hands offs and"
  },
  {
    "startMs": 543519,
    "text": "a trail you can trust."
  },
  {
    "startMs": 548880,
    "text": "The real outcome"
  },
  {
    "startMs": 551200,
    "text": "uh wasn't the agent in the system. It"
  },
  {
    "startMs": 553920,
    "text": "was the [clears throat] methodology we"
  },
  {
    "startMs": 555680,
    "text": "built around it. If you want the"
  },
  {
    "startMs": 558399,
    "text": "blueprint, then it's uh these five"
  },
  {
    "startMs": 560640,
    "text": "moves. Make work representable. Make exe"
  },
  {
    "startMs": 564320,
    "text": "execution bounded. Make behavior"
  },
  {
    "startMs": 566880,
    "text": "observable for every agent and make"
  },
  {
    "startMs": 569839,
    "text": "correction cheap. And last thing is make"
  },
  {
    "startMs": 572959,
    "text": "improvement compound. So gradually"
  },
  {
    "startMs": 575519,
    "text": "systematically improve the quality of"
  },
  {
    "startMs": 577440,
    "text": "the system."
  },
  {
    "startMs": 581920,
    "text": "AI native um operation is more than"
  },
  {
    "startMs": 585519,
    "text": "agents in workflow. It's a system that"
  },
  {
    "startMs": 588320,
    "text": "learns from what works and fold folds it"
  },
  {
    "startMs": 591920,
    "text": "back into code as new composite tools"
  },
  {
    "startMs": 594720,
    "text": "adapting to the applications and the"
  },
  {
    "startMs": 596480,
    "text": "people around it. The best AI models um"
  },
  {
    "startMs": 600480,
    "text": "oriented intelligence for us. The"
  },
  {
    "startMs": 603200,
    "text": "adaptive architecture we built is the"
  },
  {
    "startMs": 605839,
    "text": "asset, the final asset"
  },
  {
    "startMs": 610720,
    "text": "and"
  },
  {
    "startMs": 612399,
    "text": "we aggregating all um repeatable"
  },
  {
    "startMs": 616079,
    "text": "sequences of steps successful scenarios"
  },
  {
    "startMs": 618720,
    "text": "and uh merging them into bigger tools"
  },
  {
    "startMs": 622320,
    "text": "which uh combine the disproven scenarios"
  },
  {
    "startMs": 625200,
    "text": "into the reusable snippets by other"
  },
  {
    "startMs": 627839,
    "text": "agents. So and then um it's possible to"
  },
  {
    "startMs": 631120,
    "text": "roll out them not only for one country"
  },
  {
    "startMs": 633040,
    "text": "but for hundreds country in one go."
  },
  {
    "startMs": 638640,
    "text": "So this is um um all for the talk and"
  },
  {
    "startMs": 643040,
    "text": "little time for questions and I'll be"
  },
  {
    "startMs": 644800,
    "text": "around afterwards. And the final"
  },
  {
    "startMs": 647040,
    "text": "reminder you know if you you know if you"
  },
  {
    "startMs": 650160,
    "text": "are AI builder if you emotionally"
  },
  {
    "startMs": 654560,
    "text": "attached to tools not MCPS"
  },
  {
    "startMs": 657360,
    "text": "we're not using MCPS because uh for us"
  },
  {
    "startMs": 659760,
    "text": "it's uh always not the best choice. So"
  },
  {
    "startMs": 662880,
    "text": "because all all systems usually really"
  },
  {
    "startMs": 665760,
    "text": "bloated and we have to distill responses"
  },
  {
    "startMs": 668480,
    "text": "and uh tune the tools through function"
  },
  {
    "startMs": 672480,
    "text": "calling uh to our agents then we can"
  },
  {
    "startMs": 675440,
    "text": "control quality of um our software and"
  },
  {
    "startMs": 679120,
    "text": "ensure that uh it's correctly"
  },
  {
    "startMs": 682720,
    "text": "processing assigned tasks."
  },
  {
    "startMs": 687200,
    "text": "Thank you. Any questions?"
  },
  {
    "startMs": 693519,
    "text": "Okay, then um thanks for your attent u"
  },
  {
    "startMs": 697279,
    "text": "attention. Then I will be around so you"
  },
  {
    "startMs": 699519,
    "text": "can ask me questions if you want."
  },
  {
    "startMs": 704133,
    "text": "[applause]"
  },
  {
    "startMs": 720508,
    "text": ">> [music]"
  }
]
