[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12960,
    "text": "Hello. All right. Uh, let's get started."
  },
  {
    "startMs": 16400,
    "text": "So, um, well, welcome to the talk. Give"
  },
  {
    "startMs": 20080,
    "text": "the agent a budget, not a not a token."
  },
  {
    "startMs": 23359,
    "text": "Um, quick intro about me. My name is"
  },
  {
    "startMs": 25199,
    "text": "Sachin. Um, I'm an engineer on the CI"
  },
  {
    "startMs": 27199,
    "text": "team at Entropic. Uh we basically build"
  },
  {
    "startMs": 29519,
    "text": "and manage all of the test machinery for"
  },
  {
    "startMs": 31840,
    "text": "all the code that people write. Uh so"
  },
  {
    "startMs": 34079,
    "text": "everything from test quarantining to"
  },
  {
    "startMs": 35600,
    "text": "merge automation to CI autoscaling to"
  },
  {
    "startMs": 38239,
    "text": "merge cues, everything and anything in"
  },
  {
    "startMs": 39760,
    "text": "between. Um it's it's basically the"
  },
  {
    "startMs": 41440,
    "text": "plumbing that allows a few thousand"
  },
  {
    "startMs": 42640,
    "text": "engineers every day to ship code safely."
  },
  {
    "startMs": 45040,
    "text": "Uh I've been at the company for a little"
  },
  {
    "startMs": 46640,
    "text": "over 10 months. I'm based out of"
  },
  {
    "startMs": 47840,
    "text": "Seattle. Um and outside of work, I love"
  },
  {
    "startMs": 50480,
    "text": "dialing my espressos and climbing some"
  },
  {
    "startMs": 51840,
    "text": "crazy mountains."
  },
  {
    "startMs": 54160,
    "text": "So a lot of cool agent demos they they"
  },
  {
    "startMs": 56719,
    "text": "start with the same way really. So"
  },
  {
    "startMs": 58719,
    "text": "someone gives an agent a god token um"
  },
  {
    "startMs": 61280,
    "text": "and gives it an access to some sort of"
  },
  {
    "startMs": 62800,
    "text": "tool list and just watch as it goes. So"
  },
  {
    "startMs": 65360,
    "text": "and it does go like in in this"
  },
  {
    "startMs": 66799,
    "text": "particular example it it'll spin up like"
  },
  {
    "startMs": 68560,
    "text": "a cute little coffee website. It'll"
  },
  {
    "startMs": 70320,
    "text": "build all the pages, deploy them, uh"
  },
  {
    "startMs": 72479,
    "text": "bring it up live and it's done in 3"
  },
  {
    "startMs": 73920,
    "text": "seconds. Uh people will nod people will"
  },
  {
    "startMs": 76080,
    "text": "clap and someone will ship it into"
  },
  {
    "startMs": 78240,
    "text": "production. And this this talk is"
  },
  {
    "startMs": 80479,
    "text": "basically about what happens after that"
  },
  {
    "startMs": 82400,
    "text": "after the demo ships once the agent is"
  },
  {
    "startMs": 84720,
    "text": "starting to do some like real work in"
  },
  {
    "startMs": 86240,
    "text": "production. [snorts]"
  },
  {
    "startMs": 88479,
    "text": "So here's what after looked like in in"
  },
  {
    "startMs": 91119,
    "text": "our one of our scenarios. So what you're"
  },
  {
    "startMs": 93040,
    "text": "looking at is like a real command. Um"
  },
  {
    "startMs": 94640,
    "text": "the agent was trying to clean up after"
  },
  {
    "startMs": 96240,
    "text": "itself. Um it it was basically listing a"
  },
  {
    "startMs": 98720,
    "text": "bunch of workloads that were no longer"
  },
  {
    "startMs": 100320,
    "text": "interesting or useful to itself. Um and"
  },
  {
    "startMs": 102880,
    "text": "then it found them and deleted them. Now"
  },
  {
    "startMs": 105119,
    "text": "that's that's completely reasonable"
  },
  {
    "startMs": 106560,
    "text": "except one stage in the pipeline"
  },
  {
    "startMs": 109040,
    "text": "basically evaluated to nothing and the"
  },
  {
    "startMs": 111600,
    "text": "filter dropped out and now the selector"
  },
  {
    "startMs": 113600,
    "text": "matched everything."
  },
  {
    "startMs": 116079,
    "text": "So you can see it took out about 200"
  },
  {
    "startMs": 118000,
    "text": "workloads uh which ended up impacting"
  },
  {
    "startMs": 120960,
    "text": "about 20 engineers worth of stuff and"
  },
  {
    "startMs": 122799,
    "text": "all of that was gone in 90 seconds."
  },
  {
    "startMs": 125040,
    "text": "Nobody was being malicious in this case"
  },
  {
    "startMs": 127280,
    "text": "like the agent genuinely thought it was"
  },
  {
    "startMs": 129520,
    "text": "tidying up after itself. uh some of"
  },
  {
    "startMs": 131599,
    "text": "these workloads were long running"
  },
  {
    "startMs": 133440,
    "text": "training jobs and stuff. Maybe some of"
  },
  {
    "startMs": 135120,
    "text": "these were not even checkpointed and it"
  },
  {
    "startMs": 137280,
    "text": "was just like hours of progress that was"
  },
  {
    "startMs": 138720,
    "text": "gone poof in like 90 seconds and stuff,"
  },
  {
    "startMs": 141440,
    "text": "right? Um the the problem in in this"
  },
  {
    "startMs": 144879,
    "text": "case is that this idea of like here's a"
  },
  {
    "startMs": 147360,
    "text": "token and here's a tool list that is"
  },
  {
    "startMs": 149760,
    "text": "just like not enough like it just"
  },
  {
    "startMs": 151120,
    "text": "doesn't scale when the agent is starting"
  },
  {
    "startMs": 152560,
    "text": "to do some some like real work in"
  },
  {
    "startMs": 154239,
    "text": "production."
  },
  {
    "startMs": 157120,
    "text": ">> So here's just a brief summary of like"
  },
  {
    "startMs": 159280,
    "text": "what we're going to talk about today. Uh"
  },
  {
    "startMs": 160720,
    "text": "I'm going to walk through three"
  },
  {
    "startMs": 162080,
    "text": "primitives um and asymmetric verbs. We"
  },
  {
    "startMs": 165920,
    "text": "we're going to look at some like rate"
  },
  {
    "startMs": 167440,
    "text": "limits. We're going to look at this idea"
  },
  {
    "startMs": 169280,
    "text": "of like trip wires over allow lists. And"
  },
  {
    "startMs": 171519,
    "text": "then I'm going to talk about this one"
  },
  {
    "startMs": 172800,
    "text": "lens that which I call the undo test"
  },
  {
    "startMs": 175440,
    "text": "that you can sort of use to size up the"
  },
  {
    "startMs": 177599,
    "text": "other three primitives. Uh so three"
  },
  {
    "startMs": 179920,
    "text": "things that you basically enforce and"
  },
  {
    "startMs": 181360,
    "text": "then one question you sort of ask about"
  },
  {
    "startMs": 182879,
    "text": "all of them."
  },
  {
    "startMs": 186879,
    "text": "So here's basically what bugged me uh"
  },
  {
    "startMs": 189840,
    "text": "after the cleanup was done for the"
  },
  {
    "startMs": 190959,
    "text": "incident that I was just showing you. Uh"
  },
  {
    "startMs": 192560,
    "text": "the the agent technically hadn't done"
  },
  {
    "startMs": 194560,
    "text": "anything that I couldn't have done. Uh"
  },
  {
    "startMs": 196800,
    "text": "it was using my token after all. The the"
  },
  {
    "startMs": 199040,
    "text": "failure wasn't the model itself. The"
  },
  {
    "startMs": 200720,
    "text": "failure was that I was giving the agent"
  },
  {
    "startMs": 202879,
    "text": "unbounded amount of power to do"
  },
  {
    "startMs": 204239,
    "text": "something that I wasn't watching super"
  },
  {
    "startMs": 206239,
    "text": "intently. Um and it was we've basically"
  },
  {
    "startMs": 210879,
    "text": "solved this the same kind of problem"
  },
  {
    "startMs": 212720,
    "text": "except not for agents but like the first"
  },
  {
    "startMs": 214799,
    "text": "time you onboarded like a junior"
  },
  {
    "startMs": 215920,
    "text": "engineer on onto your own teams, right?"
  },
  {
    "startMs": 217440,
    "text": "So just think about it like we don't we"
  },
  {
    "startMs": 219599,
    "text": "don't basically sit around watching um"
  },
  {
    "startMs": 221840,
    "text": "every engineer or new boarded person"
  },
  {
    "startMs": 223920,
    "text": "just like type out every keystroke on"
  },
  {
    "startMs": 226000,
    "text": "their keyboard, right? Like we're not"
  },
  {
    "startMs": 227040,
    "text": "behind their chairs just watching"
  },
  {
    "startMs": 228319,
    "text": "everything that they're doing. Uh there"
  },
  {
    "startMs": 230159,
    "text": "is always a path on whenever a new"
  },
  {
    "startMs": 232000,
    "text": "engineer wants to ask for something. Um"
  },
  {
    "startMs": 233840,
    "text": "there is always an escalation path. uh"
  },
  {
    "startMs": 236799,
    "text": "the catastrophic stuff is just"
  },
  {
    "startMs": 238480,
    "text": "structurally out of reach for them. So"
  },
  {
    "startMs": 240720,
    "text": "the the other side of the coin is that"
  },
  {
    "startMs": 242319,
    "text": "the agents are very different. They they"
  },
  {
    "startMs": 244000,
    "text": "never get tired. They never sleep. And"
  },
  {
    "startMs": 245840,
    "text": "every so often they're just like very"
  },
  {
    "startMs": 247200,
    "text": "confidently wrong. Um and if I'm being"
  },
  {
    "startMs": 249280,
    "text": "honest, that's that's maybe most of us"
  },
  {
    "startMs": 250879,
    "text": "on the first month of our jobs anyways."
  },
  {
    "startMs": 252959,
    "text": "And so what what we're going to talk"
  },
  {
    "startMs": 254959,
    "text": "about in a little bit in is just this"
  },
  {
    "startMs": 257440,
    "text": "idea of like your onboarding checklist,"
  },
  {
    "startMs": 259040,
    "text": "but written down as policy for for"
  },
  {
    "startMs": 260720,
    "text": "agents."
  },
  {
    "startMs": 263600,
    "text": "So that and that brings me to the word"
  },
  {
    "startMs": 265360,
    "text": "in the title of my my talk here is that"
  },
  {
    "startMs": 267840,
    "text": "the the the standard fix for an incident"
  },
  {
    "startMs": 270160,
    "text": "like that is basically that you narrow"
  },
  {
    "startMs": 272560,
    "text": "the token scope, right? Like you just"
  },
  {
    "startMs": 274160,
    "text": "take the deletes away effectively. You"
  },
  {
    "startMs": 276720,
    "text": "would technically never do that for like"
  },
  {
    "startMs": 278080,
    "text": "a new hire, right? Like you would you"
  },
  {
    "startMs": 279600,
    "text": "wouldn't take the whole verb away from"
  },
  {
    "startMs": 281199,
    "text": "them. And it also doesn't work for an"
  },
  {
    "startMs": 283199,
    "text": "agent either. Maybe it works for about a"
  },
  {
    "startMs": 285440,
    "text": "week, maybe two, but then you eventually"
  },
  {
    "startMs": 287840,
    "text": "end up in a situation where the agent is"
  },
  {
    "startMs": 289360,
    "text": "genuinely trying to delete something"
  },
  {
    "startMs": 290639,
    "text": "that it feels like is just not in the"
  },
  {
    "startMs": 292560,
    "text": "infrastructure and you will just be"
  },
  {
    "startMs": 295280,
    "text": "there sitting and pressing enter by hand"
  },
  {
    "startMs": 297680,
    "text": "all over again. The the core concept"
  },
  {
    "startMs": 300479,
    "text": "with a token that I feel like is wrong"
  },
  {
    "startMs": 302080,
    "text": "is that a token is a boolean. It's just"
  },
  {
    "startMs": 303919,
    "text": "a yes or no. It's a static list of"
  },
  {
    "startMs": 305600,
    "text": "scopes. Um you either have it or you"
  },
  {
    "startMs": 308320,
    "text": "don't have it. So if the token list is"
  },
  {
    "startMs": 310639,
    "text": "too tight, then your agent is"
  },
  {
    "startMs": 312400,
    "text": "effectively useless. If the token list"
  },
  {
    "startMs": 314400,
    "text": "is too wide, then you're maybe writing"
  },
  {
    "startMs": 316720,
    "text": "writing like a postmortm. A budget is a"
  },
  {
    "startMs": 319520,
    "text": "very different shape. A budget is is um"
  },
  {
    "startMs": 322160,
    "text": "not just like one number. It just has"
  },
  {
    "startMs": 323840,
    "text": "like four different four different"
  },
  {
    "startMs": 325039,
    "text": "dimensions, right? How much can the"
  },
  {
    "startMs": 327120,
    "text": "agent do? Uh how fast can it do it? What"
  },
  {
    "startMs": 330080,
    "text": "can it undo on its own? And then who's"
  },
  {
    "startMs": 332080,
    "text": "noticing while it's actually taking"
  },
  {
    "startMs": 333840,
    "text": "those actions? And that's roughly the"
  },
  {
    "startMs": 336000,
    "text": "umbrella for like my four primitives. Uh"
  },
  {
    "startMs": 337840,
    "text": "is that each of the things that follow,"
  },
  {
    "startMs": 339360,
    "text": "they're basically being um one of the"
  },
  {
    "startMs": 341840,
    "text": "things along these dimensions. and"
  },
  {
    "startMs": 343440,
    "text": "they're going to replace a yes or no"
  },
  {
    "startMs": 344960,
    "text": "question with a budget."
  },
  {
    "startMs": 347680,
    "text": "So, I'm going to start off with this"
  },
  {
    "startMs": 348800,
    "text": "first verb which I call asymmetric"
  },
  {
    "startMs": 350800,
    "text": "verbs. Uh, and by verbs I simply means"
  },
  {
    "startMs": 353520,
    "text": "operations or actions that an agent can"
  },
  {
    "startMs": 355199,
    "text": "take. So, like it could be API calls, it"
  },
  {
    "startMs": 356800,
    "text": "could be uh CLI commands, it could be"
  },
  {
    "startMs": 358800,
    "text": "really anything. And they're asymmetric"
  },
  {
    "startMs": 361120,
    "text": "because the same sized action even"
  },
  {
    "startMs": 363680,
    "text": "though it looks same size, they're they"
  },
  {
    "startMs": 365360,
    "text": "can have very different blast radius in"
  },
  {
    "startMs": 367199,
    "text": "actuality depending on which direction"
  },
  {
    "startMs": 368800,
    "text": "it goes. So the the core point is that"
  },
  {
    "startMs": 371360,
    "text": "you need to stop thinking about"
  },
  {
    "startMs": 372400,
    "text": "resources for a second and think about"
  },
  {
    "startMs": 374000,
    "text": "verbs. So specifically think about what"
  },
  {
    "startMs": 377120,
    "text": "happens when one of the verbs goes"
  },
  {
    "startMs": 379360,
    "text": "wrong. So some verbs they fail out loud."
  },
  {
    "startMs": 382000,
    "text": "So in this particular example, let's say"
  },
  {
    "startMs": 383680,
    "text": "if an agent decides to unskip a test and"
  },
  {
    "startMs": 386560,
    "text": "say it's the wrong call, the worst that"
  },
  {
    "startMs": 388560,
    "text": "would happen is CI would go red uh for a"
  },
  {
    "startMs": 390880,
    "text": "bunch of people. Um same with paging."
  },
  {
    "startMs": 392960,
    "text": "Like if an agent decides to page a human"
  },
  {
    "startMs": 394960,
    "text": "and if it's the wrong call, the worst"
  },
  {
    "startMs": 396639,
    "text": "that's happening is that it's a nuisance"
  },
  {
    "startMs": 398080,
    "text": "for the on call. But there's always a"
  },
  {
    "startMs": 399440,
    "text": "human to correct it. There are other"
  },
  {
    "startMs": 402080,
    "text": "verbs that fail silently, right? So if"
  },
  {
    "startMs": 403759,
    "text": "the agent decides to skip a test uh due"
  },
  {
    "startMs": 406479,
    "text": "to whatever reason, it shouldn't have."
  },
  {
    "startMs": 408639,
    "text": "Nothing technically turns red. A real"
  },
  {
    "startMs": 410880,
    "text": "bug can actually walk into production"
  },
  {
    "startMs": 412720,
    "text": "with green checks and nobody would"
  },
  {
    "startMs": 414160,
    "text": "notice it until much later. So unskip"
  },
  {
    "startMs": 416560,
    "text": "and skip in this example, they're"
  },
  {
    "startMs": 418160,
    "text": "effectively the they're the same kind of"
  },
  {
    "startMs": 420240,
    "text": "action, but the difference is which of"
  },
  {
    "startMs": 422000,
    "text": "the failures would show up on a"
  },
  {
    "startMs": 423039,
    "text": "dashboard and which one wouldn't. And so"
  },
  {
    "startMs": 424800,
    "text": "the core idea is that you give um access"
  },
  {
    "startMs": 427520,
    "text": "to verbs that can fail out loud loudly"
  },
  {
    "startMs": 429759,
    "text": "on a dashboard to your agent and for the"
  },
  {
    "startMs": 431360,
    "text": "other ones just involve a human."
  },
  {
    "startMs": 434319,
    "text": "Uh just a bit of context on CI stuff and"
  },
  {
    "startMs": 438000,
    "text": "how this plays out for us is that we"
  },
  {
    "startMs": 439360,
    "text": "have a test quarantining service behind"
  },
  {
    "startMs": 440800,
    "text": "the scenes and it basically holds a list"
  },
  {
    "startMs": 442800,
    "text": "of um all the tests that are currently"
  },
  {
    "startMs": 444960,
    "text": "skipped because an on call decided that"
  },
  {
    "startMs": 446400,
    "text": "it had to uh they had to like break"
  },
  {
    "startMs": 448400,
    "text": "glass a certain situation whenever there"
  },
  {
    "startMs": 449759,
    "text": "was like an incident or something. Now"
  },
  {
    "startMs": 451680,
    "text": "the agent has the ability to reenable"
  },
  {
    "startMs": 454560,
    "text": "any one of these own depending on like"
  },
  {
    "startMs": 455919,
    "text": "when it evaluates when the tests are uh"
  },
  {
    "startMs": 458000,
    "text": "working fine and stuff. Again if it's"
  },
  {
    "startMs": 460560,
    "text": "the wrong call the worst that would"
  },
  {
    "startMs": 461680,
    "text": "happen is a bunch of different builds"
  },
  {
    "startMs": 462960,
    "text": "and tests would just like start showing"
  },
  {
    "startMs": 464319,
    "text": "up red and then a human can actually put"
  },
  {
    "startMs": 465919,
    "text": "them back very cheaply. The skip is a"
  },
  {
    "startMs": 468639,
    "text": "break glass verb itself right like it's"
  },
  {
    "startMs": 470319,
    "text": "basically as I mentioned what our on"
  },
  {
    "startMs": 471599,
    "text": "call would reach for uh during a very"
  },
  {
    "startMs": 473440,
    "text": "critical situation during under pressure"
  },
  {
    "startMs": 474879,
    "text": "and stuff. And so in that scenario if"
  },
  {
    "startMs": 477599,
    "text": "like uh an agent actually gets it wrong"
  },
  {
    "startMs": 480080,
    "text": "a real production bug can actually a"
  },
  {
    "startMs": 481840,
    "text": "real bug can walk into production. So"
  },
  {
    "startMs": 483280,
    "text": "this needs a human and it will always"
  },
  {
    "startMs": 485199,
    "text": "leave an audit trail. Uh and the key"
  },
  {
    "startMs": 487280,
    "text": "detail is that the agent itself is not"
  },
  {
    "startMs": 489120,
    "text": "responsible for writing the row or the"
  },
  {
    "startMs": 490560,
    "text": "audit trail itself. There is a proxy in"
  },
  {
    "startMs": 492319,
    "text": "the middle that I've highlighted that"
  },
  {
    "startMs": 493280,
    "text": "I'm going to talk about in a little bit."
  },
  {
    "startMs": 494800,
    "text": "Uh which is responsible for stamping the"
  },
  {
    "startMs": 496800,
    "text": "caller's main identity on every call"
  },
  {
    "startMs": 498639,
    "text": "whether it's a skip or an unskip. um the"
  },
  {
    "startMs": 501599,
    "text": "agent technically never holds uh the pen"
  },
  {
    "startMs": 504080,
    "text": "on its own provenence."
  },
  {
    "startMs": 506960,
    "text": "The second primitive is about rate"
  },
  {
    "startMs": 508319,
    "text": "limits. Uh it's pretty standard concept"
  },
  {
    "startMs": 510240,
    "text": "but and this is the most concrete form"
  },
  {
    "startMs": 512320,
    "text": "of like the budget idea as a whole. So a"
  },
  {
    "startMs": 514479,
    "text": "ceiling that refills. So every caller"
  },
  {
    "startMs": 516719,
    "text": "gets a small amount of disruptive"
  },
  {
    "startMs": 518640,
    "text": "actions uh per time window. So you can"
  },
  {
    "startMs": 521039,
    "text": "spend them however you want. Uh there's"
  },
  {
    "startMs": 522959,
    "text": "no approval, there's no waiting. And if"
  },
  {
    "startMs": 524720,
    "text": "you cross the line, the request simply"
  },
  {
    "startMs": 526160,
    "text": "bounces back with a count saying that"
  },
  {
    "startMs": 527839,
    "text": "you're actually exceeding your count um"
  },
  {
    "startMs": 529279,
    "text": "your budget. You wait a bit and then the"
  },
  {
    "startMs": 531519,
    "text": "limit essentially refills. And that's"
  },
  {
    "startMs": 533279,
    "text": "that's the whole thing. So the agent"
  },
  {
    "startMs": 535120,
    "text": "gets full autonomy within the limit. And"
  },
  {
    "startMs": 537600,
    "text": "there is a hard ceiling on how bad a"
  },
  {
    "startMs": 539200,
    "text": "single loop can get. And every write"
  },
  {
    "startMs": 541839,
    "text": "effectively gets a rate limit. There are"
  },
  {
    "startMs": 543680,
    "text": "no exceptions to that. What changes is"
  },
  {
    "startMs": 545680,
    "text": "the size of the rate limit. So if if if"
  },
  {
    "startMs": 547360,
    "text": "I'm if I'm trying to delete a bunch of"
  },
  {
    "startMs": 548720,
    "text": "workloads in my own name space like my"
  },
  {
    "startMs": 550480,
    "text": "my rate limit might be higher but if I'm"
  },
  {
    "startMs": 552480,
    "text": "trying to touch resources in a shared"
  },
  {
    "startMs": 553680,
    "text": "namespace my budget or my rate limits"
  },
  {
    "startMs": 555680,
    "text": "might be smaller."
  },
  {
    "startMs": 559440,
    "text": "So this is uh effectively the"
  },
  {
    "startMs": 561600,
    "text": "solutioning for the incident that I was"
  },
  {
    "startMs": 563279,
    "text": "showing earlier. Um after the incident"
  },
  {
    "startMs": 564880,
    "text": "the team sort of that sits next to mine"
  },
  {
    "startMs": 566560,
    "text": "they they built an admission web hook of"
  },
  {
    "startMs": 568320,
    "text": "sorts uh whose sole job is to cap the"
  },
  {
    "startMs": 571360,
    "text": "number of deletes at a fixed number per"
  },
  {
    "startMs": 573120,
    "text": "hour per resource kind per name space."
  },
  {
    "startMs": 575600,
    "text": "There is always a bypass flag because"
  },
  {
    "startMs": 577200,
    "text": "sometimes you genuinely want to delete"
  },
  {
    "startMs": 579200,
    "text": "more than you're allowed for and maybe"
  },
  {
    "startMs": 580720,
    "text": "some on call scenario or whatever. And"
  },
  {
    "startMs": 582560,
    "text": "the part that I absolutely love in this"
  },
  {
    "startMs": 584640,
    "text": "case is that inside a cloud code session"
  },
  {
    "startMs": 586480,
    "text": "or inside an agent session effectively"
  },
  {
    "startMs": 588160,
    "text": "the the bypass flag simply refuses to do"
  },
  {
    "startMs": 590399,
    "text": "anything. All it's going to do is tell"
  },
  {
    "startMs": 592080,
    "text": "the agent to ask the human to run the"
  },
  {
    "startMs": 593839,
    "text": "command itself. So the agent effectively"
  },
  {
    "startMs": 596720,
    "text": "gets the rate limit and the human keeps"
  },
  {
    "startMs": 598480,
    "text": "the override and nobody effectively has"
  },
  {
    "startMs": 600320,
    "text": "to file a ticket for uh the limit"
  },
  {
    "startMs": 602080,
    "text": "because it just like refills."
  },
  {
    "startMs": 605680,
    "text": "The third primitive is about this"
  },
  {
    "startMs": 607839,
    "text": "concept of trip wires over allow lists."
  },
  {
    "startMs": 610160,
    "text": "So in my mind like an allow list is"
  },
  {
    "startMs": 612080,
    "text": "effectively a guess that you're making"
  },
  {
    "startMs": 613600,
    "text": "up front about what the agent needs or"
  },
  {
    "startMs": 616160,
    "text": "about model behavior or agent behavior"
  },
  {
    "startMs": 617760,
    "text": "itself. uh I feel like it's pretty"
  },
  {
    "startMs": 619279,
    "text": "static and you write it up front before"
  },
  {
    "startMs": 621519,
    "text": "before you have any data on like how the"
  },
  {
    "startMs": 622959,
    "text": "agent is behaving in different"
  },
  {
    "startMs": 624000,
    "text": "situations. A trip wire on the other"
  },
  {
    "startMs": 626079,
    "text": "hand is how you get that data like after"
  },
  {
    "startMs": 628399,
    "text": "the fact. So for cheap actions you let"
  },
  {
    "startMs": 630640,
    "text": "the agent act and every action gets"
  },
  {
    "startMs": 632160,
    "text": "recorded with the actor sort of stamp"
  },
  {
    "startMs": 634480,
    "text": "identity stamp and these two kind of go"
  },
  {
    "startMs": 636640,
    "text": "well together like rate limits are the"
  },
  {
    "startMs": 638480,
    "text": "enforcement um they they put a hard"
  },
  {
    "startMs": 640880,
    "text": "limit on uh the rate itself. Trip wires"
  },
  {
    "startMs": 643360,
    "text": "are how you find out what actually"
  },
  {
    "startMs": 644880,
    "text": "happened so you've got something to"
  },
  {
    "startMs": 646560,
    "text": "react to. So you effectively watch the"
  },
  {
    "startMs": 649120,
    "text": "aggregate and not like individual calls."
  },
  {
    "startMs": 651360,
    "text": "And usually when a trip wire goes off"
  },
  {
    "startMs": 652959,
    "text": "the fix is like maybe one or two lines"
  },
  {
    "startMs": 654320,
    "text": "in the agents context and not really a"
  },
  {
    "startMs": 656399,
    "text": "big code change. Uh the core point is"
  },
  {
    "startMs": 658320,
    "text": "that allow lists don't really get better"
  },
  {
    "startMs": 659920,
    "text": "over time. They can get stale but trip"
  },
  {
    "startMs": 661360,
    "text": "wires do get better over time."
  },
  {
    "startMs": 664480,
    "text": "So here's like a loopin practice for us."
  },
  {
    "startMs": 666800,
    "text": "So like we track one number uh which is"
  },
  {
    "startMs": 668720,
    "text": "the number of investigation threads that"
  },
  {
    "startMs": 670079,
    "text": "our agent is launching per hour for a"
  },
  {
    "startMs": 672240,
    "text": "given test job failure. So one morning"
  },
  {
    "startMs": 674240,
    "text": "say the number was way above the"
  },
  {
    "startMs": 675519,
    "text": "baseline and the trip fire page is on"
  },
  {
    "startMs": 677680,
    "text": "call and that part's important because a"
  },
  {
    "startMs": 679760,
    "text": "trip wire that nobody sees is"
  },
  {
    "startMs": 681040,
    "text": "practically useless. Uh it pages after"
  },
  {
    "startMs": 683360,
    "text": "the right has already happened after the"
  },
  {
    "startMs": 685040,
    "text": "the limit has been crossed not before."
  },
  {
    "startMs": 687040,
    "text": "It's effectively the smoke detector not"
  },
  {
    "startMs": 689040,
    "text": "the lock on the door. Uh the agent has"
  },
  {
    "startMs": 691440,
    "text": "spun up in this particular scenario a"
  },
  {
    "startMs": 693200,
    "text": "bunch of like investigation threads for"
  },
  {
    "startMs": 694959,
    "text": "dozens of job that were all failing with"
  },
  {
    "startMs": 696880,
    "text": "the same kind of error signature. Each"
  },
  {
    "startMs": 699120,
    "text": "thread effectively looked reasonable on"
  },
  {
    "startMs": 700720,
    "text": "its own, but like if you took them in"
  },
  {
    "startMs": 702079,
    "text": "aggregate, you would realize that it was"
  },
  {
    "startMs": 703440,
    "text": "actually an infrastructure failure that"
  },
  {
    "startMs": 704880,
    "text": "was causing the same test failure"
  },
  {
    "startMs": 706880,
    "text": "signatures across the board. So the fix"
  },
  {
    "startMs": 708880,
    "text": "in this case was as simply as telling"
  },
  {
    "startMs": 711040,
    "text": "the agent on like, hey, the next time"
  },
  {
    "startMs": 712320,
    "text": "you encounter something like this, maybe"
  },
  {
    "startMs": 713680,
    "text": "try to correlate a bunch of different"
  },
  {
    "startMs": 715680,
    "text": "failures and test jobs and stuff before"
  },
  {
    "startMs": 717920,
    "text": "launching a separate investigation"
  },
  {
    "startMs": 719200,
    "text": "thread. So that correlation is"
  },
  {
    "startMs": 720399,
    "text": "important. Um and the next time when"
  },
  {
    "startMs": 722240,
    "text": "this happened, uh it did exactly that"
  },
  {
    "startMs": 724079,
    "text": "because that that example and that line"
  },
  {
    "startMs": 725760,
    "text": "of like how it should react and like how"
  },
  {
    "startMs": 727279,
    "text": "it should like uh debug things was was"
  },
  {
    "startMs": 729600,
    "text": "right there to guide it."
  },
  {
    "startMs": 732880,
    "text": "So that's three rough primitives. Now"
  },
  {
    "startMs": 734959,
    "text": "the lens I mentioned at the start is"
  },
  {
    "startMs": 736480,
    "text": "this idea of undo test. And this one's"
  },
  {
    "startMs": 738480,
    "text": "slightly different. It's it's not"
  },
  {
    "startMs": 740000,
    "text": "something you effectively enforce in"
  },
  {
    "startMs": 741360,
    "text": "code. It's the question you ask when"
  },
  {
    "startMs": 742959,
    "text": "you're sizing the other three. So it's"
  },
  {
    "startMs": 745279,
    "text": "two questions really. One, can the agent"
  },
  {
    "startMs": 747120,
    "text": "put it back by itself? and how bad would"
  },
  {
    "startMs": 749839,
    "text": "the impact be if it actually got it"
  },
  {
    "startMs": 751680,
    "text": "wrong. And this this this sounds like"
  },
  {
    "startMs": 754000,
    "text": "asymmetric verbs in a sense, but it's"
  },
  {
    "startMs": 755760,
    "text": "kind of difference because the verbs ask"
  },
  {
    "startMs": 758160,
    "text": "whether you would notice the failure and"
  },
  {
    "startMs": 760399,
    "text": "undo asks whether you can recover from"
  },
  {
    "startMs": 762320,
    "text": "it. So if if if the agent can"
  },
  {
    "startMs": 765760,
    "text": "effectively roll back its own change and"
  },
  {
    "startMs": 767920,
    "text": "the blast radius is acceptable to you,"
  },
  {
    "startMs": 769600,
    "text": "you effectively log it and you let it"
  },
  {
    "startMs": 770959,
    "text": "go. If either of the answers is no, then"
  },
  {
    "startMs": 773680,
    "text": "you effectively need a second key and"
  },
  {
    "startMs": 775760,
    "text": "the second key is not something that the"
  },
  {
    "startMs": 777680,
    "text": "agent holds itself. It has to be someone"
  },
  {
    "startMs": 779440,
    "text": "else. Um, and there has to be an audit"
  },
  {
    "startMs": 781760,
    "text": "record so that you can you can track on"
  },
  {
    "startMs": 783200,
    "text": "like what happened, why the second key"
  },
  {
    "startMs": 784480,
    "text": "was involved and stuff like that."
  },
  {
    "startMs": 787760,
    "text": "Let me let me just show you this another"
  },
  {
    "startMs": 789279,
    "text": "example of like how this works outside"
  },
  {
    "startMs": 790639,
    "text": "of like the CI domain specifically. So"
  },
  {
    "startMs": 792639,
    "text": "our agent has a key for one of our"
  },
  {
    "startMs": 794720,
    "text": "feature flag services. So on the canary"
  },
  {
    "startMs": 796959,
    "text": "side which is basically our staging"
  },
  {
    "startMs": 798160,
    "text": "traffic and a bunch of like dog fooding"
  },
  {
    "startMs": 799600,
    "text": "customers the agent effectively has the"
  },
  {
    "startMs": 801200,
    "text": "full dial. It has the ability to um ramp"
  },
  {
    "startMs": 804160,
    "text": "up a specific feature flag roll it out"
  },
  {
    "startMs": 805920,
    "text": "to uh all of the canary traffic and it"
  },
  {
    "startMs": 808320,
    "text": "can take it all the way from zero to 100"
  },
  {
    "startMs": 810160,
    "text": "and it also has the ability to sort of"
  },
  {
    "startMs": 812000,
    "text": "uh look at any bugs that are being filed"
  },
  {
    "startMs": 813600,
    "text": "and toggle it back off and on. What the"
  },
  {
    "startMs": 815839,
    "text": "agents key is not scoped to do is for"
  },
  {
    "startMs": 817440,
    "text": "promoting the flag to real production."
  },
  {
    "startMs": 819519,
    "text": "The best that an agent can do for now in"
  },
  {
    "startMs": 821200,
    "text": "a lot of cases is that it can propose uh"
  },
  {
    "startMs": 823600,
    "text": "that an that someone actually promote"
  },
  {
    "startMs": 825680,
    "text": "the feature flag in production because"
  },
  {
    "startMs": 826800,
    "text": "it's been tested out in Canary. But"
  },
  {
    "startMs": 828480,
    "text": "that's pretty much it. Um the second key"
  },
  {
    "startMs": 831279,
    "text": "in this scenario is not necessarily a"
  },
  {
    "startMs": 833040,
    "text": "new O system. It's a scoped key for"
  },
  {
    "startMs": 835519,
    "text": "production and a scoped key for Canary."
  },
  {
    "startMs": 839839,
    "text": "So this is what it looks like day-to-day"
  },
  {
    "startMs": 841760,
    "text": "for for me personally. is like I I ask"
  },
  {
    "startMs": 843760,
    "text": "cla tag in our slack channel to to own"
  },
  {
    "startMs": 846079,
    "text": "the loop the entire loop uh on on"
  },
  {
    "startMs": 848320,
    "text": "rolling out a flag. It can it can"
  },
  {
    "startMs": 849839,
    "text": "basically ask me a clarifying question"
  },
  {
    "startMs": 851120,
    "text": "on whether this is for internal dog"
  },
  {
    "startMs": 852480,
    "text": "fooding group or early access or"
  },
  {
    "startMs": 853760,
    "text": "whatever. Uh and the important part is"
  },
  {
    "startMs": 856000,
    "text": "that I'm not in the middle of any of"
  },
  {
    "startMs": 857360,
    "text": "these things. Every action that the"
  },
  {
    "startMs": 859199,
    "text": "agent is taking is stamped with its own"
  },
  {
    "startMs": 861199,
    "text": "identity and not mine. And that stamp is"
  },
  {
    "startMs": 863199,
    "text": "basically what ties the whole thing the"
  },
  {
    "startMs": 864720,
    "text": "whole thing together."
  },
  {
    "startMs": 867120,
    "text": "So the these these primitives they"
  },
  {
    "startMs": 869519,
    "text": "compose together nicely because each one"
  },
  {
    "startMs": 871279,
    "text": "asks us a different question right the"
  },
  {
    "startMs": 872880,
    "text": "verbs ask what the agent is touching the"
  },
  {
    "startMs": 875360,
    "text": "rate limit asks how often and the trip"
  },
  {
    "startMs": 877920,
    "text": "wires catch what happened afterwards and"
  },
  {
    "startMs": 879600,
    "text": "the undo test is sort of the length that"
  },
  {
    "startMs": 880959,
    "text": "I personally used to to size all these"
  },
  {
    "startMs": 882720,
    "text": "three now the original sort of cold open"
  },
  {
    "startMs": 885279,
    "text": "delete that I was showing you the the"
  },
  {
    "startMs": 886480,
    "text": "incident itself the rate limit itself"
  },
  {
    "startMs": 888480,
    "text": "would have capped it at a few couple of"
  },
  {
    "startMs": 890560,
    "text": "tens of workloads itself the undo test"
  },
  {
    "startMs": 893120,
    "text": "if you would be thinking about it it"
  },
  {
    "startMs": 894399,
    "text": "will basically tell you that you can't"
  },
  {
    "startMs": 895519,
    "text": "unddelete a running job in someone"
  },
  {
    "startMs": 896800,
    "text": "else's namespace. Um so so anything past"
  },
  {
    "startMs": 899360,
    "text": "the cap basically need a human with the"
  },
  {
    "startMs": 901360,
    "text": "with a with a second key. So the the"
  },
  {
    "startMs": 904399,
    "text": "important part is that you don't need"
  },
  {
    "startMs": 905600,
    "text": "all the checks in every right scenario."
  },
  {
    "startMs": 907440,
    "text": "Only some of these might be relevant for"
  },
  {
    "startMs": 908880,
    "text": "like whatever kind of action you're"
  },
  {
    "startMs": 910160,
    "text": "trying to evaluate. Uh and if any of"
  },
  {
    "startMs": 912639,
    "text": "these really sound familiar, it it"
  },
  {
    "startMs": 914480,
    "text": "should because this is effectively some"
  },
  {
    "startMs": 916480,
    "text": "sort of onboarding checklist for your"
  },
  {
    "startMs": 917760,
    "text": "engineers like what can a new engineer"
  },
  {
    "startMs": 919760,
    "text": "touch, how much rope do they get, who"
  },
  {
    "startMs": 921360,
    "text": "signs off on their operations, and how"
  },
  {
    "startMs": 923199,
    "text": "do we know it's effectively working? We"
  },
  {
    "startMs": 925279,
    "text": "just wrote it for people. It's now the"
  },
  {
    "startMs": 927199,
    "text": "same checklist that we want for agents."
  },
  {
    "startMs": 930959,
    "text": "So where does all the policy that I've"
  },
  {
    "startMs": 932639,
    "text": "been talking about really live? Uh it's"
  },
  {
    "startMs": 934079,
    "text": "two places and I feel like you need"
  },
  {
    "startMs": 935440,
    "text": "both. The first one is text. Uh that's"
  },
  {
    "startMs": 938160,
    "text": "prompts. That's your context files"
  },
  {
    "startMs": 940240,
    "text": "markdown the agent reads before it can"
  },
  {
    "startMs": 942079,
    "text": "act. And this is where you can explain"
  },
  {
    "startMs": 943519,
    "text": "your why the the intent. We've we got"
  },
  {
    "startMs": 945920,
    "text": "exactly the same sentence that I was"
  },
  {
    "startMs": 947120,
    "text": "showing earlier um written down in a"
  },
  {
    "startMs": 948880,
    "text": "markdown file. It works about 80% of the"
  },
  {
    "startMs": 951600,
    "text": "time. Uh the the upside is that it's"
  },
  {
    "startMs": 953839,
    "text": "very cheap to change and you can explain"
  },
  {
    "startMs": 955360,
    "text": "the reasoning. The downside is you have"
  },
  {
    "startMs": 957440,
    "text": "to garden it because the the files can"
  },
  {
    "startMs": 959120,
    "text": "grow over time and at the end of the day"
  },
  {
    "startMs": 960959,
    "text": "it's just advice. Uh text can shape the"
  },
  {
    "startMs": 963680,
    "text": "intent but there is no enforcement"
  },
  {
    "startMs": 965040,
    "text": "anywhere. The the second place is"
  },
  {
    "startMs": 967519,
    "text": "infrastructure uh which is the proxy"
  },
  {
    "startMs": 969759,
    "text": "layer for us. So the proxy is is not"
  },
  {
    "startMs": 971920,
    "text": "reading the prompt. It doesn't know why"
  },
  {
    "startMs": 973360,
    "text": "the agent wants to do something and it"
  },
  {
    "startMs": 975519,
    "text": "doesn't really care. It will see a"
  },
  {
    "startMs": 977279,
    "text": "delete happening. It'll see like a"
  },
  {
    "startMs": 978720,
    "text": "budget being crossed and it'll simply"
  },
  {
    "startMs": 980240,
    "text": "return a 403 and that's the whole"
  },
  {
    "startMs": 981759,
    "text": "conversation really. It's narrow. It's"
  },
  {
    "startMs": 983680,
    "text": "deterministic. It it counts, compares,"
  },
  {
    "startMs": 986480,
    "text": "it can allow a delete or deny. What it"
  },
  {
    "startMs": 988959,
    "text": "can't do is explain the why and a clever"
  },
  {
    "startMs": 991199,
    "text": "like prompt injection cannot really talk"
  },
  {
    "startMs": 992880,
    "text": "it out of the rule itself. So you need"
  },
  {
    "startMs": 995040,
    "text": "both. Uh the text shapes what an agent"
  },
  {
    "startMs": 997839,
    "text": "is trying to do and infra is bounding"
  },
  {
    "startMs": 999600,
    "text": "how wrong can it go."
  },
  {
    "startMs": 1002959,
    "text": "So zooming out, this is where uh the the"
  },
  {
    "startMs": 1005600,
    "text": "infrastructure sort of really lives for"
  },
  {
    "startMs": 1007040,
    "text": "us. So like every agent session has its"
  },
  {
    "startMs": 1008720,
    "text": "own proxy running right next to it. The"
  },
  {
    "startMs": 1011040,
    "text": "agent starts off by reading its own"
  },
  {
    "startMs": 1012639,
    "text": "context file. The the markdown files"
  },
  {
    "startMs": 1014000,
    "text": "that I was showing earlier that's that's"
  },
  {
    "startMs": 1015279,
    "text": "the text layer and every outbound call"
  },
  {
    "startMs": 1017519,
    "text": "goes through the proxy after that point"
  },
  {
    "startMs": 1019199,
    "text": "which is the infrastructure layer. So"
  },
  {
    "startMs": 1020959,
    "text": "the proxy is not like whatever the agent"
  },
  {
    "startMs": 1023519,
    "text": "is trying to do the the proxy is only"
  },
  {
    "startMs": 1025039,
    "text": "responsible for stamping that action"
  },
  {
    "startMs": 1026559,
    "text": "with the agent's identity. Uh like if"
  },
  {
    "startMs": 1028880,
    "text": "the if the if the agent is trying to"
  },
  {
    "startMs": 1030319,
    "text": "launch a bunch of jobs in a Kubernetes"
  },
  {
    "startMs": 1031760,
    "text": "cluster the the proxy is not following"
  },
  {
    "startMs": 1033280,
    "text": "it. what is following is the stamp"
  },
  {
    "startMs": 1035600,
    "text": "itself, the identity itself. So the in"
  },
  {
    "startMs": 1038559,
    "text": "this example, the cluster would write"
  },
  {
    "startMs": 1040000,
    "text": "the stamp onto the job as a label. Um"
  },
  {
    "startMs": 1042480,
    "text": "and every child job or anything that's"
  },
  {
    "startMs": 1044240,
    "text": "happening afterwards simply inherits the"
  },
  {
    "startMs": 1045760,
    "text": "same identity and the every safeguard"
  },
  {
    "startMs": 1049200,
    "text": "that is there in the rest of our systems"
  },
  {
    "startMs": 1050720,
    "text": "that that are simply they're simply"
  },
  {
    "startMs": 1052160,
    "text": "reading that like one label. Uh whether"
  },
  {
    "startMs": 1054160,
    "text": "it's ownership, whether it's kotaas,"
  },
  {
    "startMs": 1055679,
    "text": "rate limits, approvals, trip wires,"
  },
  {
    "startMs": 1057919,
    "text": "whatever it is, they're all keyed on the"
  },
  {
    "startMs": 1059520,
    "text": "same stamp and the agent never got to"
  },
  {
    "startMs": 1061120,
    "text": "touch it."
  },
  {
    "startMs": 1063679,
    "text": "So that's roughly the shape of it. But"
  },
  {
    "startMs": 1065679,
    "text": "before I wrap up, I want to spend like"
  },
  {
    "startMs": 1067120,
    "text": "30 seconds on why it has to be a proxy"
  },
  {
    "startMs": 1069440,
    "text": "layer, like a different layer that is"
  },
  {
    "startMs": 1070799,
    "text": "doing the stamping on this identity and"
  },
  {
    "startMs": 1072160,
    "text": "not the caller itself. Because without"
  },
  {
    "startMs": 1074240,
    "text": "it, the agent can roughly pick a"
  },
  {
    "startMs": 1075679,
    "text": "different name. Um, and every limit that"
  },
  {
    "startMs": 1077679,
    "text": "you've basically set resets."
  },
  {
    "startMs": 1081120,
    "text": "So say the agent has the ability to set"
  },
  {
    "startMs": 1084000,
    "text": "its own identity in a header, right? So"
  },
  {
    "startMs": 1086240,
    "text": "let's say it hits a specific limit and"
  },
  {
    "startMs": 1088240,
    "text": "what's the easiest fix from an agent's"
  },
  {
    "startMs": 1090400,
    "text": "point of view that it can do? it'll just"
  },
  {
    "startMs": 1092080,
    "text": "change the header. In this case, it'll"
  },
  {
    "startMs": 1093679,
    "text": "just say instead of such in such in two"
  },
  {
    "startMs": 1095919,
    "text": "and voila, you just have a fresh budget"
  },
  {
    "startMs": 1098000,
    "text": "to work with. Now, in this case, you"
  },
  {
    "startMs": 1100559,
    "text": "technically don't have a rate limit. You"
  },
  {
    "startMs": 1101919,
    "text": "just have a suggestion. Uh with the"
  },
  {
    "startMs": 1103840,
    "text": "proxy in the path, the agent never gets"
  },
  {
    "startMs": 1105520,
    "text": "to say who it is. The proxy already"
  },
  {
    "startMs": 1107200,
    "text": "knows. It's the thing that's holding"
  },
  {
    "startMs": 1108960,
    "text": "real credentials and it stamps every"
  },
  {
    "startMs": 1111039,
    "text": "call with the identity that it already"
  },
  {
    "startMs": 1112640,
    "text": "knows, not the one that agent claims."
  },
  {
    "startMs": 1115120,
    "text": "And because proxy is the one that is"
  },
  {
    "startMs": 1116960,
    "text": "stamping, you also get this per session"
  },
  {
    "startMs": 1119440,
    "text": "ID so that you're able to differentiate"
  },
  {
    "startMs": 1120960,
    "text": "different sessions that are running all"
  },
  {
    "startMs": 1122400,
    "text": "for you and see which one's overreacting"
  },
  {
    "startMs": 1124640,
    "text": "or which one's not acting as it's"
  },
  {
    "startMs": 1126000,
    "text": "supposed to be."
  },
  {
    "startMs": 1129039,
    "text": "U so again just stuffs that I want you"
  },
  {
    "startMs": 1131679,
    "text": "to take home with is give the agent"
  },
  {
    "startMs": 1134320,
    "text": "access to verbs that would fail out loud"
  },
  {
    "startMs": 1136400,
    "text": "uh and keep the human on the ones that"
  },
  {
    "startMs": 1138240,
    "text": "can fail out quietly. Two, you you put a"
  },
  {
    "startMs": 1140799,
    "text": "ceiling on every right and that has the"
  },
  {
    "startMs": 1142559,
    "text": "ability to refill on its own so nobody's"
  },
  {
    "startMs": 1144240,
    "text": "effectively filing tickets. Three, you"
  },
  {
    "startMs": 1146960,
    "text": "watch the aggregate and not the"
  },
  {
    "startMs": 1148320,
    "text": "individual calls that help you"
  },
  {
    "startMs": 1149600,
    "text": "understand the agents behavior. Um, and"
  },
  {
    "startMs": 1152000,
    "text": "then you fix whatever you can with the"
  },
  {
    "startMs": 1153440,
    "text": "sentence. And four, you use the undo"
  },
  {
    "startMs": 1156080,
    "text": "test to sort of size all of these three"
  },
  {
    "startMs": 1158080,
    "text": "or any other u primitive that you have"
  },
  {
    "startMs": 1160320,
    "text": "for your own write operations. And"
  },
  {
    "startMs": 1162000,
    "text": "underneath all of this uh is is this"
  },
  {
    "startMs": 1164320,
    "text": "concept of identity which has to come"
  },
  {
    "startMs": 1165840,
    "text": "from the infrastructure, not from the"
  },
  {
    "startMs": 1167679,
    "text": "request. I feel like that's one rule. If"
  },
  {
    "startMs": 1169760,
    "text": "you get that one rule right, everything"
  },
  {
    "startMs": 1171200,
    "text": "else is uh just just tuning."
  },
  {
    "startMs": 1174000,
    "text": "Thank you."
  },
  {
    "startMs": 1175738,
    "text": "[applause]"
  },
  {
    "startMs": 1190148,
    "text": "[music]"
  }
]
