[
  {
    "startMs": 0,
    "text": "Picture a pull request. It changes a"
  },
  {
    "startMs": 3280,
    "text": "schema in one of our data pipelines. It"
  },
  {
    "startMs": 6000,
    "text": "goes green in CI. The tests pass. The"
  },
  {
    "startMs": 8960,
    "text": "linder is happy. And everything about it"
  },
  {
    "startMs": 11360,
    "text": "looks clean. So, is it safe to merge?"
  },
  {
    "startMs": 15280,
    "text": "Because a change like this can still"
  },
  {
    "startMs": 17199,
    "text": "break something downstream. Maybe it"
  },
  {
    "startMs": 19520,
    "text": "blows a freshness SLA. Or maybe it"
  },
  {
    "startMs": 22080,
    "text": "breaks a consumer three teams away who"
  },
  {
    "startMs": 24080,
    "text": "never saw it coming. Or maybe it exposes"
  },
  {
    "startMs": 26880,
    "text": "data that never supposed to be"
  },
  {
    "startMs": 28400,
    "text": "reachable. That gap between CI past and"
  },
  {
    "startMs": 32640,
    "text": "this change was actually safe is the gap"
  },
  {
    "startMs": 34960,
    "text": "my team has to close every day. Hello"
  },
  {
    "startMs": 38239,
    "text": "everyone, I'm Jas. I'm an engineer on"
  },
  {
    "startMs": 40800,
    "text": "the data platform team at Robin Hood. I"
  },
  {
    "startMs": 43200,
    "text": "design and build the distributed data"
  },
  {
    "startMs": 45360,
    "text": "injection and processing frameworks"
  },
  {
    "startMs": 46800,
    "text": "behind our data lake. The ones teams"
  },
  {
    "startMs": 49840,
    "text": "across the company rely on to make"
  },
  {
    "startMs": 51760,
    "text": "decisions with data."
  },
  {
    "startMs": 54000,
    "text": "Notice the title of this talk. The bots"
  },
  {
    "startMs": 56640,
    "text": "in it are the ones opening the pull"
  },
  {
    "startMs": 58559,
    "text": "requests and as you will see the ones"
  },
  {
    "startMs": 61840,
    "text": "reviewing them as well."
  },
  {
    "startMs": 64559,
    "text": "Today I want to show you how we answer"
  },
  {
    "startMs": 66479,
    "text": "that question when the PRs are coming in"
  },
  {
    "startMs": 68960,
    "text": "faster than any of us can read them."
  },
  {
    "startMs": 74000,
    "text": "So here's the problem in one sentence."
  },
  {
    "startMs": 76880,
    "text": "Pull requests to our production data"
  },
  {
    "startMs": 78960,
    "text": "pipelines now arrive at machine speed."
  },
  {
    "startMs": 81920,
    "text": "But reviewing them safely still runs at"
  },
  {
    "startMs": 84240,
    "text": "human speed."
  },
  {
    "startMs": 86320,
    "text": "First, a quick word on the scope of this"
  },
  {
    "startMs": 88479,
    "text": "talk because it shapes everything else"
  },
  {
    "startMs": 90640,
    "text": "in this talk."
  },
  {
    "startMs": 92799,
    "text": "AI coding assistants have made pull"
  },
  {
    "startMs": 94720,
    "text": "requests cheap to open. A new query, a"
  },
  {
    "startMs": 98240,
    "text": "schema change, or an access grant now"
  },
  {
    "startMs": 100799,
    "text": "takes just a few minutes. And an"
  },
  {
    "startMs": 103520,
    "text": "assistant can open pull requests far"
  },
  {
    "startMs": 105920,
    "text": "faster than any team of humans ever"
  },
  {
    "startMs": 108320,
    "text": "could."
  },
  {
    "startMs": 110479,
    "text": "The PRs I'm talking about today are the"
  },
  {
    "startMs": 112399,
    "text": "PRs to our data pipelines like schema"
  },
  {
    "startMs": 115119,
    "text": "changes, pipeline code, configuration"
  },
  {
    "startMs": 118320,
    "text": "changes, and grants that give someone"
  },
  {
    "startMs": 120640,
    "text": "access to data."
  },
  {
    "startMs": 123280,
    "text": "Data platform is the team that owns"
  },
  {
    "startMs": 125119,
    "text": "reviewing that kind of PR. So when those"
  },
  {
    "startMs": 127520,
    "text": "PRs pile up, they pile up on us. And the"
  },
  {
    "startMs": 130879,
    "text": "reviewer I'm about to describe is the"
  },
  {
    "startMs": 132800,
    "text": "system we built to handle them."
  },
  {
    "startMs": 135440,
    "text": "So the people who feel this are a small"
  },
  {
    "startMs": 137599,
    "text": "group. The engineers who carry the"
  },
  {
    "startMs": 139840,
    "text": "context to say what is safe to merge"
  },
  {
    "startMs": 142239,
    "text": "into production and everyone who is"
  },
  {
    "startMs": 144160,
    "text": "waiting on them."
  },
  {
    "startMs": 147760,
    "text": "So why is this a problem?"
  },
  {
    "startMs": 150800,
    "text": "AI open PRs just keep arriving."
  },
  {
    "startMs": 154560,
    "text": "And to be fair, the safety net we"
  },
  {
    "startMs": 156800,
    "text": "already had works. CI checks together"
  },
  {
    "startMs": 160239,
    "text": "with a human review have always caught a"
  },
  {
    "startMs": 162640,
    "text": "lot of real bugs like broken code,"
  },
  {
    "startMs": 165120,
    "text": "inefficient queries and risks like"
  },
  {
    "startMs": 168080,
    "text": "downstream breakage or sensitive data"
  },
  {
    "startMs": 170239,
    "text": "leaks. But the human half only works at"
  },
  {
    "startMs": 173120,
    "text": "the human speed and only with the"
  },
  {
    "startMs": 175680,
    "text": "context a person happens to carry in"
  },
  {
    "startMs": 177760,
    "text": "their head. Who reads this table? How"
  },
  {
    "startMs": 180879,
    "text": "big is it? And how sensitive is the data"
  },
  {
    "startMs": 183360,
    "text": "in it? The PRs are coming in at the"
  },
  {
    "startMs": 186319,
    "text": "machine speed, but the number of people"
  },
  {
    "startMs": 188319,
    "text": "who understand what is safe to merge"
  },
  {
    "startMs": 190080,
    "text": "into a production data pipeline did not"
  },
  {
    "startMs": 192319,
    "text": "grow at all. That's the mismatch. The"
  },
  {
    "startMs": 195920,
    "text": "queue builds up in front of the few"
  },
  {
    "startMs": 197760,
    "text": "people with the context and that's where"
  },
  {
    "startMs": 199840,
    "text": "things start to slip."
  },
  {
    "startMs": 202959,
    "text": "So the problem we set out to solve was"
  },
  {
    "startMs": 204800,
    "text": "this. Build a PR reviewer using AI that"
  },
  {
    "startMs": 208879,
    "text": "can keep up with the speed at which PRs"
  },
  {
    "startMs": 211120,
    "text": "are being opened."
  },
  {
    "startMs": 214239,
    "text": "So here's the whole answer before we get"
  },
  {
    "startMs": 216319,
    "text": "into the details. Three duties, three"
  },
  {
    "startMs": 219120,
    "text": "owners."
  },
  {
    "startMs": 220879,
    "text": "Scripts prove what can be proven. An"
  },
  {
    "startMs": 223840,
    "text": "agent judges what needs context and"
  },
  {
    "startMs": 226799,
    "text": "people own the calls only they should"
  },
  {
    "startMs": 229200,
    "text": "make. We took a hybrid approach. Scripts"
  },
  {
    "startMs": 232560,
    "text": "for the facts and AI for judgment. We"
  },
  {
    "startMs": 235760,
    "text": "did not want AI reviewing every change"
  },
  {
    "startMs": 238640,
    "text": "when some questions already have an"
  },
  {
    "startMs": 240640,
    "text": "authoritative answer."
  },
  {
    "startMs": 242959,
    "text": "And that's layer one. Deterministic"
  },
  {
    "startMs": 246080,
    "text": "scripts check a source of truth like the"
  },
  {
    "startMs": 248720,
    "text": "schema registry or a catalog. A failed"
  },
  {
    "startMs": 251680,
    "text": "check turns CI red and blocks the PR"
  },
  {
    "startMs": 254319,
    "text": "until it is fixed."
  },
  {
    "startMs": 257040,
    "text": "Layer two is the agent. The agents rules"
  },
  {
    "startMs": 260639,
    "text": "tell it what layer 1 already checks. So"
  },
  {
    "startMs": 263440,
    "text": "it focuses on the risks those checks"
  },
  {
    "startMs": 265759,
    "text": "cannot catch. It reviews the PR, scores"
  },
  {
    "startMs": 268880,
    "text": "the risk and approves or escalates to a"
  },
  {
    "startMs": 271520,
    "text": "person. Both these layers review the"
  },
  {
    "startMs": 274560,
    "text": "same PR. Merge requires passing checks"
  },
  {
    "startMs": 277680,
    "text": "and an approving review."
  },
  {
    "startMs": 280320,
    "text": "And this also keeps AI costs down. We"
  },
  {
    "startMs": 283600,
    "text": "use AI for judgment, not to rediscover"
  },
  {
    "startMs": 286160,
    "text": "facts a script can look up."
  },
  {
    "startMs": 290240,
    "text": "So here is where we are going today."
  },
  {
    "startMs": 293120,
    "text": "We have covered the problem and the"
  },
  {
    "startMs": 294639,
    "text": "approach. Now I'll walk you through the"
  },
  {
    "startMs": 296880,
    "text": "hybrid reviewer we built and the"
  },
  {
    "startMs": 298960,
    "text": "decisions we deliberately keep with the"
  },
  {
    "startMs": 300800,
    "text": "people. After that how the reviewer"
  },
  {
    "startMs": 304000,
    "text": "learns from the human feedback and what"
  },
  {
    "startMs": 306160,
    "text": "results we have seen so far. And we'll"
  },
  {
    "startMs": 308720,
    "text": "finish with the lessons we would pass on"
  },
  {
    "startMs": 310400,
    "text": "to you."
  },
  {
    "startMs": 312400,
    "text": "Three things to listen for as we go."
  },
  {
    "startMs": 315199,
    "text": "Scripts for facts and agent for the"
  },
  {
    "startMs": 317840,
    "text": "judgment and people for the protected"
  },
  {
    "startMs": 320320,
    "text": "calls. I'll come back to all three at"
  },
  {
    "startMs": 322720,
    "text": "the end."
  },
  {
    "startMs": 324479,
    "text": "So to make this concrete, let me show"
  },
  {
    "startMs": 327120,
    "text": "you the one PR will follow through the"
  },
  {
    "startMs": 329039,
    "text": "rest of this talk."
  },
  {
    "startMs": 333120,
    "text": "This PR makes two changes to a table in"
  },
  {
    "startMs": 336160,
    "text": "our data lake. It adds a new column and"
  },
  {
    "startMs": 339039,
    "text": "widens an existing column from integer"
  },
  {
    "startMs": 341039,
    "text": "to big integer. It's the kind of PR we"
  },
  {
    "startMs": 343360,
    "text": "see all the time. On the left is an"
  },
  {
    "startMs": 346400,
    "text": "upstream source table. Our injection"
  },
  {
    "startMs": 348960,
    "text": "pipeline reads from it and writes into a"
  },
  {
    "startMs": 351360,
    "text": "table in the data lake. call it our"
  },
  {
    "startMs": 353600,
    "text": "table."
  },
  {
    "startMs": 355520,
    "text": "The first change, the source has gained"
  },
  {
    "startMs": 357919,
    "text": "a new column and this PR makes injection"
  },
  {
    "startMs": 361039,
    "text": "bring that new column into our lake"
  },
  {
    "startMs": 363120,
    "text": "table."
  },
  {
    "startMs": 365280,
    "text": "The second change is to a column of a"
  },
  {
    "startMs": 367680,
    "text": "table already has"
  },
  {
    "startMs": 369840,
    "text": "its type gets widened from integer to"
  },
  {
    "startMs": 372720,
    "text": "big integer so it can hold bigger values"
  },
  {
    "startMs": 376000,
    "text": "on its own. That sounds completely"
  },
  {
    "startMs": 377759,
    "text": "harmless. Widening a type is about the"
  },
  {
    "startMs": 380639,
    "text": "safest schema change you can make,"
  },
  {
    "startMs": 382639,
    "text": "right?"
  },
  {
    "startMs": 384319,
    "text": "Well, whenever a column changes, the"
  },
  {
    "startMs": 386639,
    "text": "real question is who reads it"
  },
  {
    "startMs": 388479,
    "text": "downstream? Some of those readers are"
  },
  {
    "startMs": 390880,
    "text": "registered in our lineage tool. So, we"
  },
  {
    "startMs": 393360,
    "text": "know about them."
  },
  {
    "startMs": 395520,
    "text": "But keep a second question in the back"
  },
  {
    "startMs": 396960,
    "text": "of your mind. Are those all of them?"
  },
  {
    "startMs": 400639,
    "text": "Hold on to this picture. We will come"
  },
  {
    "startMs": 403199,
    "text": "back to this exact diagram each time the"
  },
  {
    "startMs": 405199,
    "text": "PR moves forward through the reviewer."
  },
  {
    "startMs": 409759,
    "text": "So let's start with layer one. Layer one"
  },
  {
    "startMs": 412319,
    "text": "is the scripts and their whole job is"
  },
  {
    "startMs": 414800,
    "text": "checking facts in the live systems."
  },
  {
    "startMs": 417520,
    "text": "Things the code alone cannot tell you."
  },
  {
    "startMs": 420560,
    "text": "Here are the two of those checks and"
  },
  {
    "startMs": 423039,
    "text": "what they said about our running PR."
  },
  {
    "startMs": 427199,
    "text": "First data contracts. When a PR changes"
  },
  {
    "startMs": 430800,
    "text": "a schema, a script compares the schema"
  },
  {
    "startMs": 433759,
    "text": "and keys against a source of truth, the"
  },
  {
    "startMs": 436240,
    "text": "schema registry."
  },
  {
    "startMs": 438080,
    "text": "for our PR. The question is simple. Does"
  },
  {
    "startMs": 440639,
    "text": "that new column actually exist upstream"
  },
  {
    "startMs": 443120,
    "text": "as registered in the registry? It does."
  },
  {
    "startMs": 446720,
    "text": "So that check passes."
  },
  {
    "startMs": 449919,
    "text": "Second require tags. Every table needs"
  },
  {
    "startMs": 453280,
    "text": "an owner and any PII has to be labeled"
  },
  {
    "startMs": 455840,
    "text": "in the catalog. The script checks the"
  },
  {
    "startMs": 458960,
    "text": "catalog directly. Does our table have an"
  },
  {
    "startMs": 461360,
    "text": "owner? Yes. Is its PII labeled? Yes."
  },
  {
    "startMs": 466000,
    "text": "pass again."
  },
  {
    "startMs": 468400,
    "text": "And if either of those had failed, there"
  },
  {
    "startMs": 470800,
    "text": "is no debate and no model involved. CI"
  },
  {
    "startMs": 473599,
    "text": "goes red like any other failing checks"
  },
  {
    "startMs": 476160,
    "text": "and the PR is blocked until it is fixed."
  },
  {
    "startMs": 480160,
    "text": "Contracts and tags are two of the layer"
  },
  {
    "startMs": 482319,
    "text": "1 checks. Next, the two that look at"
  },
  {
    "startMs": 485840,
    "text": "downstream impact and performance. So"
  },
  {
    "startMs": 488400,
    "text": "these are the other two layer 1 checks."
  },
  {
    "startMs": 490960,
    "text": "They look past the table itself, what"
  },
  {
    "startMs": 493280,
    "text": "happens downstream and whether the job"
  },
  {
    "startMs": 495599,
    "text": "will still run in time."
  },
  {
    "startMs": 499039,
    "text": "First, the lineage blast radius. When a"
  },
  {
    "startMs": 502400,
    "text": "column changes, like the one our PR"
  },
  {
    "startMs": 504479,
    "text": "widens, a script walks every downstream"
  },
  {
    "startMs": 507280,
    "text": "job the lineage tool has on record for"
  },
  {
    "startMs": 509280,
    "text": "that column and asks a simple question."
  },
  {
    "startMs": 512240,
    "text": "Does any of them break? It's a real"
  },
  {
    "startMs": 514880,
    "text": "check against a real system. But notice"
  },
  {
    "startMs": 517839,
    "text": "the boundary I have drawn. It can only"
  },
  {
    "startMs": 520719,
    "text": "see what is recorded in lineage. Its"
  },
  {
    "startMs": 523519,
    "text": "view of downstream is exactly as"
  },
  {
    "startMs": 526160,
    "text": "complete as that record. No more."
  },
  {
    "startMs": 530000,
    "text": "Second, the performance. A correct"
  },
  {
    "startMs": 532640,
    "text": "change can still be too slow or too"
  },
  {
    "startMs": 534560,
    "text": "expensive. So the CI performance test"
  },
  {
    "startMs": 537519,
    "text": "runs the change in staging at the same"
  },
  {
    "startMs": 540000,
    "text": "data volume as production using mock"
  },
  {
    "startMs": 542160,
    "text": "data."
  },
  {
    "startMs": 544320,
    "text": "So that tells us two things before the"
  },
  {
    "startMs": 546240,
    "text": "PR merges. whether the cluster is sized"
  },
  {
    "startMs": 548720,
    "text": "right for the job and whether the job"
  },
  {
    "startMs": 550720,
    "text": "will still make its freshness SLA. Those"
  },
  {
    "startMs": 554160,
    "text": "are the kind of problems that otherwise"
  },
  {
    "startMs": 556480,
    "text": "show up weeks later as an incident. So"
  },
  {
    "startMs": 559600,
    "text": "that's the full set of layer 1 checks."
  },
  {
    "startMs": 562560,
    "text": "Let's see what they said about our PR."
  },
  {
    "startMs": 567839,
    "text": "So back to our running PR. Same picture"
  },
  {
    "startMs": 570399,
    "text": "as before. Here's what the layer one"
  },
  {
    "startMs": 572720,
    "text": "said about it."
  },
  {
    "startMs": 575279,
    "text": "The data contract check went to the"
  },
  {
    "startMs": 577519,
    "text": "schema registry and asked whether the"
  },
  {
    "startMs": 579760,
    "text": "new column really exists upstream as"
  },
  {
    "startMs": 581920,
    "text": "register. It does and the check passes."
  },
  {
    "startMs": 586640,
    "text": "The leanage blast radius check walked"
  },
  {
    "startMs": 588720,
    "text": "every registered consumer of the widen"
  },
  {
    "startMs": 590640,
    "text": "column. The jobs the lenage tool has on"
  },
  {
    "startMs": 593120,
    "text": "record. None of them break when the"
  },
  {
    "startMs": 595279,
    "text": "column goes from integer to big integer."
  },
  {
    "startMs": 597440,
    "text": "They all look fine. Along with that tags"
  },
  {
    "startMs": 601120,
    "text": "pass, performance passes. Every check is"
  },
  {
    "startMs": 604000,
    "text": "green."
  },
  {
    "startMs": 605600,
    "text": "As far as the scripts can see, this PR"
  },
  {
    "startMs": 607680,
    "text": "is safe. But I chose those words"
  },
  {
    "startMs": 610000,
    "text": "carefully. As far as the scripts can"
  },
  {
    "startMs": 612720,
    "text": "see, everything on the slide is a"
  },
  {
    "startMs": 615279,
    "text": "statement about the registered picture,"
  },
  {
    "startMs": 617600,
    "text": "the jobs and the columns our systems"
  },
  {
    "startMs": 619760,
    "text": "have a record of. And keep that in mind."
  },
  {
    "startMs": 623120,
    "text": "It is not the whole story."
  },
  {
    "startMs": 626079,
    "text": "Now let's look at the agent with the"
  },
  {
    "startMs": 628399,
    "text": "diff. What actually changed?"
  },
  {
    "startMs": 631839,
    "text": "The PR description and the commit"
  },
  {
    "startMs": 633519,
    "text": "messages explain the author's intent,"
  },
  {
    "startMs": 635920,
    "text": "but intent never explains away a real"
  },
  {
    "startMs": 638480,
    "text": "finding. If the description says it is"
  },
  {
    "startMs": 641279,
    "text": "safe and the code says otherwise, the"
  },
  {
    "startMs": 643600,
    "text": "code wins."
  },
  {
    "startMs": 646320,
    "text": "File tools let the agent read, search,"
  },
  {
    "startMs": 648480,
    "text": "and list the files across the"
  },
  {
    "startMs": 649920,
    "text": "repository. It follows changes beyond"
  },
  {
    "startMs": 652720,
    "text": "the diff into pipeline definitions and"
  },
  {
    "startMs": 654959,
    "text": "access grants looking for the"
  },
  {
    "startMs": 656880,
    "text": "relationships the author may have"
  },
  {
    "startMs": 658320,
    "text": "missed."
  },
  {
    "startMs": 661040,
    "text": "These are the repository repository"
  },
  {
    "startMs": 663040,
    "text": "tools only. The agent cannot query the"
  },
  {
    "startMs": 665920,
    "text": "live catalog lineage tool or production"
  },
  {
    "startMs": 668480,
    "text": "state. Life facts stay layer 1's job."
  },
  {
    "startMs": 672560,
    "text": "The agent reasons over the code. So why"
  },
  {
    "startMs": 675600,
    "text": "walk that code for a schema change?"
  },
  {
    "startMs": 678000,
    "text": "Because registered lineage is not the"
  },
  {
    "startMs": 680560,
    "text": "whole picture."
  },
  {
    "startMs": 683760,
    "text": "The rules define where the judgment"
  },
  {
    "startMs": 686320,
    "text": "belongs. They are configured by the"
  },
  {
    "startMs": 688320,
    "text": "change type in the PR knowing what layer"
  },
  {
    "startMs": 690560,
    "text": "1 already covers. They target query anti"
  },
  {
    "startMs": 693760,
    "text": "patterns, sensitive data asks and"
  },
  {
    "startMs": 696399,
    "text": "consumers outside lineage."
  },
  {
    "startMs": 699279,
    "text": "For example, an unpartitioned cross join"
  },
  {
    "startMs": 701680,
    "text": "can run correctly and still hammer a"
  },
  {
    "startMs": 703440,
    "text": "cluster. A grant can look ordinary and"
  },
  {
    "startMs": 706320,
    "text": "still open sensitive data."
  },
  {
    "startMs": 708880,
    "text": "And for the schema changes, the agent"
  },
  {
    "startMs": 711200,
    "text": "walks the pipeline definitions and"
  },
  {
    "startMs": 713120,
    "text": "access grants in the repository to find"
  },
  {
    "startMs": 715519,
    "text": "consumers that were not registered. It's"
  },
  {
    "startMs": 718640,
    "text": "a tedious search uh a person would"
  },
  {
    "startMs": 720560,
    "text": "otherwise do by hand."
  },
  {
    "startMs": 722959,
    "text": "Let's return to our PR and see what that"
  },
  {
    "startMs": 724959,
    "text": "search found."
  },
  {
    "startMs": 727839,
    "text": "So back to our running PR. Same picture."
  },
  {
    "startMs": 730800,
    "text": "Layer 1 said every registered consumer"
  },
  {
    "startMs": 733120,
    "text": "of the vid column looks fine. Now the"
  },
  {
    "startMs": 736160,
    "text": "agent walks the code"
  },
  {
    "startMs": 738560,
    "text": "and it finds a job that lineage never"
  },
  {
    "startMs": 741120,
    "text": "knew about. It's right there in the"
  },
  {
    "startMs": 743360,
    "text": "repository. It reads our widen column"
  },
  {
    "startMs": 746240,
    "text": "from our late table but it was never"
  },
  {
    "startMs": 748399,
    "text": "registered in the lineage which is"
  },
  {
    "startMs": 750240,
    "text": "exactly why the layer 1 check could not"
  },
  {
    "startMs": 752320,
    "text": "see it."
  },
  {
    "startMs": 754079,
    "text": "And that job takes the value and writes"
  },
  {
    "startMs": 756639,
    "text": "it into another table. And in that other"
  },
  {
    "startMs": 759519,
    "text": "table, the column is still an integer."
  },
  {
    "startMs": 763120,
    "text": "Now follow it through. Our column is a"
  },
  {
    "startMs": 765680,
    "text": "big integer. Now the first time a value"
  },
  {
    "startMs": 768480,
    "text": "comes through that's bigger than an"
  },
  {
    "startMs": 770880,
    "text": "integer can hold that right fails. A"
  },
  {
    "startMs": 774320,
    "text": "valid schema change green in CI passes"
  },
  {
    "startMs": 777760,
    "text": "every registered check and a downstream"
  },
  {
    "startMs": 780000,
    "text": "job still breaks."
  },
  {
    "startMs": 783040,
    "text": "The agent flagged exactly this chain in"
  },
  {
    "startMs": 785519,
    "text": "its review comment with the reasoning."
  },
  {
    "startMs": 787920,
    "text": "So a person could see the whole path"
  },
  {
    "startMs": 789920,
    "text": "from our column to the failing right."
  },
  {
    "startMs": 792240,
    "text": "We'll get to what happened next in a"
  },
  {
    "startMs": 794000,
    "text": "moment."
  },
  {
    "startMs": 796399,
    "text": "So why did the lineage miss that job?"
  },
  {
    "startMs": 799279,
    "text": "Because the enterprise data world is"
  },
  {
    "startMs": 800959,
    "text": "messy. Workloads run across different"
  },
  {
    "startMs": 803440,
    "text": "query engines and each engine has to"
  },
  {
    "startMs": 805760,
    "text": "publish its own lineage. If one does not"
  },
  {
    "startMs": 808639,
    "text": "publish, that relationship is missing"
  },
  {
    "startMs": 810639,
    "text": "from the picture. There's a single"
  },
  {
    "startMs": 812639,
    "text": "catalog that automatically sees"
  },
  {
    "startMs": 814320,
    "text": "everything. So the registered picture is"
  },
  {
    "startMs": 816800,
    "text": "useful but it is not the whole picture."
  },
  {
    "startMs": 821360,
    "text": "Now I don't want to oversell the"
  },
  {
    "startMs": 823279,
    "text": "codewalk. It's genuinely useful but it"
  },
  {
    "startMs": 826160,
    "text": "cannot see everything."
  },
  {
    "startMs": 828720,
    "text": "The agent can search what is in the"
  },
  {
    "startMs": 830800,
    "text": "repo. Pipeline code access grants and"
  },
  {
    "startMs": 833440,
    "text": "the jobs defined there that read our"
  },
  {
    "startMs": 836000,
    "text": "tables. That's where it found our"
  },
  {
    "startMs": 838320,
    "text": "unregistered job. But plenty of data use"
  },
  {
    "startMs": 842079,
    "text": "lives outside the codebase. Ad hawk"
  },
  {
    "startMs": 844320,
    "text": "queries that someone runs by hand."
  },
  {
    "startMs": 846560,
    "text": "Dashboards, those are completely"
  },
  {
    "startMs": 848480,
    "text": "invisible to a repository search. A"
  },
  {
    "startMs": 851120,
    "text": "consumer out there never shows up, no"
  },
  {
    "startMs": 853120,
    "text": "matter how carefully the agent looks. So"
  },
  {
    "startMs": 856000,
    "text": "we make the agent honest about that"
  },
  {
    "startMs": 857839,
    "text": "limit. If it does not find anything, it"
  },
  {
    "startMs": 860560,
    "text": "does not get to say no downstream"
  },
  {
    "startMs": 862240,
    "text": "consumers exist. It says it found none"
  },
  {
    "startMs": 865519,
    "text": "in the places it searched. None found is"
  },
  {
    "startMs": 868800,
    "text": "not the same as none exist and the agent"
  },
  {
    "startMs": 871600,
    "text": "never gets to claim an allcle."
  },
  {
    "startMs": 876240,
    "text": "So the agent has walked the code. What"
  },
  {
    "startMs": 878480,
    "text": "does it actually produce? A review the"
  },
  {
    "startMs": 881199,
    "text": "same shape a good human reviewer would"
  },
  {
    "startMs": 883519,
    "text": "give you."
  },
  {
    "startMs": 885279,
    "text": "First review comments on the specific"
  },
  {
    "startMs": 887760,
    "text": "lines that matter each with its"
  },
  {
    "startMs": 890240,
    "text": "reasoning. Not awake looks risky but"
  },
  {
    "startMs": 893839,
    "text": "this line changes this column. And here"
  },
  {
    "startMs": 896000,
    "text": "is the job that reads it."
  },
  {
    "startMs": 898800,
    "text": "Second, a risk score. How severe is what"
  },
  {
    "startMs": 901839,
    "text": "it found. That's what drives the next"
  },
  {
    "startMs": 904240,
    "text": "step. Then it makes a call. If there is"
  },
  {
    "startMs": 907440,
    "text": "no serious risk, it can approve. If"
  },
  {
    "startMs": 909839,
    "text": "there is a risk that needs a person, it"
  },
  {
    "startMs": 912160,
    "text": "escalates to a human and the comments"
  },
  {
    "startMs": 914560,
    "text": "and reasoning go with it. So the person"
  },
  {
    "startMs": 917600,
    "text": "has the context to investigate instead"
  },
  {
    "startMs": 919839,
    "text": "of starting from scratch."
  },
  {
    "startMs": 922480,
    "text": "And to be clear about what it takes to"
  },
  {
    "startMs": 924720,
    "text": "merge, layer 1's checks have to pass and"
  },
  {
    "startMs": 928160,
    "text": "the review has to approve either the"
  },
  {
    "startMs": 930399,
    "text": "agent or after an escalation a person."
  },
  {
    "startMs": 933920,
    "text": "Both have to say yes. Now let's see what"
  },
  {
    "startMs": 936959,
    "text": "that looked like for our PR."
  },
  {
    "startMs": 940480,
    "text": "Here is how our running PR actually"
  },
  {
    "startMs": 942320,
    "text": "ended. Same picture. The registered"
  },
  {
    "startMs": 945279,
    "text": "consumers look fine and the core walk"
  },
  {
    "startMs": 948000,
    "text": "found an unregistered job that writes"
  },
  {
    "startMs": 950399,
    "text": "our widened column into a table where it"
  },
  {
    "startMs": 952800,
    "text": "is still an integer"
  },
  {
    "startMs": 955279,
    "text": "because the agent found a real"
  },
  {
    "startMs": 956720,
    "text": "downstream break. It scored the risk"
  },
  {
    "startMs": 958880,
    "text": "high and escalated to a human with the"
  },
  {
    "startMs": 961360,
    "text": "whole chain spelled out in its review"
  },
  {
    "startMs": 963360,
    "text": "comment. Now the human does the part the"
  },
  {
    "startMs": 966240,
    "text": "agent cannot. The agent only reasons"
  },
  {
    "startMs": 969040,
    "text": "over the repo. It cannot see whether the"
  },
  {
    "startMs": 971440,
    "text": "job is actually running in production."
  },
  {
    "startMs": 973920,
    "text": "human reviewer checked and confirmed"
  },
  {
    "startMs": 976079,
    "text": "that the consumer the agent found was"
  },
  {
    "startMs": 978000,
    "text": "live and active. So this was not a"
  },
  {
    "startMs": 981040,
    "text": "theoretical risk. The PR waited"
  },
  {
    "startMs": 985120,
    "text": "the f the fix was about order not about"
  },
  {
    "startMs": 988560,
    "text": "rejecting the change. The downstream"
  },
  {
    "startMs": 991120,
    "text": "table was updated first so its column"
  },
  {
    "startMs": 993600,
    "text": "could handle the wider type"
  },
  {
    "startMs": 996399,
    "text": "and only then did our PR merge and"
  },
  {
    "startMs": 998800,
    "text": "deploy. No incident. The scripts in"
  },
  {
    "startMs": 1001759,
    "text": "layer 1 said safe. The agent found the"
  },
  {
    "startMs": 1004399,
    "text": "gap and a human made the call."
  },
  {
    "startMs": 1009279,
    "text": "And that was not a one-time thing. Most"
  },
  {
    "startMs": 1012000,
    "text": "changes can be cleared by the agent."
  },
  {
    "startMs": 1014160,
    "text": "When the checks pass and the agent is"
  },
  {
    "startMs": 1016079,
    "text": "confident in its verdict, it approves on"
  },
  {
    "startMs": 1018240,
    "text": "its own. But there's a whole category of"
  },
  {
    "startMs": 1020720,
    "text": "changes where the agent never gets to"
  },
  {
    "startMs": 1022240,
    "text": "say approved. And no matter how"
  },
  {
    "startMs": 1024240,
    "text": "confident it sounds,"
  },
  {
    "startMs": 1026959,
    "text": "grants touching sensitive data are one."
  },
  {
    "startMs": 1030160,
    "text": "Who can reach sensitive data is a"
  },
  {
    "startMs": 1032000,
    "text": "decision we want a person to own every"
  },
  {
    "startMs": 1034319,
    "text": "time."
  },
  {
    "startMs": 1036400,
    "text": "A migration that looks like it changes"
  },
  {
    "startMs": 1038480,
    "text": "intent, not just mechanics, is another."
  },
  {
    "startMs": 1041679,
    "text": "If a change alters what the data means"
  },
  {
    "startMs": 1044319,
    "text": "rather than just how it is moved or"
  },
  {
    "startMs": 1046400,
    "text": "stored, a person decides."
  },
  {
    "startMs": 1050160,
    "text": "And if the agent itself is not confident"
  },
  {
    "startMs": 1052320,
    "text": "in its own verdict, that's the third. It"
  },
  {
    "startMs": 1055440,
    "text": "does not get to round up to approve just"
  },
  {
    "startMs": 1057360,
    "text": "because nothing concrete came back"
  },
  {
    "startMs": 1059200,
    "text": "wrong. In all the all the three cases,"
  },
  {
    "startMs": 1062720,
    "text": "its job is only able to surface the risk"
  },
  {
    "startMs": 1065280,
    "text": "and its reasoning. The decision stays"
  },
  {
    "startMs": 1067919,
    "text": "with a person."
  },
  {
    "startMs": 1070400,
    "text": "That is one PR. Now, how does this hold"
  },
  {
    "startMs": 1073360,
    "text": "up across every PR we see? And how does"
  },
  {
    "startMs": 1075760,
    "text": "the reviewer get better over time?"
  },
  {
    "startMs": 1079679,
    "text": "How do we know the flags are useful or"
  },
  {
    "startMs": 1082000,
    "text": "whether a rule change helps? We start"
  },
  {
    "startMs": 1085280,
    "text": "with the agents verdict and its"
  },
  {
    "startMs": 1086960,
    "text": "reasoning in the review comments."
  },
  {
    "startMs": 1089600,
    "text": "A person replies agreeing or disagreeing"
  },
  {
    "startMs": 1092400,
    "text": "and explaining why. That explanation is"
  },
  {
    "startMs": 1095200,
    "text": "what makes the feedback useful rather"
  },
  {
    "startMs": 1097360,
    "text": "than just a vote."
  },
  {
    "startMs": 1099760,
    "text": "Those replies are collected every day."
  },
  {
    "startMs": 1103280,
    "text": "Once a week, the on call reads the"
  },
  {
    "startMs": 1105280,
    "text": "feedback and updates the review rules by"
  },
  {
    "startMs": 1107520,
    "text": "hand. Those rules shape the next review."
  },
  {
    "startMs": 1111120,
    "text": "I know this is manual today. Nothing"
  },
  {
    "startMs": 1113679,
    "text": "retrains itself and no rule changes"
  },
  {
    "startMs": 1115840,
    "text": "automatically. But we have an idea to"
  },
  {
    "startMs": 1118320,
    "text": "let an incident investigation agent help"
  },
  {
    "startMs": 1121120,
    "text": "with that weekly pass. But it is not"
  },
  {
    "startMs": 1123280,
    "text": "built. Without the feedback, every rule"
  },
  {
    "startMs": 1126320,
    "text": "change would be a guess. So let me show"
  },
  {
    "startMs": 1128720,
    "text": "you an example where a human corrected"
  },
  {
    "startMs": 1130799,
    "text": "the agent and we use that feedback to"
  },
  {
    "startMs": 1133679,
    "text": "improve its next review."
  },
  {
    "startMs": 1137600,
    "text": "So here's a real one. And note the label"
  },
  {
    "startMs": 1140400,
    "text": "in the corner. This is a separate"
  },
  {
    "startMs": 1142640,
    "text": "example, not our schema change PR."
  },
  {
    "startMs": 1146400,
    "text": "A column was marked as personal"
  },
  {
    "startMs": 1148400,
    "text": "information in the source database."
  },
  {
    "startMs": 1150960,
    "text": "But by the time it lands in the data"
  },
  {
    "startMs": 1152720,
    "text": "lake, the injection pipeline has already"
  },
  {
    "startMs": 1154720,
    "text": "encrypted it. So the PR proposed"
  },
  {
    "startMs": 1157360,
    "text": "labeling it nonsensitive in the data"
  },
  {
    "startMs": 1159679,
    "text": "lake even though the upstream data is"
  },
  {
    "startMs": 1161679,
    "text": "still PII."
  },
  {
    "startMs": 1163840,
    "text": "The agent flagged this. From where it"
  },
  {
    "startMs": 1166480,
    "text": "stood, the upstream says PII, so a"
  },
  {
    "startMs": 1168960,
    "text": "nonsensitive label looked wrong. The"
  },
  {
    "startMs": 1171840,
    "text": "author pushed back and asked the agent"
  },
  {
    "startMs": 1174160,
    "text": "to look at how similar columns on the"
  },
  {
    "startMs": 1175760,
    "text": "other tables were already classified"
  },
  {
    "startMs": 1178000,
    "text": "instead of going purely off the upstream"
  },
  {
    "startMs": 1180320,
    "text": "tagging."
  },
  {
    "startMs": 1182640,
    "text": "A human agreed with the author. The"
  },
  {
    "startMs": 1185280,
    "text": "encryption before write made the"
  },
  {
    "startMs": 1186880,
    "text": "nonsensitive label look okay for this"
  },
  {
    "startMs": 1189120,
    "text": "column. And that kind of correction is"
  },
  {
    "startMs": 1191760,
    "text": "exactly the signal that teaches the"
  },
  {
    "startMs": 1193840,
    "text": "rules to get smarter because the next"
  },
  {
    "startMs": 1196320,
    "text": "time a similar column comes through that"
  },
  {
    "startMs": 1198720,
    "text": "context can stop a needless escalation."
  },
  {
    "startMs": 1202640,
    "text": "One caution though, this depends on the"
  },
  {
    "startMs": 1204880,
    "text": "specific pipeline encrypting before the"
  },
  {
    "startMs": 1207039,
    "text": "right. It's not a blanket rule that"
  },
  {
    "startMs": 1209039,
    "text": "encryption always makes data"
  },
  {
    "startMs": 1211039,
    "text": "nonsensitive."
  },
  {
    "startMs": 1214640,
    "text": "So how do we know whether the reviewer"
  },
  {
    "startMs": 1216240,
    "text": "is actually doing a good job?"
  },
  {
    "startMs": 1218640,
    "text": "We look at two kinds of mistakes. Bad"
  },
  {
    "startMs": 1221440,
    "text": "PRs it approved and fine PRs it"
  },
  {
    "startMs": 1224240,
    "text": "unnecessarily send to a person. That"
  },
  {
    "startMs": 1227280,
    "text": "matrix separates four possibilities."
  },
  {
    "startMs": 1231360,
    "text": "A fine PR approved and a bad PR"
  },
  {
    "startMs": 1233440,
    "text": "escalated are the two correct outcomes."
  },
  {
    "startMs": 1236799,
    "text": "A bad PR approved is a miss. That's"
  },
  {
    "startMs": 1239840,
    "text": "dangerous because the change ships."
  },
  {
    "startMs": 1242720,
    "text": "A fine PR escalated is a false"
  },
  {
    "startMs": 1244880,
    "text": "escalation that costs a person's time. A"
  },
  {
    "startMs": 1248880,
    "text": "gate that escalates everything may avoid"
  },
  {
    "startMs": 1251200,
    "text": "misses, but it has not reduced the"
  },
  {
    "startMs": 1253360,
    "text": "review load."
  },
  {
    "startMs": 1256000,
    "text": "As of August 31st, 2026, our miss rate"
  },
  {
    "startMs": 1258640,
    "text": "was about 2%."
  },
  {
    "startMs": 1261679,
    "text": "Mostly from the gaps in the rules rather"
  },
  {
    "startMs": 1263600,
    "text": "than the failures of the judgment. We"
  },
  {
    "startMs": 1266159,
    "text": "don't have a clean single false"
  },
  {
    "startMs": 1267679,
    "text": "escalation number. But the dominant"
  },
  {
    "startMs": 1270000,
    "text": "cause is missing context like the"
  },
  {
    "startMs": 1272000,
    "text": "encrypted column example that I just"
  },
  {
    "startMs": 1274080,
    "text": "gave you."
  },
  {
    "startMs": 1276320,
    "text": "Misses give us new rules and false"
  },
  {
    "startMs": 1278720,
    "text": "escalation show us context the reviewer"
  },
  {
    "startMs": 1280960,
    "text": "needs. So here is one of those misses."
  },
  {
    "startMs": 1284080,
    "text": "Again know the label. This is a separate"
  },
  {
    "startMs": 1286240,
    "text": "example, not our schema change PR."
  },
  {
    "startMs": 1289840,
    "text": "We have an open copy of the sum of some"
  },
  {
    "startMs": 1292080,
    "text": "data. some of its columns are about to"
  },
  {
    "startMs": 1294880,
    "text": "be rettagged and there is a downstream"
  },
  {
    "startMs": 1297039,
    "text": "reader that reads those columns from the"
  },
  {
    "startMs": 1299039,
    "text": "open copy."
  },
  {
    "startMs": 1301360,
    "text": "Privacy retag those columns as personal"
  },
  {
    "startMs": 1303840,
    "text": "information. When that happens, they"
  },
  {
    "startMs": 1306559,
    "text": "move into a more restricted copy of the"
  },
  {
    "startMs": 1308799,
    "text": "data and they are gone from the open"
  },
  {
    "startMs": 1310640,
    "text": "copy."
  },
  {
    "startMs": 1312320,
    "text": "The retag looked fine and it merged. But"
  },
  {
    "startMs": 1315520,
    "text": "what nobody checked was whether anything"
  },
  {
    "startMs": 1317440,
    "text": "downstream was still reading those"
  },
  {
    "startMs": 1318960,
    "text": "columns from the old open copy."
  },
  {
    "startMs": 1321360,
    "text": "something was. And once the columns were"
  },
  {
    "startMs": 1323760,
    "text": "gone, that reader broke. That's a miss."
  },
  {
    "startMs": 1327200,
    "text": "A PR that should have been escalated got"
  },
  {
    "startMs": 1329520,
    "text": "approved."
  },
  {
    "startMs": 1331520,
    "text": "It wasn't a failure of judgment. It was"
  },
  {
    "startMs": 1333600,
    "text": "a gap in the rules. A coroner case"
  },
  {
    "startMs": 1336159,
    "text": "nobody had written down. So, it became a"
  },
  {
    "startMs": 1338720,
    "text": "new rule. When a column gets retagged as"
  },
  {
    "startMs": 1341360,
    "text": "sensitive, search the codebase for"
  },
  {
    "startMs": 1343280,
    "text": "anyone still reading it from the copy"
  },
  {
    "startMs": 1345120,
    "text": "that's about to lose it. And that's how"
  },
  {
    "startMs": 1347679,
    "text": "the rules get better. the misses get"
  },
  {
    "startMs": 1350320,
    "text": "found and get turned into new rules."
  },
  {
    "startMs": 1354640,
    "text": "So let's pull it all together. You"
  },
  {
    "startMs": 1357360,
    "text": "remember the pull request from the"
  },
  {
    "startMs": 1358720,
    "text": "start, the one that was green in CI."
  },
  {
    "startMs": 1362000,
    "text": "Three things happened to it."
  },
  {
    "startMs": 1364640,
    "text": "First, scripts proved what a source of"
  },
  {
    "startMs": 1367440,
    "text": "truth could prove. The contracts, the"
  },
  {
    "startMs": 1370320,
    "text": "tags, the registered lineage and"
  },
  {
    "startMs": 1372480,
    "text": "performance all passed."
  },
  {
    "startMs": 1374720,
    "text": "Second, the agent read the code and"
  },
  {
    "startMs": 1377120,
    "text": "found what the scripts could not see. A"
  },
  {
    "startMs": 1379440,
    "text": "job that Lineage never knew about,"
  },
  {
    "startMs": 1381520,
    "text": "quietly waiting to fail on a wider"
  },
  {
    "startMs": 1383520,
    "text": "column, it flagged the whole chain and"
  },
  {
    "startMs": 1385919,
    "text": "escalated. Third, a person confirmed the"
  },
  {
    "startMs": 1389280,
    "text": "job was live. Uh, the downstream table"
  },
  {
    "startMs": 1391360,
    "text": "was fixed first and only then did the PR"
  },
  {
    "startMs": 1394000,
    "text": "merge. No incident. And behind all of"
  },
  {
    "startMs": 1397280,
    "text": "that, the reviewer keeps getting better"
  },
  {
    "startMs": 1400400,
    "text": "because every correction a person makes"
  },
  {
    "startMs": 1402720,
    "text": "can become a rule. These results are as"
  },
  {
    "startMs": 1405760,
    "text": "of August 31st, 2026. So you might ask,"
  },
  {
    "startMs": 1409039,
    "text": "Jas, does it all work?"
  },
  {
    "startMs": 1412240,
    "text": "Yes, about 60% of the data pipeline PRs"
  },
  {
    "startMs": 1415440,
    "text": "merged without a human review."
  },
  {
    "startMs": 1418080,
    "text": "We have seen about 75% less manual"
  },
  {
    "startMs": 1420960,
    "text": "review effort. The agent specific"
  },
  {
    "startMs": 1423520,
    "text": "comments and reasoning help focus the"
  },
  {
    "startMs": 1425520,
    "text": "review rather than making a person"
  },
  {
    "startMs": 1428000,
    "text": "reconstruct every risk from scratch."
  },
  {
    "startMs": 1431840,
    "text": "The time saved comes to about 132"
  },
  {
    "startMs": 1435120,
    "text": "engineering hours a month. Incidents"
  },
  {
    "startMs": 1437600,
    "text": "tied to these PRs are down about 30%."
  },
  {
    "startMs": 1441919,
    "text": "And that does not mean human free"
  },
  {
    "startMs": 1443840,
    "text": "review. The protected decisions still"
  },
  {
    "startMs": 1446480,
    "text": "reach a person by design. The goal is"
  },
  {
    "startMs": 1449679,
    "text": "less review work, not removing"
  },
  {
    "startMs": 1451919,
    "text": "accountability. And those are the"
  },
  {
    "startMs": 1454159,
    "text": "results. Now, let's look at the"
  },
  {
    "startMs": 1456080,
    "text": "architecture choices we got wrong."
  },
  {
    "startMs": 1460720,
    "text": "Here is what we got wrong today. Layer 1"
  },
  {
    "startMs": 1463760,
    "text": "and layer 2 are separate systems"
  },
  {
    "startMs": 1466320,
    "text": "triggered independently and running in"
  },
  {
    "startMs": 1468240,
    "text": "parallel. Each produces its own result"
  },
  {
    "startMs": 1472000,
    "text": "that seemed like an obvious design when"
  },
  {
    "startMs": 1473679,
    "text": "we built it,"
  },
  {
    "startMs": 1475760,
    "text": "but neither sees the others pass fail or"
  },
  {
    "startMs": 1478240,
    "text": "outage state. Layer 2 knows what the"
  },
  {
    "startMs": 1481200,
    "text": "scripts are configured to cover, not"
  },
  {
    "startMs": 1483760,
    "text": "what happened when they ran. Merge still"
  },
  {
    "startMs": 1486880,
    "text": "needs passing checks and an approving"
  },
  {
    "startMs": 1488799,
    "text": "review, but no single system has a full"
  },
  {
    "startMs": 1491360,
    "text": "runtime picture."
  },
  {
    "startMs": 1493760,
    "text": "The agent also cannot call live layer 1"
  },
  {
    "startMs": 1496880,
    "text": "checks. Its tools only read, search and"
  },
  {
    "startMs": 1499840,
    "text": "list the repository files. If a judgment"
  },
  {
    "startMs": 1502559,
    "text": "needs a live fact from a live production"
  },
  {
    "startMs": 1505120,
    "text": "system, it cannot fetch that fact"
  },
  {
    "startMs": 1507360,
    "text": "itself."
  },
  {
    "startMs": 1509520,
    "text": "When one site cannot finish, we wait for"
  },
  {
    "startMs": 1512240,
    "text": "a human. There is no automatic fallback."
  },
  {
    "startMs": 1515440,
    "text": "These are current limitations and they"
  },
  {
    "startMs": 1517760,
    "text": "motivate the proposal. On the next"
  },
  {
    "startMs": 1519440,
    "text": "slide, platform team is exploring one"
  },
  {
    "startMs": 1522400,
    "text": "entry point. This is a proposal and not"
  },
  {
    "startMs": 1525039,
    "text": "what's running today. The left panel is"
  },
  {
    "startMs": 1527760,
    "text": "the current design and the right is"
  },
  {
    "startMs": 1530080,
    "text": "where we want to go. A classifier looks"
  },
  {
    "startMs": 1533120,
    "text": "at the whole PR then decides which"
  },
  {
    "startMs": 1535360,
    "text": "changes to review separately and which"
  },
  {
    "startMs": 1537360,
    "text": "to review together."
  },
  {
    "startMs": 1539440,
    "text": "This is a whole PR decision, not"
  },
  {
    "startMs": 1541360,
    "text": "independent routing of each file."
  },
  {
    "startMs": 1544640,
    "text": "The agent would also get the layer 1"
  },
  {
    "startMs": 1546960,
    "text": "checks as collable tools alongside its"
  },
  {
    "startMs": 1549360,
    "text": "repository tools. When judgment needs a"
  },
  {
    "startMs": 1552159,
    "text": "live fact, it could request the"
  },
  {
    "startMs": 1554000,
    "text": "appropriate check instead of being"
  },
  {
    "startMs": 1555279,
    "text": "limited to code alone."
  },
  {
    "startMs": 1558080,
    "text": "The aim here is one system with the"
  },
  {
    "startMs": 1560320,
    "text": "whole PR in view. We haven't built this"
  },
  {
    "startMs": 1562640,
    "text": "yet. So, it's a direction, not a result."
  },
  {
    "startMs": 1566240,
    "text": "Let me close with what we would carry"
  },
  {
    "startMs": 1568480,
    "text": "forward."
  },
  {
    "startMs": 1570159,
    "text": "So, here is the whole talk in three"
  },
  {
    "startMs": 1571760,
    "text": "lines."
  },
  {
    "startMs": 1573600,
    "text": "Deterministic evaluations. First, if"
  },
  {
    "startMs": 1576240,
    "text": "something can be proven against a source"
  },
  {
    "startMs": 1577840,
    "text": "of truth, let a script prove it. Don't"
  },
  {
    "startMs": 1580720,
    "text": "spend a judgment call on a yes or no"
  },
  {
    "startMs": 1582559,
    "text": "fact. It's not just cleaner, it's"
  },
  {
    "startMs": 1585120,
    "text": "cheaper, too. Every fact a script proves"
  },
  {
    "startMs": 1588559,
    "text": "is review work that never needs AI. So,"
  },
  {
    "startMs": 1590960,
    "text": "the cost of running this stays down."
  },
  {
    "startMs": 1594799,
    "text": "Flag never clear. Not for every PR, but"
  },
  {
    "startMs": 1598559,
    "text": "for the changes that actually scare"
  },
  {
    "startMs": 1600080,
    "text": "people, like sensitive grants, migration"
  },
  {
    "startMs": 1602559,
    "text": "intent, and low confidence verdicts."
  },
  {
    "startMs": 1605919,
    "text": "There, the agent's job stops at"
  },
  {
    "startMs": 1608080,
    "text": "surfacing the risk and the reasoning"
  },
  {
    "startMs": 1610240,
    "text": "behind it. The decision stays with the"
  },
  {
    "startMs": 1613039,
    "text": "person every time."
  },
  {
    "startMs": 1616000,
    "text": "Count the false escalations. A gate that"
  },
  {
    "startMs": 1619039,
    "text": "escalates everything is not actually"
  },
  {
    "startMs": 1621039,
    "text": "saving anyone any time. It's just moving"
  },
  {
    "startMs": 1623919,
    "text": "the work around. measure that cost the"
  },
  {
    "startMs": 1626799,
    "text": "same way you measure a miss. So I"
  },
  {
    "startMs": 1629679,
    "text": "started with a pull request that was"
  },
  {
    "startMs": 1631120,
    "text": "green in CI and a problem PRs arriving"
  },
  {
    "startMs": 1634320,
    "text": "at machine speed and too few people with"
  },
  {
    "startMs": 1636640,
    "text": "the context to review them safely. We"
  },
  {
    "startMs": 1639039,
    "text": "built a hybrid reviewer scripts for the"
  },
  {
    "startMs": 1641440,
    "text": "facts and agent for judgment and people"
  },
  {
    "startMs": 1644000,
    "text": "for the protected calls. We walked one"
  },
  {
    "startMs": 1646720,
    "text": "PR through all three saw it get lighter"
  },
  {
    "startMs": 1649520,
    "text": "without going human free and saw how it"
  },
  {
    "startMs": 1652320,
    "text": "learns from the corrections. We told you"
  },
  {
    "startMs": 1654640,
    "text": "what we got wrong and we left you with"
  },
  {
    "startMs": 1656720,
    "text": "three lessons. Prove what you can"
  },
  {
    "startMs": 1659440,
    "text": "cheaply. Flag and never clear the scary"
  },
  {
    "startMs": 1662559,
    "text": "changes. And count the false"
  },
  {
    "startMs": 1664640,
    "text": "escalations."
  },
  {
    "startMs": 1666559,
    "text": "So who reviews the bots? Scripts for"
  },
  {
    "startMs": 1669360,
    "text": "everything a source of truth can prove,"
  },
  {
    "startMs": 1671440,
    "text": "an agent for everything that needs"
  },
  {
    "startMs": 1673120,
    "text": "context and judgment, and humans for the"
  },
  {
    "startMs": 1676320,
    "text": "calls only they get to make. Thank you"
  },
  {
    "startMs": 1679279,
    "text": "for listening."
  }
]
