# 60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood

Chetas Joshi, Robinhood

[Talk page](https://ai.engineer/talks/MQiojLnK-E0-60-our-data-prs-now-merge-no) · [Video](https://www.youtube.com/watch?v=MQiojLnK-E0)

## 60% of Our Data PRs Now Merge With No Human Review

At Robinhood, a PR to widen an integer column passed every CI check. Following the value one table farther explains what a review agent can catch and why this particular merge still needed an engineer.

### The queue forms around the people who know the table

[0:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=0s)

A schema change passes its tests and the linter. Is it safe to merge? A pipeline can produce correct data too late to meet its freshness deadline. A change can break a consumer three teams away or give someone access to sensitive data. For Chetas Joshi’s data platform team at Robinhood, answering the merge question means understanding who depends on the changed code and what they expect from it.

Coding assistants could open a PR for a new query, schema change or access grant in minutes. Reviews of pipeline code and configuration also landed with the data platform team. CI and human review already caught broken code, inefficient queries and data leaks. What slowed the queue was waiting for one of the few engineers who knew who read a table, how large it was and what sensitive data it held. The number of people carrying that context stayed flat as the PRs piled up.

Robinhood split the work by the kind of answer each reviewer could give. A script can look up a fact in a schema registry or catalog and block a PR if the check fails. An agent can read the code, assess risk, and either approve or ask a person to decide. Its rules tell it which questions the scripts cover, saving the cost of asking a model to reconstruct a fact the team can look up. A merge needs both passing checks and an approving review.

### What four green checks establish about one schema change

[5:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=324s)

An ingestion pipeline reads an upstream source table and writes to a table in the data lake. One PR makes two changes. The source has gained a column, and the pipeline will bring it into the lake. A different column, already in the lake table, will widen from `INTEGER` to `BIGINT` so it can hold larger values. The first edit raises a question about the source. The second raises a question about every job that reads the widened column.

- **Data contract.** A script compares the proposed schema and keys with the schema registry. The registry confirms that the new column exists upstream.

- **Required tags.** Every table needs an owner, and personally identifiable information (PII) needs a label in the catalog. A direct catalog lookup confirms both for this table.

- **Downstream dependencies.** The lineage tool records which jobs read a column. A script walks every job registered for the widened column and checks whether the type change breaks it.

- **Performance.** CI runs the change in staging with mock data at production volume. The run checks whether the cluster is sized for the job and whether the job can finish in time to meet its freshness service-level agreement.

A failed check blocks the PR until someone fixes it. Here all four passed. How far downstream did that green result reach? Every registered consumer could handle the wider type. The word *registered* matters. The script starts with the lineage tool’s list, so a job missing from that list contributes nothing to the result.

![The schema registry confirms the new column exists upstream. The data catalog confirms ownership and PII labeling. Both checks pass for this pull request. A banner says CI blocks the pull request if either check fails.](https://ai.engineer/api/talk-essay-media/6f9725c5e15241d8b0f6c2c6eacf9ce3d2703e0bac8c8105d7c6186a84e7aee3)

The registry and catalog answer separate questions about the new column and ownership of the table.

### A repository search finds the narrower destination

[10:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=626s)

The agent starts with the diff. The PR description and commit messages explain what the author intended, but a finding in the code takes precedence over a claim that the change is safe. The agent can list, search and read files across the repository. It has no direct access to the live catalog, lineage tool or production state.

The type of change tells the agent what to look for. In a query, an unpartitioned cross join can return the right answer while hammering a cluster. In an access grant, an ordinary-looking edit can open sensitive data. For a schema change, the rules direct the agent into pipeline definitions and access grants to find consumers beyond those recorded in lineage. For this PR, the question was where a value would go after a job read it from the `BIGINT` column.

The search found a job defined in the repository that had never been registered in lineage. It read the widened column and copied the value to another table, where the receiving column was still `INTEGER`. Values within the original range would continue to fit. The first value large enough to need the wider range would reach that last write and fail. The agent’s review comment traced the path from the changed column through the job to the destination.

Each query engine has to publish its own lineage. If an engine omits a relationship, a repository search can still recover it from a job definition. The search has limits of its own. An ad hoc query someone runs by hand, or a dashboard defined outside the codebase, leaves the agent nothing to read. When a search comes up empty, the agent reports where it looked and that it found no consumers there.

Where would the first out-of-range value fail?
Follow the data out of the lake table. The lineage check covered the registered branch. Following the other branch through repository code revealed a write to a narrower column.
- Lake table — The PR widens an existing column from INTEGER to BIGINT.
- Registered consumers — Every reader listed for the column passes the lineage check.
- Unregistered job — The repository contains the job that copies this column onward.
- Downstream table — Its INTEGER column cannot store a value outside the original range.
- Lake table → Registered consumers (Read the widened column)
- Lake table → Unregistered job (Reads the same column)
- Unregistered job → Downstream table (Writes the value)

### The reviewer changes the rollout order

[14:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=876s)

The agent attaches its reasoning to specific lines and scores the severity of what it found. For an ordinary change, a confident review with no serious finding can supply the approval needed alongside passing CI. For the widening PR, the agent scored the downstream write as high risk and escalated. The engineer who picked it up already knew which job to investigate. The open question was whether that job actually ran in production.

The human reviewer confirmed the consumer was live. The PR waited while the team widened the receiving column in the downstream table. Once that destination could hold the larger values, the original PR merged and deployed without incident.

Some decisions always go to a person. In each case, the agent passes along its findings and reasoning.

- **Sensitive access grants.** A person decides who may reach sensitive data, however confident the agent sounds.

- **Migrations that appear to change meaning.** If a migration changes what the data represents, a person decides whether that change of intent is acceptable.

- **Low-confidence verdicts.** Even when the search turns up no concrete defect, uncertainty sends the review to a person.

![A diagram traces a widened column to an unregistered job. Three boxes show the agent flagging a write-failure risk, a person confirming the job runs in production, and the downstream table’s column being widened to bigint.](https://ai.engineer/api/talk-essay-media/b189cea2cb4d778a31b53b7a5073a69071fee7e4ad6c0c3491acf14758a6baa9)

The order of operations matters: widen the downstream destination before merging the upstream schema change.

### An encrypted column changes the review question

[17:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1070s)

A person can reply to the agent’s comment, agree or disagree, and explain why. The explanation gives the on-call engineer something specific to change. The team collects replies daily. Once a week the on-call engineer reads them and edits the review rules by hand. Those edits shape subsequent reviews without retraining the model. Using an incident-investigation agent to help with the weekly pass was still an idea.

In a separate PR, a source database marked a column as PII. The ingestion pipeline encrypted the column before writing it to the lake, and the author proposed a nonsensitive label for the lake copy. The agent saw a disagreement between the upstream and downstream labels and flagged it. The author pointed to comparable columns in other tables and asked the agent to consider how those columns were classified.

A human reviewer agreed with the author about this column. What happened between the source and the lake mattered to the decision. A future review could check where the pipeline applies encryption and how the team classifies comparable output before escalating a label mismatch. The decision here depended on encryption before this particular write. Encryption alone cannot settle the classification of another dataset.

### A false alarm costs time. A missed reader breaks.

[20:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1214s)

How can the team tell whether a review rule helps? Cross the agent’s decision with the decision the PR needed. Approving a safe change and escalating a risky one are both correct. Approving a risky change is a miss that lets the change ship. Escalating a safe change unnecessarily consumes an engineer’s time. A reviewer could avoid every unsafe approval by forwarding every PR, leaving the original queue intact.

As of August 31, 2026, Joshi reported a miss rate of about 2%, without specifying the denominator. The team traced most misses to gaps in the review rules. It had no single reliable rate for false escalations, but the most common cause was missing context of the kind the encrypted-column example exposed.

One miss started as a privacy correction. A downstream job read several columns from an open copy of a dataset. When the privacy team retagged the columns as personal information, they moved to a restricted copy and disappeared from the open one. The retag itself looked reasonable. The reviewer approved it, the PR merged, and the job broke when it tried to read the missing columns.

The team added a rule with a specific trigger and search target. When a column is retagged as sensitive, search the codebase for readers that still reference that column in the copy about to lose it. That gives the reviewer a way to flag a broken read even when the privacy classification is correct.

![A two-by-two matrix compares the agent’s decision with the quality of a pull request. Approving a bad change is a miss. Escalating a sound change costs a reviewer time. The other two cells show correct approvals and escalations.](https://ai.engineer/api/talk-essay-media/0b0637e3170bac66f7af9e57e9372384810395d374cf19bb33eb971dc2253193)

The four cells distinguish useful escalations, false alarms, missed issues and changes safe to pass.

### Even escalated PRs took less work to review

[22:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1354s)

As of August 31, 2026, about 60% of Robinhood’s data pipeline PRs merged without human review. The team reported about 75% less manual review effort, saving roughly 132 engineering hours a month. The widening PR illustrates how an escalation could also save work. The engineer could go straight to checking whether a particular consumer was live because the agent had already traced the dependency.

The team also observed about 30% fewer incidents tied to these PRs. That figure describes the team’s experience with the hybrid reviewer in place, including the engineers who decided protected cases and investigated escalations.

### Two independent reviewers and a proposal for one entry point

[24:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1460s)

The two layers in the deployed design run in parallel. Separate triggers start the scripts and the agent on the same PR, and each produces its own result. The agent knows which checks are configured, but it cannot see whether a particular check passed, failed or hit an outage. Nor can it call a check when its investigation turns up a question about production. Merge still requires passing CI and approval. If either system cannot finish, the team waits for a human.

What would a single entry point change? In the proposed design, a classifier would examine the whole PR before deciding which changes to review together and which to handle separately. That would let the grouping decision account for relationships across files. The agent would also get callable deterministic checks alongside its repository tools, so it could request a live fact when the code raised a question. The team was exploring this design. The reported savings came from the parallel systems already in use.

Joshi would carry three decisions into that redesign. Keep yes-or-no lookups in scripts, where each lookup saves model work. Give people the final call on sensitive grants, migrations that change meaning and uncertain verdicts. And count unnecessary escalations alongside misses. Each needless referral spends the time of the same few engineers the reviewer was built to free up.

## Key takeaways

- A passing lineage check establishes compatibility for registered readers. A repository search can reveal another reader, and an engineer can check whether it is active in production.
- An escalation can save engineering time when it names the changed line, traces the dependency and leaves the reviewer a specific question to resolve.
- Unsafe approvals can reveal a missing search rule. Unnecessary escalations can reveal missing context. A human explanation helps the on-call engineer decide which change to make.

## Slides

### Who Reviews the Bots?

A schema change can pass CI and still break a consumer three teams away. Robinhood’s Chetas Joshi introduces a hybrid PR reviewer to help decide whether a bot’s change is safe to merge.

[0:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=0s) [Beside the subtitle “Building a Hybrid PR Reviewer Agent,” a schema-change pull request marked “CI passed” leads to the question “Safe to merge?”](https://assets.ai.engineer/sites/aiecode2025/sha256/fc70edc9398480928c0fcc9a5f3351dbbe4f46f31713f4787c006baa683812c2.webp)

Who Reviews the Bots?
Building a Hybrid PR Reviewer Agent

Chetas Joshi
Engineer, Data Platform · Robinhood
AIE CODE Summit 2026

PULL REQUEST
Schema change
CI passed

?
Safe to merge?

### AI now opens PRs at machine speed

An AI coding assistant can open a data-pipeline pull request in minutes. At Robinhood, schema changes and data access grants land in the same review queue. When requests pile up, the Data Platform team handles the backlog.

[2:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=121s) [Arrows connect an AI coding assistant to a stack of data-pipeline pull requests and then to the Data Platform review team. The requests cover schema changes, pipeline code, configuration changes and data access grants.](https://assets.ai.engineer/sites/aiecode2025/sha256/7d96bc721380216c6b876bc882362ef7216d30a9a7a43036d0d5e6b1b9dffceb.webp)

THE PROBLEM
AI now opens PRs at machine speed

WHAT
AI coding assistant
opens a PR in minutes

Data-pipeline PR
Schema changes
Pipeline code
Config changes
Data access grants

WHO
REVIEW OWNED BY
Data Platform
My team at Robinhood reviews these changes

Who Reviews the Bots?  2/26

### Platform context is the bottleneck

CI checks catch bugs as AI keeps opening pull requests. Deciding whether a change is safe to merge still depends on engineers who know who reads the affected table and how sensitive its data is.

[2:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=145s) [A grid of incoming AI-opened pull requests points to a box labeled “CI checks,” with the caption “catch real bugs.”](https://assets.ai.engineer/sites/aiecode2025/sha256/d7aa885ec2fc0d30f2276cdd72fc73b1a3debd684e3a2f7c465350ae81e48deb.webp)

THE PROBLEM · WHY
Platform context is the bottleneck

PR PR PR PR PR
PR PR PR PR PR
PR PR PR PR PR
PR PR PR PR PR
AI-opened PRs
keep arriving

CI checks
catch real bugs

Who Reviews the Bots?  3/26

### Platform context is the bottleneck

CI catches bugs, but AI-opened pull requests still queue up for the few reviewers who know who reads a table and how sensitive its data is. The team wants an AI reviewer that can keep pace with incoming requests.

[2:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=170s) [AI-opened pull requests pass through CI checks and enter a funnel labeled “the queue” before reaching reviewers with platform context. A goal box calls for an AI reviewer that keeps pace with incoming requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f7b38d94070958f7d3aa672b39bb867695b35f3fea4ddf87cb8e366b6e3ff9c.webp)

THE PROBLEM · WHY
Platform context is the bottleneck

PR PR PR PR PR
PR PR PR PR PR
PR PR PR PR PR
PR PR PR PR PR
AI-opened PRs
keep arriving

CI checks
catch real bugs

the queue

Reviewers with platform context
catch real bugs too, at human speed

PRs now arrive at machine speed. Review that needs platform context still moves at human speed.

THE GOAL
Build a PR reviewer, with AI, that keeps pace with how fast PRs are opened.

Who Reviews the Bots?  3/26

### Use AI only where proof runs out

Scripts check a pull request against sources like the schema registry. The agent judges the remaining risks and either approves the request or sends it to a person. Merging requires both passing checks and approval.

[4:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=279s) [Scripts check a pull request against the schema registry or data catalog and block it on failure. An agent judges the remaining questions and approves or escalates to a person. A callout links this division of work to lower AI costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/49e9310df3733349460e05a68e61acde6d3e7df43497629338fbb2bf1574f7cc.webp)

THE HYBRID DESIGN
Use AI only where proof runs out

Scripts for facts. An agent for judgment. People for the protected calls.

PR

LAYER 1 · DETERMINISTIC SCRIPTS
Prove facts against a source of truth
like the schema registry or the data catalog
Fail: CI goes red, the PR is blocked

LAYER 2 · AGENT REVIEW
Judge what scripts can't prove
rules scoped to what Layer 1 doesn't cover
Approve, or escalate to a human

AI does only the judgment work, not checks a script can prove, which keeps AI cost lower.

Who Reviews the Bots?  4/26

### Roadmap

Scripts check facts, an agent exercises judgment, and people make the protected decisions. Joshi then traces how the reviewer learns from human feedback, what results the team has seen, and what it learned.

[4:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=287s) [Five connected markers list the reviewer, human boundaries, how it learns, results, and lessons. Below them, the slide assigns fact checks to scripts, judgment to an agent, and protected decisions to people.](https://assets.ai.engineer/sites/aiecode2025/sha256/e504f239b50e0ba5b688f5dc45f2a005cc13345d34c4dffa6d92a3eceecba118.webp)

WHERE WE'RE GOING
Roadmap

We've covered the problem and the approach.

The reviewer
Human boundaries
How it learns
Results
Lessons

Three things to listen for: scripts for facts, an agent for judgment, and people for the protected calls.

Who Reviews the Bots?  5/26

### One PR, two schema changes

The pull request updates the ingestion pipeline to bring a new upstream column into the data lake table. It also widens an existing column in that table from int to bigint so it can hold larger values.

[5:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=330s) [An ingestion pipeline carries a new column from an upstream table into a data lake table. The same pull request also widens an existing lake table column from int to bigint.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d5d4fb846e9763dffaba3d8024394a9730540433841bc1a839fa68dc93d58fd.webp)

One PR, two schema changes
THE RUNNING EXAMPLE
RUNNING PR · PR OPENED

SOURCE TABLE
Upstream system
+ new column

Ingestion pipeline

DATA LAKE TABLE
Our table
+ new column
widened: int → bigint

CHANGE 1 · ADD A COLUMN
The source gains a new column, and ingestion now brings it into our lake table.

CHANGE 2 · WIDEN A COLUMN
A column the lake table already has goes from integer to big integer.

Who Reviews the Bots?  6/26

### One PR, two schema changes

The PR brings a new upstream column into the lake table and widens an existing column from int to bigint. The lineage tool lists registered readers of the widened column. Who else might read it?

[6:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=383s) [The pipeline diagram shows a new column arriving in the lake table and an existing column widening from int to bigint. Arrows lead from the lake table to registered consumers recorded in lineage and to a dashed box asking whether anyone else reads the widened column.](https://assets.ai.engineer/sites/aiecode2025/sha256/752c727b0f3cca0b6c98a500f213092d5e5eac9fb81a16de8771196efa1c74d5.webp)

One PR, two schema changes
THE RUNNING EXAMPLE
RUNNING PR · PR OPENED

SOURCE TABLE
Upstream system
+ new column

Ingestion pipeline

DATA LAKE TABLE
Our table
+ new column
widened: int → bigint

WHO READS THE WIDENED COLUMN?
Registered consumers
recorded in lineage

Any others?

CHANGE 1 · ADD A COLUMN
The source gains a new column, and ingestion now brings it into our lake table.

CHANGE 2 · WIDEN A COLUMN
A column the lake table already has goes from integer to big integer.

Who Reviews the Bots?  6/26

### Layer 1 checks facts in live systems

Scripts use the schema registry to confirm the new column exists upstream and the data catalog to verify the table’s owner and PII labels. Both checks pass for this PR. If either fails, CI blocks the merge until the issue is fixed.

[6:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=407s) [A pipeline diagram traces the new column from the upstream system to the lake table and marks the widened integer there. Below it are three headings for the check, the source of truth, and the running pull request. The space beneath the headings is empty.](https://assets.ai.engineer/sites/aiecode2025/sha256/f47907dff876fb2f2f496cbaff3a31f2293d2972b1079d0b3e71f4159d34c788.webp)

Layer 1 checks facts in live systems
LAYER 1 · SCRIPTS
RUNNING PR · CONTRACT + TAGS

Upstream system  + new column → Ingestion pipeline → Our table  + new column  widened: int → bigint

THE CHECK    SOURCE OF TRUTH    OUR RUNNING PR

Who Reviews the Bots?  7/26

[7:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=466s) [The schema registry confirms the new column exists upstream. The data catalog confirms ownership and PII labeling. Both checks pass for this pull request. A banner says CI blocks the pull request if either check fails.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f9725c5e15241d8b0f6c2c6eacf9ce3d2703e0bac8c8105d7c6186a84e7aee3.webp)

Layer 1 checks facts in live systems
LAYER 1 · SCRIPTS
RUNNING PR · CONTRACT + TAGS

Upstream system  + new column → Ingestion pipeline → Our table  + new column  widened: int → bigint

THE CHECK    SOURCE OF TRUTH    OUR RUNNING PR

DATA CONTRACTS
Schema and keys
compared with the registry
Schema registry
PASS
The new column already exists upstream, as registered

REQUIRED TAGS
Owner and PII labels
must be present on the table
Data catalog
PASS
Our table has an owner, and its PII is labeled

If either check fails, CI goes red and the PR is blocked until it's fixed. No judgment, no AI.

Who Reviews the Bots?  7/26

### Checking downstream impact and performance

When a column widens, the script checks only the downstream jobs recorded in lineage. Before merge, a staging run uses mock data at production volume to check cluster sizing and whether the job meets its freshness deadline.

[8:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=529s) [A widened column points to two jobs recorded in lineage. Alongside it, a staging test uses mocked data at production volume to check cluster sizing and whether the job meets its freshness target.](https://assets.ai.engineer/sites/aiecode2025/sha256/25b00c91c6044a338adca85f5f4f30dfe5e3f3e858a7234596d67b58a68146d7.webp)

Checking downstream impact and performance
LAYER 1 · SCRIPTS
RUNNING PR · LINEAGE CHECK

Upstream system   + new column → Ingestion pipeline → Our table   + new column   widened: int → bigint

LINEAGE BLAST RADIUS
What breaks downstream?
Widened column
RECORDED IN LINEAGE
Registered job
Registered job

Walks every downstream job the lineage tool has on record. Its view is only as complete as that record.

STAGING PERFORMANCE TEST
Will it run in time?
Runs in staging with:
Production data volume
Mocked data

Cluster sized right?
Freshness SLA met?

Catches cluster sizing and freshness SLA risk before the PR merges.

Who Reviews the Bots?   8/26

### The agent starts with intent, then reads the code

The PR description and commit messages help the agent interpret the diff. It traces the change into pipeline code and access grants across the repo. If the code contradicts the author’s account, the code wins. Layer 1 handles live queries.

[10:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=658s) [The diff, PR description and commit messages feed an agent that reads, searches and lists repository files. Two callouts explain that intent cannot dismiss a finding and that Layer 1 handles live checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/dae88669c97e72967165605cebfd143aa94580e372b86498cf702c8161d0cf8d.webp)

LAYER 2 · THE AGENT
The agent starts with intent, then reads the code

The diff
what changed

PR description
what the author meant

Commit messages
how it got there

Agent

FILE TOOLS · THE REPO ONLY
read
search
list

Pipeline code and access grants across the repo, so it can follow a change beyond the lines in the diff

Intent explains a change.
It never explains away a real finding.

No live queries
Not the catalog, lineage tool, or production state. Live facts stay Layer 1's job.

Who Reviews the Bots?  10/26

### The rules cover gaps, not facts scripts prove

The agent uses different rules for different changes. For schema changes, it searches pipeline definitions and access grants in the repository to find downstream consumers missing from lineage records.

[11:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=707s) [Below the three rule categories, an arrow labeled “code walk” leads from pipeline definitions and access grants in the repository to consumers that lineage never recorded.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5128cd5d2f3479f4b035112a44a1bdcc6ecc4a9ae36b994187636b3635811ea.webp)

LAYER 2 · THE AGENT
The rules cover gaps, not facts scripts prove

AGENT RULES · SET PER CHANGE TYPE, KNOWING WHAT LAYER 1 COVERS
Query anti-patterns
Sensitive data asks
Consumers outside lineage

WHAT THE AGENT DOES INSTEAD
THE REPO
Pipeline definitions
Access grants

code walk
Consumers lineage never recorded

Slow, tedious work for a person. The agent does it on every relevant diff.

Who Reviews the Bots?  11/26

### Why lineage missed it

Query engines A and B publish their dependencies to the lineage tool. Engine C’s dashed red arrow marks a dependency it never published, so a downstream job that relies on the table can be missing from the catalog.

[13:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=794s) [Solid arrows connect query engines A and B to a lineage tool. Engine C has a dashed red arrow marked “not published.” Each engine must publish its own lineage.](https://assets.ai.engineer/sites/aiecode2025/sha256/f95ceb03f42ebd9ef95d24f08aee41b49bbf4aea22fac2699aa7477cc5ed3936.webp)

Why lineage missed it
LAYER 2 · THE AGENT

WHY LINEAGE HAS GAPS
Query engine A
Query engine B
Query engine C
Lineage tool
not published

Each engine has to publish its own lineage. No single catalog tracks everything.

Who Reviews the Bots? 13/26

### “None found” doesn't mean “none exist”

The agent searched the repository and found an unregistered job that reads the table. Ad hoc queries and dashboards can read the same table outside the repo, so an empty search result applies only to the places the agent checked.

[13:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=839s) [For a table change, the agent can search pipeline code, access grants and jobs in the repository. External ad hoc queries and dashboards remain outside its view. The green callout limits a negative finding to the places searched.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4e1cfe45b534c866f520f03793734f0dbeaae135b3b25c25a555620321b5d5e.webp)

“None found” doesn't mean “none exist”
LAYER 2 · LIMITS
RUNNING PR · CODE WALK LIMITS

Upstream system  + new column  →  Ingestion pipeline  →  Our table  + new column  widened: int → bigint

INSIDE THE REPO · THE AGENT CAN SEARCH
Pipeline code
Access grants
Jobs that read our tables
This is where it found our unregistered job.

OUTSIDE THE REPO · INVISIBLE TO IT
Ad hoc queries
Dashboards
A consumer here never shows up in a repo search.

WHAT THE AGENT SAYS
“No consumers found in the places I searched.”

WHAT IT NEVER CLAIMS
“All clear. No consumers exist.”

Who Reviews the Bots?  14/26

[14:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=862s) [Two panels show what a repository search covers. The agent can inspect pipeline code, access grants and jobs that read the tables. Ad hoc queries and dashboards can use the same tables outside its view.](https://assets.ai.engineer/sites/aiecode2025/sha256/11e4c0f06f362f1aaef52d4977ad6fbd7e2644bc5f9926152b502dce27d39465.webp)

“None found” doesn't mean “none exist”
LAYER 2 · LIMITS
RUNNING PR · CODE WALK LIMITS

Upstream system  + new column  →  Ingestion pipeline  →  Our table  + new column  widened: int → bigint

INSIDE THE REPO · THE AGENT CAN SEARCH
Pipeline code
Access grants
Jobs that read our tables
This is where it found our unregistered job.

OUTSIDE THE REPO · INVISIBLE TO IT
Ad hoc queries
Dashboards
A consumer here never shows up in a repo search.

WHAT THE AGENT SAYS
“No consumers found in the places I searched.”

WHAT IT NEVER CLAIMS
“All clear. No consumers exist.”

Who Reviews the Bots?  14/26

### A review ends with a decision

The agent scores the risk. If it finds no serious risk, it can approve. Otherwise, a human gets the line comments and the agent’s reasoning. The pull request merges when Layer 1 checks pass and either the agent or the human approves.

[15:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=904s) [The agent comments on specific lines, explains its reasoning and scores the risk. It either approves or escalates to a person. A pull request merges when Layer 1 checks pass and the agent or a human reviewer approves.](https://assets.ai.engineer/sites/aiecode2025/sha256/5006e71d7f6e634258091c46aa4c715cbdb1df2f71ed61e74dd45faa28784c48.webp)

A review ends with a decision
LAYER 2 · THE AGENT'S OUTPUT

Agent
reviews the change

REVIEW COMMENTS
On the specific lines that matter
each with its reasoning

RISK SCORE
How severe is what it found?

APPROVE
no serious risk found

ESCALATE TO A HUMAN
with context to investigate

WHEN DOES A PR MERGE?
Layer 1 checks pass
+
The review approves
by the agent, or by a person after escalation
→ PR merges

Who Reviews the Bots?  15/26

### Update the downstream first, then merge

The agent flagged an unregistered job that could fail when writing bigint values to an int column. A reviewer confirmed the job ran in production, so the team widened the destination column to bigint before merging the PR.

[15:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=937s) [A diagram traces a widened column to an unregistered job. Three boxes show the agent flagging a write-failure risk, a person confirming the job runs in production, and the downstream table’s column being widened to bigint.](https://assets.ai.engineer/sites/aiecode2025/sha256/b189cea2cb4d778a31b53b7a5073a69071fee7e4ad6c0c3491acf14758a6baa9.webp)

Update the downstream first, then merge
THE RUNNING EXAMPLE
RUNNING PR · HUMAN DECISION

SOURCE TABLE
Upstream system
+ new column

Ingestion pipeline

DATA LAKE TABLE
Our table
+ new column
widened: int → bigint

WHO READS THE WIDENED COLUMN?
Registered consumers
all look fine

Unregistered job
target widened first

AGENT ESCALATES
Write-failure risk, flagged with its reasoning

A PERSON CHECKS
The job is live in production. The agent can't see that.

DOWNSTREAM FIRST
The other table's column is widened to bigint

Who Reviews the Bots?  16/26

### Update the downstream first, then merge

The agent flagged an unregistered job whose writes could fail. A reviewer confirmed the job ran in production. The team widened its target column to bigint first, then merged the pull request and deployed without incident.

[16:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=995s) [Four boxes trace the example from a flagged write-failure risk through confirmation that the downstream job is live. Its target column is widened to bigint first. The final box says the pull request then merges and deployment finishes without incident.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ffb98603fbf495d1d5951c7c17320633fe2c19f5b93e7f6b30c861df7cd1706.webp)

Update the downstream first, then merge
THE RUNNING EXAMPLE
RUNNING PR · HUMAN DECISION

SOURCE TABLE
Upstream system
+ new column

Ingestion pipeline

DATA LAKE TABLE
Our table
+ new column
widened: int → bigint

WHO READS THE WIDENED COLUMN?
Registered consumers
all look fine

Unregistered job
target widened first

AGENT ESCALATES
Write-failure risk, flagged with its reasoning

A PERSON CHECKS
The job is live in production. The agent can't see that.

DOWNSTREAM FIRST
The other table's column is widened to bigint

THEN OUR PR MERGES
Deployed safely. No incident.

Who Reviews the Bots?  16/26

### Some calls always stay with a person

The agent flags grants of access to sensitive data and migrations that change what the data means. A person decides both. The agent can approve other changes when the checks pass and it is confident in its verdict.

[17:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1043s) [Two amber cards assign sensitive data grants and changes to the meaning of data to a human reviewer. A green panel says the agent can approve other changes when checks pass and it is confident in its verdict.](https://assets.ai.engineer/sites/aiecode2025/sha256/54ed0c3e526c748110dc5576ea375553e0a1c956c1aaaad84f5de315f6b0e813.webp)

HUMAN BOUNDARIES
Some calls always stay with a person

The reviewer
Human boundaries
How it learns
Results
Lessons

THE AGENT CAN APPROVE
Everything else
when the checks pass and the agent is confident in its verdict

ALWAYS A PERSON · THE AGENT FLAGS, NEVER CLEARS
Sensitive data grants
Who can reach sensitive data

Migration intent
Changes what the data means, not just the mechanics

Who Reviews the Bots?  17/26

### Some calls always stay with a person

Even when checks pass, a person decides who gets sensitive data access, whether a migration should change what the data means, and how to proceed when the agent is unsure. The agent sends its risk assessment and reasoning with each case.

[17:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1048s) [Three cards identify decisions that require a person: grants of access to sensitive data, migrations that change what data means, and verdicts the agent lacks confidence in. The agent sends its risk assessment and reasoning with each case.](https://assets.ai.engineer/sites/aiecode2025/sha256/6cd5f4ac7c91c5e6192d8fa073a3b92078dfc045a1ced79d388280c8110f8790.webp)

HUMAN BOUNDARIES
Some calls always stay with a person

The reviewer    Human boundaries    How it learns    Results    Lessons

THE AGENT CAN APPROVE
Everything else
when the checks pass and the agent is confident in its verdict

ALWAYS A PERSON · THE AGENT FLAGS, NEVER CLEARS

Sensitive data grants
Who can reach sensitive data

Migration intent
Changes what the data means, not just the mechanics

Low-confidence verdicts
Unsure never gets rounded up to approve

Even when nothing concrete comes back wrong, these go to a person, with the agent's risk and reasoning attached.

Who Reviews the Bots?  17/26

### Humans teach the rules, by hand

The agent explains its verdict in a review comment. People reply with why they agree or disagree. Once a week, the person on call reads the replies collected each day and edits the rules the agent uses for its next review.

[18:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1101s) [Four connected boxes show the agent posting a verdict and reason, a person explaining their agreement or disagreement, daily collection of replies, and weekly rule edits by the person on call. An arrow carries the updated rules back to the next review.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcc85823e57190913826357fca08b23fa5eff03ef5551cba1d4cf8f5ea7ba0a2.webp)

LEARNING FROM FEEDBACK
Humans teach the rules, by hand

The reviewer
Human boundaries
How it learns
Results
Lessons

THE AGENT
Verdict + reason
in its review comments

A PERSON REPLIES
Agree or disagree
and says why

EVERY DAY
Feedback collected
replies gathered

ONCE A WEEK
Oncall updates rules
reviewed and edited by hand

Updated rules shape the next review

MANUAL TODAY
Nothing retrains itself. A person reads the feedback and edits the rules each week.

Who Reviews the Bots?  18/26

### Context can remove a false alarm

The agent flagged a proposed non-sensitive label because the source column contained personal information. The missing context was where encryption happened: the pipeline encrypted the column before writing it to the data lake.

[18:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1135s) [A three-box diagram follows a column tagged as personal information from the source database through an ingestion pipeline. The pipeline encrypts the column before writing it to the data lake, where the proposed label is non-sensitive.](https://assets.ai.engineer/sites/aiecode2025/sha256/e029e01aecb68d03b72ce6c77abc05f7422b97226c7b9422e5558613552a192c.webp)

Context can remove a false alarm

LEARNING FROM FEEDBACK
SEPARATE EXAMPLE · NOT THE RUNNING PR

SOURCE DATABASE
A column tagged PII
personal information upstream

INGESTION PIPELINE
Encrypts the column
before it writes to the lake

DATA LAKE
Encrypted column
proposed label: non-sensitive

Who Reviews the Bots?  19/26

### Grade misses and false escalations

A miss lets a bad change ship. A false escalation sends a sound pull request to a human reviewer. Escalating every pull request may avoid misses, but it leaves reviewers with the same workload.

[20:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1241s) [A two-by-two matrix compares the agent’s decision with the quality of a pull request. Approving a bad change is a miss. Escalating a sound change costs a reviewer time. The other two cells show correct approvals and escalations.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b0637e3170bac66f7af9e57e9372384810395d374cf19bb33eb971dc2253193.webp)

Grade misses and false escalations
GRADING THE REVIEWER
As of Aug 31, 2026

	PR was fine	PR was bad
Agent approved	Correct approve — nobody's time spent	MISS — a bad change ships: the dangerous one
Agent escalated	FALSE ESCALATION — a fine PR costs a person's time	Correct escalate — a person catches it

Who Reviews the Bots?  20/26

### Grade misses and false escalations

A miss lets a bad PR ship. As of Aug 31, 2026, the miss rate was about 2%, mostly from gaps in the rules. Missing context caused most false escalations, sending sound PRs to human reviewers.

[20:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1254s) [A matrix compares agent approvals and escalations with whether a PR was fine or bad. Approving a bad PR is a miss. Escalating a fine PR costs reviewer time. The slide reports about 2% misses, mostly from gaps in the rules, and attributes most false escalations to missing context.](https://assets.ai.engineer/sites/aiecode2025/sha256/cbf7648b532075f413be003d5abead278974ef0c4caef95bf6e91d4d25bec09e.webp)

Grade misses and false escalations
GRADING THE REVIEWER
As of Aug 31, 2026

	PR was fine	PR was bad
Agent approved	Correct approve — nobody's time spent	MISS — a bad change ships: the dangerous one
Agent escalated	FALSE ESCALATION — a fine PR costs a person's time	Correct escalate — a person catches it

~2%
miss rate, mostly from gaps in the rules

False escalations: no single clean number.
Mostly missing context.

Misses become new rules. False escalations shrink when the agent gets the context it was missing.

Who Reviews the Bots?  20/26

### A miss becomes a new rule

A privacy retag moved columns from the open copy to a restricted copy, breaking a downstream reader. The new rule tells reviewers to search the code for readers that still use those columns in the open copy before the retag merges.

[21:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1311s) [Retagging columns as personal information moved them from an open copy to a restricted copy. A downstream reader still needed them in the open copy. The new review rule calls for searching the code for those readers before the change merges.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4ada87d0e603e01fc49886a696824d7cb36bf2a43cd336d918000c2f1bbff78.webp)

A miss becomes a new rule
GRADING THE REVIEWER
SEPARATE MISS · NOT THE RUNNING PR

OPEN COPY
Other columns
Gone from the open copy

privacy retag moves them

RESTRICTED COPY · AFTER THE RETAG
Retagged as personal information

DOWNSTREAM READER
Reads those columns from the open copy

THE MISS
The retag looked fine and merged. The reader's columns were gone.

THE NEW RULE
When a column is retagged sensitive, search the code for anyone still reading it from the copy that's losing it.

Who Reviews the Bots?  21/26

### The PR in three beats

The scripts passed a change from int to bigint. The agent traced the code and found an unregistered job that still wrote to an int column. A reviewer confirmed the job was live. The downstream table was fixed before the PR merged.

[23:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1385s) [Script checks passed for a PR that widened a column from int to bigint. The agent found an unregistered job writing to an int column and escalated. A person confirmed the job was active, and the downstream table was updated before the PR merged.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b79c5db270e83bbca7efb1d635526bde406de2a1ea832f3d7067058ead63558.webp)

The PR in three beats
KEY TAKEAWAYS
RUNNING PR · THREE BEATS

SOURCE TABLE
Upstream system
+ new column

Ingestion pipeline

DATA LAKE TABLE
Our table
+ new column
widened: int → bigint

WHO READS THE WIDENED COLUMN?
Registered consumers
all look fine

Unregistered job
writes to an int column

SCRIPTS
The contract, the tags, the registered lineage, and performance all passed.

AGENT
The code walk found a job that lineage never knew about. It flagged the whole chain and escalated.

PERSON
A person confirmed the job was live. The downstream table was updated first, and only then did the PR merge. No incident.

Every correction a person makes can become a rule.

Who Reviews the Bots?   22/26

## Full transcript

[0:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=0s) Picture a pull request. It changes a

[0:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=3s) schema in one of our data pipelines. It

[0:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=6s) goes green in CI. The tests pass. The

[0:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=8s) linder is happy. And everything about it

[0:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=11s) looks clean. So, is it safe to merge?

[0:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=15s) Because a change like this can still

[0:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=17s) break something downstream. Maybe it

[0:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=19s) blows a freshness SLA. Or maybe it

[0:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=22s) breaks a consumer three teams away who

[0:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=24s) never saw it coming. Or maybe it exposes

[0:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=26s) data that never supposed to be

[0:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=28s) reachable. That gap between CI past and

[0:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=32s) this change was actually safe is the gap

[0:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=34s) my team has to close every day. Hello

[0:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=38s) everyone, I'm Jas. I'm an engineer on

[0:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=40s) the data platform team at Robin Hood. I

[0:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=43s) design and build the distributed data

[0:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=45s) injection and processing frameworks

[0:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=46s) behind our data lake. The ones teams

[0:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=49s) across the company rely on to make

[0:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=51s) decisions with data.

[0:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=54s) Notice the title of this talk. The bots

[0:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=56s) in it are the ones opening the pull

[0:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=58s) requests and as you will see the ones

[1:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=61s) reviewing them as well.

[1:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=64s) Today I want to show you how we answer

[1:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=66s) that question when the PRs are coming in

[1:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=68s) faster than any of us can read them.

[1:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=74s) So here's the problem in one sentence.

[1:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=76s) Pull requests to our production data

[1:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=78s) pipelines now arrive at machine speed.

[1:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=81s) But reviewing them safely still runs at

[1:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=84s) human speed.

[1:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=86s) First, a quick word on the scope of this

[1:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=88s) talk because it shapes everything else

[1:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=90s) in this talk.

[1:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=92s) AI coding assistants have made pull

[1:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=94s) requests cheap to open. A new query, a

[1:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=98s) schema change, or an access grant now

[1:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=100s) takes just a few minutes. And an

[1:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=103s) assistant can open pull requests far

[1:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=105s) faster than any team of humans ever

[1:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=108s) could.

[1:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=110s) The PRs I'm talking about today are the

[1:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=112s) PRs to our data pipelines like schema

[1:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=115s) changes, pipeline code, configuration

[1:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=118s) changes, and grants that give someone

[2:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=120s) access to data.

[2:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=123s) Data platform is the team that owns

[2:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=125s) reviewing that kind of PR. So when those

[2:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=127s) PRs pile up, they pile up on us. And the

[2:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=130s) reviewer I'm about to describe is the

[2:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=132s) system we built to handle them.

[2:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=135s) So the people who feel this are a small

[2:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=137s) group. The engineers who carry the

[2:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=139s) context to say what is safe to merge

[2:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=142s) into production and everyone who is

[2:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=144s) waiting on them.

[2:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=147s) So why is this a problem?

[2:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=150s) AI open PRs just keep arriving.

[2:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=154s) And to be fair, the safety net we

[2:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=156s) already had works. CI checks together

[2:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=160s) with a human review have always caught a

[2:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=162s) lot of real bugs like broken code,

[2:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=165s) inefficient queries and risks like

[2:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=168s) downstream breakage or sensitive data

[2:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=170s) leaks. But the human half only works at

[2:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=173s) the human speed and only with the

[2:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=175s) context a person happens to carry in

[2:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=177s) their head. Who reads this table? How

[3:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=180s) big is it? And how sensitive is the data

[3:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=183s) in it? The PRs are coming in at the

[3:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=186s) machine speed, but the number of people

[3:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=188s) who understand what is safe to merge

[3:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=190s) into a production data pipeline did not

[3:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=192s) grow at all. That's the mismatch. The

[3:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=195s) queue builds up in front of the few

[3:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=197s) people with the context and that's where

[3:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=199s) things start to slip.

[3:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=202s) So the problem we set out to solve was

[3:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=204s) this. Build a PR reviewer using AI that

[3:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=208s) can keep up with the speed at which PRs

[3:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=211s) are being opened.

[3:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=214s) So here's the whole answer before we get

[3:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=216s) into the details. Three duties, three

[3:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=219s) owners.

[3:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=220s) Scripts prove what can be proven. An

[3:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=223s) agent judges what needs context and

[3:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=226s) people own the calls only they should

[3:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=229s) make. We took a hybrid approach. Scripts

[3:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=232s) for the facts and AI for judgment. We

[3:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=235s) did not want AI reviewing every change

[3:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=238s) when some questions already have an

[4:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=240s) authoritative answer.

[4:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=242s) And that's layer one. Deterministic

[4:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=246s) scripts check a source of truth like the

[4:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=248s) schema registry or a catalog. A failed

[4:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=251s) check turns CI red and blocks the PR

[4:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=254s) until it is fixed.

[4:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=257s) Layer two is the agent. The agents rules

[4:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=260s) tell it what layer 1 already checks. So

[4:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=263s) it focuses on the risks those checks

[4:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=265s) cannot catch. It reviews the PR, scores

[4:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=268s) the risk and approves or escalates to a

[4:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=271s) person. Both these layers review the

[4:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=274s) same PR. Merge requires passing checks

[4:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=277s) and an approving review.

[4:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=280s) And this also keeps AI costs down. We

[4:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=283s) use AI for judgment, not to rediscover

[4:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=286s) facts a script can look up.

[4:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=290s) So here is where we are going today.

[4:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=293s) We have covered the problem and the

[4:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=294s) approach. Now I'll walk you through the

[4:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=296s) hybrid reviewer we built and the

[4:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=298s) decisions we deliberately keep with the

[5:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=300s) people. After that how the reviewer

[5:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=304s) learns from the human feedback and what

[5:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=306s) results we have seen so far. And we'll

[5:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=308s) finish with the lessons we would pass on

[5:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=310s) to you.

[5:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=312s) Three things to listen for as we go.

[5:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=315s) Scripts for facts and agent for the

[5:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=317s) judgment and people for the protected

[5:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=320s) calls. I'll come back to all three at

[5:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=322s) the end.

[5:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=324s) So to make this concrete, let me show

[5:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=327s) you the one PR will follow through the

[5:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=329s) rest of this talk.

[5:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=333s) This PR makes two changes to a table in

[5:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=336s) our data lake. It adds a new column and

[5:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=339s) widens an existing column from integer

[5:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=341s) to big integer. It's the kind of PR we

[5:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=343s) see all the time. On the left is an

[5:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=346s) upstream source table. Our injection

[5:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=348s) pipeline reads from it and writes into a

[5:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=351s) table in the data lake. call it our

[5:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=353s) table.

[5:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=355s) The first change, the source has gained

[5:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=357s) a new column and this PR makes injection

[6:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=361s) bring that new column into our lake

[6:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=363s) table.

[6:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=365s) The second change is to a column of a

[6:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=367s) table already has

[6:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=369s) its type gets widened from integer to

[6:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=372s) big integer so it can hold bigger values

[6:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=376s) on its own. That sounds completely

[6:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=377s) harmless. Widening a type is about the

[6:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=380s) safest schema change you can make,

[6:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=382s) right?

[6:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=384s) Well, whenever a column changes, the

[6:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=386s) real question is who reads it

[6:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=388s) downstream? Some of those readers are

[6:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=390s) registered in our lineage tool. So, we

[6:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=393s) know about them.

[6:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=395s) But keep a second question in the back

[6:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=396s) of your mind. Are those all of them?

[6:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=400s) Hold on to this picture. We will come

[6:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=403s) back to this exact diagram each time the

[6:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=405s) PR moves forward through the reviewer.

[6:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=409s) So let's start with layer one. Layer one

[6:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=412s) is the scripts and their whole job is

[6:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=414s) checking facts in the live systems.

[6:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=417s) Things the code alone cannot tell you.

[7:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=420s) Here are the two of those checks and

[7:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=423s) what they said about our running PR.

[7:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=427s) First data contracts. When a PR changes

[7:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=430s) a schema, a script compares the schema

[7:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=433s) and keys against a source of truth, the

[7:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=436s) schema registry.

[7:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=438s) for our PR. The question is simple. Does

[7:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=440s) that new column actually exist upstream

[7:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=443s) as registered in the registry? It does.

[7:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=446s) So that check passes.

[7:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=449s) Second require tags. Every table needs

[7:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=453s) an owner and any PII has to be labeled

[7:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=455s) in the catalog. The script checks the

[7:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=458s) catalog directly. Does our table have an

[7:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=461s) owner? Yes. Is its PII labeled? Yes.

[7:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=466s) pass again.

[7:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=468s) And if either of those had failed, there

[7:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=470s) is no debate and no model involved. CI

[7:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=473s) goes red like any other failing checks

[7:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=476s) and the PR is blocked until it is fixed.

[8:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=480s) Contracts and tags are two of the layer

[8:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=482s) 1 checks. Next, the two that look at

[8:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=485s) downstream impact and performance. So

[8:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=488s) these are the other two layer 1 checks.

[8:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=490s) They look past the table itself, what

[8:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=493s) happens downstream and whether the job

[8:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=495s) will still run in time.

[8:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=499s) First, the lineage blast radius. When a

[8:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=502s) column changes, like the one our PR

[8:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=504s) widens, a script walks every downstream

[8:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=507s) job the lineage tool has on record for

[8:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=509s) that column and asks a simple question.

[8:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=512s) Does any of them break? It's a real

[8:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=514s) check against a real system. But notice

[8:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=517s) the boundary I have drawn. It can only

[8:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=520s) see what is recorded in lineage. Its

[8:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=523s) view of downstream is exactly as

[8:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=526s) complete as that record. No more.

[8:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=530s) Second, the performance. A correct

[8:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=532s) change can still be too slow or too

[8:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=534s) expensive. So the CI performance test

[8:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=537s) runs the change in staging at the same

[9:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=540s) data volume as production using mock

[9:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=542s) data.

[9:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=544s) So that tells us two things before the

[9:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=546s) PR merges. whether the cluster is sized

[9:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=548s) right for the job and whether the job

[9:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=550s) will still make its freshness SLA. Those

[9:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=554s) are the kind of problems that otherwise

[9:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=556s) show up weeks later as an incident. So

[9:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=559s) that's the full set of layer 1 checks.

[9:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=562s) Let's see what they said about our PR.

[9:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=567s) So back to our running PR. Same picture

[9:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=570s) as before. Here's what the layer one

[9:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=572s) said about it.

[9:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=575s) The data contract check went to the

[9:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=577s) schema registry and asked whether the

[9:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=579s) new column really exists upstream as

[9:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=581s) register. It does and the check passes.

[9:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=586s) The leanage blast radius check walked

[9:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=588s) every registered consumer of the widen

[9:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=590s) column. The jobs the lenage tool has on

[9:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=593s) record. None of them break when the

[9:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=595s) column goes from integer to big integer.

[9:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=597s) They all look fine. Along with that tags

[10:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=601s) pass, performance passes. Every check is

[10:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=604s) green.

[10:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=605s) As far as the scripts can see, this PR

[10:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=607s) is safe. But I chose those words

[10:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=610s) carefully. As far as the scripts can

[10:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=612s) see, everything on the slide is a

[10:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=615s) statement about the registered picture,

[10:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=617s) the jobs and the columns our systems

[10:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=619s) have a record of. And keep that in mind.

[10:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=623s) It is not the whole story.

[10:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=626s) Now let's look at the agent with the

[10:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=628s) diff. What actually changed?

[10:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=631s) The PR description and the commit

[10:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=633s) messages explain the author's intent,

[10:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=635s) but intent never explains away a real

[10:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=638s) finding. If the description says it is

[10:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=641s) safe and the code says otherwise, the

[10:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=643s) code wins.

[10:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=646s) File tools let the agent read, search,

[10:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=648s) and list the files across the

[10:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=649s) repository. It follows changes beyond

[10:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=652s) the diff into pipeline definitions and

[10:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=654s) access grants looking for the

[10:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=656s) relationships the author may have

[10:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=658s) missed.

[11:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=661s) These are the repository repository

[11:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=663s) tools only. The agent cannot query the

[11:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=665s) live catalog lineage tool or production

[11:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=668s) state. Life facts stay layer 1's job.

[11:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=672s) The agent reasons over the code. So why

[11:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=675s) walk that code for a schema change?

[11:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=678s) Because registered lineage is not the

[11:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=680s) whole picture.

[11:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=683s) The rules define where the judgment

[11:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=686s) belongs. They are configured by the

[11:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=688s) change type in the PR knowing what layer

[11:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=690s) 1 already covers. They target query anti

[11:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=693s) patterns, sensitive data asks and

[11:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=696s) consumers outside lineage.

[11:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=699s) For example, an unpartitioned cross join

[11:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=701s) can run correctly and still hammer a

[11:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=703s) cluster. A grant can look ordinary and

[11:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=706s) still open sensitive data.

[11:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=708s) And for the schema changes, the agent

[11:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=711s) walks the pipeline definitions and

[11:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=713s) access grants in the repository to find

[11:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=715s) consumers that were not registered. It's

[11:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=718s) a tedious search uh a person would

[12:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=720s) otherwise do by hand.

[12:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=722s) Let's return to our PR and see what that

[12:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=724s) search found.

[12:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=727s) So back to our running PR. Same picture.

[12:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=730s) Layer 1 said every registered consumer

[12:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=733s) of the vid column looks fine. Now the

[12:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=736s) agent walks the code

[12:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=738s) and it finds a job that lineage never

[12:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=741s) knew about. It's right there in the

[12:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=743s) repository. It reads our widen column

[12:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=746s) from our late table but it was never

[12:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=748s) registered in the lineage which is

[12:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=750s) exactly why the layer 1 check could not

[12:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=752s) see it.

[12:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=754s) And that job takes the value and writes

[12:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=756s) it into another table. And in that other

[12:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=759s) table, the column is still an integer.

[12:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=763s) Now follow it through. Our column is a

[12:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=765s) big integer. Now the first time a value

[12:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=768s) comes through that's bigger than an

[12:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=770s) integer can hold that right fails. A

[12:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=774s) valid schema change green in CI passes

[12:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=777s) every registered check and a downstream

[13:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=780s) job still breaks.

[13:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=783s) The agent flagged exactly this chain in

[13:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=785s) its review comment with the reasoning.

[13:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=787s) So a person could see the whole path

[13:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=789s) from our column to the failing right.

[13:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=792s) We'll get to what happened next in a

[13:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=794s) moment.

[13:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=796s) So why did the lineage miss that job?

[13:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=799s) Because the enterprise data world is

[13:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=800s) messy. Workloads run across different

[13:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=803s) query engines and each engine has to

[13:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=805s) publish its own lineage. If one does not

[13:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=808s) publish, that relationship is missing

[13:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=810s) from the picture. There's a single

[13:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=812s) catalog that automatically sees

[13:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=814s) everything. So the registered picture is

[13:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=816s) useful but it is not the whole picture.

[13:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=821s) Now I don't want to oversell the

[13:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=823s) codewalk. It's genuinely useful but it

[13:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=826s) cannot see everything.

[13:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=828s) The agent can search what is in the

[13:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=830s) repo. Pipeline code access grants and

[13:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=833s) the jobs defined there that read our

[13:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=836s) tables. That's where it found our

[13:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=838s) unregistered job. But plenty of data use

[14:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=842s) lives outside the codebase. Ad hawk

[14:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=844s) queries that someone runs by hand.

[14:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=846s) Dashboards, those are completely

[14:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=848s) invisible to a repository search. A

[14:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=851s) consumer out there never shows up, no

[14:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=853s) matter how carefully the agent looks. So

[14:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=856s) we make the agent honest about that

[14:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=857s) limit. If it does not find anything, it

[14:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=860s) does not get to say no downstream

[14:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=862s) consumers exist. It says it found none

[14:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=865s) in the places it searched. None found is

[14:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=868s) not the same as none exist and the agent

[14:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=871s) never gets to claim an allcle.

[14:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=876s) So the agent has walked the code. What

[14:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=878s) does it actually produce? A review the

[14:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=881s) same shape a good human reviewer would

[14:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=883s) give you.

[14:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=885s) First review comments on the specific

[14:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=887s) lines that matter each with its

[14:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=890s) reasoning. Not awake looks risky but

[14:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=893s) this line changes this column. And here

[14:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=896s) is the job that reads it.

[14:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=898s) Second, a risk score. How severe is what

[15:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=901s) it found. That's what drives the next

[15:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=904s) step. Then it makes a call. If there is

[15:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=907s) no serious risk, it can approve. If

[15:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=909s) there is a risk that needs a person, it

[15:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=912s) escalates to a human and the comments

[15:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=914s) and reasoning go with it. So the person

[15:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=917s) has the context to investigate instead

[15:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=919s) of starting from scratch.

[15:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=922s) And to be clear about what it takes to

[15:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=924s) merge, layer 1's checks have to pass and

[15:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=928s) the review has to approve either the

[15:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=930s) agent or after an escalation a person.

[15:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=933s) Both have to say yes. Now let's see what

[15:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=936s) that looked like for our PR.

[15:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=940s) Here is how our running PR actually

[15:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=942s) ended. Same picture. The registered

[15:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=945s) consumers look fine and the core walk

[15:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=948s) found an unregistered job that writes

[15:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=950s) our widened column into a table where it

[15:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=952s) is still an integer

[15:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=955s) because the agent found a real

[15:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=956s) downstream break. It scored the risk

[15:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=958s) high and escalated to a human with the

[16:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=961s) whole chain spelled out in its review

[16:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=963s) comment. Now the human does the part the

[16:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=966s) agent cannot. The agent only reasons

[16:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=969s) over the repo. It cannot see whether the

[16:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=971s) job is actually running in production.

[16:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=973s) human reviewer checked and confirmed

[16:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=976s) that the consumer the agent found was

[16:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=978s) live and active. So this was not a

[16:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=981s) theoretical risk. The PR waited

[16:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=985s) the f the fix was about order not about

[16:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=988s) rejecting the change. The downstream

[16:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=991s) table was updated first so its column

[16:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=993s) could handle the wider type

[16:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=996s) and only then did our PR merge and

[16:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=998s) deploy. No incident. The scripts in

[16:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1001s) layer 1 said safe. The agent found the

[16:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1004s) gap and a human made the call.

[16:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1009s) And that was not a one-time thing. Most

[16:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1012s) changes can be cleared by the agent.

[16:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1014s) When the checks pass and the agent is

[16:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1016s) confident in its verdict, it approves on

[16:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1018s) its own. But there's a whole category of

[17:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1020s) changes where the agent never gets to

[17:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1022s) say approved. And no matter how

[17:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1024s) confident it sounds,

[17:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1026s) grants touching sensitive data are one.

[17:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1030s) Who can reach sensitive data is a

[17:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1032s) decision we want a person to own every

[17:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1034s) time.

[17:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1036s) A migration that looks like it changes

[17:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1038s) intent, not just mechanics, is another.

[17:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1041s) If a change alters what the data means

[17:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1044s) rather than just how it is moved or

[17:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1046s) stored, a person decides.

[17:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1050s) And if the agent itself is not confident

[17:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1052s) in its own verdict, that's the third. It

[17:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1055s) does not get to round up to approve just

[17:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1057s) because nothing concrete came back

[17:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1059s) wrong. In all the all the three cases,

[17:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1062s) its job is only able to surface the risk

[17:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1065s) and its reasoning. The decision stays

[17:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1067s) with a person.

[17:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1070s) That is one PR. Now, how does this hold

[17:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1073s) up across every PR we see? And how does

[17:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1075s) the reviewer get better over time?

[17:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1079s) How do we know the flags are useful or

[18:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1082s) whether a rule change helps? We start

[18:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1085s) with the agents verdict and its

[18:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1086s) reasoning in the review comments.

[18:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1089s) A person replies agreeing or disagreeing

[18:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1092s) and explaining why. That explanation is

[18:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1095s) what makes the feedback useful rather

[18:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1097s) than just a vote.

[18:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1099s) Those replies are collected every day.

[18:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1103s) Once a week, the on call reads the

[18:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1105s) feedback and updates the review rules by

[18:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1107s) hand. Those rules shape the next review.

[18:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1111s) I know this is manual today. Nothing

[18:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1113s) retrains itself and no rule changes

[18:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1115s) automatically. But we have an idea to

[18:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1118s) let an incident investigation agent help

[18:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1121s) with that weekly pass. But it is not

[18:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1123s) built. Without the feedback, every rule

[18:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1126s) change would be a guess. So let me show

[18:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1128s) you an example where a human corrected

[18:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1130s) the agent and we use that feedback to

[18:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1133s) improve its next review.

[18:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1137s) So here's a real one. And note the label

[19:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1140s) in the corner. This is a separate

[19:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1142s) example, not our schema change PR.

[19:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1146s) A column was marked as personal

[19:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1148s) information in the source database.

[19:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1150s) But by the time it lands in the data

[19:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1152s) lake, the injection pipeline has already

[19:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1154s) encrypted it. So the PR proposed

[19:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1157s) labeling it nonsensitive in the data

[19:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1159s) lake even though the upstream data is

[19:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1161s) still PII.

[19:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1163s) The agent flagged this. From where it

[19:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1166s) stood, the upstream says PII, so a

[19:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1168s) nonsensitive label looked wrong. The

[19:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1171s) author pushed back and asked the agent

[19:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1174s) to look at how similar columns on the

[19:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1175s) other tables were already classified

[19:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1178s) instead of going purely off the upstream

[19:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1180s) tagging.

[19:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1182s) A human agreed with the author. The

[19:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1185s) encryption before write made the

[19:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1186s) nonsensitive label look okay for this

[19:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1189s) column. And that kind of correction is

[19:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1191s) exactly the signal that teaches the

[19:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1193s) rules to get smarter because the next

[19:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1196s) time a similar column comes through that

[19:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1198s) context can stop a needless escalation.

[20:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1202s) One caution though, this depends on the

[20:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1204s) specific pipeline encrypting before the

[20:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1207s) right. It's not a blanket rule that

[20:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1209s) encryption always makes data

[20:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1211s) nonsensitive.

[20:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1214s) So how do we know whether the reviewer

[20:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1216s) is actually doing a good job?

[20:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1218s) We look at two kinds of mistakes. Bad

[20:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1221s) PRs it approved and fine PRs it

[20:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1224s) unnecessarily send to a person. That

[20:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1227s) matrix separates four possibilities.

[20:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1231s) A fine PR approved and a bad PR

[20:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1233s) escalated are the two correct outcomes.

[20:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1236s) A bad PR approved is a miss. That's

[20:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1239s) dangerous because the change ships.

[20:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1242s) A fine PR escalated is a false

[20:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1244s) escalation that costs a person's time. A

[20:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1248s) gate that escalates everything may avoid

[20:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1251s) misses, but it has not reduced the

[20:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1253s) review load.

[20:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1256s) As of August 31st, 2026, our miss rate

[20:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1258s) was about 2%.

[21:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1261s) Mostly from the gaps in the rules rather

[21:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1263s) than the failures of the judgment. We

[21:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1266s) don't have a clean single false

[21:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1267s) escalation number. But the dominant

[21:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1270s) cause is missing context like the

[21:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1272s) encrypted column example that I just

[21:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1274s) gave you.

[21:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1276s) Misses give us new rules and false

[21:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1278s) escalation show us context the reviewer

[21:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1280s) needs. So here is one of those misses.

[21:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1284s) Again know the label. This is a separate

[21:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1286s) example, not our schema change PR.

[21:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1289s) We have an open copy of the sum of some

[21:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1292s) data. some of its columns are about to

[21:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1294s) be rettagged and there is a downstream

[21:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1297s) reader that reads those columns from the

[21:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1299s) open copy.

[21:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1301s) Privacy retag those columns as personal

[21:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1303s) information. When that happens, they

[21:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1306s) move into a more restricted copy of the

[21:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1308s) data and they are gone from the open

[21:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1310s) copy.

[21:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1312s) The retag looked fine and it merged. But

[21:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1315s) what nobody checked was whether anything

[21:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1317s) downstream was still reading those

[21:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1318s) columns from the old open copy.

[22:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1321s) something was. And once the columns were

[22:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1323s) gone, that reader broke. That's a miss.

[22:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1327s) A PR that should have been escalated got

[22:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1329s) approved.

[22:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1331s) It wasn't a failure of judgment. It was

[22:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1333s) a gap in the rules. A coroner case

[22:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1336s) nobody had written down. So, it became a

[22:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1338s) new rule. When a column gets retagged as

[22:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1341s) sensitive, search the codebase for

[22:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1343s) anyone still reading it from the copy

[22:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1345s) that's about to lose it. And that's how

[22:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1347s) the rules get better. the misses get

[22:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1350s) found and get turned into new rules.

[22:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1354s) So let's pull it all together. You

[22:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1357s) remember the pull request from the

[22:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1358s) start, the one that was green in CI.

[22:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1362s) Three things happened to it.

[22:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1364s) First, scripts proved what a source of

[22:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1367s) truth could prove. The contracts, the

[22:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1370s) tags, the registered lineage and

[22:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1372s) performance all passed.

[22:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1374s) Second, the agent read the code and

[22:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1377s) found what the scripts could not see. A

[22:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1379s) job that Lineage never knew about,

[23:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1381s) quietly waiting to fail on a wider

[23:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1383s) column, it flagged the whole chain and

[23:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1385s) escalated. Third, a person confirmed the

[23:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1389s) job was live. Uh, the downstream table

[23:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1391s) was fixed first and only then did the PR

[23:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1394s) merge. No incident. And behind all of

[23:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1397s) that, the reviewer keeps getting better

[23:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1400s) because every correction a person makes

[23:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1402s) can become a rule. These results are as

[23:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1405s) of August 31st, 2026. So you might ask,

[23:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1409s) Jas, does it all work?

[23:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1412s) Yes, about 60% of the data pipeline PRs

[23:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1415s) merged without a human review.

[23:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1418s) We have seen about 75% less manual

[23:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1420s) review effort. The agent specific

[23:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1423s) comments and reasoning help focus the

[23:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1425s) review rather than making a person

[23:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1428s) reconstruct every risk from scratch.

[23:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1431s) The time saved comes to about 132

[23:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1435s) engineering hours a month. Incidents

[23:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1437s) tied to these PRs are down about 30%.

[24:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1441s) And that does not mean human free

[24:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1443s) review. The protected decisions still

[24:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1446s) reach a person by design. The goal is

[24:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1449s) less review work, not removing

[24:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1451s) accountability. And those are the

[24:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1454s) results. Now, let's look at the

[24:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1456s) architecture choices we got wrong.

[24:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1460s) Here is what we got wrong today. Layer 1

[24:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1463s) and layer 2 are separate systems

[24:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1466s) triggered independently and running in

[24:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1468s) parallel. Each produces its own result

[24:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1472s) that seemed like an obvious design when

[24:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1473s) we built it,

[24:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1475s) but neither sees the others pass fail or

[24:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1478s) outage state. Layer 2 knows what the

[24:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1481s) scripts are configured to cover, not

[24:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1483s) what happened when they ran. Merge still

[24:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1486s) needs passing checks and an approving

[24:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1488s) review, but no single system has a full

[24:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1491s) runtime picture.

[24:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1493s) The agent also cannot call live layer 1

[24:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1496s) checks. Its tools only read, search and

[24:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1499s) list the repository files. If a judgment

[25:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1502s) needs a live fact from a live production

[25:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1505s) system, it cannot fetch that fact

[25:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1507s) itself.

[25:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1509s) When one site cannot finish, we wait for

[25:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1512s) a human. There is no automatic fallback.

[25:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1515s) These are current limitations and they

[25:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1517s) motivate the proposal. On the next

[25:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1519s) slide, platform team is exploring one

[25:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1522s) entry point. This is a proposal and not

[25:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1525s) what's running today. The left panel is

[25:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1527s) the current design and the right is

[25:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1530s) where we want to go. A classifier looks

[25:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1533s) at the whole PR then decides which

[25:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1535s) changes to review separately and which

[25:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1537s) to review together.

[25:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1539s) This is a whole PR decision, not

[25:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1541s) independent routing of each file.

[25:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1544s) The agent would also get the layer 1

[25:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1546s) checks as collable tools alongside its

[25:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1549s) repository tools. When judgment needs a

[25:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1552s) live fact, it could request the

[25:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1554s) appropriate check instead of being

[25:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1555s) limited to code alone.

[25:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1558s) The aim here is one system with the

[26:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1560s) whole PR in view. We haven't built this

[26:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1562s) yet. So, it's a direction, not a result.

[26:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1566s) Let me close with what we would carry

[26:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1568s) forward.

[26:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1570s) So, here is the whole talk in three

[26:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1571s) lines.

[26:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1573s) Deterministic evaluations. First, if

[26:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1576s) something can be proven against a source

[26:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1577s) of truth, let a script prove it. Don't

[26:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1580s) spend a judgment call on a yes or no

[26:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1582s) fact. It's not just cleaner, it's

[26:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1585s) cheaper, too. Every fact a script proves

[26:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1588s) is review work that never needs AI. So,

[26:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1590s) the cost of running this stays down.

[26:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1594s) Flag never clear. Not for every PR, but

[26:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1598s) for the changes that actually scare

[26:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1600s) people, like sensitive grants, migration

[26:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1602s) intent, and low confidence verdicts.

[26:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1605s) There, the agent's job stops at

[26:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1608s) surfacing the risk and the reasoning

[26:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1610s) behind it. The decision stays with the

[26:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1613s) person every time.

[26:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1616s) Count the false escalations. A gate that

[26:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1619s) escalates everything is not actually

[27:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1621s) saving anyone any time. It's just moving

[27:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1623s) the work around. measure that cost the

[27:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1626s) same way you measure a miss. So I

[27:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1629s) started with a pull request that was

[27:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1631s) green in CI and a problem PRs arriving

[27:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1634s) at machine speed and too few people with

[27:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1636s) the context to review them safely. We

[27:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1639s) built a hybrid reviewer scripts for the

[27:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1641s) facts and agent for judgment and people

[27:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1644s) for the protected calls. We walked one

[27:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1646s) PR through all three saw it get lighter

[27:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1649s) without going human free and saw how it

[27:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1652s) learns from the corrections. We told you

[27:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1654s) what we got wrong and we left you with

[27:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1656s) three lessons. Prove what you can

[27:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1659s) cheaply. Flag and never clear the scary

[27:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1662s) changes. And count the false

[27:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1664s) escalations.

[27:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1666s) So who reviews the bots? Scripts for

[27:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1669s) everything a source of truth can prove,

[27:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1671s) an agent for everything that needs

[27:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1673s) context and judgment, and humans for the

[27:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1676s) calls only they get to make. Thank you

[27:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1679s) for listening.
