# AI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, Uber

Preetam Dwivedi, Uber · Manjari Akella, Uber

[Talk page](https://ai.engineer/talks/hX140a5klW0-ai-writes-code-faster-than-uber-merge) · [Video](https://www.youtube.com/watch?v=hX140a5klW0)

## AI Writes Code Faster Than Uber Can Merge It

A 44% rise in hourly merge requests pushed Uber’s 95th-percentile land time from 30 minutes to three or four hours. To bring it back down, the developer platform team changed which requests could run together and which checks could wait until after merge.

### How two green pull requests break trunk

[0:04](https://www.youtube.com/watch?v=hX140a5klW0&t=4s)

Uber’s merge queue has to answer a question that a passing pull request leaves open. Will this change still build after someone else’s change lands? Preetam Dwivedi and Manjari Akella work on that problem across six language-based monorepos. More than 5,000 developers contribute over 36,000 commits and about 27,000 pull requests a week across those repositories. The Go monorepo is the largest and the focus of their account.

Take two developers who open PRs A and B against the same commit on trunk. Both builds pass. A merges, and the build for its merge commit passes too. What did B’s green check establish? CI tested B on the original base. When B merges, trunk contains A and B together for the first time, and that build fails. Both developers had good reasons to expect their changes to work. Neither had a test result for the state they actually produced.

SubmitQueue, Uber’s merge queue, brings that combined state into CI before it reaches trunk. With two requests, there are three candidate futures worth considering: A alone, B alone, and A plus B. A speculative build checks one of those futures while the changes are still queued. If the combined build passes, the queue can land the pair with evidence for the state it will create. If it fails, the two individual passes are insufficient to land that combination safely.

![Pull requests A and B branch from base H. H+A and H+B each pass CI. Neither pull request has landed, and their combination remains untested.](https://ai.engineer/api/talk-essay-media/fd0914c9c73cede9bc032ec7ead64c6ca23e1927913859da4dfd027aae059fd7)

Each pull request passes against its own base. Neither test run exercises the combined change.

### How SubmitQueue kept a green trunk moving

[3:51](https://www.youtube.com/watch?v=hX140a5klW0&t=231s)

Testing possible futures creates its own scheduling problem. As more requests arrive, which ones need a combined build? Which speculative build should get resources first? The original queue used three optimizations.

- **Target analysis.** The queue identifies the build and test targets a change affects. Two developers working on different services can have disjoint sets of targets. The queue can build and test those changes independently and merge them without making one wait for the other.
- **Probabilistic speculation.** Build history and characteristics of the incoming change provide a score for how likely it is to succeed. That score helps prioritize the speculative builds most likely to be needed to land the change. The build itself still provides the pass or fail result.
- **Bypassing large changes.** A slow build can already be in progress when a smaller change arrives. If the smaller change can build and land first, the queue lets it pass. Arrival order alone needn’t hold a ready change behind a long build.

### At 130 requests an hour, the long wait was measured in hours

[5:33](https://www.youtube.com/watch?v=hX140a5klW0&t=333s)

How much more traffic did it take to turn a half-hour wait into an afternoon? In January 2026, the queue handled roughly 90 requests an hour, with a 95th-percentile land time of 30 minutes. By March it was handling 130 requests an hour, a 44% increase that the team attributed primarily to AI-generated changes. At that rate, P95 land time reached three to four hours. The queue still kept trunk green. The cost showed up in how long developers at the slow end of the distribution waited to merge.

### Admitting cheap changes without making library owners schedule their own merges

[8:04](https://www.youtube.com/watch?v=hX140a5klW0&t=484s)

The first change put a gate at the entrance to the active queue. One way to estimate the cost of a request was already available from target analysis. A change affecting a few targets leaves room to build unrelated changes alongside it. A change to a widely used library can affect much of the build graph. It can hold resources while the queue validates it against many other requests. A small edit to that library can therefore be an expensive merge.

During peak hours, the gate admitted cheap changes for the usual builds and tests. It rejected expensive ones and left their authors to retry outside peak hours. The business-hours land rate rose from 130 to about 160 an hour, a gain of over 20%. That policy worked particularly well for developers changing individual services. For a library maintainer, routine work could repeatedly trigger a rejection. The gate had effectively given that maintainer a second job as the scheduler.

A secondary queue took over the retry. When an expensive request arrived during peak hours, the gate persisted it in the deferred queue. Once capacity permitted, for example after the peak, the system replayed that same request into active SubmitQueue. The developer could submit once and go home. The expensive change still had to wait for an opening, but nobody had to wake up early or stay late to press the button.

### Two weak links tie six requests to one queue

[12:19](https://www.youtube.com/watch?v=hX140a5klW0&t=739s)

Admission gating controlled what entered the queue. A separate constraint governed what the admitted requests could do. Imagine six requests, A through F, each touching a set of targets. Draw a link between two requests when their sets overlap. A shares a target with B, B shares one with C, and further overlaps connect the rest. A and F can end up in the same connected component without sharing a target themselves. Under the strict conflict rule, the queue schedules the whole component as one serial group.

Conflict relaxation gives each link a strength. SubmitQueue uses Jaccard scoring for dependency overlap, which measures the fraction of distinct targets across two sets that they share. It also accounts for new dependency edges introduced by the changes. The combined score ranges from zero to one, with higher values indicating stronger overlap. The scheduler keeps the stronger links and can drop weaker ones to meet a configured parallelism target. The score measures overlap. An actual build is still needed to find out whether the combined changes work.

Which links would free enough work to run in parallel? In the six-request example, a target of at least 50% parallelism calls for at least three independently working queues. The two weakest edges are B–C and B–E. Dropping those two scheduling constraints splits the single component into three components of two requests each. Each pair still has an ordering constraint inside its own queue, while the three queues can make progress at the same time. The shared targets still exist in the code.

![Jaccard similarity divides shared targets by distinct targets. The ranking score takes the largest of three values: direct target similarity and two half-weighted similarities comparing each side’s new dependency edges with the other side’s targets.](https://ai.engineer/api/talk-essay-media/17cddaaf1dcbdf9ed2da61f80de815ea534131a8c52cf43d5e8590fa5e96a554)

The score divides the number of shared targets by the number of distinct targets across the two builds.

### Assign conflicts on arrival to preserve speculative builds

[16:19](https://www.youtube.com/watch?v=hX140a5klW0&t=979s)

When should the scheduler choose which edges to relax? A global calculation could reconsider the entire conflict graph whenever work arrived or finished. But suppose a speculative build was already running for a combination chosen under the previous graph. Reassigning the requests could make a different combination the one that matters. Frequent updates would make it harder to use work the queue had already scheduled.

Uber chose a local decision at entry. On arrival, a request is compared with everything already in the queue. The scheduler decides which of the arriving request’s edges to keep, then reuses that assignment. The next arrival gets its own comparison. Existing requests keep the relationships that their speculative builds assumed, so those builds can still help the requests land.

Conflict relaxation delivered the largest throughput gain of the three changes. Peak throughput rose from about 160 lands an hour to nearly 250, sometimes higher, an increase of over 50%. It also changed the promise SubmitQueue could make. Two requests separated by a dropped edge could each pass pre-merge validation and land before anything tested their combination. Uber had deliberately reopened the possibility of the A-plus-B failure.

### Stovepipe tracks the last trunk state that passed

[19:15](https://www.youtube.com/watch?v=hX140a5klW0&t=1155s)

Return to A and B. SubmitQueue relaxes their conflict, tests each independently, then lands A followed by B. The combined build fails on trunk. Stovepipe, Uber’s post-merge validation system, checks the code as it actually landed. That gives consumers two references on the same history. `main` follows the latest landed commit. Verified main follows the latest trunk state that Stovepipe has validated successfully.

How far apart can the two references move? In a fresh example, both start at known-good commit A. Then B, C and D land, moving `main` to D. Stovepipe checks the state produced by B through D on top of A. The build fails, so verified main stays at A. When E, F and G land, the next check covers B through G on top of A, including the changes from the failed check. This time the build passes. Verified main can advance directly from A to G even though the earlier state at D failed.

Keeping a usable pointer takes one action on failure: leave it at the last passing commit. Repairing trunk takes human work. Stovepipe pages the on-call engineer, who prepares a fix or a revert and sends it through SubmitQueue. Once the repair lands, Stovepipe validates the resulting state. A passing check lets verified main move forward again.

![On a history labeled A through G, both main HEAD and verified/main HEAD point to the validated commit A. Commits B through G are faded.](https://ai.engineer/api/talk-essay-media/cd2ae7b7a3e2787ef267ef8dfab3fe3a62dd4b115fa207e2dd8609f3ec3e4dcd)

The last verified trunk commit and the current trunk head are separate references. Validation moves the verified reference forward.

Where do the two references point after each build?
Follow the top row for what has landed and the bottom row for what consumers can use as verified. At D the references diverge. The passing check at G brings them back together.
- main at A — Known-good starting commit.
- main at D — B, C and D have landed on A.
- main at G — E, F and G have since landed.
- Verified main at A — Both references start together.
- Still at A — The integrated state at D fails.
- Verified main at G — The integrated state at G passes.
- main at A → main at D (Land B through D)
- main at D → main at G (Land E through G)
- main at D → Still at A (Check B–D on A · fail)
- Verified main at A → Still at A (Hold pointer)
- main at G → Verified main at G (Check B–G on A · pass)
- Still at A → Verified main at G (Advance pointer)

### Back to a 30-minute P95, with recovery work still to do

[23:42](https://www.youtube.com/watch?v=hX140a5klW0&t=1422s)

The team’s final peak figure was roughly 300 lands an hour while maintaining a P95 land time of 30 minutes. That is about 233% more lands per hour than January’s baseline of 90, at the same reported P95. SubmitQueue and Stovepipe were both being developed in the open. The next engineering decisions concerned build cost, the scope of a green result, and the time spent recovering from a red one.

- **Batch related changes.** SubmitQueue could group related requests so that fewer builds cover the same work. The aim is to land more changes with the available build resources.
- **Track green states by service or project.** A single verified-main pointer describes the whole repository. A service-level result could tell a team that its service still passes at a commit where another part of the repository fails. Rollouts of healthy services could then proceed while the affected project recovers.
- **Automate fixes and reverts.** The on-call engineer still owns the repair after a failed post-merge check. The team wanted to automate preparing a fix or revert and landing it through SubmitQueue to shorten the time trunk spends broken.

Uber’s hybrid approach gives a merge and a verified commit different meanings. Before merge, SubmitQueue still checks individual changes and the combinations linked by stronger conflicts. After merge, Stovepipe runs exhaustive validation on the state those changes actually produced. A developer can see a change land while consumers who need a passing build continue to use the last verified commit.

## Key takeaways

- An edit’s merge cost depends on how much of the build graph it affects. Deferring broad changes protected peak-hour capacity, and persisting the request relieved its author of the retry while leaving the wait for capacity in place.
- Weak overlaps can tie many requests into one serial group. Relaxing two edges in the six-request example yielded three independent queues. Choosing edges at entry let existing speculative builds remain useful.
- A verified-main pointer gives consumers a passing commit during a trunk failure. The pointer can advance as soon as a later integrated state passes, while repairing the failure remains a separate operational job.

## Slides

### Merging at AI Scale

Green branches merge into a shared trunk. Just past the last merge, a circled X marks where the trunk turns red. Preetam Dwivedi and Manjari Akella trace how Uber’s once-green trunk began to fail.

[0:00](https://www.youtube.com/watch?v=hX140a5klW0&t=0s) [The subtitle reads “From Green to Broken Trunk.” Green branches converge on a trunk that turns red at a circled X.](https://assets.ai.engineer/sites/aiecode2025/sha256/41dc8944660e412fe306171e1813d35d25a9ae075d3fb1e37b923ae5c4740496.webp)

Merging at AI Scale
From Green to Broken Trunk

Preetam Dwivedi
Manjari Akella

01 / 30

Interface text:
Cascades | SEA

### About us

Preetam Dwivedi (left), a staff software engineer, and Manjari Akella (right), a senior software engineer, work together on Uber’s Developer Platform.

[0:19](https://www.youtube.com/watch?v=hX140a5klW0&t=19s) [Side-by-side portraits introduce Preetam Dwivedi, Staff Software Engineer, and Manjari Akella, Senior Software Engineer. Both work on Uber’s Developer Platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/ec785c0380068eed9af71103709dacc10a3aa0dd4c4c473a92afd0b032802dc8.webp)

SPEAKERS
About us

Preetam Dwivedi
Staff Software Engineer
Developer Platform, Uber

Manjari Akella
Senior Software Engineer
Developer Platform, Uber

02 / 30

Interface text:
Cascades | SEA

### Outline

The speakers start with how Uber merges code, then explain how the team adapted as AI sped up development. They close with the results and what they plan to do next.

[0:32](https://www.youtube.com/watch?v=hX140a5klW0&t=32s) [The agenda covers merging at Uber, how AI changed the pace, how the team adapted, and results and next steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f5def0e7840d06966b97ad0c771c897e5a7bf71b9a314a38ac2339d8921b6b6.webp)

Outline

• Merging at Uber
• AI changed the pace
• How we adapted
• Results and what’s next

03 / 30

Interface text:
Cascades | SEA

### Two green PRs. One untested combination.

Pull requests A and B branch from the same commit, H. Each passes CI on its own. If A lands first, B’s passing result still describes H + B. CI has yet to test H + A + B.

[0:52](https://www.youtube.com/watch?v=hX140a5klW0&t=52s) [Pull requests A and B branch from base H. H+A and H+B each pass CI. Neither pull request has landed, and their combination remains untested.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd0914c9c73cede9bc032ec7ead64c6ca23e1927913859da4dfd027aae059fd7.webp)

THE INTEGRATION PROBLEM
Two green PRs. One untested combination.

Both PRs pass CI against the same base. Neither has landed.

main
H
Base H
A   PR A   User A
B   PR B   User B

ISOLATED CI
Both PRs pass
H + A ✓
H + B ✓

05 / 30

Interface text:
Cascades | SEA

### SubmitQueue speculates over futures

Before either pull request lands, SubmitQueue builds and tests three possibilities from the same base H: A alone, B alone, and B on top of A. In this example the combined build passes, so the queue merges A then B into main.

[1:55](https://www.youtube.com/watch?v=hX140a5klW0&t=115s) [Ready pull requests A and B enter SubmitQueue, which builds and tests H+A, H+B and H+A+B. In the passing example, A then B land on main after H.](https://assets.ai.engineer/sites/aiecode2025/sha256/3b7fde7c461cd9b10bc17fe1fdc60b4a33be4c62be9f093dabd31ed38d3d1427.webp)

UBER’S MERGE QUEUE
SubmitQueue speculates over futures
Coordinate the order. Speculate futures. Land on green.

READY PRs
A   PR A
B   PR B

SUBMITQUEUE
Speculate futures
H — A
H — B
H — A — B
Build + test possible states

MAIN
H — A — B
Land on green
Example: both pass

07 / 30

Interface text:
Cascades | SEA

[3:01](https://www.youtube.com/watch?v=hX140a5klW0&t=181s) [Starting from head H, SubmitQueue tests pull request A alone, B alone, and A followed by B. In the illustrated passing case, main advances from H to A to B.](https://assets.ai.engineer/sites/aiecode2025/sha256/de700f82c6fce9b2a5717b596bcc29a2eb9e87a1f597d04a5df82e291e11af0f.webp)

UBER’S MERGE QUEUE
SubmitQueue speculates over futures
Coordinate the order. Speculate futures. Land on green.

READY PRs
A   PR A
B   PR B

SUBMITQUEUE
Speculate futures
H — A
H — B
H — A — B
Build + test possible states

MAIN
H — A — B
Land on green
Example: both pass

07 / 30

Interface text:
Cascades | SEA

### How SubmitQueue scales

SubmitQueue uses build targets to find changes it can test independently. It runs the builds most likely to be needed first. A ready change can merge ahead of a large one when all required speculative outcomes agree.

[3:49](https://www.youtube.com/watch?v=hX140a5klW0&t=229s) [Target analysis splits independent work into virtual queues. Probabilistic speculation tests the most likely states first. Ready work can bypass a large change when all required speculative outcomes agree.](https://assets.ai.engineer/sites/aiecode2025/sha256/adffbe546c641060c3e99e4d260f0038ca5ced06d777e15c3026e695feb1a1fa.webp)

SUBMITQUEUE OPTIMIZATIONS
How SubmitQueue scales

01
Target analysis
Separate independent work
Build-target relationships split work into virtual queues.

02
Probabilistic speculation
Build likely futures first
Prioritize speculative states most likely to be needed.

03
Bypass large changes
Let ready work move ahead
Finalize later work when all required speculative outcomes agree.

08 / 30

Interface text:
Cascades | SEA

### Then came AI

Uber’s merge queue had kept pace when developers wrote the code themselves. With AI, pull requests started arriving faster than Uber could merge them.

[5:32](https://www.youtube.com/watch?v=hX140a5klW0&t=332s) [Large green letters emphasize AI on a dark background. The subtitle reads, “Changes arrived faster than we could land them.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e2b13923d19bec60738e766bc436e781030309353d4c3453aba93af8dd558eab.webp)

Then came
AI

Changes arrived faster than we could land them.

09 / 30

Interface text:
Cascades | SEA

### AI changed the pace of integration

At peak, Uber merged about 90 code changes per hour in January 2026 and about 130 per hour in March. The speaker attributes the roughly 44% increase over those two months primarily to AI-generated changes.

[5:44](https://www.youtube.com/watch?v=hX140a5klW0&t=344s) [Peak lands per hour rose from about 90 in January 2026 to about 130 in March 2026, an increase of about 44% in two months.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b7362a8ac36a899d304d2c2457a933ba994ca52b8509bc50be728cd67aa5759.webp)

JANUARY TO MARCH 2026
AI changed the pace of integration

January 2026
~90
Peak lands / hour

→

March 2026
~130
Peak lands / hour

~44% increase in two months

March was the tipping point.

10 / 30

Interface text:
Cascades | SEA

### Green trunk. Hours to land

Uber kept its main branch green by making developers wait. As traffic rose from about 90 to 130 requests per hour, the 95th-percentile time to land a change rose from about 30 minutes to 3 to 4 hours at peak.

[6:17](https://www.youtube.com/watch?v=hX140a5klW0&t=377s) [The 95th-percentile time to land a change rose from about 30 minutes to 3–4 hours at peak. Developers waited longer to keep main green.](https://assets.ai.engineer/sites/aiecode2025/sha256/1bafa93282b269c117b8968365f4b7a8497657216228da258d04d76bf4efc1db.webp)

THE DEVELOPER IMPACT
Green trunk. Hours to land

Land-time p95
~30 minutes → 3 to 4 hours at peak

We were protecting main by making developers wait.

11 / 30

Interface text:
Cascades | SEA

### What we changed

Uber first controlled which changes entered the merge queue and used a second queue to hold deferred changes. It then relaxed conflict rules so more changes could proceed in parallel.

[6:47](https://www.youtube.com/watch?v=hX140a5klW0&t=407s) [Admission gating shapes traffic, deferrals automate the wait, and conflict relaxation allows more work to proceed in parallel. The slide places demand management before changes to the merge model.](https://assets.ai.engineer/sites/aiecode2025/sha256/7376ef5a14d9ccee18397158a8047a6930a7c83ae7e3dde67cd110a3e5c50c1e.webp)

THE RESPONSE
What we changed

01 Shape the traffic
Admission gating

02 Automate the wait
Deferrals

03 Unlock parallelism
Conflict relaxation

Manage demand first. Then change the merge model.

12 / 30

Interface text:
Cascades | SEA

### Admission gating

During peak hours, smaller changes enter the submit queue for builds and tests. The gate rejects expensive changes that could tie up resources, so developers must resubmit them outside peak hours.

[7:55](https://www.youtube.com/watch?v=hX140a5klW0&t=475s) [A smaller change passes the peak-hour gate and enters the active queue for build and test. The gate rejects an expensive change, and the developer retries outside peak hours.](https://assets.ai.engineer/sites/aiecode2025/sha256/e74cbb3de9752d5137313288175beee6c5c90c2da761d87ff0331f566a1c9560.webp)

INTERVENTION 1
Admission gating

Submitted changes
Peak-hour gate

Smaller change
Admit
Active SQ
A   B   S
Build + test

Expensive change
Rejected during peak hours
Developer retries outside peak hours

First, control what enters the active queue during peak hours

13 / 30

Interface text:
Cascades | SEA

### Admission gating: the tradeoff

After Uber introduced peak-hour gating, its business-hour merge rate rose 23%, from about 130 to about 160 changes per hour. Developers whose expensive changes hit the gate had to resubmit them outside peak hours.

[9:28](https://www.youtube.com/watch?v=hX140a5klW0&t=568s) [After peak-hour gating, business-hour throughput rose from about 130 to 160 lands per hour, an observed 23% increase. Developers had to resubmit expensive changes outside peak hours.](https://assets.ai.engineer/sites/aiecode2025/sha256/89cc3ed0f6cbcaf1cfa4924f31aa71d869ef05a9c56414fcee7cbdd788100cf8.webp)

INTERVENTION 1
Admission gating: the tradeoff

After peak-hour gating
~130 → ~160
Lands / hour
Business hours
+23% observed increase

Developer cost
Developers become the schedulers
“Submit early. Or try again tonight.”
Resubmit expensive changes outside peak hours.

We protected capacity during peak hours, but made developers manage the wait

14 / 30

Interface text:
Cascades | SEA

### Deferrals

A developer submits an expensive change once during peak hours. The platform holds it in a deferred queue before validation, then automatically moves it into the active submit queue when capacity allows.

[10:32](https://www.youtube.com/watch?v=hX140a5klW0&t=632s) [During peak hours, request R waits in a deferred queue before validation. The platform re-evaluates R automatically and releases it into the active queue when capacity allows.](https://assets.ai.engineer/sites/aiecode2025/sha256/1acfebc07ae53f9b80585d83d05c137df40011b7758ba910b2bc4f3159027d72.webp)

INTERVENTION 2
Deferrals

DURING PEAK HOURS
Request R

Deferred queue
R   Persisted
Accepted, waiting
No active validation yet

Re-evaluate automatically

When gates permit release
No resubmission needed

R
Release
Active SQ
A   B   R

Submit once. Release automatically when capacity allows.

15 / 30

Interface text:
Cascades | SEA

### Deferrals: the tradeoff

Developers submit once. The platform holds eligible changes until the queue has room, then releases them automatically. Deferred changes still incur validation costs. Dependencies can still force changes to run in sequence.

[11:43](https://www.youtube.com/watch?v=hX140a5klW0&t=703s) [Developers submit once, and the platform releases eligible work automatically. Deferred work still waits and incurs validation costs. Dependencies can still force changes to run in sequence.](https://assets.ai.engineer/sites/aiecode2025/sha256/74eb3d336a4213645d4f3d71424ce299075ff8545a81c46159e4fb92865ed0df.webp)

INTERVENTION 2
Deferrals: the tradeoff

Good
Submit once.
Release automatically.
The platform holds and releases eligible work without developer resubmission.

Bad
The waiting still exists
Demand moves to another time. Validation cost and dependency coupling remain.

We moved the wait. We had not removed serialization.

16 / 30

Interface text:
Cascades | SEA

### Any target overlap could serialize work

Each neighboring pair of changes from A through F shares at least one target. The overlap between C and D joins the two halves, so the strict queue rule could force all six changes to run in sequence.

[12:15](https://www.youtube.com/watch?v=hX140a5klW0&t=735s) [Six changes labeled A through F form one connected chain. Under the strict overlap rule, a single overlap can join two groups so that all connected changes share an ordering constraint.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e509aae1af31b2ccf86958af9f6896001d0b4ca65f1126576ed0c1abeb399ed.webp)

THE REMAINING BOTTLENECK
Any target overlap could serialize work

STRICT · 1 CONNECTED COMPONENT

A   B   C   D   E   F

A single overlap can join otherwise separate groups.

Connected changes share an ordering constraint

17 / 30

Interface text:
Cascades | SEA

### Jaccard scoring for dependency overlap

Direct target overlap gets full weight. Each change’s new dependencies get half weight when compared with the other change’s targets. The largest of the three scores ranks the link and helps determine which links to keep.

[13:37](https://www.youtube.com/watch?v=hX140a5klW0&t=817s) [Jaccard similarity divides shared targets by distinct targets. The ranking score takes the largest of three values: direct target similarity and two half-weighted similarities comparing each side’s new dependency edges with the other side’s targets.](https://assets.ai.engineer/sites/aiecode2025/sha256/17cddaaf1dcbdf9ed2da61f80de815ea534131a8c52cf43d5e8590fa5e96a554.webp)

CONFLICT RELAXATION
Jaccard scoring for dependency overlap

Jaccard similarity = shared targets ÷ distinct targets

score(A, B) = max(
  1 × Jaccard(A, B),
  0.5 × Jaccard(E_A, B),
  0.5 × Jaccard(E_B, A)
)

A, B
Overlapping targets

E_A, E_B
New Dependency Edge

A ranking signal.
Not a failure probability.

19 / 30

Interface text:
Cascades | SEA

### Buy parallelism by dropping the weakest bridges

For six changes, a 50% parallelism target means three queues. Dropping the two links with the lowest overlap scores, B–C (0.08) and D–E (0.04), lets the remaining pairs A–B, C–D and E–F run in parallel.

[15:04](https://www.youtube.com/watch?v=hX140a5klW0&t=904s) [Dropping B–C (0.08) and D–E (0.04) leaves three queues of two changes each: A–B (0.90), C–D (0.80) and E–F (0.70).](https://assets.ai.engineer/sites/aiecode2025/sha256/2259d9a0e084179a0acb3d4c1d7c9ebada7ad855505458facaf537a78e55e0b5.webp)

CONFLICT RELAXATION
Buy parallelism by dropping the weakest bridges

6 changes · 50% desired parallelism    target = 3 queues
RELAXED · 3 VIRTUAL QUEUES

A–B: 0.90
B–C: 0.08 (drop)
C–D: 0.80
D–E: 0.04 (drop)
E–F: 0.70

Keep the strongest links. Stop joining independent queues at the target.

20 / 30

Interface text:
Cascades | SEA

### Assign new arrivals. Leave in-flight work alone.

When a change arrives, Uber checks it for conflicts with queued work and assigns it to a queue once. Changes already under validation stay in place, preserving the speculative builds already underway.

[16:17](https://www.youtube.com/watch?v=hX140a5klW0&t=977s) [Two columns show when placement happens. The conflict policy assigns new work once. Work already undergoing validation keeps its queue placement.](https://assets.ai.engineer/sites/aiecode2025/sha256/31eb2c4e1f210254c370960b7f07fe9fc398dbeb0e48cdd52ee4932ef65d8a14.webp)

CONFLICT RELAXATION
Assign new arrivals. Leave in-flight work alone.

NEW ARRIVALS
Assign once
Place new work using the conflict policy.

IN-FLIGHT WORK
Preserve placement
Do not reshuffle work already being validated.

Preserve useful speculative builds

21 / 30

Interface text:
Cascades | SEA

### Conflict relaxation: the tradeoff

During business hours in July 2026, Uber saw throughput rise from about 160 to about 250 lands per hour, a roughly 56% gain. Changes that pass separately can still fail together, so Uber needs to test the combination that lands.

[17:59](https://www.youtube.com/watch?v=hX140a5klW0&t=1079s) [During business hours in July 2026, observed throughput rose from about 160 to 250 lands per hour, roughly 56%. Changes that pass individually can interact after landing and cause trunk validation to fail.](https://assets.ai.engineer/sites/aiecode2025/sha256/c39843b08c20d328705dceb8cc534006dcc642503799cf54a170d82e8fb83d98.webp)

INTERVENTION 3
Conflict relaxation: the tradeoff

Early throughput
~160 → ~250
Lands / hour
Business hours, July 2026
~56% observed increase

The tradeoff
Some combinations land untested
Individually green changes can interact after landing. Trunk can turn red.

Higher throughput creates a new obligation: validate what actually lands

22 / 30

Interface text:
Cascades | SEA

### The cost: trunk can turn red

Changes A and B each pass when tested separately against the same starting commit, H. When A lands and B follows, trunk contains both changes. That combination can fail even though both pre-merge checks passed.

[19:14](https://www.youtube.com/watch?v=hX140a5klW0&t=1154s) [H + A and H + B each pass independent pre-merge validation. The post-merge diagram shows one possible outcome in which H + A + B fails.](https://assets.ai.engineer/sites/aiecode2025/sha256/581ac96124e867e8f89a6879e93ee9b389c4a4bba0b2775ddb835147fb84ed0e.webp)

THE CONSEQUENCE
The cost: trunk can turn red

PRE-MERGE · INDEPENDENT VALIDATION
H       A       H + A passes
        B       H + B passes

POST-MERGE · ONE POSSIBLE OUTCOME
H       A       B
H + A + B fails

Which landed state can we actually trust?

23 / 30

Interface text:
Cascades | SEA

### Stovepipe: one history, two moving refs

Both pointers start at commit A. As new commits land, main advances through D, where validation fails, to G. verified/main stays at A until validation passes at G, then catches up to main.

[20:02](https://www.youtube.com/watch?v=hX140a5klW0&t=1202s) [On a history labeled A through G, both main HEAD and verified/main HEAD point to the validated commit A. Commits B through G are faded.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd2ae7b7a3e2787ef267ef8dfab3fe3a62dd4b115fa207e2dd8609f3ec3e4dcd.webp)

POST-MERGE CONFIDENCE
Stovepipe: one history, two moving refs

main advances on landing. verified/main advances on passing validation.

A is known green
Both refs point to the known-good baseline.

main HEAD
A   B   C   D   E   F   G
verified/main HEAD

24 / 30

Interface text:
Cascades | SEA

[22:25](https://www.youtube.com/watch?v=hX140a5klW0&t=1345s) [A single line connects commits A through G. D remains red. A green bracket marks validation from A to G as passed, and the main and verified/main arrows both point to G.](https://assets.ai.engineer/sites/aiecode2025/sha256/025566c26986ef051507618c35036135a2115314be000c8a3822269efce0ed1c.webp)

POST-MERGE CONFIDENCE
Stovepipe: one history, two moving refs

main advances on landing. verified/main advances on passing validation.

G passes validation
verified/main advances to the passing state at G.

main HEAD
A   B   C   D   E   F   G
verified/main HEAD

Validate A–G: passed

24 / 30

Interface text:
Cascades | SEA

### Recovering a broken trunk

When commit D fails validation, Stovepipe pages the on-call engineer. The engineer gets a fix or revert approved and lands it through SubmitQueue. Stovepipe advances verified/main only after the repaired trunk passes validation.

[22:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1375s) [Stovepipe alerts the on-call engineer when D fails validation. The engineer prepares a fix or revert, gets approval and lands it through SubmitQueue. Stovepipe validates the repair and advances verified/main on success.](https://assets.ai.engineer/sites/aiecode2025/sha256/a04bc76973211fd3906339437e7582ec401b404125aa8dec35b261d77c7e1da6.webp)

POST-MERGE CONFIDENCE
Recovering a broken trunk

STOVEPIPE
Detect and alert
D fails validation. On-call is alerted.

ON-CALL
Prepare a repair
Create a fix or revert.

ON-CALL
Review and land
Get approvals and land the repair through SubmitQueue.

STOVEPIPE
Verify recovery
Validate the repaired state. A passing result advances verified/main.

25 / 30

Interface text:
Cascades | SEA

### More throughput. Shorter waits.

By October 2026, Uber’s peak merge rate had risen from about 90 changes per hour in January to about 300, a roughly 233% increase. About 95% of changes landed within roughly 30 minutes.

[23:40](https://www.youtube.com/watch?v=hX140a5klW0&t=1420s) [As of October 2026, peak throughput is about 300 lands per hour, up about 233% from roughly 90 per hour in January. The 95th-percentile land time is about 30 minutes.](https://assets.ai.engineer/sites/aiecode2025/sha256/2efdcb74e9a1507f0f10e5fc72f4de5c2fd87239ddb86acdbd09805779842881.webp)

RESULTS · AS OF OCTOBER 2026
More throughput. Shorter waits.

~300
Peak lands / hour
~233% increase from January’s ~90/hour

~30 min
Land-time p95

26 / 30

Interface text:
Cascades | SEA

### SubmitQueue and Stovepipe in the open

SubmitQueue tests what the code would look like after a proposed merge and lands changes that pass. Stovepipe then checks the merged code and tracks which revisions passed. Uber develops both tools in the open.

[24:12](https://www.youtube.com/watch?v=hX140a5klW0&t=1452s) [Before merge, SubmitQueue validates proposed changes and lands those that pass. After merge, Stovepipe validates landed states and tracks known-good states. A QR code and the project repository address appear on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/0afab361afd484d32104d7d93ab6349ca4180fce4b5b0b909b3481e3fae7f996.webp)

OPEN SOURCE
SubmitQueue and Stovepipe in the open

PRE-MERGE
SubmitQueue
Validate speculative futures.
Land passing changes.

POST-MERGE
Stovepipe
Validate landed states.
Track known-good states.

Explore the project
github.com/uber/submitqueue

27 / 30

Interface text:
Cascades | SEA

### What comes next

Uber wants SubmitQueue to batch related changes so it runs fewer builds. Project-level checks would show which projects still pass when trunk breaks. Automated fixes and reverts would restore trunk health faster.

[24:35](https://www.youtube.com/watch?v=hX140a5klW0&t=1475s) [Three columns outline plans to batch changes in SubmitQueue, report validation results for individual projects, and automate fixes and reverts to restore trunk health faster.](https://assets.ai.engineer/sites/aiecode2025/sha256/4de422d99aa7c1c9c310538fdd8e0e289c4f7f30a7ad08ad269270f7c1730092.webp)

OPEN SOURCE & NEXT
What comes next

BETTER THROUGHPUT
Batching for SubmitQueue
Validate groups of changes together to reduce repeated work.

BETTER CONFIDENCE
Validate project scope
Answer “is my project green?” without treating every failure as global.

FASTER RECOVERY
Automate fixes and reverts
Invest in automated diagnosis and corrective actions to restore trunk health faster.

28 / 30

Interface text:
Cascades | SEA

### A hybrid approach to validation

Uber runs required checks and enforces stronger dependency constraints before merge. It continuously tests the integrated trunk after merge, so more changes can land while the wait for each change stays reasonable.

[26:13](https://www.youtube.com/watch?v=hX140a5klW0&t=1573s) [Required checks and stronger dependency constraints apply before merge. Continuous checks validate the integrated trunk after merge. Highlighted text says this supports higher landing throughput while keeping land times reasonable.](https://assets.ai.engineer/sites/aiecode2025/sha256/b272214f3e8ffca6fc3b9b313616170feea8e19b7b9493c9fd909affd0a619c4.webp)

CLOSING
A hybrid approach to validation

We retain required checks and stronger dependency constraints before merge, while continuously validating integrated trunk states after merge.

This gives us the flexibility to support higher landing throughput while keeping land times reasonable.

29 / 30

Interface text:
Cascades | SEA

### Thank you

Explore Uber’s SubmitQueue repository, or scan a QR code to watch Merging at Scale for a closer look at how the queue works. The left code links to the WeAreDevelopers talk and the right to Seattle Systems.

[27:07](https://www.youtube.com/watch?v=hX140a5klW0&t=1627s) [The SubmitQueue repository address appears above two QR codes for the talk “Merging at Scale,” subtitled “From Broken Builds to Green Mainline.” The left code is labeled WeAreDevelopers and the right code Seattle Systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/87827eb2cb0d13a6b529c37ac7c9a73373a7156a7c17cbcdf3b5d845e005f6e4.webp)

Thank you
github.com/uber/submitqueue

WeAreDevelopers
Merging at Scale
From Broken Builds to Green Mainline
Watch talk ↗

Seattle Systems
Merging at Scale
From Broken Builds to Green Mainline
Watch talk ↗

30 / 30

Interface text:
Cascades | SEA

## Full transcript

[0:04](https://www.youtube.com/watch?v=hX140a5klW0&t=4s) Uh hello everyone. Uh we are here to

[0:06](https://www.youtube.com/watch?v=hX140a5klW0&t=6s) present our talk uh merging at um AI

[0:10](https://www.youtube.com/watch?v=hX140a5klW0&t=10s) scale. Uh this is essentially uh Uber's

[0:13](https://www.youtube.com/watch?v=hX140a5klW0&t=13s) journey from going to a fully green

[0:17](https://www.youtube.com/watch?v=hX140a5klW0&t=17s) trunk to somewhat broken trunk.

[0:20](https://www.youtube.com/watch?v=hX140a5klW0&t=20s) Um I'm Priam Devi. I'm a staff engineer

[0:23](https://www.youtube.com/watch?v=hX140a5klW0&t=23s) at Uber working in developer platform.

[0:26](https://www.youtube.com/watch?v=hX140a5klW0&t=26s) Hi, I'm Maji. I'm a senior engineer

[0:28](https://www.youtube.com/watch?v=hX140a5klW0&t=28s) working alongside Priam. uh in the

[0:30](https://www.youtube.com/watch?v=hX140a5klW0&t=30s) developer at at Uber.

[0:32](https://www.youtube.com/watch?v=hX140a5klW0&t=32s) Great. Um so here's the outline for the

[0:36](https://www.youtube.com/watch?v=hX140a5klW0&t=36s) talk. Uh we're going to go over like how

[0:38](https://www.youtube.com/watch?v=hX140a5klW0&t=38s) the merging um works at Uber at the

[0:40](https://www.youtube.com/watch?v=hX140a5klW0&t=40s) moment and then how AI changed the

[0:44](https://www.youtube.com/watch?v=hX140a5klW0&t=44s) landscape and then I'll respond um to

[0:47](https://www.youtube.com/watch?v=hX140a5klW0&t=47s) that how we adapted and then discuss the

[0:49](https://www.youtube.com/watch?v=hX140a5klW0&t=49s) results and um talk about what are we

[0:52](https://www.youtube.com/watch?v=hX140a5klW0&t=52s) planning next.

[0:54](https://www.youtube.com/watch?v=hX140a5klW0&t=54s) So at Uber uh we have a six language

[0:59](https://www.youtube.com/watch?v=hX140a5klW0&t=59s) based monor repos where more than 5,000

[1:02](https://www.youtube.com/watch?v=hX140a5klW0&t=62s) uh developers uh are contributing over

[1:05](https://www.youtube.com/watch?v=hX140a5klW0&t=65s) 36,000 commits uh 27,000

[1:09](https://www.youtube.com/watch?v=hX140a5klW0&t=69s) PRs uh every week and uh pretty much

[1:13](https://www.youtube.com/watch?v=hX140a5klW0&t=73s) every changes uh to the services

[1:15](https://www.youtube.com/watch?v=hX140a5klW0&t=75s) libraries are being done in one of these

[1:18](https://www.youtube.com/watch?v=hX140a5klW0&t=78s) monor repos. uh for the purpose of this

[1:21](https://www.youtube.com/watch?v=hX140a5klW0&t=81s) talk we're going to focus mostly on go

[1:23](https://www.youtube.com/watch?v=hX140a5klW0&t=83s) code when we talk about anything because

[1:25](https://www.youtube.com/watch?v=hX140a5klW0&t=85s) that is our the biggest one and that's

[1:27](https://www.youtube.com/watch?v=hX140a5klW0&t=87s) where the most of the scaling challenges

[1:29](https://www.youtube.com/watch?v=hX140a5klW0&t=89s) uh we face.

[1:30](https://www.youtube.com/watch?v=hX140a5klW0&t=90s) So let's understand the problem a little

[1:32](https://www.youtube.com/watch?v=hX140a5klW0&t=92s) bit and um and understand like why we

[1:37](https://www.youtube.com/watch?v=hX140a5klW0&t=97s) need a merge cube here. So uh given here

[1:41](https://www.youtube.com/watch?v=hX140a5klW0&t=101s) basically we have two users uh user A

[1:43](https://www.youtube.com/watch?v=hX140a5klW0&t=103s) and B creating PR A and B um on a base

[1:47](https://www.youtube.com/watch?v=hX140a5klW0&t=107s) edge uh from the trunk and when the PRs

[1:50](https://www.youtube.com/watch?v=hX140a5klW0&t=110s) are created and CI is run uh the CI is

[1:53](https://www.youtube.com/watch?v=hX140a5klW0&t=113s) green builds are passing everything is

[1:54](https://www.youtube.com/watch?v=hX140a5klW0&t=114s) good.

[1:56](https://www.youtube.com/watch?v=hX140a5klW0&t=116s) Now uh when a is merged into the trunk,

[2:00](https://www.youtube.com/watch?v=hX140a5klW0&t=120s) we can see the bills for the merge

[2:02](https://www.youtube.com/watch?v=hX140a5klW0&t=122s) commit are still green. Um bills are

[2:05](https://www.youtube.com/watch?v=hX140a5klW0&t=125s) still passing. But when the PRB is

[2:08](https://www.youtube.com/watch?v=hX140a5klW0&t=128s) merged into the trunk, um bills are no

[2:11](https://www.youtube.com/watch?v=hX140a5klW0&t=131s) longer green. And the reason is that B

[2:13](https://www.youtube.com/watch?v=hX140a5klW0&t=133s) was never tested uh with the changes

[2:16](https://www.youtube.com/watch?v=hX140a5klW0&t=136s) that were merged by A. So uh we never

[2:20](https://www.youtube.com/watch?v=hX140a5klW0&t=140s) tested the the combination of those two

[2:22](https://www.youtube.com/watch?v=hX140a5klW0&t=142s) things before merging and hence the the

[2:27](https://www.youtube.com/watch?v=hX140a5klW0&t=147s) when B is merged it's broken.

[2:30](https://www.youtube.com/watch?v=hX140a5klW0&t=150s) Um now imagine when we have thousands of

[2:33](https://www.youtube.com/watch?v=hX140a5klW0&t=153s) uh engineers working in a monor repo and

[2:35](https://www.youtube.com/watch?v=hX140a5klW0&t=155s) everyone is making changes um like

[2:38](https://www.youtube.com/watch?v=hX140a5klW0&t=158s) hundreds and thousands of those every

[2:40](https://www.youtube.com/watch?v=hX140a5klW0&t=160s) day. Uh this is a constant problem uh

[2:44](https://www.youtube.com/watch?v=hX140a5klW0&t=164s) where trunk is broken um very frequently

[2:47](https://www.youtube.com/watch?v=hX140a5klW0&t=167s) and there is a burden on fixing and

[2:50](https://www.youtube.com/watch?v=hX140a5klW0&t=170s) there is a toll across the company to

[2:52](https://www.youtube.com/watch?v=hX140a5klW0&t=172s) sort of um figure it out and uh fix it.

[2:56](https://www.youtube.com/watch?v=hX140a5klW0&t=176s) Um so here uh we're going to present a

[2:59](https://www.youtube.com/watch?v=hX140a5klW0&t=179s) solution uh to that problem uh and that

[3:02](https://www.youtube.com/watch?v=hX140a5klW0&t=182s) solution is summit Q. Um that is our

[3:04](https://www.youtube.com/watch?v=hX140a5klW0&t=184s) Uber's merge Q um that we deploy uh to

[3:08](https://www.youtube.com/watch?v=hX140a5klW0&t=188s) solve this problem. And in this case we

[3:10](https://www.youtube.com/watch?v=hX140a5klW0&t=190s) can see our two examples here P A and

[3:13](https://www.youtube.com/watch?v=hX140a5klW0&t=193s) PRB. And here we essentially what we are

[3:16](https://www.youtube.com/watch?v=hX140a5klW0&t=196s) doing is we are predicting the futures

[3:18](https://www.youtube.com/watch?v=hX140a5klW0&t=198s) that would be u that could be true on

[3:20](https://www.youtube.com/watch?v=hX140a5klW0&t=200s) main. So there are like three things

[3:23](https://www.youtube.com/watch?v=hX140a5klW0&t=203s) here. A could land on on its own to the

[3:26](https://www.youtube.com/watch?v=hX140a5klW0&t=206s) main B could land on its own to the main

[3:29](https://www.youtube.com/watch?v=hX140a5klW0&t=209s) and then the combination of A and B. In

[3:32](https://www.youtube.com/watch?v=hX140a5klW0&t=212s) general I think uh what we want to do is

[3:34](https://www.youtube.com/watch?v=hX140a5klW0&t=214s) that we want to land the change that is

[3:35](https://www.youtube.com/watch?v=hX140a5klW0&t=215s) going to keep the trunk green as a

[3:37](https://www.youtube.com/watch?v=hX140a5klW0&t=217s) whole. So in this case when we land A

[3:39](https://www.youtube.com/watch?v=hX140a5klW0&t=219s) and B together the trunk is green and uh

[3:43](https://www.youtube.com/watch?v=hX140a5klW0&t=223s) and that's how basically uh sort of

[3:45](https://www.youtube.com/watch?v=hX140a5klW0&t=225s) summit Q works and keeps the trunk green

[3:48](https://www.youtube.com/watch?v=hX140a5klW0&t=228s) for our monor repos. Now how does it

[3:51](https://www.youtube.com/watch?v=hX140a5klW0&t=231s) scale in this uh version of it? Um there

[3:54](https://www.youtube.com/watch?v=hX140a5klW0&t=234s) are three things that summit key

[3:56](https://www.youtube.com/watch?v=hX140a5klW0&t=236s) employs. Uh the very first thing is

[3:58](https://www.youtube.com/watch?v=hX140a5klW0&t=238s) target analysis. Uh what it means is

[4:00](https://www.youtube.com/watch?v=hX140a5klW0&t=240s) when two users are working across two

[4:02](https://www.youtube.com/watch?v=hX140a5klW0&t=242s) different services uh they don't really

[4:05](https://www.youtube.com/watch?v=hX140a5klW0&t=245s) have any overlapping changes. Um so

[4:07](https://www.youtube.com/watch?v=hX140a5klW0&t=247s) those two things can be built and tested

[4:10](https://www.youtube.com/watch?v=hX140a5klW0&t=250s) independently of each other and then can

[4:12](https://www.youtube.com/watch?v=hX140a5klW0&t=252s) be merged. So this allows us to

[4:14](https://www.youtube.com/watch?v=hX140a5klW0&t=254s) basically get higher throughput and not

[4:16](https://www.youtube.com/watch?v=hX140a5klW0&t=256s) serialize every change that is entering

[4:18](https://www.youtube.com/watch?v=hX140a5klW0&t=258s) into the queue.

[4:20](https://www.youtube.com/watch?v=hX140a5klW0&t=260s) Uh second is probabistic speculation. Um

[4:23](https://www.youtube.com/watch?v=hX140a5klW0&t=263s) what it means is uh we are using

[4:26](https://www.youtube.com/watch?v=hX140a5klW0&t=266s) historical um the build information and

[4:29](https://www.youtube.com/watch?v=hX140a5klW0&t=269s) also the the change characteristics to

[4:32](https://www.youtube.com/watch?v=hX140a5klW0&t=272s) predict uh what is the success score for

[4:35](https://www.youtube.com/watch?v=hX140a5klW0&t=275s) a given change and then we are using

[4:37](https://www.youtube.com/watch?v=hX140a5klW0&t=277s) that to basically decide what all builds

[4:40](https://www.youtube.com/watch?v=hX140a5klW0&t=280s) are likely needed in order to land that

[4:43](https://www.youtube.com/watch?v=hX140a5klW0&t=283s) change. And so in that case we

[4:45](https://www.youtube.com/watch?v=hX140a5klW0&t=285s) prioritize the build that are most

[4:47](https://www.youtube.com/watch?v=hX140a5klW0&t=287s) likely needed and are important for that

[4:49](https://www.youtube.com/watch?v=hX140a5klW0&t=289s) change to be uh merged to the trunk

[4:51](https://www.youtube.com/watch?v=hX140a5klW0&t=291s) rather than like every other build that

[4:53](https://www.youtube.com/watch?v=hX140a5klW0&t=293s) uh comes from the speculation. The the

[4:56](https://www.youtube.com/watch?v=hX140a5klW0&t=296s) third part is bypassing the large

[4:58](https://www.youtube.com/watch?v=hX140a5klW0&t=298s) changes. Uh what it means is that if we

[5:00](https://www.youtube.com/watch?v=hX140a5klW0&t=300s) have a large chain sitting in the queue

[5:01](https://www.youtube.com/watch?v=hX140a5klW0&t=301s) which is taking longer to build and at

[5:04](https://www.youtube.com/watch?v=hX140a5klW0&t=304s) the same time if we have a smaller

[5:05](https://www.youtube.com/watch?v=hX140a5klW0&t=305s) changes entering into the queue um those

[5:08](https://www.youtube.com/watch?v=hX140a5klW0&t=308s) can be built faster and so though those

[5:10](https://www.youtube.com/watch?v=hX140a5klW0&t=310s) changes don't need to wait for large

[5:12](https://www.youtube.com/watch?v=hX140a5klW0&t=312s) changes to land before they can land. So

[5:14](https://www.youtube.com/watch?v=hX140a5klW0&t=314s) in this case we can land the smaller

[5:16](https://www.youtube.com/watch?v=hX140a5klW0&t=316s) changes without landing the larger

[5:18](https://www.youtube.com/watch?v=hX140a5klW0&t=318s) changes first and this optimization

[5:20](https://www.youtube.com/watch?v=hX140a5klW0&t=320s) allows us to basically move fast and get

[5:22](https://www.youtube.com/watch?v=hX140a5klW0&t=322s) a higher throughput of the queue. So

[5:25](https://www.youtube.com/watch?v=hX140a5klW0&t=325s) this is what has worked um when it was

[5:28](https://www.youtube.com/watch?v=hX140a5klW0&t=328s) just humans uh landing changes creating

[5:31](https://www.youtube.com/watch?v=hX140a5klW0&t=331s) PRs

[5:33](https://www.youtube.com/watch?v=hX140a5klW0&t=333s) but um then we had AI and the changes at

[5:38](https://www.youtube.com/watch?v=hX140a5klW0&t=338s) which the uh PRs were created and landed

[5:42](https://www.youtube.com/watch?v=hX140a5klW0&t=342s) it all changed. So to understand it a

[5:44](https://www.youtube.com/watch?v=hX140a5klW0&t=344s) better let's look at some of the

[5:46](https://www.youtube.com/watch?v=hX140a5klW0&t=346s) numbers. So uh in January 2026 we were

[5:50](https://www.youtube.com/watch?v=hX140a5klW0&t=350s) doing uh roughly 90 requests uh every

[5:54](https://www.youtube.com/watch?v=hX140a5klW0&t=354s) hour and by March we were doing 130

[5:58](https://www.youtube.com/watch?v=hX140a5klW0&t=358s) requests every hour. So that's roughly a

[6:01](https://www.youtube.com/watch?v=hX140a5klW0&t=361s) 44% increase in two months and the

[6:05](https://www.youtube.com/watch?v=hX140a5klW0&t=365s) reason for this increase is primarily

[6:06](https://www.youtube.com/watch?v=hX140a5klW0&t=366s) because of the AI generated changes. Now

[6:10](https://www.youtube.com/watch?v=hX140a5klW0&t=370s) given the way the system was built um uh

[6:13](https://www.youtube.com/watch?v=hX140a5klW0&t=373s) it had the rate of uh these changes had

[6:17](https://www.youtube.com/watch?v=hX140a5klW0&t=377s) effect on it. So um looking into those

[6:21](https://www.youtube.com/watch?v=hX140a5klW0&t=381s) like we had 30 minutes of uh P95 line

[6:23](https://www.youtube.com/watch?v=hX140a5klW0&t=383s) times when we were doing 90 requests

[6:26](https://www.youtube.com/watch?v=hX140a5klW0&t=386s) every hour and that went to roughly 3 to

[6:30](https://www.youtube.com/watch?v=hX140a5klW0&t=390s) four hours when we were doing uh 130 uh

[6:33](https://www.youtube.com/watch?v=hX140a5klW0&t=393s) requests every hour. So in this case we

[6:36](https://www.youtube.com/watch?v=hX140a5klW0&t=396s) were still keeping the trunk green uh at

[6:39](https://www.youtube.com/watch?v=hX140a5klW0&t=399s) all times but at the cost of developers

[6:42](https://www.youtube.com/watch?v=hX140a5klW0&t=402s) essentially waiting for 3 to four hours

[6:44](https://www.youtube.com/watch?v=hX140a5klW0&t=404s) to land their changes to the trunk. Um

[6:47](https://www.youtube.com/watch?v=hX140a5klW0&t=407s) now I'm going to hand over to Mry uh and

[6:49](https://www.youtube.com/watch?v=hX140a5klW0&t=409s) she's going to talk about our response

[6:51](https://www.youtube.com/watch?v=hX140a5klW0&t=411s) to it and how we adapted uh to the

[6:54](https://www.youtube.com/watch?v=hX140a5klW0&t=414s) situation.

[6:55](https://www.youtube.com/watch?v=hX140a5klW0&t=415s) Great. Thanks Pritham. So, Priam just uh

[6:59](https://www.youtube.com/watch?v=hX140a5klW0&t=419s) walked us through a few optimizations

[7:01](https://www.youtube.com/watch?v=hX140a5klW0&t=421s) that Submit Q had that worked so far in

[7:03](https://www.youtube.com/watch?v=hX140a5klW0&t=423s) keeping up with the demand uh that

[7:06](https://www.youtube.com/watch?v=hX140a5klW0&t=426s) humans generated. But once AI came into

[7:08](https://www.youtube.com/watch?v=hX140a5klW0&t=428s) the picture, uh the optimizations we had

[7:11](https://www.youtube.com/watch?v=hX140a5klW0&t=431s) were no longer sufficient. So we our

[7:14](https://www.youtube.com/watch?v=hX140a5klW0&t=434s) response essentially was another set of

[7:17](https://www.youtube.com/watch?v=hX140a5klW0&t=437s) optimizations to our existing submit Q

[7:20](https://www.youtube.com/watch?v=hX140a5klW0&t=440s) and we approached the response in three

[7:22](https://www.youtube.com/watch?v=hX140a5klW0&t=442s) different ways. The first was we

[7:24](https://www.youtube.com/watch?v=hX140a5klW0&t=444s) introduced this concept of admission

[7:26](https://www.youtube.com/watch?v=hX140a5klW0&t=446s) gating which helped us to shape the

[7:29](https://www.youtube.com/watch?v=hX140a5klW0&t=449s) traffic that we accept into the queue.

[7:31](https://www.youtube.com/watch?v=hX140a5klW0&t=451s) Uh next we introduced the concept of

[7:34](https://www.youtube.com/watch?v=hX140a5klW0&t=454s) deferrals which is a secondary queue

[7:37](https://www.youtube.com/watch?v=hX140a5klW0&t=457s) that we introduced that helps with

[7:39](https://www.youtube.com/watch?v=hX140a5klW0&t=459s) developer experience uh from the pain

[7:42](https://www.youtube.com/watch?v=hX140a5klW0&t=462s) that comes with admission gating. And

[7:44](https://www.youtube.com/watch?v=hX140a5klW0&t=464s) lastly, we implemented what we call

[7:46](https://www.youtube.com/watch?v=hX140a5klW0&t=466s) conflict relaxation which helps us buy

[7:49](https://www.youtube.com/watch?v=hX140a5klW0&t=469s) back more parallelism uh within the

[7:53](https://www.youtube.com/watch?v=hX140a5klW0&t=473s) queue itself. Uh next we will look at

[7:56](https://www.youtube.com/watch?v=hX140a5klW0&t=476s) each of these uh interventions in a

[7:59](https://www.youtube.com/watch?v=hX140a5klW0&t=479s) little bit of depth and understand

[8:01](https://www.youtube.com/watch?v=hX140a5klW0&t=481s) exactly what each of them are doing. So

[8:04](https://www.youtube.com/watch?v=hX140a5klW0&t=484s) the first one admission gating uh is

[8:06](https://www.youtube.com/watch?v=hX140a5klW0&t=486s) basically what it sounds like. There is

[8:08](https://www.youtube.com/watch?v=hX140a5klW0&t=488s) a gate at the admission of every request

[8:11](https://www.youtube.com/watch?v=hX140a5klW0&t=491s) into the main submit que. We look at the

[8:14](https://www.youtube.com/watch?v=hX140a5klW0&t=494s) request that comes in and we categorize

[8:17](https://www.youtube.com/watch?v=hX140a5klW0&t=497s) it as one of two things. Either it's a

[8:19](https://www.youtube.com/watch?v=hX140a5klW0&t=499s) small change or it's an expensive

[8:21](https://www.youtube.com/watch?v=hX140a5klW0&t=501s) change. And the definition of this uh

[8:24](https://www.youtube.com/watch?v=hX140a5klW0&t=504s) could be anything. One of the

[8:25](https://www.youtube.com/watch?v=hX140a5klW0&t=505s) definitions that we use is the target

[8:28](https://www.youtube.com/watch?v=hX140a5klW0&t=508s) analysis that Pitham talked about uh

[8:31](https://www.youtube.com/watch?v=hX140a5klW0&t=511s) earlier in the slides. Uh a small change

[8:34](https://www.youtube.com/watch?v=hX140a5klW0&t=514s) touches fewer targets. An expensive

[8:36](https://www.youtube.com/watch?v=hX140a5klW0&t=516s) change might be something that spans

[8:38](https://www.youtube.com/watch?v=hX140a5klW0&t=518s) across uh multiple targets, the entire

[8:42](https://www.youtube.com/watch?v=hX140a5klW0&t=522s) build graph, something like that. So a

[8:44](https://www.youtube.com/watch?v=hX140a5klW0&t=524s) smaller change is more likely to give

[8:47](https://www.youtube.com/watch?v=hX140a5klW0&t=527s) you higher parallelism and we basically

[8:51](https://www.youtube.com/watch?v=hX140a5klW0&t=531s) say admit it during peak hours and add

[8:53](https://www.youtube.com/watch?v=hX140a5klW0&t=533s) it to the active uh submit Q build and

[8:56](https://www.youtube.com/watch?v=hX140a5klW0&t=536s) test as usual uh and move along. But an

[8:59](https://www.youtube.com/watch?v=hX140a5klW0&t=539s) expensive change has the capability to

[9:02](https://www.youtube.com/watch?v=hX140a5klW0&t=542s) basically be a bottleneck by holding

[9:06](https://www.youtube.com/watch?v=hX140a5klW0&t=546s) resources

[9:07](https://www.youtube.com/watch?v=hX140a5klW0&t=547s) uh needing to validate against

[9:09](https://www.youtube.com/watch?v=hX140a5klW0&t=549s) everything else that's in the queue. So

[9:11](https://www.youtube.com/watch?v=hX140a5klW0&t=551s) to protect the rest of the queue, what

[9:14](https://www.youtube.com/watch?v=hX140a5klW0&t=554s) we do uh in this uh phase is just to

[9:18](https://www.youtube.com/watch?v=hX140a5klW0&t=558s) reject any expensive change during peak

[9:21](https://www.youtube.com/watch?v=hX140a5klW0&t=561s) hours. What this means is that the

[9:23](https://www.youtube.com/watch?v=hX140a5klW0&t=563s) developer has to come back and retry

[9:25](https://www.youtube.com/watch?v=hX140a5klW0&t=565s) their expensive change outside peak

[9:28](https://www.youtube.com/watch?v=hX140a5klW0&t=568s) hours. Uh so this gives us control into

[9:31](https://www.youtube.com/watch?v=hX140a5klW0&t=571s) what enters the queue by moving the

[9:34](https://www.youtube.com/watch?v=hX140a5klW0&t=574s) burden onto a very subset

[9:37](https://www.youtube.com/watch?v=hX140a5klW0&t=577s) of changes and developers.

[9:41](https://www.youtube.com/watch?v=hX140a5klW0&t=581s) Uh let's look at the trade-off of

[9:42](https://www.youtube.com/watch?v=hX140a5klW0&t=582s) admission gating. Uh after we

[9:44](https://www.youtube.com/watch?v=hX140a5klW0&t=584s) implemented this, we definitely observed

[9:47](https://www.youtube.com/watch?v=hX140a5klW0&t=587s) an increase in our throughput. We were

[9:49](https://www.youtube.com/watch?v=hX140a5klW0&t=589s) able to achieve over 20% increase. Uh

[9:52](https://www.youtube.com/watch?v=hX140a5klW0&t=592s) our land rate went from 130 to about 160

[9:56](https://www.youtube.com/watch?v=hX140a5klW0&t=596s) during business hours. But it came at a

[9:59](https://www.youtube.com/watch?v=hX140a5klW0&t=599s) cost and the cost is that developers

[10:02](https://www.youtube.com/watch?v=hX140a5klW0&t=602s) become the scheduleulers. uh not all

[10:05](https://www.youtube.com/watch?v=hX140a5klW0&t=605s) developers since we do protect the

[10:07](https://www.youtube.com/watch?v=hX140a5klW0&t=607s) smaller changes. But you can imagine if

[10:09](https://www.youtube.com/watch?v=hX140a5klW0&t=609s) you are a developer who owns any types

[10:12](https://www.youtube.com/watch?v=hX140a5klW0&t=612s) of libraries uh that span the entire

[10:16](https://www.youtube.com/watch?v=hX140a5klW0&t=616s) code, you will be particularly targeted

[10:19](https://www.youtube.com/watch?v=hX140a5klW0&t=619s) in this because the work that you do uh

[10:22](https://www.youtube.com/watch?v=hX140a5klW0&t=622s) naturally falls into this expensive

[10:24](https://www.youtube.com/watch?v=hX140a5klW0&t=624s) change bucket. Uh so we managed to

[10:27](https://www.youtube.com/watch?v=hX140a5klW0&t=627s) protect the capacity during peak hours

[10:29](https://www.youtube.com/watch?v=hX140a5klW0&t=629s) but had to make developers wait. Uh next

[10:33](https://www.youtube.com/watch?v=hX140a5klW0&t=633s) slide. So the next thing that we do is

[10:36](https://www.youtube.com/watch?v=hX140a5klW0&t=636s) targeted more towards bettering the

[10:38](https://www.youtube.com/watch?v=hX140a5klW0&t=638s) developer experience uh of uh the cons

[10:42](https://www.youtube.com/watch?v=hX140a5klW0&t=642s) that come from admission gating. Uh what

[10:45](https://www.youtube.com/watch?v=hX140a5klW0&t=645s) if instead of outright rejecting

[10:48](https://www.youtube.com/watch?v=hX140a5klW0&t=648s) expensive changes, we are able to record

[10:50](https://www.youtube.com/watch?v=hX140a5klW0&t=650s) them and replay them back into the main

[10:53](https://www.youtube.com/watch?v=hX140a5klW0&t=653s) queue uh based on the status of the

[10:57](https://www.youtube.com/watch?v=hX140a5klW0&t=657s) queue uh when we have let's say a better

[10:59](https://www.youtube.com/watch?v=hX140a5klW0&t=659s) uh handle on the capacity. And that's

[11:02](https://www.youtube.com/watch?v=hX140a5klW0&t=662s) exactly what deferrals implement. During

[11:04](https://www.youtube.com/watch?v=hX140a5klW0&t=664s) peak hours, if an expensive request

[11:06](https://www.youtube.com/watch?v=hX140a5klW0&t=666s) comes in, we no longer reject it, but

[11:09](https://www.youtube.com/watch?v=hX140a5klW0&t=669s) put it into a deferred queue where it is

[11:11](https://www.youtube.com/watch?v=hX140a5klW0&t=671s) persisted.

[11:13](https://www.youtube.com/watch?v=hX140a5klW0&t=673s) And once the gates permit, usually upon

[11:17](https://www.youtube.com/watch?v=hX140a5klW0&t=677s) uh for example, we can say after peak

[11:20](https://www.youtube.com/watch?v=hX140a5klW0&t=680s) hours uh we put this persisted request

[11:24](https://www.youtube.com/watch?v=hX140a5klW0&t=684s) back onto the active submit queue. So

[11:26](https://www.youtube.com/watch?v=hX140a5klW0&t=686s) essentially we uh shape the traffic in a

[11:30](https://www.youtube.com/watch?v=hX140a5klW0&t=690s) way where we release more expensive

[11:32](https://www.youtube.com/watch?v=hX140a5klW0&t=692s) changes when the queue is able to handle

[11:35](https://www.youtube.com/watch?v=hX140a5klW0&t=695s) it. So what this solves is a developer

[11:38](https://www.youtube.com/watch?v=hX140a5klW0&t=698s) submits once and then we automatically

[11:41](https://www.youtube.com/watch?v=hX140a5klW0&t=701s) release it for them when the capacity

[11:43](https://www.youtube.com/watch?v=hX140a5klW0&t=703s) allows. Next slide. So the trade-off

[11:46](https://www.youtube.com/watch?v=hX140a5klW0&t=706s) here is it solves the problem for the

[11:49](https://www.youtube.com/watch?v=hX140a5klW0&t=709s) developer. They don't have to wake up

[11:51](https://www.youtube.com/watch?v=hX140a5klW0&t=711s) early, stay up late, try to find a

[11:53](https://www.youtube.com/watch?v=hX140a5klW0&t=713s) particular window where the queue looks

[11:55](https://www.youtube.com/watch?v=hX140a5klW0&t=715s) good and hit the button on their end. We

[11:59](https://www.youtube.com/watch?v=hX140a5klW0&t=719s) will handle that for them. But the other

[12:01](https://www.youtube.com/watch?v=hX140a5klW0&t=721s) side of this, the bad that comes with

[12:03](https://www.youtube.com/watch?v=hX140a5klW0&t=723s) this with this is that it doesn't really

[12:06](https://www.youtube.com/watch?v=hX140a5klW0&t=726s) solve the waiting for the developer. You

[12:09](https://www.youtube.com/watch?v=hX140a5klW0&t=729s) still have to wait the same amount of

[12:11](https://www.youtube.com/watch?v=hX140a5klW0&t=731s) time for your expensive change to go in

[12:15](https://www.youtube.com/watch?v=hX140a5klW0&t=735s) to the queue. Uh, next slide. So before

[12:19](https://www.youtube.com/watch?v=hX140a5klW0&t=739s) we look at the next uh and final

[12:21](https://www.youtube.com/watch?v=hX140a5klW0&t=741s) optimization that we did uh I want to

[12:24](https://www.youtube.com/watch?v=hX140a5klW0&t=744s) touch a little bit about the uh on the

[12:27](https://www.youtube.com/watch?v=hX140a5klW0&t=747s) concept of what a target overlap is uh

[12:30](https://www.youtube.com/watch?v=hX140a5klW0&t=750s) and how that serializes work in

[12:33](https://www.youtube.com/watch?v=hX140a5klW0&t=753s) something. Um so recall that Fam

[12:35](https://www.youtube.com/watch?v=hX140a5klW0&t=755s) mentioned the target analysis that we

[12:37](https://www.youtube.com/watch?v=hX140a5klW0&t=757s) perform uh to determine whether two

[12:40](https://www.youtube.com/watch?v=hX140a5klW0&t=760s) requests should be serialized or can

[12:43](https://www.youtube.com/watch?v=hX140a5klW0&t=763s) they go in parallel. So an example that

[12:46](https://www.youtube.com/watch?v=hX140a5klW0&t=766s) we are looking at here uh is a strict

[12:48](https://www.youtube.com/watch?v=hX140a5klW0&t=768s) one connected component. What this means

[12:50](https://www.youtube.com/watch?v=hX140a5klW0&t=770s) is that each of these requests A through

[12:53](https://www.youtube.com/watch?v=hX140a5klW0&t=773s) F touch some subset of targets and all

[12:57](https://www.youtube.com/watch?v=hX140a5klW0&t=777s) of there's at least one overlap of

[13:00](https://www.youtube.com/watch?v=hX140a5klW0&t=780s) targets between each pair of views. So

[13:02](https://www.youtube.com/watch?v=hX140a5klW0&t=782s) like A and B overlap on at least one

[13:05](https://www.youtube.com/watch?v=hX140a5klW0&t=785s) target, B and C overlap at least one

[13:07](https://www.youtube.com/watch?v=hX140a5klW0&t=787s) target. Similarly for C, D, E, and F.

[13:10](https://www.youtube.com/watch?v=hX140a5klW0&t=790s) And what this makes uh submit Q do is

[13:13](https://www.youtube.com/watch?v=hX140a5klW0&t=793s) group all of these requests into a

[13:16](https://www.youtube.com/watch?v=hX140a5klW0&t=796s) single component. And that means that

[13:18](https://www.youtube.com/watch?v=hX140a5klW0&t=798s) all of these have to go together. And

[13:22](https://www.youtube.com/watch?v=hX140a5klW0&t=802s) this is basically what prevents uh us

[13:26](https://www.youtube.com/watch?v=hX140a5klW0&t=806s) from doing work faster. Uh and the next

[13:31](https://www.youtube.com/watch?v=hX140a5klW0&t=811s) optimization we do is how do we break

[13:33](https://www.youtube.com/watch?v=hX140a5klW0&t=813s) this? Is there a way where we can break

[13:35](https://www.youtube.com/watch?v=hX140a5klW0&t=815s) this up in a smart way? Um next slide.

[13:39](https://www.youtube.com/watch?v=hX140a5klW0&t=819s) So conflict relaxation is what helps us

[13:42](https://www.youtube.com/watch?v=hX140a5klW0&t=822s) break these links uh in an informed

[13:45](https://www.youtube.com/watch?v=hX140a5klW0&t=825s) fashion. Uh before is basically the

[13:48](https://www.youtube.com/watch?v=hX140a5klW0&t=828s) status quo. Any relevant overlap between

[13:52](https://www.youtube.com/watch?v=hX140a5klW0&t=832s) two requests uh makes them conflict with

[13:55](https://www.youtube.com/watch?v=hX140a5klW0&t=835s) each other and makes uh them serialize

[13:58](https://www.youtube.com/watch?v=hX140a5klW0&t=838s) one behind the other. But with

[14:00](https://www.youtube.com/watch?v=hX140a5klW0&t=840s) relaxation we essentially do three

[14:03](https://www.youtube.com/watch?v=hX140a5klW0&t=843s) steps. First we rank the overlap. So

[14:05](https://www.youtube.com/watch?v=hX140a5klW0&t=845s) it's no longer a binary decision of is

[14:09](https://www.youtube.com/watch?v=hX140a5klW0&t=849s) there at least one overlap between these

[14:11](https://www.youtube.com/watch?v=hX140a5klW0&t=851s) two requests. Think of it more as a how

[14:15](https://www.youtube.com/watch?v=hX140a5klW0&t=855s) strong is this overlap. Uh once we have

[14:18](https://www.youtube.com/watch?v=hX140a5klW0&t=858s) that we keep the stronger links and

[14:21](https://www.youtube.com/watch?v=hX140a5klW0&t=861s) relax the weaker ones to buy back

[14:24](https://www.youtube.com/watch?v=hX140a5klW0&t=864s) parallelism based on a config value that

[14:28](https://www.youtube.com/watch?v=hX140a5klW0&t=868s) we set. Um next slide. So the first step

[14:31](https://www.youtube.com/watch?v=hX140a5klW0&t=871s) is essentially scoring these edges uh or

[14:34](https://www.youtube.com/watch?v=hX140a5klW0&t=874s) these overlaps. We use a jakard scoring

[14:37](https://www.youtube.com/watch?v=hX140a5klW0&t=877s) uh for computing this dependency

[14:39](https://www.youtube.com/watch?v=hX140a5klW0&t=879s) overlap. We basically have overlapping

[14:41](https://www.youtube.com/watch?v=hX140a5klW0&t=881s) targets. Uh we check for that and any

[14:44](https://www.youtube.com/watch?v=hX140a5klW0&t=884s) new dependency edges that are

[14:46](https://www.youtube.com/watch?v=hX140a5klW0&t=886s) introduced. We compute a combination of

[14:49](https://www.youtube.com/watch?v=hX140a5klW0&t=889s) these two into a continuous valued score

[14:51](https://www.youtube.com/watch?v=hX140a5klW0&t=891s) between zero and one where a higher

[14:54](https://www.youtube.com/watch?v=hX140a5klW0&t=894s) value of score indicates a stronger

[14:57](https://www.youtube.com/watch?v=hX140a5klW0&t=897s) overlap and a lower value of the score

[15:00](https://www.youtube.com/watch?v=hX140a5klW0&t=900s) indicates a weaker edge with fewer

[15:03](https://www.youtube.com/watch?v=hX140a5klW0&t=903s) targets that overlap between requests.

[15:06](https://www.youtube.com/watch?v=hX140a5klW0&t=906s) Next slide. Once we are armed with this

[15:08](https://www.youtube.com/watch?v=hX140a5klW0&t=908s) knowledge, we basically uh make this

[15:11](https://www.youtube.com/watch?v=hX140a5klW0&t=911s) conflict graph again. Right? So taking

[15:13](https://www.youtube.com/watch?v=hX140a5klW0&t=913s) the same example and adding some numbers

[15:16](https://www.youtube.com/watch?v=hX140a5klW0&t=916s) to the uh conflicts that we have uh from

[15:19](https://www.youtube.com/watch?v=hX140a5klW0&t=919s) the earlier slide. Let's say we have

[15:21](https://www.youtube.com/watch?v=hX140a5klW0&t=921s) these six changes and the config value

[15:24](https://www.youtube.com/watch?v=hX140a5klW0&t=924s) we set says we want at least 50%

[15:27](https://www.youtube.com/watch?v=hX140a5klW0&t=927s) parallelism in the queue at all times.

[15:29](https://www.youtube.com/watch?v=hX140a5klW0&t=929s) What this means is that we need at least

[15:33](https://www.youtube.com/watch?v=hX140a5klW0&t=933s) three separate cues that are working uh

[15:37](https://www.youtube.com/watch?v=hX140a5klW0&t=937s) independent of each other and based on

[15:41](https://www.youtube.com/watch?v=hX140a5klW0&t=941s) the u overlap scores that we have

[15:45](https://www.youtube.com/watch?v=hX140a5klW0&t=945s) computed we are able to make this choice

[15:48](https://www.youtube.com/watch?v=hX140a5klW0&t=948s) of dropping certain edges in order to be

[15:51](https://www.youtube.com/watch?v=hX140a5klW0&t=951s) able to achieve that target uh Q size

[15:55](https://www.youtube.com/watch?v=hX140a5klW0&t=955s) that we have. So in this case uh the

[15:57](https://www.youtube.com/watch?v=hX140a5klW0&t=957s) weakest edges are between B and C and B

[16:00](https://www.youtube.com/watch?v=hX140a5klW0&t=960s) and E and we drop those two which gives

[16:03](https://www.youtube.com/watch?v=hX140a5klW0&t=963s) us three separate connected components.

[16:06](https://www.youtube.com/watch?v=hX140a5klW0&t=966s) Uh and this is how we essentially go

[16:09](https://www.youtube.com/watch?v=hX140a5klW0&t=969s) from a serial chain of six requests to a

[16:14](https://www.youtube.com/watch?v=hX140a5klW0&t=974s) parallelized chain where two each exist

[16:16](https://www.youtube.com/watch?v=hX140a5klW0&t=976s) in three separate cues.

[16:19](https://www.youtube.com/watch?v=hX140a5klW0&t=979s) Next slide. So next comes a question of

[16:23](https://www.youtube.com/watch?v=hX140a5klW0&t=983s) um where exactly do we make this

[16:25](https://www.youtube.com/watch?v=hX140a5klW0&t=985s) determination of uh dropping the edges

[16:29](https://www.youtube.com/watch?v=hX140a5klW0&t=989s) right so there are two places where we

[16:31](https://www.youtube.com/watch?v=hX140a5klW0&t=991s) can do this uh one is at a global level

[16:35](https://www.youtube.com/watch?v=hX140a5klW0&t=995s) every time uh we are trying to assign uh

[16:38](https://www.youtube.com/watch?v=hX140a5klW0&t=998s) config graphs and building them uh we

[16:41](https://www.youtube.com/watch?v=hX140a5klW0&t=1001s) can basically recomputee this every time

[16:44](https://www.youtube.com/watch?v=hX140a5klW0&t=1004s) and what that does is it introduces some

[16:48](https://www.youtube.com/watch?v=hX140a5klW0&t=1008s) level of instability into the graph

[16:51](https://www.youtube.com/watch?v=hX140a5klW0&t=1011s) which is determined by any new requests

[16:53](https://www.youtube.com/watch?v=hX140a5klW0&t=1013s) that enter the queue, any work that

[16:56](https://www.youtube.com/watch?v=hX140a5klW0&t=1016s) finishes and exits the queue. So uh

[17:00](https://www.youtube.com/watch?v=hX140a5klW0&t=1020s) essentially what we we can do that uh

[17:02](https://www.youtube.com/watch?v=hX140a5klW0&t=1022s) but what we chose to do instead was a

[17:05](https://www.youtube.com/watch?v=hX140a5klW0&t=1025s) more uh local assignment uh of overlaps

[17:09](https://www.youtube.com/watch?v=hX140a5klW0&t=1029s) and the place that we choose to do this

[17:12](https://www.youtube.com/watch?v=hX140a5klW0&t=1032s) at is only at the new arrival uh

[17:15](https://www.youtube.com/watch?v=hX140a5klW0&t=1035s) checkpoint. So once a new request comes

[17:19](https://www.youtube.com/watch?v=hX140a5klW0&t=1039s) in, we essentially take that and compute

[17:22](https://www.youtube.com/watch?v=hX140a5klW0&t=1042s) the edges of this with everything that

[17:24](https://www.youtube.com/watch?v=hX140a5klW0&t=1044s) exists in the queue, make the decisions

[17:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1047s) and that's how it stays. The next

[17:30](https://www.youtube.com/watch?v=hX140a5klW0&t=1050s) decision making will only happen when

[17:32](https://www.youtube.com/watch?v=hX140a5klW0&t=1052s) another new arrival comes in. So the

[17:35](https://www.youtube.com/watch?v=hX140a5klW0&t=1055s) concept is simple. We assign once at

[17:37](https://www.youtube.com/watch?v=hX140a5klW0&t=1057s) entry time uh and basically reuse this.

[17:40](https://www.youtube.com/watch?v=hX140a5klW0&t=1060s) Um what this helps us do is any work

[17:43](https://www.youtube.com/watch?v=hX140a5klW0&t=1063s) that we schedule based on previous

[17:47](https://www.youtube.com/watch?v=hX140a5klW0&t=1067s) conflicts gra conflict graphs that we

[17:49](https://www.youtube.com/watch?v=hX140a5klW0&t=1069s) have created is preserved uh and it

[17:52](https://www.youtube.com/watch?v=hX140a5klW0&t=1072s) helps us hang on to speculative builds

[17:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1075s) that can help us move faster uh towards

[17:58](https://www.youtube.com/watch?v=hX140a5klW0&t=1078s) a finalization of request.

[18:01](https://www.youtube.com/watch?v=hX140a5klW0&t=1081s) Next slide. So the tradeoff of this

[18:03](https://www.youtube.com/watch?v=hX140a5klW0&t=1083s) final um intervention was the biggest

[18:07](https://www.youtube.com/watch?v=hX140a5klW0&t=1087s) that we saw. uh we saw over 50% increase

[18:11](https://www.youtube.com/watch?v=hX140a5klW0&t=1091s) in the throughput. So we went from about

[18:13](https://www.youtube.com/watch?v=hX140a5klW0&t=1093s) 160 lines per hour to almost 250

[18:17](https://www.youtube.com/watch?v=hX140a5klW0&t=1097s) sometimes even greater uh during peak

[18:19](https://www.youtube.com/watch?v=hX140a5klW0&t=1099s) hours. Uh so it worked in that it not

[18:22](https://www.youtube.com/watch?v=hX140a5klW0&t=1102s) only helped us uh significantly improve

[18:25](https://www.youtube.com/watch?v=hX140a5klW0&t=1105s) our throughputs while giving us uh

[18:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1109s) extremely good uh P95 land times but it

[18:33](https://www.youtube.com/watch?v=hX140a5klW0&t=1113s) came at a cost as everything does and

[18:36](https://www.youtube.com/watch?v=hX140a5klW0&t=1116s) the trade-off that we have given away

[18:38](https://www.youtube.com/watch?v=hX140a5klW0&t=1118s) here is basically what Prium alluded to

[18:41](https://www.youtube.com/watch?v=hX140a5klW0&t=1121s) in the very first slide uh talking about

[18:43](https://www.youtube.com/watch?v=hX140a5klW0&t=1123s) the integration testing some

[18:46](https://www.youtube.com/watch?v=hX140a5klW0&t=1126s) combinations of changes will land

[18:49](https://www.youtube.com/watch?v=hX140a5klW0&t=1129s) untested

[18:50](https://www.youtube.com/watch?v=hX140a5klW0&t=1130s) because we are dropping edges from our

[18:53](https://www.youtube.com/watch?v=hX140a5klW0&t=1133s) conflict graph. What this means is main

[18:56](https://www.youtube.com/watch?v=hX140a5klW0&t=1136s) can now be broken. Uh and this creates

[19:00](https://www.youtube.com/watch?v=hX140a5klW0&t=1140s) essentially a new obligation where we

[19:04](https://www.youtube.com/watch?v=hX140a5klW0&t=1144s) have to validate uh what lands after it

[19:08](https://www.youtube.com/watch?v=hX140a5klW0&t=1148s) has landed. Uh and to talk a little bit

[19:10](https://www.youtube.com/watch?v=hX140a5klW0&t=1150s) more about how we do this, I will hand

[19:13](https://www.youtube.com/watch?v=hX140a5klW0&t=1153s) it back to Kam.

[19:15](https://www.youtube.com/watch?v=hX140a5klW0&t=1155s) Great. Um thanks Mandre uh for walking

[19:18](https://www.youtube.com/watch?v=hX140a5klW0&t=1158s) us through the uh those three um sort of

[19:22](https://www.youtube.com/watch?v=hX140a5klW0&t=1162s) the solutions that we employed. So uh

[19:25](https://www.youtube.com/watch?v=hX140a5klW0&t=1165s) taking back this example that we had

[19:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1167s) initially we had basically two changes

[19:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1169s) here A and B and say we do a premerge

[19:33](https://www.youtube.com/watch?v=hX140a5klW0&t=1173s) validation independently after relaxing

[19:36](https://www.youtube.com/watch?v=hX140a5klW0&t=1176s) those uh conflict between those two

[19:38](https://www.youtube.com/watch?v=hX140a5klW0&t=1178s) things and then say we land it. So now

[19:41](https://www.youtube.com/watch?v=hX140a5klW0&t=1181s) our the landed state is essentially a

[19:44](https://www.youtube.com/watch?v=hX140a5klW0&t=1184s) lands first then b and now the

[19:47](https://www.youtube.com/watch?v=hX140a5klW0&t=1187s) combination of a plus b uh fails on the

[19:50](https://www.youtube.com/watch?v=hX140a5klW0&t=1190s) trunk. Right? So this is where we

[19:52](https://www.youtube.com/watch?v=hX140a5klW0&t=1192s) started um at the start of this talk and

[19:56](https://www.youtube.com/watch?v=hX140a5klW0&t=1196s) so now we have a broken trunk again. So

[20:00](https://www.youtube.com/watch?v=hX140a5klW0&t=1200s) now how are we dealing with this?

[20:03](https://www.youtube.com/watch?v=hX140a5klW0&t=1203s) So we have a system uh here called

[20:06](https://www.youtube.com/watch?v=hX140a5klW0&t=1206s) stowpipe and stowpipe is our like post

[20:10](https://www.youtube.com/watch?v=hX140a5klW0&t=1210s) uh land uh validation system. Um it's

[20:13](https://www.youtube.com/watch?v=hX140a5klW0&t=1213s) responsible for validating any commit

[20:14](https://www.youtube.com/watch?v=hX140a5klW0&t=1214s) that lands on to the main and then

[20:17](https://www.youtube.com/watch?v=hX140a5klW0&t=1217s) making sure that uh it it detects

[20:20](https://www.youtube.com/watch?v=hX140a5klW0&t=1220s) basically if bills are broken and if so

[20:23](https://www.youtube.com/watch?v=hX140a5klW0&t=1223s) then it creates a uh recovery uh

[20:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1227s) strategy for that. Now uh here we have

[20:31](https://www.youtube.com/watch?v=hX140a5klW0&t=1231s) basically two things. One is every

[20:34](https://www.youtube.com/watch?v=hX140a5klW0&t=1234s) change that is being landed by summit Q

[20:36](https://www.youtube.com/watch?v=hX140a5klW0&t=1236s) um lands onto the main trunk and then we

[20:39](https://www.youtube.com/watch?v=hX140a5klW0&t=1239s) have the second reference here or the uh

[20:42](https://www.youtube.com/watch?v=hX140a5klW0&t=1242s) trunk um the pointer if you call is the

[20:45](https://www.youtube.com/watch?v=hX140a5klW0&t=1245s) verified main and the verified main is

[20:47](https://www.youtube.com/watch?v=hX140a5klW0&t=1247s) essentially the one that is being uh

[20:50](https://www.youtube.com/watch?v=hX140a5klW0&t=1250s) managed uh by store. So let's take an

[20:53](https://www.youtube.com/watch?v=hX140a5klW0&t=1253s) example to understand it better. So

[20:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1255s) imagine we have this comet A u where

[20:59](https://www.youtube.com/watch?v=hX140a5klW0&t=1259s) main and verified main both are pointing

[21:01](https://www.youtube.com/watch?v=hX140a5klW0&t=1261s) to it. This is a good known comet. Uh

[21:04](https://www.youtube.com/watch?v=hX140a5klW0&t=1264s) this is a building uh this is green on

[21:06](https://www.youtube.com/watch?v=hX140a5klW0&t=1266s) the trunk. Now let's say we have uh

[21:10](https://www.youtube.com/watch?v=hX140a5klW0&t=1270s) another set of comet that land onto the

[21:12](https://www.youtube.com/watch?v=hX140a5klW0&t=1272s) main say B C and D. Now main moves to

[21:16](https://www.youtube.com/watch?v=hX140a5klW0&t=1276s) point uh to the D uh commit. So at this

[21:20](https://www.youtube.com/watch?v=hX140a5klW0&t=1280s) point a STO pipe is going to take this

[21:22](https://www.youtube.com/watch?v=hX140a5klW0&t=1282s) range of commits essentially B, C and D

[21:25](https://www.youtube.com/watch?v=hX140a5klW0&t=1285s) on top of A and validate this newly uh

[21:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1289s) landed changes to see if this new state

[21:32](https://www.youtube.com/watch?v=hX140a5klW0&t=1292s) is still buildable and is still passing

[21:35](https://www.youtube.com/watch?v=hX140a5klW0&t=1295s) or not. In this case we can see it

[21:38](https://www.youtube.com/watch?v=hX140a5klW0&t=1298s) fails. And so now store pipe is not

[21:41](https://www.youtube.com/watch?v=hX140a5klW0&t=1301s) going to move the verified main pointer

[21:43](https://www.youtube.com/watch?v=hX140a5klW0&t=1303s) in this case. it will just keep it at a

[21:45](https://www.youtube.com/watch?v=hX140a5klW0&t=1305s) because that's what the good known last

[21:48](https://www.youtube.com/watch?v=hX140a5klW0&t=1308s) uh commit is on the trunk. Now over time

[21:53](https://www.youtube.com/watch?v=hX140a5klW0&t=1313s) more commits uh land onto the trunk and

[21:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1315s) main moves to commit G. So now between

[21:59](https://www.youtube.com/watch?v=hX140a5klW0&t=1319s) uh the new state that we have G and the

[22:02](https://www.youtube.com/watch?v=hX140a5klW0&t=1322s) previously uh good known state which is

[22:05](https://www.youtube.com/watch?v=hX140a5klW0&t=1325s) A we have more commits in between. So we

[22:08](https://www.youtube.com/watch?v=hX140a5klW0&t=1328s) have B C D E F and G commits in between.

[22:12](https://www.youtube.com/watch?v=hX140a5klW0&t=1332s) So now store pipe is going to take those

[22:15](https://www.youtube.com/watch?v=hX140a5klW0&t=1335s) commit uh range and basically validate

[22:18](https://www.youtube.com/watch?v=hX140a5klW0&t=1338s) and see if those are buildable and if

[22:20](https://www.youtube.com/watch?v=hX140a5klW0&t=1340s) bills are passing. So now say in this

[22:24](https://www.youtube.com/watch?v=hX140a5klW0&t=1344s) case builds are passing and so the store

[22:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1347s) pipe will basically take the verified

[22:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1349s) main pointer and point it to the newly

[22:32](https://www.youtube.com/watch?v=hX140a5klW0&t=1352s) uh validated uh good known uh commit uh

[22:36](https://www.youtube.com/watch?v=hX140a5klW0&t=1356s) g in this case. So that's what a store

[22:38](https://www.youtube.com/watch?v=hX140a5klW0&t=1358s) pipe does is it is essentially observing

[22:41](https://www.youtube.com/watch?v=hX140a5klW0&t=1361s) the newly uh landed commits onto the

[22:43](https://www.youtube.com/watch?v=hX140a5klW0&t=1363s) main uh verifying if they can build or

[22:45](https://www.youtube.com/watch?v=hX140a5klW0&t=1365s) not and if they pass move the pointer if

[22:49](https://www.youtube.com/watch?v=hX140a5klW0&t=1369s) not then uh continuously look for more

[22:52](https://www.youtube.com/watch?v=hX140a5klW0&t=1372s) changes and keep validating.

[22:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1375s) So now let's look at like when uh a

[22:59](https://www.youtube.com/watch?v=hX140a5klW0&t=1379s) comet landed on main u breaks the trunk.

[23:02](https://www.youtube.com/watch?v=hX140a5klW0&t=1382s) So what happens in that case as I said

[23:05](https://www.youtube.com/watch?v=hX140a5klW0&t=1385s) earlier stopipe is essentially

[23:06](https://www.youtube.com/watch?v=hX140a5klW0&t=1386s) validating and then detecting that it

[23:09](https://www.youtube.com/watch?v=hX140a5klW0&t=1389s) failed and on failure it basically

[23:11](https://www.youtube.com/watch?v=hX140a5klW0&t=1391s) alerts. So on call gets paged uh and as

[23:15](https://www.youtube.com/watch?v=hX140a5klW0&t=1395s) a response to this um broken trunk on

[23:18](https://www.youtube.com/watch?v=hX140a5klW0&t=1398s) call is responsible to prepare a fix or

[23:21](https://www.youtube.com/watch?v=hX140a5klW0&t=1401s) revert that change um get it removed and

[23:24](https://www.youtube.com/watch?v=hX140a5klW0&t=1404s) then land through summit cube and then

[23:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1407s) eventually it merges back into the trunk

[23:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1409s) and then a stow pipe continuously

[23:32](https://www.youtube.com/watch?v=hX140a5klW0&t=1412s) validates the new state to see if it has

[23:34](https://www.youtube.com/watch?v=hX140a5klW0&t=1414s) been fixed or not and then accordingly

[23:37](https://www.youtube.com/watch?v=hX140a5klW0&t=1417s) move the pointer um to the newly landed

[23:40](https://www.youtube.com/watch?v=hX140a5klW0&t=1420s) comet.

[23:42](https://www.youtube.com/watch?v=hX140a5klW0&t=1422s) So uh with all of this um I want to just

[23:46](https://www.youtube.com/watch?v=hX140a5klW0&t=1426s) recap on basically what we have

[23:48](https://www.youtube.com/watch?v=hX140a5klW0&t=1428s) achieved. So uh at peak times we are

[23:51](https://www.youtube.com/watch?v=hX140a5klW0&t=1431s) able to do roughly uh 300 uh lands per

[23:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1435s) hour. we are still able to maintain our

[23:58](https://www.youtube.com/watch?v=hX140a5klW0&t=1438s) P95 land time to 30 minutes and this is

[24:01](https://www.youtube.com/watch?v=hX140a5klW0&t=1441s) essentially a 233% increase uh since the

[24:06](https://www.youtube.com/watch?v=hX140a5klW0&t=1446s) January uh numbers that we had which

[24:08](https://www.youtube.com/watch?v=hX140a5klW0&t=1448s) were like 90 lines uh per hour.

[24:12](https://www.youtube.com/watch?v=hX140a5klW0&t=1452s) So u with this u we are doing all this

[24:16](https://www.youtube.com/watch?v=hX140a5klW0&t=1456s) work um in the open u we have summit Q

[24:19](https://www.youtube.com/watch?v=hX140a5klW0&t=1459s) and store pipe being uh developed and uh

[24:24](https://www.youtube.com/watch?v=hX140a5klW0&t=1464s) everything is in the open source. So if

[24:26](https://www.youtube.com/watch?v=hX140a5klW0&t=1466s) you want to explore and contribute um do

[24:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1469s) scan the QR code explore the project and

[24:32](https://www.youtube.com/watch?v=hX140a5klW0&t=1472s) uh feel feel free to uh contribute to

[24:35](https://www.youtube.com/watch?v=hX140a5klW0&t=1475s) the project. Um what are we doing next?

[24:39](https://www.youtube.com/watch?v=hX140a5klW0&t=1479s) And um as we have already done this work

[24:42](https://www.youtube.com/watch?v=hX140a5klW0&t=1482s) uh we are still uh looking ways to uh

[24:46](https://www.youtube.com/watch?v=hX140a5klW0&t=1486s) make it better uh optimize for

[24:48](https://www.youtube.com/watch?v=hX140a5klW0&t=1488s) throughput handle the uh the recovery

[24:52](https://www.youtube.com/watch?v=hX140a5klW0&t=1492s) process in better way and also like

[24:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1495s) improve on our like how we how we sort

[24:58](https://www.youtube.com/watch?v=hX140a5klW0&t=1498s) of tag what is green or not. So as a

[25:02](https://www.youtube.com/watch?v=hX140a5klW0&t=1502s) first thing we are thinking about is on

[25:04](https://www.youtube.com/watch?v=hX140a5klW0&t=1504s) summit Q side basically batching uh

[25:06](https://www.youtube.com/watch?v=hX140a5klW0&t=1506s) related changes together so that way we

[25:08](https://www.youtube.com/watch?v=hX140a5klW0&t=1508s) run less number of builds and we can do

[25:10](https://www.youtube.com/watch?v=hX140a5klW0&t=1510s) more work uh with the same uh same

[25:14](https://www.youtube.com/watch?v=hX140a5klW0&t=1514s) amount of resources. Second is uh today

[25:17](https://www.youtube.com/watch?v=hX140a5klW0&t=1517s) we have a single pointer verified main

[25:19](https://www.youtube.com/watch?v=hX140a5klW0&t=1519s) which points whether a repository head

[25:22](https://www.youtube.com/watch?v=hX140a5klW0&t=1522s) is green or not or a given commit uh is

[25:24](https://www.youtube.com/watch?v=hX140a5klW0&t=1524s) green or not. We want to do better where

[25:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1527s) basically we can say a given service is

[25:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1529s) green, given project is green. So even

[25:31](https://www.youtube.com/watch?v=hX140a5klW0&t=1531s) if a comet on the trunk is broken, uh we

[25:36](https://www.youtube.com/watch?v=hX140a5klW0&t=1536s) can essentially tell which part of the

[25:38](https://www.youtube.com/watch?v=hX140a5klW0&t=1538s) trunk is broken and not all of it. So

[25:40](https://www.youtube.com/watch?v=hX140a5klW0&t=1540s) that would allow us to basically do a

[25:43](https://www.youtube.com/watch?v=hX140a5klW0&t=1543s) better um uh sort of rollouts of those

[25:46](https://www.youtube.com/watch?v=hX140a5klW0&t=1546s) services and changes um and we don't

[25:49](https://www.youtube.com/watch?v=hX140a5klW0&t=1549s) have to block everything. And the third

[25:51](https://www.youtube.com/watch?v=hX140a5klW0&t=1551s) one is basically uh as I said the the

[25:53](https://www.youtube.com/watch?v=hX140a5klW0&t=1553s) recovery process from a broken trunk is

[25:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1555s) manual. So uh the most of the investment

[25:58](https://www.youtube.com/watch?v=hX140a5klW0&t=1558s) is around automating the fixes or

[26:01](https://www.youtube.com/watch?v=hX140a5klW0&t=1561s) reverting the fixes and then landing

[26:03](https://www.youtube.com/watch?v=hX140a5klW0&t=1563s) those uh to summit queue uh in automated

[26:06](https://www.youtube.com/watch?v=hX140a5klW0&t=1566s) way so that our time to recovery can be

[26:09](https://www.youtube.com/watch?v=hX140a5klW0&t=1569s) uh reduced and we can fix the broken

[26:12](https://www.youtube.com/watch?v=hX140a5klW0&t=1572s) trunks faster.

[26:14](https://www.youtube.com/watch?v=hX140a5klW0&t=1574s) And so as a closing point uh I want to

[26:17](https://www.youtube.com/watch?v=hX140a5klW0&t=1577s) call out that we chose a hybrid approach

[26:20](https://www.youtube.com/watch?v=hX140a5klW0&t=1580s) uh to validating the changes. Um one we

[26:24](https://www.youtube.com/watch?v=hX140a5klW0&t=1584s) are still doing the premerge um uh

[26:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1587s) validations. Um we are still uh making

[26:30](https://www.youtube.com/watch?v=hX140a5klW0&t=1590s) sure that we test the changes uh where

[26:34](https://www.youtube.com/watch?v=hX140a5klW0&t=1594s) it matters and relax the changes uh

[26:37](https://www.youtube.com/watch?v=hX140a5klW0&t=1597s) where we can take that hit and valid

[26:40](https://www.youtube.com/watch?v=hX140a5klW0&t=1600s) validate them later uh after they land

[26:42](https://www.youtube.com/watch?v=hX140a5klW0&t=1602s) to the uh trunk and then we are running

[26:45](https://www.youtube.com/watch?v=hX140a5klW0&t=1605s) the whole exhaustive uh validation with

[26:47](https://www.youtube.com/watch?v=hX140a5klW0&t=1607s) the help of a store pipe um uh post uh

[26:51](https://www.youtube.com/watch?v=hX140a5klW0&t=1611s) landing to the trunk and that allows us

[26:53](https://www.youtube.com/watch?v=hX140a5klW0&t=1613s) to basically have this balance where we

[26:55](https://www.youtube.com/watch?v=hX140a5klW0&t=1615s) are validating uh whatever whatever

[26:57](https://www.youtube.com/watch?v=hX140a5klW0&t=1617s) matters before and then validating

[26:59](https://www.youtube.com/watch?v=hX140a5klW0&t=1619s) everything at the end to make sure it is

[27:01](https://www.youtube.com/watch?v=hX140a5klW0&t=1621s) still good. Um and uh our consumers can

[27:04](https://www.youtube.com/watch?v=hX140a5klW0&t=1624s) consume the the good estate

[27:07](https://www.youtube.com/watch?v=hX140a5klW0&t=1627s) and with that um that would be our end

[27:10](https://www.youtube.com/watch?v=hX140a5klW0&t=1630s) of the talk. Um um if you want to learn

[27:13](https://www.youtube.com/watch?v=hX140a5klW0&t=1633s) more about Summit Q and inner workings

[27:16](https://www.youtube.com/watch?v=hX140a5klW0&t=1636s) of it, feel free to scan these QR codes.

[27:19](https://www.youtube.com/watch?v=hX140a5klW0&t=1639s) Um these are the previously uh talks

[27:21](https://www.youtube.com/watch?v=hX140a5klW0&t=1641s) that we have done um on summit Q in

[27:23](https://www.youtube.com/watch?v=hX140a5klW0&t=1643s) depth and they go over like in detail on

[27:27](https://www.youtube.com/watch?v=hX140a5klW0&t=1647s) how how it functions internally and

[27:29](https://www.youtube.com/watch?v=hX140a5klW0&t=1649s) again thank you.

[27:30](https://www.youtube.com/watch?v=hX140a5klW0&t=1650s) Thank you.
