# Give Your On-Call Its Own AI Teammate - Swaapnika Motapothula, Chinmay Gandhi & Vlad Tsvang, Meta

Chinmay Gandhi, Meta · Swaapnika Motapothula, Meta · Vlad Tsvang, Meta

[Talk page](https://ai.engineer/talks/_NBx2NNOuA4-give-your-on-call-its-own-ai) · [Video](https://www.youtube.com/watch?v=_NBx2NNOuA4)

## Give Your On-Call Its Own AI Teammate

At Meta, participating on-call rotations each have a shared agent. It can trace a page to a configuration change, bring the engineer a proposed fix, and remember the decision for the next shift. What it can do on its own depends on the cost of a mistake and its record on that particular action.

### Twelve alerts at 2 a.m.

[0:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=1s)

A change can pass code review and still wake someone at 2 a.m. At Meta, Swaapnika Motapothula, Chinmay Gandhi and Vlad Tsvang build the tools engineers use when that happens. They saw AI accelerate coding while the work of operating the resulting software still fell to the engineer carrying the pager. Their team runs an AI teammate on real rotations around the clock. A simplified night with 12 alerts illustrates the job they gave it.

In the first version, you sort through checkout, payments and disk alerts, pick checkout, and ask an assistant why its 99th-percentile latency is high. It pulls the metrics and finds a deployment at 1:49 a.m. You ask what changed. The assistant finds that the change lowered a timing setting and checks the runbook, which recommends a rollback. By 2:39 you know the likely cause. You chose each question, and the rollback and follow-up work are still yours.

What would be waiting for you if the investigation started before you woke up? Given the same 12 alerts, the teammate ranks checkout first, groups payments with it because the two share a cause, and handles the other ten separately. It reads the dashboards, inspects the 1:49 configuration change and finds two similar incidents. You get a hypothesis, a quick way to check it and a fix ready for review. In this version of the night, you verify the finding and go back to sleep.

### The clues belonged to four different systems

[3:18](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=198s)

Replay the checkout page from the perspective of the existing tools. The alert script saw P99 latency exceed two seconds. The triage bot saw three tickets reporting slow checkout. The deployment bot knew about the 1:49 configuration push. The incident history contained two precedents. Each tool had enough context to do its own job. Connecting a user report to a latency spike, a specific change and a possible remedy was a separate job, and it belonged to whoever answered the page.

The team chose one teammate per rotation. It can investigate across those signals, keep a team memory across shifts, and act under its own identity. Underneath, the agent is a model that calls tools, reads what happened and uses the results to choose its next step. Giving that loop enough reach to investigate checkout raises a practical question. Who decides whether the proposed remedy should touch production?

### How an action earns its place on the autonomy ladder

[5:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=308s)

A new on-call engineer starts by shadowing someone experienced. The newcomer investigates incidents and suggests fixes. Later the pair can divide the work of writing and approving a change. The team gives the agent a similar progression.

- **Level one — investigate.** The agent searches in layers, drawing on team memory and the runbook. It stops at the first answer it considers confident and shares its root-cause analysis. A human applies the fix.
- **Level two — draft a fix and follow it through review.** The agent writes the change, explains the cause and supplies a test plan. It answers reviewer comments and addresses continuous integration (CI) failures. A change needs human approval to reach production. Most drafts never land.
- **Level three — clear selected routine work.** The agent can close a noisy alert on its own. High-criticality work stays with the engineer, including the decision to apply a production mitigation.

Promotion applies to a particular action. What would a mistake cost? How readily would tests, CI or review catch it? How has the agent performed on that same action? A history of correctly closing noisy alerts earns trust for closing noisy alerts. Rolling checkout back carries different consequences. At the first two levels, the agent can do extensive homework while leaving the engineer with just as many items to grade. The third level changes how many items reach the engineer.

![Three ascending blocks show an agent advising, drafting fixes for approval, and handling lower-criticality work autonomously. Advancement requires low-cost mistakes, ways to catch them, and a track record on the specific action.](https://ai.engineer/api/talk-essay-media/2b2cfbf323b047f7f775ecbb033324ba7e61d31624a1e98fa07fb7aa7c1c717c)

The ladder distinguishes offering advice, acting with approval and acting autonomously.

### Of 23 incoming items, which three need a decision?

[7:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=435s)

The example board groups work by who needs to act next. Of 23 incoming items, the agent has handled 18, including noisy alerts, duplicates and routine fixes that went through normal review and CI. Two are still in progress with the agent. The remaining three form the engineer’s decision queue. Each item says what it needs and offers a button to take that action. A piece of unfinished work can stay visible on the board without becoming another task for the engineer to drive.

Checkout still belongs in the human queue. The investigation can be ready while the critical decision is pending. The board records the work, and the teammate also participates in chat and in the incident itself. The engineer can ask a question or act on a finding wherever they encounter it.

![An illustrative board shows 23 incoming items. The agent is handling 2 and has resolved 18. The 3 needing a person are a checkout incident, a quota request and an alert that may belong to another team.](https://ai.engineer/api/talk-essay-media/6d58b6982e9f7e05cc955921149270eb368c950ce585966d51a3a11b00de62ef)

The triage screen separates a larger incoming queue from the three items that need the on-call engineer.

### From the morning digest to a checkout rollback

[8:23](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=503s)

At 9 each morning, the teammate posts a digest. In the 23-item example, the 18 handled items get a collective summary. The two in progress get a description of what the agent is doing, and each of the three decisions gets a one-click next step. An engineer can approve a quota increase from a phone. A decision that deserves a closer look can continue in the thread.

What would justify a checkout rollback? In the incident-thread example, the engineer asks for evidence. Errors began at 8:41, two minutes after a configuration push shortened the payment client timeout. Two previous incidents followed the same pattern. The engineer asks whether the problem is confined to the EU and gets a regional breakdown. With the timing, precedents and scope available in one conversation, the engineer approves the drafted rollback there. Errors begin to subside.

### What starts an investigation, and what catches a stalled one?

[9:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=572s)

The rotation sees one teammate. Behind it, three automations run under different conditions. A new signal can start an investigation at any hour. Recovery has its own schedule, and a handoff happens even while some investigations are still open.

- **On a new signal, triage and route it.** The system classifies the signal and checks its criticality. An alert goes to an incident investigator, a task to the team’s own agent, a test failure to a coding agent, and a lookup to memory and code search.
- **On a schedule, check the automation itself.** A health check looks for stuck or failed runs, attempts recovery and repairs broken schedules. If recovery fails, it pages a human. Even an investigation that produces no answer has a way to get attention.
- **At handoff, tell the incoming engineer what needs a decision.** Every shift change and every morning at 9 trigger a summary in team chat, with prepared fixes and an account of work already handled. At handoff, the teammate tags the incoming engineer. An urgent issue can trigger its own chat notification with findings and a draft fix.

A team that already trusts an agent with its runbooks and knowledge base can use it to handle tasks alongside the incident investigator and coding agent. Work can arrive through alerts, tickets, incidents, team posts or chat. The playbook, skills and memory give the generic agent local expertise. Every run is logged, so another engineer can inspect the steps it took and resume the conversation partway through an investigation.

![Alerts, tickets, incidents and messages feed a shared board backed by a scheduler and team memory. Sandboxed agents and skills produce chat updates, code changes for review and CI, and shift summaries.](https://ai.engineer/api/talk-essay-media/170597b048264d390372784c23b633b8109d40efc884b1e8883c9c1fbdd3aece)

Team-specific agents and skills sit beneath shared integrations for on-call work.

### How Monday’s rollback advice reaches the next engineer

[11:44](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=704s)

One rotation shares one board. Whoever takes the pager can question the teammate, correct it or pick up its work. At 2 a.m. on Monday, Sam tells it to remember a piece of advice. If checkout starts timing out after a configuration push, roll back the configuration first. Three days later, Jordan sees the same alert and asks for a fix. The teammate retrieves Sam’s advice, tells Jordan where it came from and has the rollback drafted.

The rotation owns the teammate’s identity and decides what access to grant it. Remembering Sam’s advice lets it prepare a change for Jordan. Applying a production mitigation still takes a human decision, and that decision can itself become part of the next shift’s memory.

Advice can age. The teammate retrieves entries by meaning, uses relevant ones in its work, then scores them on whether they helped. It can promote a useful entry or retire one automatically. A correction has two effects on different timescales. If an engineer says the answer was wrong, the work item reopens immediately. The correction also enters memory, where a later investigation can retrieve it.

### Adding rotations and measuring the time engineers get back

[13:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=804s)

Taking the system beyond its original team depended on keeping that correction path open and giving each rotation control of its expertise. Teams supply their own playbooks and skills while sharing the underlying implementation. Identity, configuration and memory are set up per team. Adding a rotation becomes a setup task, and teams can bring an existing workflow with them without maintaining a fork.

The change engineers can describe in practice is where their work starts. They have a hypothesis with linked evidence to examine and a decision to make. Teams running the agent autonomously report early signs of a lighter load. How much on-call time they get back remains open. The planned study will measure engineers’ time directly, since usage counts alone cannot answer that question.

Code changes still go through human review and CI, and most drafts never land. That makes approval a useful outcome to learn from. The team grades every proposed mitigation and feeds the outcome back, aiming to produce more fixes that engineers will accept. It also wants teams to be able to turn a runbook into a skill from chat without writing code. The next autonomy decision concerns medium-criticality work. To earn it, the teammate will need a record that justifies taking those particular actions.

## Key takeaways

- A rotation-wide investigation can connect a latency alert, three checkout tickets, a configuration push and two past incidents before the engineer chooses a first query.
- Grant autonomy action by action. The cost of a mistake, the checks that can catch it and the agent’s record on that action determine whether it can proceed alone.
- A correction can help two engineers. It reopens the item for the person on call now and gives the next person’s investigation new evidence to retrieve.

## Slides

### 3 Levels of Autonomy: Building an AI On-Call Teammate at Meta

Meta’s AI teammate runs on-call rotations around the clock. Three rings show how its role expands from advising engineers to acting with their approval and then acting on its own.

[0:00](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=0s) [The talk title and speaker credits appear beside a robot surrounded by three rings labeled Advise, Act with Approval, and Autonomous.](https://assets.ai.engineer/sites/aiecode2025/sha256/573cb0cab6f0ec19113dd3d8ad6db8a9461485d55346a0d6b32d5f5ad4a606cd.webp)

3 Levels of Autonomy: Building an AI On-Call Teammate at Meta

> agent://oncall-teammate

Chinmay Gandhi · Swaapnika Motapothula · Vlad Tsvang
AI Engineer CODE Summit 2026

1 · ADVISE
2 · ACT WITH APPROVAL
3 · AUTONOMOUS

tasks
chat
alerts
incidents
test failures

Meta

AIE CODE SUMMIT · SF 2026 · 01/16

### Meet the speakers

Software engineer Swaapnika Motapothula, front end engineer Vlad Tsvang, and senior engineering manager Chinmay Gandhi work on the Meta team that builds tools for on-call engineers to respond to incidents and mitigate their impact.

[0:16](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=16s) [From left to right, the cards introduce Swaapnika Motapothula, software engineer, Vlad Tsvang, front end engineer, and Chinmay Gandhi, senior engineering manager, all at Meta.](https://assets.ai.engineer/sites/aiecode2025/sha256/79e26bc1c10b83ac162441bf56a53d75f53fd4235fca8f683378fa7e58cfb410.webp)

Meet the speakers

Swaapnika Motapothula
Software Engineer @ Meta

Vlad Tsvang
Front End Engineer @ Meta

Chinmay Gandhi
Sr. Engineering Manager @ Meta

AIE CODE SUMMIT · SF 2026 · 02/16

### Code got faster. On-call didn’t.

AI helps teams ship code sooner. In a May 2026 CloudBees survey, 81% of enterprise tech leaders reported production failures from AI code. A change can pass review and still wake the on-call engineer at 2 a.m.

[0:30](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=30s) [Three cards labeled Faster, More, and Same cite figures about AI-generated code, reported production failures, and code-review ratings. A banner says the bottleneck moved from writing code to operating it.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9d7658bab7db47044fe2ff1f8dd7d8a9ea28c91e8624815a97019c93e2a938e.webp)

Code got faster. On-call didn’t.

Faster
AI writes more of the code, and it ships sooner

Up to 75% of new code at Big Tech is now AI-generated.
Source: Alphabet CEO, April 2026

More
Failures in production are traced back to AI changes.

81% of enterprise tech leaders report production failures from AI code
Source: CloudBees, May 2026 (vendor survey)

Same
on-call: someone still gets paged when it breaks at 2am

94% rate AI code higher quality in review, yet production incidents rose
Source: New Relic, State of AI Coding 2026 (vendor survey)

The bottleneck moved from writing code to operating it.
On-call is where it lands.

AIE CODE SUMMIT · SF 2026 · 03/16 · OPTION B (no numbers)

### 2:00 AM, 12 alerts: you sort and drive, or wake up to a ranked fix.

In this mock incident, the on-call engineer prompts each step and still owns the fix at 2:39. The AI teammate investigates before the engineer wakes, flags a shorter payment timeout, and offers a 30-second check and a draft fix for review.

[1:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=77s) [Two mock on-call panels show the same checkout latency alert. On the left, an engineer prompts each investigation step. On the right, a teammate presents a suspected timeout change, a verification step, and a draft fix awaiting review.](https://assets.ai.engineer/sites/aiecode2025/sha256/d43db193b364ae9653e6b3d4226c4b18d163927beeafda2bcc55f3889132e004.webp)

// COLD_OPEN
2:00 AM, 12 alerts: you sort and drive, or wake up to a ranked fix.

🔔 checkout p99 > 2s
⚠️ payments 5xx ↑
+10
which one first?

#oncall · 2:00 AM
🔔 PAGE: checkout latency p99 above 2s

you · 2:07
why is checkout p99 latency high?
✕ waits to be asked

AI assistant · 2:07
p99 rose 0.4s → 2.3s at 1:50. Closest change: a payments-client deploy at 1:49. Want me to dig in?
✕ you pick every next step

you · 2:15
asks what the deploy changed + what the runbook says
✕ you are the glue

AI assistant · 2:16
Timeout cut 5s → 800ms. Runbook step 3: roll back.
✕ then goes dormant

2:39 · cause found, but the fix is still yours
Today: an AI assistant

1 🔔 checkout p99 · investigating
2–12 · handled, none need you

#oncall · 2:00 AM
● started before you woke up
🔔 PAGE: checkout latency p99 above 2s

🤖 Hypothesis (high confidence)
Config change at 1:49 lowered a payment-client timeout.
Verify in 30s: compare p99 before and after 1:49
Fix drafted, waiting for your review →

✓ read 5 dashboards
✓ diffed the 1:49 change
✓ matched 2 past incidents

2:01 · verified in 30s → back to sleep
Next: an AI teammate

AIE CODE SUMMIT · SF 2026 · 04/16
Illustrative mock · synthetic data

### The autonomy ladder

An on-call agent earns autonomy for each action when it has a track record on that task and mistakes are cheap and easy to catch. It can close a noisy alert on its own while a production mitigation still needs human approval.

[5:09](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=309s) [Three ascending blocks show an agent advising, drafting fixes for approval, and handling lower-criticality work autonomously. Advancement requires low-cost mistakes, ways to catch them, and a track record on the specific action.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b2cfbf323b047f7f775ecbb033324ba7e61d31624a1e98fa07fb7aa7c1c717c.webp)

The autonomy ladder
// THE_LADDER

WHAT EARNS THE NEXT RUNG
✓ Mistakes are cheap
✓ Mistakes are easy to catch (tests, CI, review)
✓ A track record on that exact action

1 ADVISE
RCA, suggestions, pointers
Agent: investigates, changes nothing
Human: decides and acts

2 ACT WITH APPROVAL
Fixes and mitigations
Agent: drafts the fix, follows review and CI
Human: approves before production

3 AUTONOMOUS
Minimal user input
Agent: clears lower-criticality work 24/7
Human: only the very important things
High criticality stays with humans

AIE CODE SUMMIT · SF 2026 · 06/16

### Levels 1 and 2 of on-call autonomy

At level 1, the agent identifies the 1:49 config change and the on-call engineer applies the fix. At level 2, the agent drafts the revert. The engineer still reviews every draft and decides which fixes reach production.

[6:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=382s) [Two panels compare on-call workflows. At level 1, the agent investigates and a person applies the fix. At level 2, the agent drafts the fix and a person approves it before it lands.](https://assets.ai.engineer/sites/aiecode2025/sha256/12371062a304cef304dee24d098c224a8082be04b05471d70b920d97674431b6.webp)

Levels 1 and 2: the agent does the homework. You still grade every paper.
// LEVEL_1 + LEVEL_2

1   LEVEL 1 · ADVISE · RCA
Agent investigates. Changes nothing.

WHAT THE AGENT DOES
reuse investigator → memory + playbook → deep dive in code
output contract: a fix · a clear ask · silence

WHAT YOU STILL DO
Apply the fix and land it. For every item.

RCA ready: config change at 1:49
→ you apply the fix, you land it
On-call shifts from sifting to deciding.

2   LEVEL 2 · ACT WITH APPROVAL · FIXES + MITIGATIONS
Agent drafts the fix. You approve it.

WHAT THE AGENT DOES
revert 1:49 → review + CI → you approve → lands
Mitigations are proposed, never applied alone.

WHAT YOU STILL DO
Review and approve every draft.

Draft fix ready: revert the 1:49 config
→ waiting for your review
Most drafts don’t land, and that’s fine. Review is the filter.

Better help, same pile. Every item still waits for a human. Level 3 shrinks the pile.

AIE CODE SUMMIT · SF 2026 · 07/16

### Level 3 autonomous triage

In this mock on-call queue, the agent has resolved 18 of 23 items and is working on 2. Only 3 need a person. For the checkout incident, the agent attaches its investigation so the on-call engineer can decide what to do.

[7:13](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=433s) [An illustrative board shows 23 incoming items. The agent is handling 2 and has resolved 18. The 3 needing a person are a checkout incident, a quota request and an alert that may belong to another team.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d58b6982e9f7e05cc955921149270eb368c950ce585966d51a3a11b00de62ef.webp)

Level 3: Autonomous. 23 came in, 3 needed you.
// LEVEL_3

● Auto triage ON · last sweep 2 min ago · interrupts only for what matters

INCOMING 23
7 tasks
9 alerts
5 test failures
1 incident
1 chat mention

every sweep: triaged, deduped, routed to a lane

AGENT HANDLING 2
Flaky test in checkout suite
LOW · TEST FAILURE
▸ Reproducing locally... race found in test setup

Disk usage alert on a batch host
LOW · ALERT
▸ Found the job that grew... drafting cleanup

NEEDS YOU 3
🔴 Incident: checkout errors
HIGH · HUMAN-OWNED · investigation attached

🔔 Quota request above team limit
MEDIUM · TASK
▶ Run this: approve +20%
▶ Run this: ask owner

Alert looks owned by another team
LOW · ALERT
▶ Run this: reroute
▶ Run this: snooze 24h

RESOLVED ON ITS OWN 18
✓ Noisy alert: known rollback window
✓ Duplicate of an open ticket
✓ Lint fix · landed after review
✓ Flaky test fixed · landed after review
✓ Stale detector · tuning drafted

Illustrative mock · synthetic data
AIE CODE SUMMIT · SF 2026 · 08/16

### Three automations for the on-call rotation

Each new signal triggers an investigation. Recovery checks retry failed runs and page a human only if recovery fails. The example 9:00 digest tells the next shift which 3 items need attention and summarizes the 18 the agent handled.

[8:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=488s) [Three rows show when each automation runs. Incoming signals trigger investigation, scheduled checks retry or repair failed runs, and each shift ends with a digest of handled work and items needing attention.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ef9f553db067b39b06f2c7ca1605f38354bed9bc065551612790897114f3c35.webp)

Under the hood: AI automations run the oncall rotation
// UNDER_THE_HOOD · AUTOMATIONS

WHEN    AUTOMATION    WHAT IT DOES    COMES BACK AS

On signal
alert · task
test · lookup
1 · Investigate
auto-triage
alert → incident investigator
task → team’s agent · BYOA
test failure → coding agent
lookup → memory + code search
RCA + drafted fix
or ▶ Run this, one click

On a schedule
+ after any failure
2 · Keep healthy
auto-recovery
1  spot stuck or failed runs
2  retry or re-dispatch
3  repair broken schedules
4  page a human only if it can’t
Nothing stalls silently
the board heals itself

Every shift
handoff
+ 9:00 digest
3 · Hand off
digest
1  summarize the shift
2  what needs you, fix ready
3  what it handled on its own
4  post to team chat
The 9:00 digest
3 need you · 18 handled

One teammate, three automations: it investigates, keeps itself healthy, and hands off every shift.

AIE CODE SUMMIT · SF 2026 · 10/16

### Bringing team agents and skills into the on-call workflow

Alerts, tickets and chat mentions feed a shared board. The team can plug its existing agents and skills into a sandbox that runs under its own identity. Code changes go through review and CI, and teammates can inspect or resume each run.

[10:46](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=646s) [Alerts, tickets, incidents and messages feed a shared board backed by a scheduler and team memory. Sandboxed agents and skills produce chat updates, code changes for review and CI, and shift summaries.](https://assets.ai.engineer/sites/aiecode2025/sha256/170597b048264d390372784c23b633b8109d40efc884b1e8883c9c1fbdd3aece.webp)

Shared agents: bring your own agents(BYOA), bring your own skills
// UNDER_THE_HOOD · SHARED_AGENTS

SOURCES
Alerts
Tickets
Incidents
Team posts
Chat mentions

THE BOARD · system of record
Queue
Agent handling
Needs you
Resolved
⏱ Scheduler: triage · follow-up · handoff · summaries
🧠 Memory: playbook, alert profiles, decisions

EXECUTION · sandboxed
Incident investigator
BYOA   bring your own agent
BYOS   bring your own skills
Deep-dive coding session

OUTPUTS
Team chat
Code changes → review + CI
Shift summaries

📜 Event log + every trajectory, inspectable and resumable

BYOA + BYOS: teams plug in the agents and skills they already trust. The agent is generic; the expertise is the team’s, and it all runs as the team’s own identity.

AIE CODE SUMMIT · SF 2026 · 11/16

### One shared agent with the team’s on-call expertise

In the mock chat, Sam teaches the shared agent to roll back a config change first if checkout times out after a push. When the same pattern recurs on Thursday, the agent cites Sam’s advice and offers Jordan a rollback to approve.

[11:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=702s) [Three panels describe a rotation-wide agent, its team memory and its limited access. In a mock chat, one responder teaches it a rollback rule. On a later shift, the agent recalls the rule and offers the next responder a rollback to approve.](https://assets.ai.engineer/sites/aiecode2025/sha256/2197b0ea0149e39fda8bf265e556461469e0b8fb136c35d6764bb7fd02ea5b7d.webp)

One shared agent, with your team’s on-call expertise

👥 Shared by the whole rotation
One teammate per rotation, on one board. Anyone on call can ask it, correct it, or pick up where it left off.

🧠 Carries the team’s expertise
The team’s playbook, alert profiles and past decisions stay with it from shift to shift.

🪪 Acts as the team, within limits
Its own identity, owned by the rotation, with only the access the team grants.

# checkout-oncall
🧠 team memory

S  Sam   on call · Mon 2:16 AM
remember: checkout timeouts after a config push → roll back the config first

AI  agent   Mon 2:16 AM
🧠 Saved to team memory
Checkout timeouts → roll back the config first · from Sam

Thursday · a different on-call

J  Jordan   on call · Thu 2:05 AM
checkout timeouts again. what do we do?

AI  agent   Thu 2:05 AM
Same pattern as Monday: config push at 1:57. Roll back the config first.
🧠 from team memory · saved by Sam, Mon
▶ Approve rollback
View pull request

Teach it once. The next on-call gets it.

AIE CODE SUMMIT · SF 2026 · 12/18
Illustrative mock · synthetic data

Interface text:
Message #checkout-oncall

### What we’re building next

The team plans to grade every mitigation the agent drafts and use the outcomes to decide whether it has earned medium-criticality autonomy. It also plans to let teams turn runbooks into skills through chat and measure on-call time saved.

[13:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=802s) [Three cards describe plans to grade drafted mitigations and feed back their outcomes, turn runbooks into skills through chat, and measure on-call time saved. A banner names the next goal as earning medium-criticality autonomy with evidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/6bac11c33ddbd1a9fdca82e12cd84ffb590c781e943c7db5334053e717c66527.webp)

// WHATS_NEXT
What we’re building next

✅ Closing the loop on mitigations
Grade every drafted mitigation and feed the outcome back, so the agent earns the next level.

🧩 Skills written from chat
Teams turn their runbooks into skills without writing code.

📏 Measuring time given back
A rigorous study of on-call time saved, not just usage.

Next rung: earning medium-criticality autonomy, with evidence.

AIE CODE SUMMIT · SF 2026 · 17/18

## Full transcript

[0:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=1s) Hi, I'm Chinmai and with me are Swapnika

[0:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=4s) and Vlad. We're excited to share our

[0:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=6s) journey as well as our learnings around

[0:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=8s) building an AI on call teammate at Meta

[0:11](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=11s) that now runs real on call rotations

[0:14](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=14s) everyday and around the clock.

[0:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=17s) We are part of the engineering team that

[0:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=19s) builds the tooling platform and

[0:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=21s) infrastructure for incident response on

[0:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=24s) call experience and mitigations at meta.

[0:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=27s) That space is evolving rapidly with the

[0:29](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=29s) increase in AI capabilities.

[0:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=32s) AI changed how fast we write code. About

[0:35](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=35s) 3/4 of new code is now written with AI

[0:38](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=38s) and more AI generated changes reach

[0:41](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=41s) production every single day and some of

[0:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=45s) it can cause production failures just

[0:48](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=48s) like any other core change can.

[0:51](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=51s) But on call did not speed up with it.

[0:54](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=54s) High quality code can pass review and

[0:56](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=56s) still break in production. And when it

[0:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=58s) does, someone still gets paged at 2 a.m.

[1:02](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=62s) The bottleneck moved from writing code

[1:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=64s) to operating it in production. And on

[1:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=68s) call is where that operational load

[1:10](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=70s) lands. So we asked if AI is writing the

[1:13](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=73s) code, can AI also help operate it?

[1:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=79s) And we think it can. Now, let's quickly

[1:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=82s) run through one of those on call pages

[1:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=84s) with a simplified example.

[1:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=87s) It's 2:00 a.m. and 12 alerts are firing.

[1:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=91s) Same night, it can go two different

[1:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=93s) ways. On the left is how most of us do

[1:35](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=95s) this today. And notice we already have

[1:38](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=98s) good AI. First, you sort the alerts by

[1:40](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=100s) yourself. Checkout, payments, disk.

[1:43](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=103s) Which one to pick first? Let's say you

[1:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=105s) pick checkout. Then, you ask your

[1:47](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=107s) assistant why P99 latency is high.

[1:51](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=111s) It's genuinely useful. It pulls the

[1:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=113s) metrics and finds the deployment at 149.

[1:56](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=116s) And when you ask, it reads what changed

[1:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=119s) and checks the runbook.

[2:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=121s) It figured out that the time config was

[2:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=124s) changed to a lower value. And based on

[2:07](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=127s) the runbook, it suggests to roll back,

[2:09](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=129s) which is all very helpful. However, if

[2:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=132s) you look at the shape of it, it waited

[2:14](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=134s) to be asked.

[2:16](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=136s) You picked every next step. You were the

[2:20](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=140s) glue between the questions and once it

[2:23](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=143s) answered, it went dormant.

[2:26](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=146s) At 239, you know the cause, but the roll

[2:30](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=150s) back, the changes, and the follow-ups

[2:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=153s) are still yours.

[2:35](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=155s) Now, on the right, the same 12 alerts,

[2:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=159s) but an AI teammate started before you

[2:43](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=163s) woke up. It had already ranked them.

[2:46](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=166s) Checkouts first, payments merged in as

[2:49](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=169s) the same cause, and the other 10 handled

[2:52](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=172s) separately.

[2:54](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=174s) It read the dashboards, diff the same

[2:56](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=176s) 149 a.m. config change, and matched two

[2:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=179s) past incidents.

[3:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=181s) You get a hypothesis, a 30-cond way to

[3:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=184s) verify it, and a fix waiting for your

[3:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=188s) review. At 2011, you verified it, and

[3:11](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=191s) you're back to sleep. That gap is what

[3:14](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=194s) this talk is about.

[3:18](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=198s) So why were you the glue?

[3:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=202s) Every team was bolting automation onto

[3:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=204s) on call, one silo at a time. An alert

[3:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=208s) script for alerts, a triage bot for

[3:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=211s) tickets, a deploy bot for deploys, a fix

[3:34](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=214s) it agent for incidents.

[3:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=216s) Each one solves its slice, keeps its own

[3:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=219s) context, and forgets it. Now replaying

[3:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=222s) your 2 AM page, the alert script saw P99

[3:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=225s) latency above 2 seconds. The triage bot

[3:49](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=229s) saw three tickets saying checkout is

[3:50](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=230s) slow. The deploy bot knew about a config

[3:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=233s) change at 149 and the incident history

[3:56](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=236s) had two similar past incidents. Every

[3:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=239s) clue existed, but nothing connected

[4:02](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=242s) them. So the on call became the glue

[4:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=246s) exactly

[4:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=248s) as we saw on the left side of the

[4:10](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=250s) previous slide.

[4:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=252s) Our bet was one AI teammate per rotation

[4:16](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=256s) instead.

[4:18](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=258s) One loop across every signal, one team

[4:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=261s) memory that learns every shift, its own

[4:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=264s) identity so it acts as itself and a

[4:26](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=266s) human gate so you approve instead of

[4:29](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=269s) operate.

[4:30](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=270s) But giving one agent all that context

[4:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=272s) and reach raises the stakes. Which is

[4:35](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=275s) why autonomy has to be earned.

[4:40](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=280s) Under the hood, the teammate or the AI

[4:43](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=283s) teammate is the simplest agent. There is

[4:46](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=286s) model using tools in a loop acting on an

[4:50](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=290s) environment and reading the feedback.

[4:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=293s) Everything else in this talk is what we

[4:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=295s) built around that loop to make it safe

[4:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=298s) to leave it running. And for that I will

[5:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=301s) hand over to Swapnikai and Vlad next to

[5:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=304s) get into the actual details of how it

[5:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=306s) works.

[5:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=308s) To achieve this we started with an

[5:10](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=310s) autonomy ladder. Think about how you

[5:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=312s) onboard a new engineer to on call. First

[5:16](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=316s) the shadow. They investigate and suggest

[5:18](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=318s) but they don't touch production. Then

[5:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=321s) the pair they write the fix and another

[5:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=324s) engineer approves it. With time they

[5:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=327s) gain knowledge. We give our agent

[5:29](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=329s) exactly the same ladder. At level one,

[5:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=332s) it investigates and tells you what it

[5:34](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=334s) found. It changes nothing. At level two,

[5:37](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=337s) it acts with approval. It drafts the

[5:40](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=340s) fix, but nothing reaches production

[5:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=342s) until a human approves. At level three,

[5:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=345s) it clears lower criticality work on its

[5:47](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=347s) own and only interrupts you for what

[5:50](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=350s) matters. High criticality work stays

[5:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=353s) with humans.

[5:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=355s) So, what earns the next run? We use a

[5:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=358s) simple test. How costly is a mistake?

[6:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=361s) And how easy is it to catch? So an

[6:03](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=363s) action climbs only when mistakes are

[6:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=366s) cheap. They're caught by tests, CI or

[6:09](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=369s) review. And the agent already has a

[6:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=372s) track record on that exact action.

[6:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=375s) That's why the agent can close a noisy

[6:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=377s) alert on its own while a production

[6:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=379s) mitigation still waits for a human.

[6:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=384s) At level one, the investigation is

[6:26](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=386s) layered and stops at the first confident

[6:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=388s) answer. The agent investigated the issue

[6:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=391s) using the team's memory and runbook. It

[6:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=393s) also shared the root cause analysis, but

[6:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=396s) human had to apply the fix.

[6:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=399s) At level two, it writes the fix with the

[6:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=402s) root cause and a test plan. It answers

[6:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=405s) reviewer's comments and fixes CI, but

[6:48](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=408s) nothing reaches production until a human

[6:50](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=410s) approves.

[6:52](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=412s) Most drafts don't land and that's fine.

[6:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=415s) Review is the filter. At both levels,

[6:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=419s) the agent does the homework, but you

[7:02](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=422s) still had to grade every single paper.

[7:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=424s) The help got much better, but the tasks

[7:07](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=427s) requiring human input didn't get

[7:10](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=430s) smaller. Level three shrinks the pile.

[7:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=435s) Look at this example. 23 items came in,

[7:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=439s) but only three need you. On the right,

[7:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=442s) the agent handled the long tail noisy

[7:25](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=445s) alerts, duplicates, routine fixes that

[7:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=448s) went through normal review and CI. Two

[7:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=451s) more are still being worked. You don't

[7:34](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=454s) have to drive each one to completion. In

[7:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=456s) the middle is your much smaller decision

[7:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=459s) queue. Each item makes clear what it

[7:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=462s) needs from you with a run this button

[7:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=465s) that has a call to action. And remember

[7:48](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=468s) our 2 a.m. checkout page that still

[7:50](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=470s) stays human owned. The agent brings the

[7:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=473s) investigation. The human owns the

[7:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=475s) critical decision. It's the new hire

[7:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=478s) ladder again. Trusted to handle routine

[8:00](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=480s) work. Ready with the homework when it

[8:02](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=482s) needs help. Not just less work per item,

[8:05](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=485s) but fewer items waiting on you.

[8:10](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=490s) That board is where the work lives. But

[8:13](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=493s) nobody stares at a board all the time,

[8:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=495s) right? So the agent is also available

[8:18](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=498s) across many surfaces like chat and

[8:20](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=500s) native incident surfaces.

[8:23](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=503s) At 9 every morning it posts a digest. 23

[8:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=507s) came in three need you. 18 handled. Each

[8:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=511s) thing that needs you come with a

[8:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=513s) one-click next step. So approving the

[8:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=516s) quota bump takes one tap from your

[8:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=519s) phone. The 18 it handled are summarized

[8:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=522s) not listed. and the two that are still

[8:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=525s) in progress tell you what it's doing

[8:47](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=527s) about them. And when you want to dig in,

[8:50](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=530s) you just reply in the thread. As you see

[8:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=533s) on the right here, you open the checkout

[8:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=535s) incident and ask for the evidence. It

[8:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=538s) already has it. Errors start at 8:41, 2

[9:02](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=542s) minutes after a config push cut the

[9:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=544s) payment client timeout. The same pattern

[9:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=546s) as two past incidents. You ask a

[9:09](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=549s) follow-up, is it only EU? And it answers

[9:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=552s) with a breakdown. It drafts the role

[9:14](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=554s) back. You approve it right there and

[9:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=557s) errors start recovering. You never

[9:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=559s) opened a dashboard and you never had to

[9:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=561s) explain the context because it was

[9:23](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=563s) already there. That's what a teammate

[9:25](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=565s) feels like. A morning handoff a

[9:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=567s) colleague you can ask.

[9:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=572s) Now under the hood of this AI teammate,

[9:34](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=574s) it's AI automations. The first

[9:37](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=577s) investigates on every new signal, auto

[9:40](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=580s) triage classifies it, checks its

[9:43](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=583s) criticality, and kicks off the right

[9:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=585s) agent. An alert goes to the incident

[9:48](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=588s) investigator, a task to the team's own

[9:51](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=591s) agent, a test failure to a coding agent,

[9:54](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=594s) a lookup to memory, and code search.

[9:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=598s) The second keeps the system healthy.

[10:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=601s) Unattended automation fails quietly. So

[10:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=604s) on a schedule, it looks for stuck or

[10:07](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=607s) failed runs, retrace and repairs broken

[10:10](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=610s) schedules, and it only pages a human if

[10:13](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=613s) it can't fix it itself.

[10:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=615s) Nothing stalls silently.

[10:18](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=618s) The third hands off at every shift

[10:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=621s) change and every morning at 9, it

[10:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=624s) summarizes the shift. What needs you

[10:26](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=626s) with the fix ready and what it handled

[10:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=628s) on its own? Posted to the team chat. Not

[10:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=631s) only in the morning but every urgent

[10:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=633s) issue that needs user attention gets

[10:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=636s) notified in chat with findings and draft

[10:38](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=638s) fixes. It also tags the next own call in

[10:41](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=641s) the chat during handoff. But the entire

[10:43](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=643s) cycle still continues.

[10:48](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=648s) And because it belongs to the team, it

[10:51](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=651s) runs the agents and the skills the team

[10:53](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=653s) already trusts. That's the slot in the

[10:56](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=656s) middle. Bring your own agent, add your

[10:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=659s) own skills. If a team already has an

[11:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=661s) agent that has the team's runbooks and

[11:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=664s) the knowledge base, the automations

[11:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=666s) leverage it. Alongside that, it has our

[11:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=668s) own incident investigator and a deep

[11:11](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=671s) dive coding session.

[11:13](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=673s) Work comes in from alerts, tickets,

[11:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=677s) incidents, team posts, and chat. The

[11:20](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=680s) agent itself is generic. What makes it

[11:23](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=683s) good at an on call rotation is the

[11:25](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=685s) team's agents skills, memory, and

[11:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=688s) playbook. And every run is logged so

[11:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=691s) anyone on the team can inspect the

[11:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=693s) trajectory or resume the conversation.

[11:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=696s) Now I'm handing off to my teammate Vlad

[11:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=699s) to talk more on the shared agent.

[11:44](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=704s) An uncle agent can't belong to just a

[11:47](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=707s) single person because the uncle itself

[11:49](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=709s) doesn't. So it belongs to the team.

[11:52](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=712s) There's one teammate per rotation on one

[11:54](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=714s) board. Whoever is on call can ask it,

[11:57](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=717s) correct it, or pick up its work mid

[11:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=719s) investigation. It carries the team's on

[12:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=721s) call expertise, the playbook of lure

[12:03](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=723s) profiles and past decisions from one

[12:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=726s) shift to the next one, the way a good

[12:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=728s) handoff should. Here's what that looks

[12:11](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=731s) like. Monday at 2 a.m., Sam is on call

[12:14](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=734s) and tells it, remember, if checkout

[12:16](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=736s) times out after a config push, roll back

[12:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=739s) the config first. It saves it to the

[12:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=741s) memory.

[12:23](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=743s) Three days later, Jordan is on call,

[12:25](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=745s) hits the exact same alert, and just asks

[12:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=748s) for a fix. The agent answers with some

[12:31](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=751s) fix, says where it came from, and has

[12:34](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=754s) the roll back drafted. Teach it once,

[12:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=756s) the next call gets it,

[12:39](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=759s) and it accesses the team its own

[12:42](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=762s) identity owned by the rotation with only

[12:44](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=764s) the access the team is comfortable

[12:46](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=766s) granting it.

[12:49](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=769s) And it learns from every uncle shift.

[12:52](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=772s) Handoffs, decisions, and corrections all

[12:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=775s) flow into memory. So shift number 30 is

[12:57](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=777s) better than the shift number one. The

[12:59](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=779s) same way a new uncle gets better every

[13:01](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=781s) week. But memory erodess over time. So

[13:05](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=785s) ours has to earn its place. It retrieves

[13:09](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=789s) semantically uses what it finds, scores

[13:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=792s) what it actually helped, and promotes or

[13:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=795s) retires entries automatically.

[13:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=797s) When someone says that was wrong, the

[13:20](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=800s) item reopens and the correction becomes

[13:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=802s) memory.

[13:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=804s) Three things made this work beyond our

[13:26](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=806s) team.

[13:28](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=808s) First, a loop that never stops

[13:30](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=810s) listening. Every piece of feedback or

[13:33](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=813s) course correction reopens the item and

[13:35](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=815s) ultimately gets absorbed into memory.

[13:38](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=818s) Feedback is an input to the system, not

[13:40](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=820s) a story read once a quarter. Second,

[13:44](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=824s) customize instead of forking. Teams

[13:47](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=827s) bring their own playbooks and skill

[13:49](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=829s) workflows and it's still the same

[13:51](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=831s) teammate underneath.

[13:54](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=834s) Third, what we call one rotation to end.

[13:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=838s) Identity, config, and memory are per

[14:00](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=840s) team. So add a rotation becomes a setup,

[14:03](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=843s) not a new project. The agent is generic

[14:06](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=846s) while the expertise is owned by the

[14:08](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=848s) team.

[14:11](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=851s) So what has changed for the people on

[14:13](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=853s) call? Well, first their job shifted from

[14:15](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=855s) driving deep investigations to making

[14:17](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=857s) key decisions. They start from a

[14:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=859s) hypothesis with linked evidence, not a

[14:22](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=862s) blank task or dashboard.

[14:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=864s) Second, the load is getting lighter.

[14:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=867s) Early signs of on call load reduction

[14:30](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=870s) are promising for teams that run it

[14:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=872s) automatic autonomously and we want to

[14:34](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=874s) measure it rigorously before we put a

[14:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=876s) number on it.

[14:38](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=878s) And the third, every change that agent

[14:40](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=880s) writes still meets a human. Fixes land

[14:44](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=884s) only through normal review and

[14:45](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=885s) continuous integration. And most drafts

[14:48](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=888s) don't land. That's the point. Review is

[14:51](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=891s) the filter.

[14:52](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=892s) Encle stops being attacks and becomes a

[14:55](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=895s) review cue.

[14:58](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=898s) Here's what we're focusing on next.

[15:00](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=900s) First, mitigations that get approved,

[15:02](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=902s) not just drafted. We grade every

[15:04](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=904s) mitigation that agent proposes and feed

[15:07](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=907s) the outcome back so it earns its next

[15:09](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=909s) level instead of being handed it.

[15:12](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=912s) Second, skills are written from the chat

[15:14](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=914s) so a team can turn its runbooks into

[15:16](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=916s) skills without writing any code. And

[15:19](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=919s) third, measuring the time we give back

[15:21](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=921s) properly with a rigorous study of call

[15:24](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=924s) time saved rather than usage counts.

[15:27](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=927s) And the next rung on the ladder is uh

[15:30](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=930s) earning medium criticality autonomy with

[15:32](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=932s) evidence. On that, thank you so much for

[15:36](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=936s) listening to us and hopefully that was

[15:38](https://www.youtube.com/watch?v=_NBx2NNOuA4&t=938s) helpful. Bye.
