AI Engineer Code 2025

The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal

Loading slides…

Read the talk

The 5 Levels of Self-Driving Production

Eric Schwartz explains Traversal’s path from manual incident response to diagnosis across an enterprise and, eventually, verified fixes. The hard part is connecting a visible failure to a distant cause quickly enough to help the on-call team.

From a talk by Eric Schwartz

At a glance

Ideas worth remembering

  • Faster code generation can increase troubleshooting work when production grows faster than engineers’ understanding of it.

  • Enterprise root cause analysis requires connecting symptoms across services and teams; a failing checkout API may be several hops from its possible cause.

  • Level four extends diagnosis across an environment. Level five adds fixes and verification, completing the proposed production loop.

  • The Pepsi and American Express examples change the order of work: investigate alerts before asking engineers to triage them, and analyze incidents before paging broadly.

  • Evaluate an AI SRE on data coverage, search cost and load, relationship mapping, autonomous knowledge maintenance, and fast multihop investigation.

Faster development leaves more production to understand

Coding agents shorten the time needed to write software. The hoped-for result is more time to design it. Eric Schwartz, product manager at Traversal, opens with a different observation from the enterprises his team serves: more engineering time is going into troubleshooting. Code reaches production faster, while the people responsible for it may understand less of what has changed.

Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 127 secondsOpen full source frame
A slide contrasts faster development with growing troubleshooting demands.

The software development life cycle has three broad jobs in this account: decide what to build and how to structure it, write the code, then fix what happens when that code meets the real world. Accelerating the middle job does not automatically accelerate the last one. More code creates more behavior to understand and more interactions to debug. Schwartz includes himself among people who can now contribute production code without formal computer science training; the accessibility is useful, but it also widens the gap between producing a change and understanding its consequences.

Schwartz cites estimates of upwards of $400 billion in annual enterprise spending, 40% of executives identifying troubleshooting as a team problem, and engineers losing seven-plus hours each week to troubleshooting while on call. Those figures come without a named study or measurement method in the recording, so they serve as motivation rather than a precise estimate of the cost attributable to coding agents.

0:121:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

A failing checkout API can be five hops from its cause

Observability tools such as Datadog, Elastic, Splunk and ServiceNow expose failures and correlations. The unresolved job is explaining why the failures happened and what to do next. Schwartz recalls the recurring complaint from his work on ServiceNow observability products: “You're telling me what's broken. You're not telling me why, and you're not telling me what to do about it.” More dashboards can display more symptoms without connecting them into an explanation.

Consider the customer-derived example of a failing checkout API. The visible symptom is at checkout, but the investigation may need five to ten hops through dozens of services and petabytes of data. An expired TLS certificate five hops away is the possible cause Schwartz names. The example does not specify the intervening services or establish the certificate as a confirmed diagnosis; its point is the distance between where a failure appears and where an investigation must look.

Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 286 secondsOpen full source frame
A checkout API failure sits within a dense map of connected services.

That distance becomes an organizational problem. In a small environment, a seasoned site reliability engineer might know the whole stack. In a large enterprise, engineers hold different slices of context. Bringing fifty of them into a war room assembles those slices, but takes time and interrupts many people. Pairing one engineer with a coding assistant still leaves the problem of obtaining context across the environment.

Traversal’s premise is that root cause analysis requires causal reasoning: finding the cause-and-effect relationships that explain the observed failure. Correlated failures tell an investigator where to look; an explanation must connect them to what produced the incident. Schwartz also cites Google and Anthropic as recognizing the difficulty, including limitations of LLMs at finding root causes. The technical direction he develops is to give agents a model of production relationships and a way to search through them.

2:593:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:59 · section reference included

From war rooms to a closed diagnosis-and-fix loop

Self-driving production is the proposed closed loop: detect an issue, find its root cause, put up a fix and verify that the issue has been fixed. The ambition is to complete that loop without paging the team. Verification matters because a diagnosis or proposed change alone does not establish that production has recovered.

The title’s five levels sit above a manual level zero. Schwartz uses the analogy of autonomous driving to describe increasing capability:

  • Level zero: manual response. Engineers assemble in Slack or Zoom and spend hours debugging.
  • Level one: rules and automations. A structured workflow responds to a known alert. Its weakness is novelty: a severe incident may have no existing runbook.
  • Levels two and three: increasing automation. Level two receives no separate operational definition in the talk. At level three, a homegrown agent may debug a particular service or set of alerts well.
  • Level four: diagnosis across the environment. The investigation spans hundreds of services and repositories and thousands of log indexes.
  • Level five: fixes and verification. The system combines enterprise-wide diagnosis with putting up fixes and verifying them.
Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 476 secondsOpen full source frame
An autonomy diagram shows increasingly broad production capabilities, from manual troubleshooting to self-driving production.

The largest jump in this framework is from a service-specific agent to one that can investigate across an enterprise. A local agent can work within a familiar slice of the system. An environment-wide investigator must cross the divisions that otherwise require many teams to pool their knowledge. Level five adds a further responsibility: carrying the diagnosis through a change and checking the outcome. Schwartz presents it as the destination, rather than claiming every customer already operates at that level.

6:256:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:25 · section reference included

Different production jobs need different outputs

Traversal serves companies including American Express, Pepsi, DigitalOcean and Capital One. Schwartz describes environments generating trillions of logs and spans and tens of billions of metrics and events, and reports “eighty percent plus” root cause performance for high-severity incidents at that scale. The recording does not define the scoring criteria or evaluation population for that result, so it should be read as a reported product outcome rather than a benchmark for comparing systems.

The product work extends beyond answering a single incident question:

  • Alert intelligence triages and prioritizes the hundreds or thousands of alerts an on-call team may receive in a day.
  • Incident root cause analysis aims to pinpoint a cause in minutes instead of requiring hours and dozens of engineers.
  • Self-healing adds a fix to address or mitigate the diagnosed issue.
  • Production support answers everyday questions about production and retrieves specific data points, including work that never becomes a major incident.
  • Code resilience examines changes before production with the goal of making the system more reliable over time.
Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 603 secondsOpen full source frame
A slide lists alert intelligence, incident RCA, self-healing, production support, and code resilience.

These jobs ask for different kinds of help. Alert intelligence changes what deserves attention; incident analysis changes what teams know about a failure; self-healing changes what the system does about it. Schwartz develops the first two through customer examples, leaving the other capabilities at the level of use-case descriptions.

8:479:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:47 · section reference included

Pepsi: turn an alert backlog into decisions

Pepsi’s supply-chain software helps move raw materials and finished products, including the critical passage from warehouse to trucks to retailers. Before Traversal, the team received thousands or tens of thousands of alerts each week. A single engineer could have a backlog of 700 alerts. The operational difficulty was deciding where to look: important failures could slip through the noise, while trying to inspect everything risked burnout.

The change is a filtered, prioritized set of alerts that Traversal has already investigated. Investigation happens before the engineer must choose what to open. That creates several possible decisions instead of treating every alert as equally urgent:

  • Investigate now. Flag alerts worth an engineer’s attention.
  • Change the alert rule. Identify opportunities to reduce noise by updating the underlying rules in code.
  • Dismiss for now. Set aside alerts that do not currently need attention.
  • Schedule later work. Create tickets for technical debt that matters but does not require immediate response.

The causal sequence is practical: a large undifferentiated queue makes attention scarce; pre-investigation supplies context; prioritization directs immediate attention; rule changes and deferred tickets address the remaining work differently. The reported outcome is a more useful queue, with no replacement backlog count given. This is progress toward autonomous production through better triage, even before the system fixes an incident itself.

11:0611:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:06 · section reference included

American Express: investigate before paging broadly

The American Express example starts with an incident response that pages five to ten teams and twenty to fifty engineers. The particular incident comparison uses sixty minutes to resolution, although Schwartz says incidents can take hours or days. He names inability to pay a credit card bill or log into the mobile app as examples of the stakes.

In the workflow Schwartz describes, Traversal becomes the first responder whenever American Express declares an incident, internally called a bridge. Declaration dispatches the system. Within three minutes, it posts a detailed root cause analysis in the incident’s Slack channel; it can also update ServiceNow tickets. The initial explanation then informs who needs to be called.

That order changes the response. Rather than assembling many teams to discover what broke, responders start with an analysis and may page one or two teams to verify it—or no teams. In the specific comparison, Schwartz describes avoiding pages to fifty-plus engineers. The three-minute figure is time to an analysis, not time to a verified repair; the demonstrated benefit here is a narrower, less chaotic response.

Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 858 secondsOpen full source frame
A slide shows an incident timeline with an analysis followed by a smaller response flow.

What changes when investigation precedes broad escalation? The flow below shows how the initial analysis becomes an input to the paging decision. Human verification remains part of the described workflow, while fewer people need to enter the incident just to locate its cause. For incidents in the middle of the night, that is also a concrete reduction in interrupted sleep.

Analysis informs escalation at American Express

Incident declaredAmerican Express declares a bridge and dispatches Traversal.
Root cause analysis in SlackTraversal posts a detailed analysis within three minutes; ServiceNow updates are also possible.
Focused paging decisionNo teams, or one or two teams to verify findings, instead of broad escalation.
The reported three-minute output is a root cause analysis. It helps decide which teams, if any, should verify the findings.
Read the diagram as text
  • Incident declared. American Express declares a bridge and dispatches Traversal.
  • Root cause analysis in Slack. Traversal posts a detailed analysis within three minutes; ServiceNow updates are also possible.
  • Focused paging decision. No teams, or one or two teams to verify findings, instead of broad escalation.
  • Incident declared → Root cause analysis in Slack: Dispatch and investigate.
  • Root cause analysis in Slack → Focused paging decision: Use the initial findings to narrow response.
13:0613:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:06 · section reference included

Five questions to ask an AI SRE

The closing evaluation questions return to the checkout example’s central difficulty: reaching a distant cause through a large, changing environment. Schwartz proposes testing the operational requirements that make such an investigation useful:

  • Data coverage: Can the AI SRE see all the production data it needs? Missing slices of context can block a detailed diagnosis.
  • Search cost and load: Can it search petabytes of data and hundreds of billions of logs without excessive cost or overwhelming observability infrastructure?
  • Relationships: Can it map how the entities in that data relate? Reading data is insufficient if the system cannot connect it.
  • Knowledge maintenance: Does its understanding improve autonomously, or does the team have to maintain Markdown files and repeatedly supply engineering help?
  • Multihop speed: Can it reach a non-obvious root cause far from the symptom in minutes?

Speed is a condition for adoption during an incident. Schwartz’s observation is that after more than five minutes, on-call engineers lose patience and return to their established habits. An investigation therefore has to be useful before the team has already committed to another way of working. The American Express three-minute analysis illustrates the kind of response window he has in mind.

14:4215:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:42 · section reference included

Transcript download
Full transcript

Loading transcript…