AI Engineer Code 2025
The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal
Loading slides…
Read the talk
The 5 Levels of Self-Driving Production
Eric Schwartz explains Traversal’s path from manual incident response to diagnosis across an enterprise and, eventually, verified fixes. The hard part is connecting a visible failure to a distant cause quickly enough to help the on-call team.
From a talk by Eric Schwartz
At a glance
Ideas worth remembering
Faster code generation can increase troubleshooting work when production grows faster than engineers’ understanding of it.
Enterprise root cause analysis requires connecting symptoms across services and teams; a failing checkout API may be several hops from its possible cause.
Level four extends diagnosis across an environment. Level five adds fixes and verification, completing the proposed production loop.
The Pepsi and American Express examples change the order of work: investigate alerts before asking engineers to triage them, and analyze incidents before paging broadly.
Evaluate an AI SRE on data coverage, search cost and load, relationship mapping, autonomous knowledge maintenance, and fast multihop investigation.
Faster development leaves more production to understand
Coding agents shorten the time needed to write software. The hoped-for result is more time to design it. Eric Schwartz, product manager at Traversal, opens with a different observation from the enterprises his team serves: more engineering time is going into troubleshooting. Code reaches production faster, while the people responsible for it may understand less of what has changed.
The software development life cycle has three broad jobs in this account: decide what to build and how to structure it, write the code, then fix what happens when that code meets the real world. Accelerating the middle job does not automatically accelerate the last one. More code creates more behavior to understand and more interactions to debug. Schwartz includes himself among people who can now contribute production code without formal computer science training; the accessibility is useful, but it also widens the gap between producing a change and understanding its consequences.
Schwartz cites estimates of upwards of $400 billion in annual enterprise spending, 40% of executives identifying troubleshooting as a team problem, and engineers losing seven-plus hours each week to troubleshooting while on call. Those figures come without a named study or measurement method in the recording, so they serve as motivation rather than a precise estimate of the cost attributable to coding agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A failing checkout API can be five hops from its cause
Observability tools such as Datadog, Elastic, Splunk and ServiceNow expose failures and correlations. The unresolved job is explaining why the failures happened and what to do next. Schwartz recalls the recurring complaint from his work on ServiceNow observability products: “You're telling me what's broken. You're not telling me why, and you're not telling me what to do about it.” More dashboards can display more symptoms without connecting them into an explanation.
Consider the customer-derived example of a failing checkout API. The visible symptom is at checkout, but the investigation may need five to ten hops through dozens of services and petabytes of data. An expired TLS certificate five hops away is the possible cause Schwartz names. The example does not specify the intervening services or establish the certificate as a confirmed diagnosis; its point is the distance between where a failure appears and where an investigation must look.
That distance becomes an organizational problem. In a small environment, a seasoned site reliability engineer might know the whole stack. In a large enterprise, engineers hold different slices of context. Bringing fifty of them into a war room assembles those slices, but takes time and interrupts many people. Pairing one engineer with a coding assistant still leaves the problem of obtaining context across the environment.
Traversal’s premise is that root cause analysis requires causal reasoning: finding the cause-and-effect relationships that explain the observed failure. Correlated failures tell an investigator where to look; an explanation must connect them to what produced the incident. Schwartz also cites Google and Anthropic as recognizing the difficulty, including limitations of LLMs at finding root causes. The technical direction he develops is to give agents a model of production relationships and a way to search through them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From war rooms to a closed diagnosis-and-fix loop
Self-driving production is the proposed closed loop: detect an issue, find its root cause, put up a fix and verify that the issue has been fixed. The ambition is to complete that loop without paging the team. Verification matters because a diagnosis or proposed change alone does not establish that production has recovered.
The title’s five levels sit above a manual level zero. Schwartz uses the analogy of autonomous driving to describe increasing capability:
- Level zero: manual response. Engineers assemble in Slack or Zoom and spend hours debugging.
- Level one: rules and automations. A structured workflow responds to a known alert. Its weakness is novelty: a severe incident may have no existing runbook.
- Levels two and three: increasing automation. Level two receives no separate operational definition in the talk. At level three, a homegrown agent may debug a particular service or set of alerts well.
- Level four: diagnosis across the environment. The investigation spans hundreds of services and repositories and thousands of log indexes.
- Level five: fixes and verification. The system combines enterprise-wide diagnosis with putting up fixes and verifying them.
The largest jump in this framework is from a service-specific agent to one that can investigate across an enterprise. A local agent can work within a familiar slice of the system. An environment-wide investigator must cross the divisions that otherwise require many teams to pool their knowledge. Level five adds a further responsibility: carrying the diagnosis through a change and checking the outcome. Schwartz presents it as the destination, rather than claiming every customer already operates at that level.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different production jobs need different outputs
Traversal serves companies including American Express, Pepsi, DigitalOcean and Capital One. Schwartz describes environments generating trillions of logs and spans and tens of billions of metrics and events, and reports “eighty percent plus” root cause performance for high-severity incidents at that scale. The recording does not define the scoring criteria or evaluation population for that result, so it should be read as a reported product outcome rather than a benchmark for comparing systems.
The product work extends beyond answering a single incident question:
- Alert intelligence triages and prioritizes the hundreds or thousands of alerts an on-call team may receive in a day.
- Incident root cause analysis aims to pinpoint a cause in minutes instead of requiring hours and dozens of engineers.
- Self-healing adds a fix to address or mitigate the diagnosed issue.
- Production support answers everyday questions about production and retrieves specific data points, including work that never becomes a major incident.
- Code resilience examines changes before production with the goal of making the system more reliable over time.
These jobs ask for different kinds of help. Alert intelligence changes what deserves attention; incident analysis changes what teams know about a failure; self-healing changes what the system does about it. Schwartz develops the first two through customer examples, leaving the other capabilities at the level of use-case descriptions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pepsi: turn an alert backlog into decisions
Pepsi’s supply-chain software helps move raw materials and finished products, including the critical passage from warehouse to trucks to retailers. Before Traversal, the team received thousands or tens of thousands of alerts each week. A single engineer could have a backlog of 700 alerts. The operational difficulty was deciding where to look: important failures could slip through the noise, while trying to inspect everything risked burnout.
The change is a filtered, prioritized set of alerts that Traversal has already investigated. Investigation happens before the engineer must choose what to open. That creates several possible decisions instead of treating every alert as equally urgent:
- Investigate now. Flag alerts worth an engineer’s attention.
- Change the alert rule. Identify opportunities to reduce noise by updating the underlying rules in code.
- Dismiss for now. Set aside alerts that do not currently need attention.
- Schedule later work. Create tickets for technical debt that matters but does not require immediate response.
The causal sequence is practical: a large undifferentiated queue makes attention scarce; pre-investigation supplies context; prioritization directs immediate attention; rule changes and deferred tickets address the remaining work differently. The reported outcome is a more useful queue, with no replacement backlog count given. This is progress toward autonomous production through better triage, even before the system fixes an incident itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
American Express: investigate before paging broadly
The American Express example starts with an incident response that pages five to ten teams and twenty to fifty engineers. The particular incident comparison uses sixty minutes to resolution, although Schwartz says incidents can take hours or days. He names inability to pay a credit card bill or log into the mobile app as examples of the stakes.
In the workflow Schwartz describes, Traversal becomes the first responder whenever American Express declares an incident, internally called a bridge. Declaration dispatches the system. Within three minutes, it posts a detailed root cause analysis in the incident’s Slack channel; it can also update ServiceNow tickets. The initial explanation then informs who needs to be called.
That order changes the response. Rather than assembling many teams to discover what broke, responders start with an analysis and may page one or two teams to verify it—or no teams. In the specific comparison, Schwartz describes avoiding pages to fifty-plus engineers. The three-minute figure is time to an analysis, not time to a verified repair; the demonstrated benefit here is a narrower, less chaotic response.
What changes when investigation precedes broad escalation? The flow below shows how the initial analysis becomes an input to the paging decision. Human verification remains part of the described workflow, while fewer people need to enter the incident just to locate its cause. For incidents in the middle of the night, that is also a concrete reduction in interrupted sleep.
Analysis informs escalation at American Express
Read the diagram as text
- Incident declared. American Express declares a bridge and dispatches Traversal.
- Root cause analysis in Slack. Traversal posts a detailed analysis within three minutes; ServiceNow updates are also possible.
- Focused paging decision. No teams, or one or two teams to verify findings, instead of broad escalation.
- Incident declared → Root cause analysis in Slack: Dispatch and investigate.
- Root cause analysis in Slack → Focused paging decision: Use the initial findings to narrow response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five questions to ask an AI SRE
The closing evaluation questions return to the checkout example’s central difficulty: reaching a distant cause through a large, changing environment. Schwartz proposes testing the operational requirements that make such an investigation useful:
- Data coverage: Can the AI SRE see all the production data it needs? Missing slices of context can block a detailed diagnosis.
- Search cost and load: Can it search petabytes of data and hundreds of billions of logs without excessive cost or overwhelming observability infrastructure?
- Relationships: Can it map how the entities in that data relate? Reading data is insufficient if the system cannot connect it.
- Knowledge maintenance: Does its understanding improve autonomously, or does the team have to maintain Markdown files and repeatedly supply engineering help?
- Multihop speed: Can it reach a non-obvious root cause far from the symptom in minutes?
Speed is a condition for adoption during an incident. Schwartz’s observation is that after more than five minutes, on-call engineers lose patience and return to their established habits. An investigation therefore has to be useful before the team has already committed to another way of working. The American Express three-minute analysis illustrates the kind of response window he has in mind.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A production world model gives causal search somewhere to go
Traversal’s final architectural sketch separates three jobs. The bottom of the stack integrates with production data and makes it available for analysis while aiming to control cost. The middle is the production world model, which maps relationships in that data. The top packages the resulting capability into useful workflows and experiences, including alert intelligence and incident analysis.
How does production data become an incident answer? The stack diagram makes the intermediate relationship model visible. Data access supplies material for an investigation; the world model organizes the connections; the causal search engine gives agents a harness for searching through the mapped data. The output reaches engineers through the use case they are working on.
The recording describes these responsibilities without explaining how relationships are learned, how causal hypotheses are tested, or how search controls infrastructure load. It also does not develop the execution and verification procedure for autonomous fixes. The production world model and causal search engine are the named foundations for Traversal’s proposed route to level five: understanding an enterprise-wide failure well enough to fix it and verify recovery.
From production data to useful investigations
Read the diagram as text
- Production data integrations. Connect to data and support analysis with attention to cost.
- Production world model. Map and make sense of relationships in integrated data.
- Causal search engine. Give agents a harness to search through the mapped data.
- Team workflows. Package findings into useful experiences such as alert intelligence and incident analysis.
- Production data integrations → Production world model: Organize relationships.
- Production world model → Causal search engine: Search mapped data.
- Causal search engine → Team workflows: Package investigation results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…