AI Engineer World's Fair 2026
Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase
Loading slides…
Read the talk
Learned Execution Graphs for API Anomalies and Drift
Representing each request as an execution graph helps separate a slow service, a changed processing path and a shifting request mix—then choose a response without delaying payments.
From a talk by Ritvik Pandya
Before you start: Familiarity with API request flows, distributed tracing and basic latency monitoring will help; the article introduces execution graphs and drift categories as they arise.
What changed in this request?
When an API request deviates from normal behavior, how do you find what changed—and decide whether to fix it? Ritvik Pandya, who introduces himself as leading a payments team at JP Morgan, approaches this through short-lived execution graphs. Unlike persistent property graphs queried through systems such as Neo4j, these graphs describe how an individual request proceeds through a system. Their purpose is to make deviations visible and reduce manual investigation.
Consider a payment request. It enters through an edge layer, passes through gateways and ingress, and reaches authentication and authorization. An orchestrator then invokes downstream services, some in parallel, before the system notifies the client. In the illustrated graph, those parallel branches are Fraud Score, Ledger Write and FX Rate; they converge at Notify. Representing the journey as a directed acyclic graph, or DAG, preserves both the sequential dependencies and the parallel work.
The graph tells you which services execute in which order, but it also identifies the context available at each node and what passes to the next one. Retries and loops need explicit representation: Pandya treats their occurrences as separate entities so they remain trackable. An execution graph therefore records the work that happened, rather than collapsing every visit to a service into one indistinguishable node.
A compact TypeScript representation of the illustrated dependencies makes the parallel branches explicit. Each array names the steps that must precede that node; it does not impose an order among Fraud Score, Ledger Write and FX Rate.
typescript
type Step =
| "Edge GW"
| "AuthN/Z"
| "Orchestrator"
| "Fraud Score"
| "Ledger Write"
| "FX Rate"
| "Notify";
const predecessors: Record<Step, readonly Step[]> = {
"Edge GW": [],
"AuthN/Z": ["Edge GW"],
"Orchestrator": ["AuthN/Z"],
"Fraud Score": ["Orchestrator"],
"Ledger Write": ["Orchestrator"],
"FX Rate": ["Orchestrator"],
"Notify": ["Fraud Score", "Ledger Write", "FX Rate"],
};
function readySteps(completed: ReadonlySet<Step>): Step[] {
return (Object.keys(predecessors) as Step[]).filter(
step => !completed.has(step) &&
predecessors[step].every(parent => completed.has(parent)),
);
}
const completed = new Set<Step>([
"Edge GW", "AuthN/Z", "Orchestrator",
]);
const ready = readySteps(completed);
// ["Fraud Score", "Ledger Write", "FX Rate"]
Timing and request context can then be attached to these execution nodes, while repeated attempts receive distinct identities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend analysis effort where the baseline breaks
A detector should not spend the same resources on every request. Pandya compares the first tier to an airport boarding-pass check: if the request matches its normal end-to-end baseline, let it pass without escalating to a deeper inspection. A delay prompts the next question: did the structure change? An unexpected node or a missing processing step can explain why execution no longer matches the reference graph.
Further analysis can use KL divergence or an exponential moving average, but the comparison must reflect the request's context. Client A may be local, while client B's request arrives from outside the country and requires additional checks. Their normal latencies need not match. A threshold built around the appropriate client baseline helps avoid treating expected differences as incidents.
The decision sequence is straightforward:
- Represent the full request as a DAG.
- Establish its baseline and detect a deviation.
- Localize the deviation to the relevant part of the graph.
- Compare it with the threshold appropriate to that system and request.
- Alert or act only when the deviation exceeds that threshold.
Detection is not automatically a mandate for remediation. A localized difference that remains within the accepted threshold requires neither an alert nor an automated intervention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Localize the slow FX service
Return to the payment graph. The request is progressing through the system, but the foreign-exchange rate service is taking longer than usual. The graph gives the investigation a specific target: FX Rate, rather than the entire payment pipeline. Once that node is identified, previous cases of FX service failure become relevant evidence for deciding what to investigate or repair.
Pandya describes a pre-live benchmark example using OpenTelemetry and DeathStarBench, with millions of traces over seven days, followed by injected anomalies to train the system. This establishes the intended preparation sequence: observe ordinary execution, introduce faults and develop the detector before putting it live. The talk does not provide the selected workload, exact trace count, injection procedure or evaluation results for that example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An incident, a new step or a new request mix?
An anomaly and drift pose different operational questions. Suppose the trip from home to the office normally takes an hour. One day it takes longer because of traffic or an accident: that is an isolated incident, and the response depends on how critical the disruption is. Now imagine noticing during a late-afternoon coffee break that the same commute routinely takes twenty minutes longer than it did a year ago. More cars and heavier Bay Area traffic have changed the pattern. The old baseline may no longer describe normal conditions.
Different kinds of drift call for different remedies:
- Structural change: A coffee shop starts asking whether you have a membership before applying discounted or regular pricing. That added question is a new processing step. Likewise, adding or removing a service node means the API's baselines and alerts must account for a changed path.
- Capacity-related degradation: Request volume grows until a service takes longer or cannot serve the expected load. Depending on the use case, the response may be to scale the service, add instances or make a call asynchronous.
Updating a baseline for an intentional extra step and addressing insufficient capacity are distinct operations, even if both first appear as increased latency.
A third case is covariate drift: the population of requests changes. Pandya's example begins with 60% local requests and 40% international requests. International payments may require currency conversion, so requests in US dollars and other currencies have different normal processing times. As the product becomes more popular abroad, the international share grows. The system can be working correctly while its aggregate latency changes.
Two responses are possible: compare local and international requests against separate execution graphs, or revise the aggregate request-time baseline to reflect the new mix. The distinction matters because a changing mix does not necessarily require a service repair. By contrast, if the same request now produces different behavior, the relevant response may be to reconsider or roll back a change. Classify the deviation before choosing the action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn diagnosis into a controlled rollout
The detection methods here form a statistical module within a larger system involving neural algorithms; the talk does not specify that larger system's learning procedure. The operational flow starts with OpenTelemetry continuously supplying data. Those observations support root-cause analysis and drift classification, which in turn inform a candidate remedy.
Before automating that remedy, assess its risk. Pandya illustrates a staged rollout: apply the remedy to 5% or 10% of machines, monitor and verify it, then expand to 100% of nodes. This separates selecting an action from authorizing its broad execution. A plausible diagnosis still needs evidence that the proposed change behaves as intended.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep analysis off the payment path
Pandya uses an illustrative normal end-to-end request time of 700 milliseconds to compare two latency patterns. One node becoming slow directs attention to that service. Delays spread across the graph instead suggest investigating something shared by the affected nodes. The graph helps distinguish these patterns even when both appear to the client as a slow payment.
The detection system must not become another source of payment latency. Telemetry is fed asynchronously into OpenTelemetry, with Kafka as a possible transport and stream assessment downstream. That processing can split into two paths:
| Path | Purpose | Tradeoff |
|---|---|---|
| Hot path | Fast decisions and possible automation | Prioritizes response speed |
| Reconciliation path | Slower, deeper assessment | May provide greater accuracy |
The architecture slide places the OTel SDK and Collector before Kafka/MSK and a Stream Assembler, then branches into TIER-0 fast gate and TIER-2 async processing. The separation lets immediate assessment coexist with analysis that needs more time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A missing event is not necessarily a missing service
Asynchronous observation introduces an ambiguity. Suppose seven nodes participate in a request, but one node's telemetry arrives late. The detector temporarily sees six nodes. Has the execution structure changed, or is the observed graph simply incomplete? Treating every absent event as a removed node creates false alarms. Structural-change detection therefore needs explicit timing criteria for when the available evidence is sufficient.
Pandya recommends a tail-based approach for this example because the analysis needs each node's start and end. More precisely, span instrumentation records those timings; tail sampling decides which traces to retain using accumulated span information. It does not guarantee completeness. The current Collector processor documentation explicitly accounts for late-arriving spans and decisions made before they arrive, so waiting and retention policies remain part of the design. A new endpoint presents a separate cold-start problem: it needs its own baseline, rather than inheriting an overly generic one.
For detection, Pandya names maximum mean discrepancy, or MMD, and KL divergence, followed by confirmation with ADWIN. Classification then identifies the kind of problem and informs automation where feasible. These are stages in a proposed detection flow; the precise integration and configuration are not supplied.
Pandya reports a substantial reduction in mean time to discovery, attributing faster issue resolution to using a single observation window instead of waiting for multiple windows. He supplies neither a numerical improvement nor the window duration. That single-window account describes his system; ADWIN itself adapts its window length as the observed data changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the baseline specific and the decision explainable
Labels help the system learn, but they require careful tuning. The baseline also needs to correspond to a meaningful operation. One baseline for all POST requests lumps together work with different behavior: real-time payments and wire payments should be considered separately. Define the observation window for structural changes, and use per-client baselines where client differences would otherwise generate noise. The relevant unit of comparison is the operation and its context, not merely its HTTP method.
An alert needs an explanation, not just a score. Pandya compares an unexplained anomaly score to a doctor reporting a health score of twenty-two: the number alone does not tell you what is wrong or what to do. Supporting observations let an operator make an informed decision. Finally, the system must know when a new deployment occurred. That context connects changed behavior to a possible cause and helps determine whether rollback is an appropriate response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source microservices workloads for experimenting with end-to-end service performance.
Further reading
Research introducing DeathStarBench and analyzing performance characteristics of cloud and IoT microservices.
Explains head versus tail sampling and demonstrates Collector policies for retaining traces.
Author-hosted manuscript explaining adaptive windows for detecting changes in streaming data.
Foundational treatment of maximum mean discrepancy for testing whether samples come from different distributions.
Pandya's separate presentation on deterministic routing, data locality and graph-based observability, with an organizer-hosted transcript.
Updates since the talk
Current configuration guidance covering sampling policies, buffering, decision timing and late spans.
Full transcript
Loading transcript…