AI Engineer World's Fair 2026
Using RL-based Agent to Detect and Remediate ETL Pipeline Failures
Loading slides…
Read the talk
Using RL-based Agent to Detect and Remediate ETL Pipeline Failures
A failed data job needs more than a retry. This AWS recovery design separates diagnosis, action selection, safety authority, and validation—and tests what learning actually contributes.
From a talk by Anna Marie Benzon
Before you start: Familiarity with ETL jobs, AWS Glue, and basic reinforcement-learning terminology will help; no prior Q-learning implementation experience is required.
A stale dashboard after midnight
A production data job failed hours ago. The dashboard is stale, and an engineer is still checking logs, schemas, and upstream data after midnight. What changed? The fault itself may be small; the expensive part is discovering it, choosing a safe repair, rerunning the job, and confirming that the repair did not make the data worse.
Anna Marie Benzon's capstone approaches this as a bounded operational decision: can an agent select a useful, explainable response that an operations team would trust? The inputs are messy—late or unavailable sources, schema drift, date-time incompatibilities, null-rate spikes, type changes, and unfamiliar runtime errors. The familiar human sequence is inspection, diagnosis, repair, rerun, and validation. Handoffs, incomplete context, and caution stretch that sequence out.
The capstone modeled manual recovery at roughly 2.5 working days, including queuing, investigation, and approval. That is an assumed incident workflow, not a measured production baseline. The objective is to compress recovery for routine, recognizable failures while escalating uncertain, novel, or high-risk cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a Glue failure to verified recovery
The AWS architecture starts with an existing Glue ETL job. Its failure event reaches Amazon EventBridge, which invokes the Lambda function running the agent. Lambda gathers evidence from two read-only sources: CloudWatch supplies error logs, and the Glue Data Catalog supplies current schema metadata. Classification, data-quality checks, and operational risk assessment turn that evidence into the state presented to the decision engine.
The policy proposes a response; it does not execute one directly. A safety layer checks the proposal before an executor can use the Glue API to retrigger the job or apply an approved remediation. S3 stores agent artifacts, audit logs, and quarantined outputs. Rerunning and validating the job close the loop: a completed action is not yet evidence of recovery.
The capstone used client-provided synthetic data. The public RL-Guided ETL Remediation Agent repository preserves the architecture through a sanitized, generalized deployment template.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One contextual decision, six possible actions
Tabular Q-learning selects among retry, coerce, rollback, quarantine, escalate, and log. With a small state and action space, the Q table is inexpensive to evaluate. An engineer can inspect the action values associated with a particular incident state and see which action won.
Despite the reinforcement-learning terminology, each incident is modeled as a single-step contextual decision, not a long-horizon control task. The system needs one bounded operational response. Learning supplies a way to update action preferences from outcomes while keeping the decision surface inspectable; sophistication is not the objective.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The policy does not grant itself permission
The safety layer evaluates a proposed action against anomaly severity and operational constraints. Critical conditions cannot receive passive responses, and high-risk or unknown cases go to humans. Because these constraints sit outside the learned policy, changing action preferences does not change the agent's authority.
Every proposal, override, execution result, and validation outcome enters an audit record. Escalation is an explicit action because recognizing the limits of evidence or authority is useful behavior. Optimizing only for non-escalation would reward avoiding human review even when review is the correct response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the experiment reviewable without client data
The public benchmark retains a generalized Lambda-style architecture while replacing the capstone's client-provided synthetic material with newly generalized schemas, records, logs, and incident scenarios. It contains no client documents, infrastructure identifiers, or business-specific values. This preserves the experimental structure without exposing the client context.
Benzon reports four controlled experiment groups and a robustness evaluation across 30 seeds, 42–71, with 95% confidence intervals for the reported aggregates. The current reproducibility script calculates those intervals across seed results using the normal approximation, mean ± 1.96 × sample standard deviation / √n. These intervals describe variation within the synthetic experiment, not uncertainty over a production incident population. Current repository details help interpret the measurements but do not establish an identical talk-era revision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Detection, recovery, and non-escalation measure different things
On the controlled detector benchmark, precision was 1.000, recall was 0.800, and F1 was 0.889. The detector was conservative: the anomalies it flagged were correct in that benchmark, but it missed some positive cases. Perfect precision did not mean perfect detection.
The recovery measurements also need distinct denominators:
| Measurement | Reported result | Scope |
|---|---|---|
| Mean resolution time | About 5.24 minutes | Successfully resolved incidents only |
| Simulated success | 74.63% ± 1.51 percentage points | Across 30 runs; reported 95% interval |
| Non-escalation | 88.63% ± 0.89 percentage points | Across 30 runs; reported 95% interval |
Success asks whether the incident was resolved. Non-escalation asks whether the agent avoided handing it to a human; an unsuccessful action can still count as non-escalation. In the current script, the recovery experiment does not explicitly apply the safety override; guarded configurations are evaluated separately. The non-escalation result should therefore not be read as a demonstrated rate of safe, successful autonomous recovery.
The reported MTTR reduction was approximately 99.85% against the benchmark's illustrative manual baseline of 216,000 seconds. Benzon describes that baseline as 2.5 working days; the benchmark implementation uses 60 elapsed hours, equivalent to 2.5 calendar days, without defining working-day hours. Comparing about 5.24 minutes with that denominator explains the rounded reduction. It remains a synthetic comparison, not a measured production improvement.
Within this controlled scope, the results support a fast recovery path for known failure conditions. They do not establish how that path will perform across production incidents; production validation remains the next boundary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What actually produced the reliability?
The ablations separate the contribution of learned preferences from the contribution of structured decisions and safety constraints:
| Comparison | Reported difference |
|---|---|
| RL versus equivalent deterministic policy | 0 percentage points in success; reported interval ±0.19 points |
| Deterministic versus random action selection | Success higher by 15.63 percentage points |
| Safety override enabled | Non-escalation lower by about 15.03 percentage points |
The compact state space allowed a hand-defined policy to match the learned policy's success level. Deliberate action selection beat random selection. The safety override reduced non-escalation intentionally by routing more cases to humans when autonomy would be inappropriate.
The demonstrated reliability came primarily from structured state, sensible decision logic, and external safety constraints—not from RL alone. Here, RL supplies an inspectable learned decision surface without an immediate success-rate advantage. Its prospective value grows when richer incident histories reveal context-dependent outcomes and maintaining every action preference by hand becomes difficult. That is a reason to investigate learning further, not a gain already established by this benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Shadow mode before execution authority
The system responds after a failure signal; it does not predict failures. Its evaluation uses synthetic scenarios, real incidents may exceed the compact state space, and some remediation actions are simulated or deliberately bounded. Those limits make this a feasibility demonstration of the architecture. Production online learning would require strict approval gates, versioned policies, rollback support, and continuous monitoring.
The proposed next step is shadow mode on representative incident traces. Recommendations can then be compared with human decisions before the agent receives execution authority. That tests whether its state representation and action preferences remain useful when the incident distribution becomes more realistic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reserve human judgment for the incidents that need it
The engineering discipline is to give deterministic logic the measurable facts and use learning only where contextual action selection adds value. Keeping safety constraints outside the policy prevents a policy update from silently redefining authority. Escalation and post-action validation belong among the system's expected outcomes, while repeated seeds and simple baselines distinguish evidence from one favorable demonstration.
A practical recovery agent does not need the largest model. It needs clear state, bounded actions, reproducible evaluation, observable decisions, and a stopping condition when uncertainty exceeds its authority.
Return to the engineer facing the stale dashboard. The intended change is from manual log inspection and schema tracing to event-triggered diagnosis, safety-constrained action, and explicit validation for supported routine cases. The unusual and high-risk failures still reach humans. The purpose is to stop spending human judgment on the same recognizable failure in the middle of the night, preserving attention for incidents whose context, trade-offs, or authority require it.
The publicly available companion repository provides the code, synthetic benchmark, experiment scripts, and reproducibility instructions. Benzon's closing invitation focuses on three concrete questions for further review: state representation, reward design, and the safety boundary. Those are the decisions that determine what the agent can recognize, what it learns to prefer, and where it must stop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Synthetic ETL remediation prototype with Q-learning, bounded actions, AWS templates, tests, and reproducibility commands. Review its non-open-source license before reuse.
Further reading
Explains scenario generation, four evaluation groups, repeated seeds, and the limits of synthetic recovery measurements.
- ETL remediation agent methodologyDocumentation
Details the contextual Q-learning formulation, reward design, state representation, and independent safety override.
AWS documentation for routing Glue job-state events to automated actions such as Lambda invocations.
Full transcript
Loading transcript…