AI Engineer World's Fair 2026

Using RL-based Agent to Detect and Remediate ETL Pipeline Failures

Loading slides…

Read the talk

Using RL-based Agent to Detect and Remediate ETL Pipeline Failures

A failed data job needs more than a retry. This AWS recovery design separates diagnosis, action selection, safety authority, and validation—and tests what learning actually contributes.

From a talk by Anna Marie Benzon

Before you start: Familiarity with ETL jobs, AWS Glue, and basic reinforcement-learning terminology will help; no prior Q-learning implementation experience is required.

A stale dashboard after midnight

A production data job failed hours ago. The dashboard is stale, and an engineer is still checking logs, schemas, and upstream data after midnight. What changed? The fault itself may be small; the expensive part is discovering it, choosing a safe repair, rerunning the job, and confirming that the repair did not make the data worse.

Anna Marie Benzon's capstone approaches this as a bounded operational decision: can an agent select a useful, explainable response that an operations team would trust? The inputs are messy—late or unavailable sources, schema drift, date-time incompatibilities, null-rate spikes, type changes, and unfamiliar runtime errors. The familiar human sequence is inspection, diagnosis, repair, rerun, and validation. Handoffs, incomplete context, and caution stretch that sequence out.

The capstone modeled manual recovery at roughly 2.5 working days, including queuing, investigation, and approval. That is an assumed incident workflow, not a measured production baseline. The objective is to compress recovery for routine, recognizable failures while escalating uncertain, novel, or high-risk cases.

The Problem slide lists six ETL failure causes, a Failure → Inspect Logs → Diagnose → Repair → Rerun → Validate workflow, and modeled manual MTTR of approximately 2.5 working days.Open full source frame
Common cloud ETL failures and the manual recovery workflow.
0:000:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From a Glue failure to verified recovery

The AWS architecture starts with an existing Glue ETL job. Its failure event reaches Amazon EventBridge, which invokes the Lambda function running the agent. Lambda gathers evidence from two read-only sources: CloudWatch supplies error logs, and the Glue Data Catalog supplies current schema metadata. Classification, data-quality checks, and operational risk assessment turn that evidence into the state presented to the decision engine.

The policy proposes a response; it does not execute one directly. A safety layer checks the proposal before an executor can use the Glue API to retrigger the job or apply an approved remediation. S3 stores agent artifacts, audit logs, and quarantined outputs. Rerunning and validating the job close the loop: a completed action is not yet evidence of recovery.

The capstone used client-provided synthetic data. The public RL-Guided ETL Remediation Agent repository preserves the architecture through a sanitized, generalized deployment template.

1:592:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:59 · section reference included

Separate facts, choices, and authority

Rules establish facts; learning selects bounded actions; guardrails determine authority. A disappeared field, changed type, or breached null threshold is an observable condition. The policy handles the contextual choice of what to do about it. An independent safety override can then replace that choice: if a critical anomaly receives a passive response such as logging, the override converts it to escalation.

Three stacked layers show deterministic anomaly rules; Q-learning choices including retry, rollback, quarantine, and escalation; and a safety override that escalates critical anomalies paired with passive actions.Open full source frame
Deterministic anomaly rules, Q-learning decisions, and a safety override.

The diagnostic components divide the evidence-gathering work into explicit responsibilities:

ComponentResponsibility
Schema profilerExtract structure, types, nesting, and null statistics
Drift detectorCompare current and baseline profiles for additions, removals, and type changes
Data-quality analyzerCheck completeness, validity, and consistency
Error classifierMap log patterns to failure families
Risk scorerConvert those signals into operational risk

These components are deterministic because their decisions need to be straightforward to validate, explain, and audit. A directly measurable condition does not require an opaque inference.

Richer, representative incident history could justify learned classifiers later. For now, each decision belongs to the simplest reliable component. The resulting policy state is compact: failure category, risk level, retry count, drift severity, and data-quality condition.

3:273:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:27 · section reference included

One contextual decision, six possible actions

Tabular Q-learning selects among retry, coerce, rollback, quarantine, escalate, and log. With a small state and action space, the Q table is inexpensive to evaluate. An engineer can inspect the action values associated with a particular incident state and see which action won.

Despite the reinforcement-learning terminology, each incident is modeled as a single-step contextual decision, not a long-horizon control task. The system needs one bounded operational response. Learning supplies a way to update action preferences from outcomes while keeping the decision surface inspectable; sophistication is not the objective.

5:195:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:19 · section reference included

The policy does not grant itself permission

The safety layer evaluates a proposed action against anomaly severity and operational constraints. Critical conditions cannot receive passive responses, and high-risk or unknown cases go to humans. Because these constraints sit outside the learned policy, changing action preferences does not change the agent's authority.

Every proposal, override, execution result, and validation outcome enters an audit record. Escalation is an explicit action because recognizing the limits of evidence or authority is useful behavior. Optimizing only for non-escalation would reward avoiding human review even when review is the correct response.

6:116:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:11 · section reference included

Safe to propose does not mean available to execute

The demonstrated failure path starts with a Glue-style job-failure event. The classifier reports a date-time format incompatibility with confidence 0.9, and the encoded state leads the policy to propose schema coercion. The anomaly is not classified as critical, so the safety override does not fire. The executor then discovers a separate problem: automatic coercion is unavailable for this particular case.

The incident moves to manual review. Its record preserves both the proposed action and the unavailable execution; it does not report a fix that never happened. In Python, that distinction can be represented explicitly:

python

incident = {
    "failure_category": "datetime_format_incompatibility",
    "classifier_confidence": 0.9,
    "critical": False,
    "proposed_action": "coerce",
    "safety_override": False,
    "execution_status": "pending",
    "validation_status": "not_started",
    "manual_review": False,
}

coercion_available = False

if incident["proposed_action"] == "coerce" and not coercion_available:
    incident.update(
        execution_status="unavailable",
        manual_review=True,
    )

assert incident["proposed_action"] == "coerce"
assert incident["execution_status"] == "unavailable"
assert incident["validation_status"] == "not_started"
assert incident["manual_review"] is True

The proposal remains visible after the capability check, and validation is not marked successful. Policy safety and implementation capability are separate controls: an action can be permissible in principle yet unsupported in the environment.

7:077:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:07 · section reference included

Make the experiment reviewable without client data

The public benchmark retains a generalized Lambda-style architecture while replacing the capstone's client-provided synthetic material with newly generalized schemas, records, logs, and incident scenarios. It contains no client documents, infrastructure identifiers, or business-specific values. This preserves the experimental structure without exposing the client context.

Benzon reports four controlled experiment groups and a robustness evaluation across 30 seeds, 42–71, with 95% confidence intervals for the reported aggregates. The current reproducibility script calculates those intervals across seed results using the normal approximation, mean ± 1.96 × sample standard deviation / √n. These intervals describe variation within the synthetic experiment, not uncertainty over a production incident population. Current repository details help interpret the measurements but do not establish an identical talk-era revision.

8:058:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:05 · section reference included

Detection, recovery, and non-escalation measure different things

On the controlled detector benchmark, precision was 1.000, recall was 0.800, and F1 was 0.889. The detector was conservative: the anomalies it flagged were correct in that benchmark, but it missed some positive cases. Perfect precision did not mean perfect detection.

The recovery measurements also need distinct denominators:

MeasurementReported resultScope
Mean resolution timeAbout 5.24 minutesSuccessfully resolved incidents only
Simulated success74.63% ± 1.51 percentage pointsAcross 30 runs; reported 95% interval
Non-escalation88.63% ± 0.89 percentage pointsAcross 30 runs; reported 95% interval

Success asks whether the incident was resolved. Non-escalation asks whether the agent avoided handing it to a human; an unsuccessful action can still count as non-escalation. In the current script, the recovery experiment does not explicitly apply the safety override; guarded configurations are evaluated separately. The non-escalation result should therefore not be read as a demonstrated rate of safe, successful autonomous recovery.

The reported MTTR reduction was approximately 99.85% against the benchmark's illustrative manual baseline of 216,000 seconds. Benzon describes that baseline as 2.5 working days; the benchmark implementation uses 60 elapsed hours, equivalent to 2.5 calendar days, without defining working-day hours. Comparing about 5.24 minutes with that denominator explains the rounded reduction. It remains a synthetic comparison, not a measured production improvement.

Evaluation Results slide reports precision 1.000, recall 0.800, F1 0.889, simulated success and non-escalation rates, alongside an MTTR chart comparing 2.5 working days with 5.24 ± 0.14 minutes.Open full source frame
Evaluation results from a controlled synthetic benchmark across 30 seeds.

Within this controlled scope, the results support a fast recovery path for known failure conditions. They do not establish how that path will perform across production incidents; production validation remains the next boundary.

9:029:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:02 · section reference included

What actually produced the reliability?

The ablations separate the contribution of learned preferences from the contribution of structured decisions and safety constraints:

ComparisonReported difference
RL versus equivalent deterministic policy0 percentage points in success; reported interval ±0.19 points
Deterministic versus random action selectionSuccess higher by 15.63 percentage points
Safety override enabledNon-escalation lower by about 15.03 percentage points

The compact state space allowed a hand-defined policy to match the learned policy's success level. Deliberate action selection beat random selection. The safety override reduced non-escalation intentionally by routing more cases to humans when autonomy would be inappropriate.

The demonstrated reliability came primarily from structured state, sensible decision logic, and external safety constraints—not from RL alone. Here, RL supplies an inspectable learned decision surface without an immediate success-rate advantage. Its prospective value grows when richer incident histories reveal context-dependent outcomes and maintaining every action preference by hand becomes difficult. That is a reason to investigate learning further, not a gain already established by this benchmark.

10:2210:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:22 · section reference included

Shadow mode before execution authority

The system responds after a failure signal; it does not predict failures. Its evaluation uses synthetic scenarios, real incidents may exceed the compact state space, and some remediation actions are simulated or deliberately bounded. Those limits make this a feasibility demonstration of the architecture. Production online learning would require strict approval gates, versioned policies, rollback support, and continuous monitoring.

The proposed next step is shadow mode on representative incident traces. Recommendations can then be compared with human decisions before the agent receives execution authority. That tests whether its state representation and action preferences remain useful when the incident distribution becomes more realistic.

11:4411:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:44 · section reference included

Reserve human judgment for the incidents that need it

The engineering discipline is to give deterministic logic the measurable facts and use learning only where contextual action selection adds value. Keeping safety constraints outside the policy prevents a policy update from silently redefining authority. Escalation and post-action validation belong among the system's expected outcomes, while repeated seeds and simple baselines distinguish evidence from one favorable demonstration.

A practical recovery agent does not need the largest model. It needs clear state, bounded actions, reproducible evaluation, observable decisions, and a stopping condition when uncertainty exceeds its authority.

Takeaways slide lists deterministic facts, contextual RL choices, safety constraints outside the policy, escalation and validation, and repeated-seed evaluation. A GitHub icon and QR code appear beside the list.Open full source frame
Five engineering takeaways for bounded, reproducible pipeline recovery.

Return to the engineer facing the stale dashboard. The intended change is from manual log inspection and schema tracing to event-triggered diagnosis, safety-constrained action, and explicit validation for supported routine cases. The unusual and high-risk failures still reach humans. The purpose is to stop spending human judgment on the same recognizable failure in the middle of the night, preserving attention for incidents whose context, trade-offs, or authority require it.

The publicly available companion repository provides the code, synthetic benchmark, experiment scripts, and reproducibility instructions. Benzon's closing invitation focuses on three concrete questions for further review: state representation, reward design, and the safety boundary. Those are the decisions that determine what the agent can recognize, what it learns to prefer, and where it must stop.

12:3212:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:32 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…