AI Engineer Code 2025

GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe

Loading slides…

Read the talk

GPU Died. Training Didn't: Self-Healing Training at Scale

Crusoe combines Slurm’s training scheduler with Kubernetes infrastructure management. The recovery path connects GPU failure detection, node replacement, job requeueing and application checkpoint loading—while researchers keep their familiar commands.

From a talk by Connor Guerrero, Young Jeong and Nikhil Gupta

At a glance

Ideas worth remembering

  • Slurm supplies training-aware scheduling and familiar job commands; Kubernetes and AutoClusters connect those jobs to infrastructure telemetry and automatic node replacement.

  • Recovery crosses layers: Slurm cancels and requeues the job, AutoClusters replaces eligible failed nodes, and application code loads the checkpoint.

  • The two-A100-node demonstration reports roughly five minutes for hardware replacement and less than fifteen minutes for end-to-end training recovery.

  • Keeping GPU nodes in Kubernetes allows capacity released by completed Slurm jobs to serve inference workloads.

Thousands of GPUs turn hardware failure into routine work

Across thousands of GPUs, hardware failures become an expected part of running training. Fixing each failure by hand makes engineers part of the recovery path, including in the middle of the night. Connor Guerrero, Young Jeong and Nikhil Gupta introduce Crusoe’s response: combine Slurm and Kubernetes so the platform can replace failed hardware and restart training without requiring researchers or platform teams to adopt unfamiliar workflows.

The layers have different jobs. Crusoe Managed Kubernetes, or CMK, manages the underlying cluster. Managed Slurm runs above it, building on Slinky, the Slurm-on-Kubernetes project identified in the talk. Slurm supplies high-performance job scheduling; Kubernetes supplies infrastructure management and observability. AutoClusters adds the hardware-specific response: detect a failed GPU node and replace it.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Slurm coordinates training; node recovery adds another job

Slurm’s roots in high-performance computing suit distributed training. A multi-node job needs its ranks—the cooperating processes—to communicate closely across the training network. Scheduling those processes requires attention to both the resources available and their arrangement. Researchers already express these jobs through scripts and familiar sbatch and srun commands.

Several existing Slurm mechanisms serve that work:

  • Gang scheduling: coordinate the resources needed by a job whose ranks must run together.
  • Topology awareness: account for how the resources are connected when placing a training job.
  • Prolog and Epilog: provide hooks for cluster validation around job execution.

The difficulty grows when the same GPU fleet must support training, post-training, evaluation and inference. These workloads change over time, and some may run outside Slurm altogether. Young describes traditional Slurm partitions and resource arrangements as relatively static. Under GPU capacity constraints, that makes shifting hardware toward the workload that currently needs it an operational concern.

Selected presentation frame from GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe at 214 secondsOpen full source frame
A slide contrasts traditional Slurm with autoscaling, health gating, node failures, and observability.

Slurm already has ways to detect bad nodes and requeue jobs. Completing recovery still requires someone or something to connect those capabilities: identify the unhealthy node, drain it, arrange replacement capacity and get the job running again. Prolog checks, open-source tools and custom scripts can do pieces of this work, but maintaining that chain adds burden for the team.

A failed job also needs an explanation. GPU failure is one cause; intermittent network faults or switch problems can degrade performance or stop a job too. Knowing that a job failed does not by itself explain why. The infrastructure needs telemetry that helps distinguish these causes, alongside the scheduler’s view of job state.

2:212:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:13 · section reference included

Keep the Slurm interface and share the Kubernetes infrastructure

Kubernetes contributes a mature ecosystem for observability, networking and security, together with service self-healing, load balancing and autoscaling. The organizational problem is that teams often arrive with different tools: inference may already run on Kubernetes, while an incoming training team knows Slurm. Building a separate stack for each preserves familiarity but duplicates infrastructure work and costs.

Crusoe places Slurm inside the existing Kubernetes environment. Its Slurm operator coordinates with CMK and Crusoe’s storage and networking, and manages Slurm users, partitions, configuration and storage. That coordination makes creating a Slurm cluster a single operation rather than a collection of separately configured components.

The same deployment presents two useful views:

  • Researchers: receive an IP address, SSH in with their user and submit jobs to a familiar Slurm cluster. Kubernetes can remain underneath without becoming part of their daily workflow.
  • Platform teams: operate Slurm as another service alongside existing services, using the infrastructure and observability tools they already maintain. Slurm retains its topology-aware scheduling for training jobs.

A shared GPU pool also makes changing demand easier to accommodate. When inference demand rises, hardware can be reallocated from training toward inference. When user load falls, those GPUs can support a larger training run. Nikhil presents this as a capability enabled by the unified stack; the talk does not specify a policy for interrupting active training or deciding which workload wins competing demand.

The shared infrastructure matters for failure recovery too. Managed Slurm can use AutoClusters rather than maintaining a separate hardware-remediation system. The operator connects the scheduler’s job decisions with the platform’s node-replacement decisions.

5:516:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:49 · section reference included

From XID 79 to a replacement node and a restarted job

The recovery example starts with an XID 79 error, described here as leaving a GPU completely unusable. The user receives a notification, but no action is required. The Slurm operator marks the affected node down and cancels its running jobs. Before termination, the process receives SIGTERM, with up to two minutes to handle the signal—for example, by saving a checkpoint or flushing logs.

The cancelled job is automatically requeued. AutoClusters then checks whether any pod on the node carries a label that disables automatic replacement. If replacement is allowed, Kubernetes cordons and drains the node: it stops placing new work there and clears the existing workloads. AutoClusters removes the unhealthy node from its pool and replaces it with a healthy node from spare capacity, recording the alert and remediation for the user.

When the healthy node comes online, Slurm starts the job again. Application code loads the model and checkpoint, then resumes training. This division of responsibility is consequential: the platform restores capacity and restarts execution; the application must implement checkpoint saving and loading. A two-minute signal handler provides an opportunity to save state, rather than a guarantee that a failed GPU can still produce a fresh checkpoint.

What connects a hardware alert to resumed training? The flow below separates scheduler actions, node replacement and application recovery. Requeueing preserves the job’s opportunity to run again, but useful training resumes only after both healthy capacity and checkpoint-loading code are ready.

Selected presentation frame from GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe at 614 secondsOpen full source frame
A flow diagram titled “Handling a node failure” shows connected steps across several rows.

Hardware recovery and training recovery meet at job restart

Detect XID 79Detection — Notify the user that a GPU is unusable.
Mark down and cancelSlurm — Slurm stops affected jobs; SIGTERM allows up to two minutes for checkpoint or log handling.
Requeue the jobSlurm — The cancelled job waits to run again.
Check replacement eligibilityAutoClusters — Verify that no pod label disables automatic replacement.
Cordon, drain and replaceAutoClusters — Remove the unhealthy node, add a healthy node from spare capacity and log remediation.
Restart and load checkpointApplication — Slurm starts the job; application code loads the model and checkpoint to resume training.
Automatic replacement depends on the pod-label check and spare capacity. The application restores training state after Slurm restarts the job.
Read the diagram as text
  • Detect XID 79. Detection — Notify the user that a GPU is unusable.
  • Mark down and cancel. Slurm — Slurm stops affected jobs; SIGTERM allows up to two minutes for checkpoint or log handling.
  • Requeue the job. Slurm — The cancelled job waits to run again.
  • Check replacement eligibility. AutoClusters — Verify that no pod label disables automatic replacement.
  • Cordon, drain and replace. AutoClusters — Remove the unhealthy node, add a healthy node from spare capacity and log remediation.
  • Restart and load checkpoint. Application — Slurm starts the job; application code loads the model and checkpoint to resume training.
  • Detect XID 79 → Mark down and cancel: Failure reaches the Slurm operator.
  • Mark down and cancel → Requeue the job: Cancelled job is requeued.
  • Requeue the job → Check replacement eligibility: AutoClusters proceeds.
  • Check replacement eligibility → Cordon, drain and replace: Replacement permitted.
  • Cordon, drain and replace → Restart and load checkpoint: Healthy node comes online.
10:0310:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:03 · section reference included

The demo’s utilization drop becomes a checkpoint restart

The demonstration follows a PyTorch training script launched with sbatch against a pool of two A100 nodes. Connor then runs a script that tells Crusoe’s internal monitoring system an XID 79 error has occurred on one GPU. This injects the failure signal into the remediation path; it does not establish that the demo physically damaged a GPU.

GPU utilization drops, and node replacement begins immediately. Connor reports roughly five minutes from detecting the critical hardware error to returning a healthy node to the pool. The remaining time is mainly application-level work. That distinction explains why replacing hardware and recovering a training run have different completion times.

The job restarts, loads its checkpoint and continues training. Utilization returns to normal, with reported end-to-end downtime of less than fifteen minutes in this example. The alerts inform the user throughout, without asking them to perform recovery steps. The timing belongs to this demonstration, rather than a recovery-time guarantee for every model or cluster.

11:3512:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:30 · section reference included

Provision the recovery path, then keep both systems in sync

One-Click Slurm packages the environment into a provisioning operation. Despite the name, the interface described is a command: one command provisions the Kubernetes cluster, Slurm controller, login nodes and storage. A second command adds the GPU node pool. Researchers can then SSH in and submit jobs, with AutoClusters enabled by default.

Selected presentation frame from GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe at 829 secondsOpen full source frame
A slide titled “1-click Slurm” shows two command examples for provisioning.

The closing architectural point is synchronization. Traditionally, Slurm and the infrastructure beneath it can be operated as separate systems. Here, a failed Kubernetes node can be cordoned automatically and that signal propagated to the Slurm operator. The scheduler and infrastructure therefore respond to the same failure, rather than leaving an engineer to reconcile their states.

That connection preserves the two teams’ working interfaces. Researchers keep their Slurm commands without having to SSH into pods or manage containers. Kubernetes teams keep the telemetry and observability they expect. The integration absorbs the coordination work beneath those interfaces.

The GPU nodes are “Kubernetes nodes first.” When a Slurm job finishes, Kubernetes can schedule inference pods on those nodes instead of leaving them idle while they wait for another training job. This is the concrete resource-sharing case at the ending: completed training releases useful capacity for a different kind of workload.

Designing for inevitable failure means arranging the whole response in advance: detect the error, stop scheduling onto broken hardware, replace the node, restart the job and restore application state. Automating that sequence lets engineers spend their attention on the application instead of becoming the mechanism that gets training running again.

13:1813:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:18 · section reference included

Transcript download
Full transcript

Loading transcript…