AI Engineer Code 2025
GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe
Loading slides…
Read the talk
GPU Died. Training Didn't: Self-Healing Training at Scale
Crusoe combines Slurm’s training scheduler with Kubernetes infrastructure management. The recovery path connects GPU failure detection, node replacement, job requeueing and application checkpoint loading—while researchers keep their familiar commands.
From a talk by Connor Guerrero, Young Jeong and Nikhil Gupta
At a glance
Ideas worth remembering
Slurm supplies training-aware scheduling and familiar job commands; Kubernetes and AutoClusters connect those jobs to infrastructure telemetry and automatic node replacement.
Recovery crosses layers: Slurm cancels and requeues the job, AutoClusters replaces eligible failed nodes, and application code loads the checkpoint.
The two-A100-node demonstration reports roughly five minutes for hardware replacement and less than fifteen minutes for end-to-end training recovery.
Keeping GPU nodes in Kubernetes allows capacity released by completed Slurm jobs to serve inference workloads.
Thousands of GPUs turn hardware failure into routine work
Across thousands of GPUs, hardware failures become an expected part of running training. Fixing each failure by hand makes engineers part of the recovery path, including in the middle of the night. Connor Guerrero, Young Jeong and Nikhil Gupta introduce Crusoe’s response: combine Slurm and Kubernetes so the platform can replace failed hardware and restart training without requiring researchers or platform teams to adopt unfamiliar workflows.
The layers have different jobs. Crusoe Managed Kubernetes, or CMK, manages the underlying cluster. Managed Slurm runs above it, building on Slinky, the Slurm-on-Kubernetes project identified in the talk. Slurm supplies high-performance job scheduling; Kubernetes supplies infrastructure management and observability. AutoClusters adds the hardware-specific response: detect a failed GPU node and replace it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Slurm coordinates training; node recovery adds another job
Slurm’s roots in high-performance computing suit distributed training. A multi-node job needs its ranks—the cooperating processes—to communicate closely across the training network. Scheduling those processes requires attention to both the resources available and their arrangement. Researchers already express these jobs through scripts and familiar sbatch and srun commands.
Several existing Slurm mechanisms serve that work:
- Gang scheduling: coordinate the resources needed by a job whose ranks must run together.
- Topology awareness: account for how the resources are connected when placing a training job.
- Prolog and Epilog: provide hooks for cluster validation around job execution.
The difficulty grows when the same GPU fleet must support training, post-training, evaluation and inference. These workloads change over time, and some may run outside Slurm altogether. Young describes traditional Slurm partitions and resource arrangements as relatively static. Under GPU capacity constraints, that makes shifting hardware toward the workload that currently needs it an operational concern.
Slurm already has ways to detect bad nodes and requeue jobs. Completing recovery still requires someone or something to connect those capabilities: identify the unhealthy node, drain it, arrange replacement capacity and get the job running again. Prolog checks, open-source tools and custom scripts can do pieces of this work, but maintaining that chain adds burden for the team.
A failed job also needs an explanation. GPU failure is one cause; intermittent network faults or switch problems can degrade performance or stop a job too. Knowing that a job failed does not by itself explain why. The infrastructure needs telemetry that helps distinguish these causes, alongside the scheduler’s view of job state.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the Slurm interface and share the Kubernetes infrastructure
Kubernetes contributes a mature ecosystem for observability, networking and security, together with service self-healing, load balancing and autoscaling. The organizational problem is that teams often arrive with different tools: inference may already run on Kubernetes, while an incoming training team knows Slurm. Building a separate stack for each preserves familiarity but duplicates infrastructure work and costs.
Crusoe places Slurm inside the existing Kubernetes environment. Its Slurm operator coordinates with CMK and Crusoe’s storage and networking, and manages Slurm users, partitions, configuration and storage. That coordination makes creating a Slurm cluster a single operation rather than a collection of separately configured components.
The same deployment presents two useful views:
- Researchers: receive an IP address, SSH in with their user and submit jobs to a familiar Slurm cluster. Kubernetes can remain underneath without becoming part of their daily workflow.
- Platform teams: operate Slurm as another service alongside existing services, using the infrastructure and observability tools they already maintain. Slurm retains its topology-aware scheduling for training jobs.
A shared GPU pool also makes changing demand easier to accommodate. When inference demand rises, hardware can be reallocated from training toward inference. When user load falls, those GPUs can support a larger training run. Nikhil presents this as a capability enabled by the unified stack; the talk does not specify a policy for interrupting active training or deciding which workload wins competing demand.
The shared infrastructure matters for failure recovery too. Managed Slurm can use AutoClusters rather than maintaining a separate hardware-remediation system. The operator connects the scheduler’s job decisions with the platform’s node-replacement decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From XID 79 to a replacement node and a restarted job
The recovery example starts with an XID 79 error, described here as leaving a GPU completely unusable. The user receives a notification, but no action is required. The Slurm operator marks the affected node down and cancels its running jobs. Before termination, the process receives SIGTERM, with up to two minutes to handle the signal—for example, by saving a checkpoint or flushing logs.
The cancelled job is automatically requeued. AutoClusters then checks whether any pod on the node carries a label that disables automatic replacement. If replacement is allowed, Kubernetes cordons and drains the node: it stops placing new work there and clears the existing workloads. AutoClusters removes the unhealthy node from its pool and replaces it with a healthy node from spare capacity, recording the alert and remediation for the user.
When the healthy node comes online, Slurm starts the job again. Application code loads the model and checkpoint, then resumes training. This division of responsibility is consequential: the platform restores capacity and restarts execution; the application must implement checkpoint saving and loading. A two-minute signal handler provides an opportunity to save state, rather than a guarantee that a failed GPU can still produce a fresh checkpoint.
What connects a hardware alert to resumed training? The flow below separates scheduler actions, node replacement and application recovery. Requeueing preserves the job’s opportunity to run again, but useful training resumes only after both healthy capacity and checkpoint-loading code are ready.
Hardware recovery and training recovery meet at job restart
Read the diagram as text
- Detect XID 79. Detection — Notify the user that a GPU is unusable.
- Mark down and cancel. Slurm — Slurm stops affected jobs; SIGTERM allows up to two minutes for checkpoint or log handling.
- Requeue the job. Slurm — The cancelled job waits to run again.
- Check replacement eligibility. AutoClusters — Verify that no pod label disables automatic replacement.
- Cordon, drain and replace. AutoClusters — Remove the unhealthy node, add a healthy node from spare capacity and log remediation.
- Restart and load checkpoint. Application — Slurm starts the job; application code loads the model and checkpoint to resume training.
- Detect XID 79 → Mark down and cancel: Failure reaches the Slurm operator.
- Mark down and cancel → Requeue the job: Cancelled job is requeued.
- Requeue the job → Check replacement eligibility: AutoClusters proceeds.
- Check replacement eligibility → Cordon, drain and replace: Replacement permitted.
- Cordon, drain and replace → Restart and load checkpoint: Healthy node comes online.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The demo’s utilization drop becomes a checkpoint restart
The demonstration follows a PyTorch training script launched with sbatch against a pool of two A100 nodes. Connor then runs a script that tells Crusoe’s internal monitoring system an XID 79 error has occurred on one GPU. This injects the failure signal into the remediation path; it does not establish that the demo physically damaged a GPU.
GPU utilization drops, and node replacement begins immediately. Connor reports roughly five minutes from detecting the critical hardware error to returning a healthy node to the pool. The remaining time is mainly application-level work. That distinction explains why replacing hardware and recovering a training run have different completion times.
The job restarts, loads its checkpoint and continues training. Utilization returns to normal, with reported end-to-end downtime of less than fifteen minutes in this example. The alerts inform the user throughout, without asking them to perform recovery steps. The timing belongs to this demonstration, rather than a recovery-time guarantee for every model or cluster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Provision the recovery path, then keep both systems in sync
One-Click Slurm packages the environment into a provisioning operation. Despite the name, the interface described is a command: one command provisions the Kubernetes cluster, Slurm controller, login nodes and storage. A second command adds the GPU node pool. Researchers can then SSH in and submit jobs, with AutoClusters enabled by default.
The closing architectural point is synchronization. Traditionally, Slurm and the infrastructure beneath it can be operated as separate systems. Here, a failed Kubernetes node can be cordoned automatically and that signal propagated to the Slurm operator. The scheduler and infrastructure therefore respond to the same failure, rather than leaving an engineer to reconcile their states.
That connection preserves the two teams’ working interfaces. Researchers keep their Slurm commands without having to SSH into pods or manage containers. Kubernetes teams keep the telemetry and observability they expect. The integration absorbs the coordination work beneath those interfaces.
The GPU nodes are “Kubernetes nodes first.” When a Slurm job finishes, Kubernetes can schedule inference pods on those nodes instead of leaving them idle while they wait for another training job. This is the concrete resource-sharing case at the ending: completed training releases useful capacity for a different kind of workload.
Designing for inevitable failure means arranging the whole response in advance: detect the error, stop scheduling onto broken hardware, replace the node, restart the job and restore application state. Automating that sequence lets engineers spend their attention on the application instead of becoming the mechanism that gets training running again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…