# GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe

Connor Guerrero, Crusoe · Young Jeong, Crusoe · Nikhil Gupta, Crusoe

AI Engineer Code 2025

[Talk page](https://ai.engineer/talks/bRGyYaE0lxI-gpu-died-training-didnt-self-healing-training) · [Video](https://www.youtube.com/watch?v=bRGyYaE0lxI)

## GPU Died. Training Didn't: Self-Healing Training at Scale

Crusoe combines Slurm’s training scheduler with Kubernetes infrastructure management. The recovery path connects GPU failure detection, node replacement, job requeueing and application checkpoint loading—while researchers keep their familiar commands.

### Thousands of GPUs turn hardware failure into routine work

[0:12](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=12s)

Across thousands of GPUs, hardware failures become an expected part of running training. Fixing each failure by hand makes engineers part of the recovery path, including in the middle of the night. Connor Guerrero, Young Jeong and Nikhil Gupta introduce Crusoe’s response: combine Slurm and Kubernetes so the platform can replace failed hardware and restart training without requiring researchers or platform teams to adopt unfamiliar workflows.

The layers have different jobs. Crusoe Managed Kubernetes, or CMK, manages the underlying cluster. Managed Slurm runs above it, building on Slinky, the Slurm-on-Kubernetes project identified in the talk. Slurm supplies high-performance job scheduling; Kubernetes supplies infrastructure management and observability. AutoClusters adds the hardware-specific response: detect a failed GPU node and replace it.

### Slurm coordinates training; node recovery adds another job

[2:13](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=133s)

Slurm’s roots in high-performance computing suit distributed training. A multi-node job needs its ranks—the cooperating processes—to communicate closely across the training network. Scheduling those processes requires attention to both the resources available and their arrangement. Researchers already express these jobs through scripts and familiar `sbatch` and `srun` commands.

Several existing Slurm mechanisms serve that work:

- **Gang scheduling:** coordinate the resources needed by a job whose ranks must run together.
- **Topology awareness:** account for how the resources are connected when placing a training job.
- **Prolog and Epilog:** provide hooks for cluster validation around job execution.

The difficulty grows when the same GPU fleet must support training, post-training, evaluation and inference. These workloads change over time, and some may run outside Slurm altogether. Young describes traditional Slurm partitions and resource arrangements as relatively static. Under GPU capacity constraints, that makes shifting hardware toward the workload that currently needs it an operational concern.

![Selected presentation frame from GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe at 214 seconds](https://ai.engineer/api/talk-essay-media/624ab118711486031a017a4487a5799b617c3d56a5cf1f8cf74a9699868634f8)

A slide contrasts traditional Slurm with autoscaling, health gating, node failures, and observability.

Slurm already has ways to detect bad nodes and requeue jobs. Completing recovery still requires someone or something to connect those capabilities: identify the unhealthy node, drain it, arrange replacement capacity and get the job running again. Prolog checks, open-source tools and custom scripts can do pieces of this work, but maintaining that chain adds burden for the team.

A failed job also needs an explanation. GPU failure is one cause; intermittent network faults or switch problems can degrade performance or stop a job too. Knowing that a job failed does not by itself explain why. The infrastructure needs telemetry that helps distinguish these causes, alongside the scheduler’s view of job state.

### Keep the Slurm interface and share the Kubernetes infrastructure

[5:49](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=349s)

Kubernetes contributes a mature ecosystem for observability, networking and security, together with service self-healing, load balancing and autoscaling. The organizational problem is that teams often arrive with different tools: inference may already run on Kubernetes, while an incoming training team knows Slurm. Building a separate stack for each preserves familiarity but duplicates infrastructure work and costs.

Crusoe places Slurm inside the existing Kubernetes environment. Its Slurm operator coordinates with CMK and Crusoe’s storage and networking, and manages Slurm users, partitions, configuration and storage. That coordination makes creating a Slurm cluster a single operation rather than a collection of separately configured components.

The same deployment presents two useful views:

- **Researchers:** receive an IP address, SSH in with their user and submit jobs to a familiar Slurm cluster. Kubernetes can remain underneath without becoming part of their daily workflow.
- **Platform teams:** operate Slurm as another service alongside existing services, using the infrastructure and observability tools they already maintain. Slurm retains its topology-aware scheduling for training jobs.

A shared GPU pool also makes changing demand easier to accommodate. When inference demand rises, hardware can be reallocated from training toward inference. When user load falls, those GPUs can support a larger training run. Nikhil presents this as a capability enabled by the unified stack; the talk does not specify a policy for interrupting active training or deciding which workload wins competing demand.

The shared infrastructure matters for failure recovery too. Managed Slurm can use AutoClusters rather than maintaining a separate hardware-remediation system. The operator connects the scheduler’s job decisions with the platform’s node-replacement decisions.

### From XID 79 to a replacement node and a restarted job

[10:03](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=603s)

The recovery example starts with an XID 79 error, described here as leaving a GPU completely unusable. The user receives a notification, but no action is required. The Slurm operator marks the affected node down and cancels its running jobs. Before termination, the process receives `SIGTERM`, with up to two minutes to handle the signal—for example, by saving a checkpoint or flushing logs.

The cancelled job is automatically requeued. AutoClusters then checks whether any pod on the node carries a label that disables automatic replacement. If replacement is allowed, Kubernetes cordons and drains the node: it stops placing new work there and clears the existing workloads. AutoClusters removes the unhealthy node from its pool and replaces it with a healthy node from spare capacity, recording the alert and remediation for the user.

When the healthy node comes online, Slurm starts the job again. Application code loads the model and checkpoint, then resumes training. This division of responsibility is consequential: the platform restores capacity and restarts execution; the application must implement checkpoint saving and loading. A two-minute signal handler provides an opportunity to save state, rather than a guarantee that a failed GPU can still produce a fresh checkpoint.

What connects a hardware alert to resumed training? The flow below separates scheduler actions, node replacement and application recovery. Requeueing preserves the job’s opportunity to run again, but useful training resumes only after both healthy capacity and checkpoint-loading code are ready.

![Selected presentation frame from GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe at 614 seconds](https://ai.engineer/api/talk-essay-media/bbb75035e924033686653882a1677b4ef72364d71cb8d59109da077f3a2a996c)

A flow diagram titled “Handling a node failure” shows connected steps across several rows.

Hardware recovery and training recovery meet at job restart
Automatic replacement depends on the pod-label check and spare capacity. The application restores training state after Slurm restarts the job.
- Detect XID 79 — Notify the user that a GPU is unusable.
- Mark down and cancel — Slurm stops affected jobs; SIGTERM allows up to two minutes for checkpoint or log handling.
- Requeue the job — The cancelled job waits to run again.
- Check replacement eligibility — Verify that no pod label disables automatic replacement.
- Cordon, drain and replace — Remove the unhealthy node, add a healthy node from spare capacity and log remediation.
- Restart and load checkpoint — Slurm starts the job; application code loads the model and checkpoint to resume training.
- Detect XID 79 → Mark down and cancel (Failure reaches the Slurm operator)
- Mark down and cancel → Requeue the job (Cancelled job is requeued)
- Requeue the job → Check replacement eligibility (AutoClusters proceeds)
- Check replacement eligibility → Cordon, drain and replace (Replacement permitted)
- Cordon, drain and replace → Restart and load checkpoint (Healthy node comes online)

### The demo’s utilization drop becomes a checkpoint restart

[11:30](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=690s)

The demonstration follows a PyTorch training script launched with `sbatch` against a pool of two A100 nodes. Connor then runs a script that tells Crusoe’s internal monitoring system an XID 79 error has occurred on one GPU. This injects the failure signal into the remediation path; it does not establish that the demo physically damaged a GPU.

GPU utilization drops, and node replacement begins immediately. Connor reports roughly five minutes from detecting the critical hardware error to returning a healthy node to the pool. The remaining time is mainly application-level work. That distinction explains why replacing hardware and recovering a training run have different completion times.

The job restarts, loads its checkpoint and continues training. Utilization returns to normal, with reported end-to-end downtime of less than fifteen minutes in this example. The alerts inform the user throughout, without asking them to perform recovery steps. The timing belongs to this demonstration, rather than a recovery-time guarantee for every model or cluster.

### Provision the recovery path, then keep both systems in sync

[13:18](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=798s)

One-Click Slurm packages the environment into a provisioning operation. Despite the name, the interface described is a command: one command provisions the Kubernetes cluster, Slurm controller, login nodes and storage. A second command adds the GPU node pool. Researchers can then SSH in and submit jobs, with AutoClusters enabled by default.

![Selected presentation frame from GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe at 829 seconds](https://ai.engineer/api/talk-essay-media/82cece75ad7545209fdca3ff0e83a831978fcc37260527776af00175d3a92ada)

A slide titled “1-click Slurm” shows two command examples for provisioning.

The closing architectural point is synchronization. Traditionally, Slurm and the infrastructure beneath it can be operated as separate systems. Here, a failed Kubernetes node can be cordoned automatically and that signal propagated to the Slurm operator. The scheduler and infrastructure therefore respond to the same failure, rather than leaving an engineer to reconcile their states.

That connection preserves the two teams’ working interfaces. Researchers keep their Slurm commands without having to SSH into pods or manage containers. Kubernetes teams keep the telemetry and observability they expect. The integration absorbs the coordination work beneath those interfaces.

The GPU nodes are “Kubernetes nodes first.” When a Slurm job finishes, Kubernetes can schedule inference pods on those nodes instead of leaving them idle while they wait for another training job. This is the concrete resource-sharing case at the ending: completed training releases useful capacity for a different kind of workload.

Designing for inevitable failure means arranging the whole response in advance: detect the error, stop scheduling onto broken hardware, replace the node, restart the job and restore application state. Automating that sequence lets engineers spend their attention on the application instead of becoming the mechanism that gets training running again.

## Key takeaways

- Slurm supplies training-aware scheduling and familiar job commands; Kubernetes and AutoClusters connect those jobs to infrastructure telemetry and automatic node replacement.
- Recovery crosses layers: Slurm cancels and requeues the job, AutoClusters replaces eligible failed nodes, and application code loads the checkpoint.
- The two-A100-node demonstration reports roughly five minutes for hardware replacement and less than fifteen minutes for end-to-end training recovery.
- Keeping GPU nodes in Kubernetes allows capacity released by completed Slurm jobs to serve inference workloads.

## Slides

### Fault-Tolerant Training at Scale

Crusoe presents how Slurm and Kubernetes support automated remediation when hardware fails during large-scale GPU training.

[0:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=21s) [Title slide introducing fault-tolerant training and the presenters.](https://assets.ai.engineer/sites/aiecode2025/sha256/05e0b32d903686e5b2169412d11bc8753b8a1193dc65cc5eb5d8c9c2181a2769.webp)

Crusoe

Fault-Tolerant Training at Scale

Making Hardware Failures a Non-Event

Nikhil Gupta, Senior Software Engineer
Young Jeong, Staff Solutions Engineer
Connor Guerrero, Developer Advocate

### Crusoe Cloud overview

Crusoe highlights managed Kubernetes and Slurm as the orchestration layer above its cloud infrastructure. AutoClusters detects and replaces failed GPU nodes.

[0:57](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=57s) [Diagram of Crusoe Cloud services, highlighting managed Kubernetes and Slurm.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d5bf4c94037a43370c9b6aa2342c298e0845be8ab4dbd0ea5234052d33aebe6.webp)

The world’s first Vertically Integrated Cloud

Crusoe Cloud

Developer Tools
CLI
Terraform Provider
Crusoe Cloud Console
MCP Server

Crusoe Command Center
Unified operations platform that automates orchestration, observability, and support

AI IaaS
(Infrastructure as a Service)

AI Compute
VM Instances
GPUs
AMD GPUs
NVIDIA GPUs

High-Performance Storage
Block Storage
File Storage
Object Storage

Accelerated Networking
VPC
NVIDIA InfiniBand
Load Balancers

AI PaaS
(Platform as a Service)
Crusoe Managed Kubernetes
Crusoe Managed Slurm
AutoClusters

Managed AI Services

Bring Your Own Model
Managed Inference API
Proprietary Inference Engine
MemoryAlloy technology

Crusoe Intelligence Foundry

Crusoe Managed Inference
Proprietary Inference Engine
MemoryAlloy technology

Crusoe Serverless Fine-Tuning
Coming soon
Proprietary Fine-tuning Engine
Curated Model Tuning Recipes

Energy + Manufacturing

Crusoe

### Traditional Slurm

Slurm supplies familiar scheduling tools for multi-node training, but static partitions and the work of checking and replacing nodes and diagnosing job failures complicate modern AI infrastructure.

[2:09](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=129s) [Traditional Slurm capabilities and challenges involving autoscaling, health gating, node failures, and observability.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5f34f8539312ad8963223fe344788ac185720d09be24d65a0a83027990e9c80.webp)

Traditional Slurm

Built for HPC, translates well to modern AI Training workloads

Large multi-node jobs need tight resource control.

Slurm already provides needed features to handle model training - Topology awareness, gang scheduling, partition management, and others - with the tools, scripts and commands familiar to most researchers.

However...

Autoscaling

Traditional Slurm was built for static, partitioned jobs — not the dynamic resource allocation that modern AI infrastructure demands

Health Gating

Can you make sure your nodes are healthy to begin with - yes

Does it add operational burden on your team - also yes

Node Failures

GPU Nodes fail - and Slurm has features to allow for job restarts - but only if there are mechanisms to replace bad nodes

Observability

Slurm knows a job failed - but not necessarily why. Diagnosing tightly-coupled training failures require comprehensive telemetry

Crusoe

### Why Slurm on Kubernetes

Crusoe Managed Slurm keeps familiar Slurm commands while Kubernetes manages component health. Storage, networking and GPUs are preconfigured; AutoClusters replaces failed nodes from a pre-validated spare pool.

[5:45](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=345s) [Benefits include familiar Slurm commands, automatic GPU topology configuration, and node remediation.](https://assets.ai.engineer/sites/aiecode2025/sha256/8291714c7c0ba5a4b05504d8050715c43bba2559e4394751ca03282e494e07b8.webp)

Crusoe Managed Slurm

Why Slurm on K8s →

Scheduling performance from Slurm, cloud-native infra resilience from Kubernetes.

Kubernetes handles Slurm component health automatically.

Every CMK platform capability (metrics, observability, node remediation) works with Slurm out-of-the-box.

Same commands, familiar interface
→ sbatch, srun, squeue, scancel — your existing scripts run as-is

GPU topology configured automatically
→ Storage, networking, and GPU configuration are all pre-configured and managed.

Automated node remediation
→ When a node fails, AutoClusters replaces it from a pre-validated spare pool.

Crusoe

### Handling a node failure

An XID error triggers a user notification, Slurm job requeue, and AutoClusters node replacement. Once a healthy node is available, the job restarts and training resumes from a checkpoint.

[9:59](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=599s) [Flowchart showing error detection, node replacement, job requeueing, and training resumption from a checkpoint.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bc49b744cd63ca14a9ee6e0ef4683a402537d778d177c84aeaccc9bf8d3c9a1.webp)

Node Remediation

Handling a node failure ↓

User notified

XID error detected → AutoClusters → Slurm job waits for healthy node → Slurm job starts → Training resumes from ckpt

AutoClusters → Verification → Node drain → Node replacement → History logging

Slurm
Node set to DOWN → SIGTERM (2 min grace) → Job requeued

Crusoe

### AutoClusters node replacement demo

The demo launches a PyTorch training job on a two-node A100 GPU pool. A preview shows AutoClusters starting node replacement after a simulated GPU error.

[11:29](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=689s) [Demo slide showing a terminal and cloud console with an annotation that node replacement starts.](https://assets.ai.engineer/sites/aiecode2025/sha256/b540b6135c1f842d8fbf1938a194369491b95f0697d815ce2f8ce4e001b58b37.webp)

Demo

AutoClusters node replacement starts →

Interface text:
Remediations
Issue Detected | Last Seen | Affected VM | Remediation Action | Remediation Status
XID 79 - GPU Fell Off the Bus | [illegible] | [illegible] | Replace Node | In Progress
XID 79 - GPU Fell Off the Bus | [illegible] | [illegible] | Replace Node | Succeeded
Replacement VM: [illegible]
Notifications
AutoClusters Remediation Started
Host Maintenance Required
All notifications
Crusoe
[Terminal output illegible]

[11:34](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=694s) [A terminal launches a Slurm training job beside the cluster node-pool view.](https://assets.ai.engineer/sites/aiecode2025/sha256/57af7bd0098f6a1ab46946900400f7953ec5696b272970d806428fd4b57d0756.webp)

← Launching Slurm training job

Interface text:
[illegible] sbatch [illegible]
Submitted batch job [illegible]
[illegible] tail -f [illegible]

cg-cluster-3
Running
Download Kubeconfig
Details | Topology | Remediations | Metrics | Logs
Version: [illegible]
Location: us-east1-a
Created: 02/24, 4:15 PM
Updated: 02/24, 4:23 PM
ID: [illegible]
Network: default-vpc-network ([illegible])
Show more

Node Pools
2 Node Pools active
Create Node Pool
Name | Instance Type | Health | # of Nodes | K8s Version | Sync Status | Actions
slurm-workers | 8x Nvidia A100 80GB SXM IB | Running | 2/2 | [illegible] | Up to Date
slurm-control | 4x General Purpose CPU | Running | 2/2 | [illegible] | Up to Date

### Injecting a simulated XID 79 error

While the training job runs, a script sends a simulated XID 79 error to the backend monitoring system to trigger AutoClusters.

[11:58](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=718s) [A terminal sends simulated error metrics while training output and cluster GPU charts remain visible.](https://assets.ai.engineer/sites/aiecode2025/sha256/41a5a107728136269dfa5ce911a6147a36259d2350460f6929956a9308b76d7c.webp)

← Injecting simulated XID 79 error

Terminal output:
[illegible] Did you mean to pass a tuple to create a subarray type? (Deprecated NumPy 2.4)
    entry = pickle.load(f, encoding='latin1')
[Epoch 0] Loss: [illegible]
Accuracy: [illegible]
[Epoch 1] Loss: [illegible]
Accuracy: [illegible]
[Epoch 2] Loss: [illegible]
Accuracy: [illegible]
[Epoch 3] Loss: [illegible]
Accuracy: [illegible]
[Epoch 4] Loss: [illegible]
Accuracy: [illegible]
[Epoch 5] Loss: [illegible]
Accuracy: [illegible]
[Epoch 6] Loss: [illegible]
Accuracy: [illegible]
[Epoch 7] Loss: [illegible]
Accuracy: [illegible]
[Epoch 8] Loss: [illegible]
Accuracy: [illegible]
[Epoch 9] Loss: [illegible]
Accuracy: [illegible]
[Epoch 10] Loss: [illegible]
Accuracy: [illegible]
[Epoch 11] Loss: [illegible]
Accuracy: [illegible]
[Epoch 12] Loss: [illegible]
Accuracy: [illegible]
[Epoch 13] Loss: [illegible]
Accuracy: [illegible]
[Epoch 14] Loss: [illegible]
Accuracy: [illegible]
[Epoch 15] Loss: [illegible]
Accuracy: [illegible]
[Epoch 16] Loss: [illegible]
Accuracy: [illegible]
[Epoch 17] Loss: [illegible]
Accuracy: [illegible]

bash inject_failure.sh
Starting Chaos Injection for Node: [illegible]
[1/100] Metric sent.
[2/100] Metric sent.
[3/100] Metric sent.

Metrics

Cluster TFLOPS (FP16)
Y-axis ticks: 0, 0.5, 1, 1.5, 2, 2.5, 3
Timeline: 7:44 AM, 7:56 AM, 8:08 AM, 8:20 AM, 8:32 AM

Average GPU Utilization (%)
Y-axis ticks: 0, 20, 40, 60, 80, 100
Timeline: 7:44 AM, 7:56 AM, 8:08 AM, 8:20 AM, 8:32 AM

Cumulative Power Usage (kW)
Visible y-axis ticks: 1.6, 1.8, 2, 2.2, 2.4

Average GPU Memory Utilization (%)
Visible y-axis ticks: 15, 20, 25, 30, 35, 40

Interface text:
Solutions Engineering Lab
cg-cluster-3
Running
Download Kubeconfig
Details  Topology  Remediations  Metrics  Logs
Last updated: 02/27, 8:38:41 AM
1 hour
Refresh
console.crusoecloud.com/[illegible]

### Node failure interrupts training

After a simulated XID 79 error, GPU utilization falls; the demo labels the node destroyed and training crashed.

[12:07](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=727s) [Cluster charts show GPU utilization falling; an annotation says the node is destroyed and training crashes.](https://assets.ai.engineer/sites/aiecode2025/sha256/3829d696c46deecee5b61b55d8258e328299cf66cccf517cdba84e160a2a0f80.webp)

Node destroyed, training crashes →

Metrics

Cluster TFLOPS (FP16)
Y-axis ticks: 3.5, 3, 2.5, 2, 1.5, 0
Timeline: 7:47 AM, 7:59 AM, 8:11 AM, 8:23 AM, 8:35 AM
Legend: [illegible]

Average GPU Utilization (%)
Y-axis ticks: 100, 80, 60, 40, 0
Timeline: 7:47 AM, 7:59 AM, 8:11 AM, 8:23 AM, 8:35 AM
Legend: [illegible]

Cumulative Power Usage (kW)
Y-axis ticks: 2.4, 2.2, 2, 1.8, 1.6

Average GPU Memory Utilization (%)
Y-axis ticks: 80, 70, 60, 50, 40, 30

Terminal output:
[Epoch 18] Loss: 0.0076
Accuracy: 88.20%
[Epoch 19] Loss: 0.0083
Accuracy: 88.26%
[Epoch 20] Loss: 0.0077
Accuracy: 88.51%
[Additional epoch loss and accuracy output: illegible]
[Metric counts: illegible] Metric sent.
Done

Interface text:
Solutions Engineering Lab
default
cg-cluster-3
Running
Download Kubeconfig
Details  Topology  Remediations  Metrics  Logs
Last updated: 02/27, 8:41:39 AM
1 hour
Refresh

### Automatic node replacement starts

After the demo reports an XID 79 GPU error, AutoClusters starts replacing the node and issues alerts. The speaker says detection to a healthy replacement takes roughly five minutes.

[12:11](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=731s) [The remediation view shows a node replacement in progress alongside a remediation notification.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ba008389801b1b7880edd067e8924bedf300321e633a174c7fac1ed2c103f11.webp)

AutoClusters node replacement starts →

Remediations

Issues Detected
Status
Remediation

Issue Detected | Last Seen | Affected VM | Remediation Action | Remediation Status
XID 79 - GPU Fell Off the Bus | 02/27, 8:41 AM | np-[illegible]-3 | Replace Node | In Progress
XID 79 - GPU Fell Off the Bus | 02/25, 11:46 AM | np-[illegible]-2 | Replace Node | Succeeded
02/25, 11:53 AM
Replacement VM: np-[illegible]-3

Notifications
Clear all
AutoClusters Remediation Started
Default > cg-cluster-3 > np-[illegible]-3 · 1 min. ago
Host Maintenance Required
Default > cg-cluster-3 > np-[illegible]-3 · 1 min. ago
All notifications

Interface text:
Chrome
console.crusoecloud.com/orchestration/clusters/[illegible]/auto-clusters
Solutions Engineering Lab
Default
Details  Topology  Remediations  Metrics  Logs
1-2 of 2
10 per page

Training terminal:
[illegible] Loss: [illegible]
Accuracy: [illegible]
[repeated training output illegible]

Metrics terminal:
[illegible] Metric sent.
[repeated metric output illegible]
Done
[redacted]

### 1-click Slurm

One command provisions the Kubernetes cluster, Slurm controller, login nodes, and storage; a second adds a GPU worker pool. AutoClusters is enabled by default.

[13:17](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=797s) [Two example commands create a Slurm environment and add GPU worker nodes.](https://assets.ai.engineer/sites/aiecode2025/sha256/80edb16298aa841a14f8fbb1a46894962fba3497f247ba70c66d8813464518fd.webp)

Setup

1-click Slurm ↓

1
One command provisions a complete Slurm environment.

crusoe slurm clusters create \
  --name my-slurm-cluster \
  --location us-southcentral1-a \
  --keyfile ~/.ssh/id_ed25519.pub

2
GPU worker nodes can be easily added.

crusoe slurm nodesets create \
  --name gpu-workers \
  --cluster-name my-slurm-cluster \
  --type h100-80gb-sxm-ib.8x \
  --count 2 \
  --ib-partition-id <ib-partition-id>

Crusoe

### Wrapping up

Slurm on Kubernetes keeps scheduling and infrastructure in sync during failures, supporting remediation through checkpoint resume. The same GPU pool can run Slurm training jobs and Kubernetes inference services.

[13:59](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=839s) [Summary of Slurm on Kubernetes, automated remediation, and shared GPU infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/f5fe20c5e1e4de297186f001b99713136a9b6c290c6e0c3e50de972b7bcb310d.webp)

Conclusion

Wrapping up

What we covered
→ Traditional Slurm handles scheduling well, but leaves failure detection, node replacement, and recovery as manual operational work
→ How Slurm on Kubernetes creates the integration surface for automated remediation without requiring either audience to change how they work
→ The full node remediation flow — from XID error detection to checkpoint resume

Key takeaways
→ Running Slurm on Kubernetes means the scheduler and the infrastructure layer stay synchronized during a failure
→ Slurm user's get topology-aware, high performance job scheduling without needing to know Kubernetes exists.
→ For the platform team, it's just another service on the cluster, managed from one control plane
→ All GPUs live in one pool — the same nodes can serve training jobs via Slurm and long-running inference services via K8s

Crusoe

### Thank you

The speaker invites questions at booth UG12 or via @crusoedev. Crusoe also lists crusoe.ai/developers.

[16:17](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=977s) [Crusoe thank-you slide inviting attendees to connect at the booth or online.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d618315bd8afc97ddde4608778dda8bb88c7ee2620c3f8b317e84c47c006cf2.webp)

Crusoe

Thank you!

Talk to us at booth UG12

crusoe.ai/developers

Social: @crusoedev

Crusoe

## Full transcript

[0:12](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=12s) All right. Hello, everyone. Uh, my name's Connor. I'm a developer advocate at Crusoe, and I'm joined by my colleagues Nikhil and Young. And we're gonna be talking today about the infrastructure we've built at Crusoe, and how it handles critical hardware errors. And so our customers, when they're running extremely large scale training workloads across thousands of GPUs, GPU failures are inevitable. And so at scale, manual remediation is completely unsustainable. And so what we'll cover today is how

[0:42](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=42s) by using both Slurm and Kubernetes, we can provide a robust platform with automated remediation that maintains ease of use for both machine learning engineers running jobs and the platform teams managing the underlying clusters.

[0:59](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=59s) So to provide a little bit more context on what Crusoe Cloud is, for those of you who haven't heard of us, um, we provide infrastructure as a service consisting of compute, storage, networking, and the latest gen GPUs from NVIDIA and AMD. But the focus of this talk will be on the tooling that we've built on top of that to orchestrate these large fleets of GPUs, including our managed Kubernetes service, called CMK, and our managed Slurm service, which is built

[1:28](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=88s) on top of CMK and built off of Slinky, which is SchedMD's official open source project for running Slurm on top of Kubernetes. And this allows us to get the high performance job scheduling of Slurm with the infrastructure resilience features and observability of Kubernetes, including one of the systems we've developed called AutoClusters, which is our system for automatically detecting and replacing failed GPU nodes when a GPU failure has been detected. And so

[1:58](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=118s) to explain how and why we landed on this architecture, it's helpful to understand where traditional Slurm both excels and falls short, which Young will get into now.

[2:13](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=133s) Thanks, Connor. Um, traditionally, I mean, looking at it from the trad- traditional perspective...

[2:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=141s) Sorry, can, can you hear me now? Okay, cool. I'll, I'll just do it like this. But basically, traditionally, Slurm was built, Slurm was built by the researchers for the researchers, uh, universities, labs, over twenty years ago, uh, built for a high performance computing, uh, which as we've observed with our customers and others that are using Slurm today, translates very well to modern AI, specifically around training workloads. This idea that when you're running a multi-node training

[2:51](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=171s) job, you need this idea of tight collective communication across the ranks that are-- that you're using, uh, including, you know, specific network built for training workloads today. Um, concepts like gang scheduling, topology awareness, uh, Prolog and Epilog for cluster validations. These are all the features that are readily available with Slurm today that helps researchers run their jobs, uh, essentially running

[3:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=201s) their jobs with tools that are already familiar to them, scripts, sbatch, srun commands, the things that are pretty standard in your Slurm cluster today. Now, where we feel like this is a bit of a shortcoming or a short-- falls a little bit short around the modern training is the things that you see on the screen right now. Um, AI training workloads today are very dynamic, and it doesn't just include training. Uh, there's a lot of other aspects that your AI

[3:51](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=231s) labs and oth- other companies are running as part of your, your AI workloads, including training, post-training, eval, inferences, some of which may or may not even land on your Slurm cluster today. But because of, of what we've observed as, as a GPU capacity constraints and other things, kinda limits you to how you need to be able to be more dynamic around the resources that you have. Uh, Slurm traditionally is pretty static, and it's partitioning in other, other, uh, functionalities. The

[4:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=261s) other thing I wanna mention is this idea of health checks and maintaining your nodes. Uh, you know, you can make sure your nodes are healthy or not healthy. GPUs do fail. How do you manage that? Uh, Slurm obviously has features around it to be able to, you know, re-queue the jobs, uh, detect bad nodes, uh, using things like Prolog and other open source tools that are now available to be able to do this. It just adds an operational burden, uh, that we've seen from customers

[4:51](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=291s) today. You know, this idea that you have to detect if nodes are bad, you have to drain the nodes when they're bad, you have to kind of work around that. That could either be manual or you need some sort of an automation or scripting to be able to do all of this. And the last thing is observability. Um, you know, jobs can fail. Jobs can also come with degraded performances for any number of reasons. I mean, GPU failure is just one of them. There's other things like network flaps and other things within your, your switches

[5:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=321s) that may be causing degraded or failed performance. Uh, Slurm knows when a job fails. It doesn't necessarily be able to tell you everything about why it failed, and that's where the observability comes into key, uh, key play. So all of this kind of encompasses the reason why we, we built this product that Nikhil's gonna go over and sort of describe some of the specifics around Managed Slurm.

[5:49](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=349s) Um,

[5:51](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=351s) all right. Uh, thanks, Young. So yeah, I'm gonna be talking about our, uh, Crusoe Managed Slurm product. And, uh, like, like we mentioned before, um, there's a lot of benefits, uh, that Slurm can bring to the, uh, training teams and, uh, researchers that are building the next bes- best models. But there are some issues with, you know, setting up Slurm, managing it, keeping it reliable. Uh, and on the other hand, you have Kubernetes, which

[6:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=381s) is the OS of the cloud. It's wide-widespread use and, uh, familiarity. It's a very mature ecosystem with lots of tools, extensions, and plugins for whatever you need to add, whether it be observability, networking, security. It helps, uh, maintain services with their, you know, built-in self-healing, uh, load balancing and auto-scaling that keeps everything running smoothly. And so we often see that, uh, customers will, uh, come with us starting with, say,

[6:51](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=411s) inference running on Kubernetes, and then they wanna expand to training, uh, or vice versa. And so maybe a Kubernetes-familiar team struggles with deploying and managing a Slurm cluster, um, or, uh, the incoming Slurm native training team is unfamiliar with the existing Kubernetes frameworks, uh, and adds, you know, additional friction that just slows down your team. And with, you know, how fast everyone's trying to get to, uh, the next best model and

[7:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=441s) next best product, um, teams get set up with the tools they're familiar with, and you end up with two separate infrastructure stacks, which has its own host of extra operational burdens, uh, and extra costs. So we decided to build out our Crusoe-managed Slurm, uh, on top of Kubernetes. So we have our Crusoe Slurm operator, uh, which coordinates with our Crusoe-managed Kubernetes platform and the broader Crusoe cloud, so including storage, networking, et cetera.

[7:51](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=471s) And, uh, CSO manages, uh, Slurm users, Slurm partitions, configuration, and storage, so you don't have to do it piecemeal. You just, you know, one, one command, you create your Slurm cluster, and there it is existing in your Kubernetes cluster. So the benefit is for the Slurm users, um, it's just another Slurm cluster. They just need, uh, an IP address. They can SSH with their, uh, user, and now to them, it's a Slurm cluster with all the bells and whistles that they need. They don't even need to know that Kubernetes is running

[8:21](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=501s) underneath. For your platform team and your, uh, infrastructure team, uh, the benefit is Slurm is now just another service that plugs into your existing infrastructure. You don't need to set up an entirely new infrastructure stack, new observability tools, new, you know, on-call teams, new runbooks, nothing like that. It's all built into the same thing. It runs alongside all your other services. And, um, resources in Slurm, so any jobs, make use of the, uh, built-in topology-aware scheduling,

[8:52](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=532s) um, and the platform team just sees it as another service. Um, and now since all your GPUs and hardware are in one cluster, they can move around seamlessly. Uh, it enables, uh, more interesting, uh, tooling and dynamic a-availability, like when you need to burst your, uh, inference service when you're reaching, uh, high, uh, users, high number of users. You can take away some of your-- or even reallocate some of your hardware from your training cluster to your inference service, or vice versa.

[9:22](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=562s) You know, when your user, uh, load goes down, you have all these GPUs sitting idle. That's the perfect time to up your training job or do a, a larger hero run across more GPUs. Um, this is something that's only possible when you have everything in a single, uh, infrastructure stack like, uh, Crusoe-managed Slurm. Um, an additional benefit is, uh, now Crusoe-managed Slurm can make use of our AutoClusters product, which is our auto-remediation for additional hardware errors.

[9:52](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=592s) So Connor is, uh, gonna walk through that and show a demo of, uh, of how that works.

[10:03](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=603s) So this is an overview of what exactly happens when AutoClusters detects a critical hardware error, in this case, an XID seventy-nine error, where one of the GPUs is completely unusable. And so the first thing that happens is that the user is notified, although there's no action required from them. Then the Slurm operator sets that, that specific node to down and cancels any running jobs. There's then a, um... The process rec-receives a sigterm signal before

[10:33](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=633s) being terminated, and you have up to two minutes to handle that sigterm to either sa-save your checkpoints or flush your logs before that node is eventually replaced. The cancel job is automatically re-queued, and then AutoClusters starts. So first, it verifies that no pods on the node have a label that disables automatic replacement. Assuming that node can be replaced, the Kubernetes node is then cordoned and drained, and then the

[11:03](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=663s) unhealthy node is removed from the node pool and replaced with a new healthy node from spare capacity. And then a record of the alert and the remediation is logged for the user's reference, uh, but there's still no action required from them. And then once that healthy node comes back online, then the Slurm job starts, your application code runs to load your model and lo-load your checkpoint, and resume training.

[11:30](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=690s) And so here's what that looks like, um,

[11:35](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=695s) in a demo. And so here on the left, I'm just using sbatch to launch a PyTorch training script. On the right is the Crusoe cloud console, where I have a GPU node pool of two A100 nodes. And I'm gonna launch this as normal, um, and then what I'm gonna do is run a script in-- to tell our back-end internal monitoring system that there's been an XID seventy-nine error on one of the GPUs, and this will trigger

[12:05](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=725s) AutoClusters. And so we can see immediately the GPU utilization drops, and so we know that that n-- that GPU has gone offline, and AutoClusters node replacement starts immediately. And the full process from detecting a critical hardware error to getting a healthy node back into the node pool takes roughly five minutes with AutoClusters. The rest of the time is mainly application-level code. And so in this example, it's a

[12:35](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=755s) little under fifteen minutes, um, from end to end. And here I'm just showing some of the alerts that, uh, the user receives when there's a critical hardware error like this. And so a-as you'll see, um, there's no action required from the user. The Slurm job will restart. It'll load the checkpoint and continue training where it left off. And we can see the GPU utilization returns back to normal, and the total downtime is less than fifteen minutes. And this is, um, much better

[13:05](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=785s) than having an engineer have to log on in the, in the middle of the night and spend hours debugging something. All of this is handled automatically on our platform.

[13:18](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=798s) Now, historically, creating this type of environment has been challenging to merge both Slurm and Kubernetes. And so we're really excited to announce what we call One-Click Slurm, where everything that we've talked about in this presentation, this entire environment can be provisioned with, with just a single command. And so this one command includes provisioning the underlying Kubernetes cluster, the Slurm controller, login nodes, and storage. And then adding a GPU node

[13:48](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=828s) pool is only one command beyond that. And so this means that engineers can simply SSH in and start running jobs immediately, and AutoClusters is, is enabled by default.

[14:02](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=842s) So wrapping up what we covered, um, starting with the limitations of traditional Slurm, it gives you the high-performance job scheduling, but not the automated, the automated healing that modern AI infrastructure requires. And so by building Slurm on top of Kubernetes, we can enable these advance, advanced features without having either team change, uh, their workflows. Then we covered the full node remediation flow from an error being detected to the Slurm job resuming and the

[14:31](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=871s) checkpoint-- and the training resuming. And so the key takeaways here, first, is that traditionally, Slurm and the infrastructure that it runs on are two separate systems. But together with Kubernetes, both systems stay in sync. And so a failed GPU node can get cordoned automatically, and that signal will propagate str- directly to the Slurm operator. Also, like I mentioned earlier, neither

[15:01](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=901s) the platform teams nor the machine learning teams have to change their workloads. Researchers can still use the same commands to launch Slurm, and although Kubernetes is running underneath, researchers never have to SSH into a pod or worry about containers. But the Kubernetes teams still get access to the same level of telemetry and observability that they're used to. And then lastly, like Nikhil mentioned, the GPU nodes are Kubernetes

[15:31](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=931s) nodes first, and that means that Slurm is just a workload running on top. And when a Slurm job finishes, those GPU nodes don't have to sit idle waiting for another job. You can schedule inference pods on those immediately through Kubernetes because from the platform's perspective, they're still just nodes in a pool.

[15:55](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=955s) And so the, the core idea here is that failures are inevitable, um, and so in today's AI landscape, the architecture of infrastructure should be designed in a way so that when there is a critical error, all the right actions are handled autonomously, and engineers can focus on building their application and not worrying about the infrastructure.

[16:20](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=980s) So hopefully, this was an informative talk. Um, if you have any questions, feel free to, um, connect with us at Booth UG-12 or on social at crusodev. And, uh, thank you for listening, and feel free to ask any questions. Thanks.
