AI Engineer World's Fair 2024

Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

Loading slides…

Read the talk

Accelerating Mixture of Experts Training with Rail-Optimized InfiniBand

GPU networking affects how much paid compute becomes useful training work. Crusoe’s two-node example shows how NCCL PXN changes communication paths, followed by a Mixtral fine-tuning comparison.

From a talk by Ievgen Vakulenko

Before you start: Familiarity with GPU training and basic network concepts such as switches and bandwidth will help; rails and PXN are explained in the article.

What does it take to keep newer GPUs productive?

New GPUs and machine-learning models create an infrastructure problem: how do you make the systems underneath them run smoothly and fast? For Ievgen Vakulenko, a Crusoe product manager responsible for infrastructure and especially GPU networking, improving the network is part of making the compute customers buy useful.

0:290:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:29 · section reference included

Performance, usability, and energy supply

Crusoe frames its cloud around the relationship between computing and energy. Growing demand for power-hungry GPUs puts pressure on data centers, grids, and energy supplies. Its proposed response is to locate computing near stranded or otherwise wasted energy and renewable sources.

That energy strategy sits alongside two requirements for a usable AI cloud:

  • High performance: Infrastructure must be optimized for training. Delays, glitches, outages, and poor performance extend training runs and increase customer costs.
  • Ease of use: AI engineers should be able to provision compute, deploy models, train, and run inference without managing the underlying infrastructure complexity.
  • Climate alignment: Vakulenko describes a company aim to power all its data centers with renewable, wasted, or stranded energy and achieve no net new carbon emissions. This is an ambition, rather than a demonstrated emissions outcome.
Green Crusoe Cloud slide with three columns labeled High-performance, Easy to Use, and Climate-Aligned, each above an illustrative image.Open full source frame
Crusoe Cloud’s three pillars: high performance, ease of use, and climate alignment.

Energy availability also shapes the regional footprint. At the time of the talk, Vakulenko describes three continental-US locations—in Texas, the north-central US, and the eastern US—and an Iceland deployment under construction that would use geothermal energy. The Iceland expansion announcement identifies colocation at atNorth’s ICE02 and both geothermal and hydro power. Iceland also offers a connectivity and latency advantage for serving customers in Europe and the wider EMEA region.

1:151:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:15 · section reference included

Assembling a training cluster

The compute interface is a VM with attached GPUs. Multiple VMs can be connected into one training cluster, while CPU instances handle preprocessing, data preparation, offload, and other general-purpose work. Storage includes ephemeral and persistent disks backed by NVMe on the local server, plus persistent block storage. Managed network file systems were still in development in the platform described here.

The network has two distinct jobs, served by separate fabrics:

NetworkTrafficDesign priority
Front-end VPCInternet and customer-environment traffic to VMsGeneral connectivity and access controls
InfiniBand cluster fabricGPU-to-GPU communicationHigh bandwidth, low latency, optimized topology

The front-end network includes firewalls; load balancers were forthcoming. The separate rail-optimized InfiniBand network carries the communication between GPUs, keeping that specialized traffic on a fabric designed for it.

CLI, API, and GUI access expose these resources to AI developers and machine-learning engineers. The intention is to make provisioning and configuration automatable while leaving the infrastructure details with the provider.

5:255:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:25 · section reference included

Infrastructure beneath other AI products

Vakulenko names Together AI and Boson AI as customers and partners using Crusoe infrastructure. Together AI’s workloads include model training, fine-tuning, and sometimes inference.

Crusoe Cloud Customers slide with customer logos and a green Together AI callout describing its open source ML training, tuning, and inference platform.Open full source frame
Together AI’s highlighted use of Crusoe Cloud for training, tuning, and inference.

He also describes C.ai as using the compute infrastructure to train new foundation models. A developer using services such as Together AI or Codeium may therefore encounter Crusoe indirectly, through the infrastructure supporting those products.

8:358:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:35 · section reference included

Paid GPUs waiting for the network

Distributed training alternates between local computation and communication among GPUs. Forward and backward computation produce data that must be exchanged at synchronization points, using collective operations such as all-reduce. Vakulenko estimates that, without optimization, networking takes about 25–30% of training time. During the waiting periods he describes, GPUs cannot proceed with useful computation, but customers still pay for the infrastructure and wait longer for the model.

Computation–communication overlap starts exchanging data while computation is still running. It can hide some communication behind useful work, but it does not necessarily eliminate the remaining wait. In customer work, Vakulenko reports about a 10% reduction with overlap, while roughly 25% of training time still remained on the network. He does not specify whether that reduction is relative or measured in percentage points, so these figures do not define a precise before-and-after calculation. The remaining communication time motivates a closer look at the fabric itself.

9:4610:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:46 · section reference included

Where a conventional fabric creates friction

A traditional fat-tree can provide a scalable, potentially non-blocking fabric. But the way servers attach to it matters. Connecting a group of servers to one leaf switch creates both a possible bandwidth choke point and a shared fault domain: losing that leaf disconnects the GPUs attached to it.

That raises a second topology question beyond simply buying more network bandwidth: can another switching path carry the backend traffic? The next example turns to switching already available inside a GPU server.

11:5212:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:52 · section reference included

Use the local switch to reach the right rail

Consider two GPU nodes. GPUs within one server can communicate through NVLink and NVSwitch, without entering the external network. Across servers, GPUs connected through the same leaf occupy a common rail: their traffic needs only that leaf to reach the destination. Communication between different rails ordinarily goes up through the spine and back down through another leaf, adding switching stages, potential bandwidth contention, and latency that matters to collective operations.

NVIDIA’s NCCL PXN mechanism uses an intermediate GPU over NVLink to reach a suitable network interface. Introduced in NCCL 2.12, PXN lets communication use the host’s internal switching to reach another rail instead of requiring the external spine route. NVIDIA also describes message aggregation as part of the mechanism; shortening the external route is not its only optimization.

In Vakulenko’s example, GPU 0 wants to communicate with GPU 8 on another host. The optimized route proceeds as follows:

  1. GPU 0 transfers the data internally through the host’s NVLink/NVSwitch fabric to an intermediate GPU with access to the appropriate network interface.
  2. That interface sends the data onto the destination’s rail.
  3. The traffic crosses one external leaf switch to reach the other host.

The source and destination remain the same; the local GPU hop changes which external path the message takes.

PXN Optimized on Crusoe Cloud topology diagram showing two GPU nodes, leaf and spine switches, and black arrows traversing the local NVSwitch and one leaf to reach the other node.Open full source frame
PXN routes communication through the local NVSwitch and a leaf switch.
Communication caseRoute
Within one serverLocal NVLink/NVSwitch
Across nodes on one railOne external leaf
Across rails through the spineLeaf → spine → leaf
Across rails with the illustrated PXN routeLocal GPU hop → one external leaf

The optimized route retains one external leaf traversal even when the original GPUs are on different rails.

13:1213:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:12 · section reference included

Network results are only the first test

Vakulenko reports a 50% improvement for small messages and a 50% improvement for large messages in NCCL tests. He interprets small-message performance primarily in terms of latency and large-message performance primarily in terms of bandwidth, because latency stays roughly constant as messages grow. The talk does not provide message sizes, precise metric definitions, or a full test configuration for these figures.

Those synthetic results describe how the network handles a test workload. They do not directly answer the customer’s question: how much sooner will a particular model finish training? That requires a model-level comparison.

14:4815:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:48 · section reference included

Mixtral fine-tuning on 240 H100 GPUs

Sparse mixture-of-experts models make a useful application test because routing work among experts creates substantial GPU-to-GPU traffic when those experts are distributed across a cluster. Tokens must reach the GPUs holding the selected experts, and their outputs must return. The network is therefore involved in the model’s execution, beyond a single synchronization at the end of a training step.

Crusoe used Mixtral 8x7B for the fine-tuning comparison. The architecture has eight feed-forward experts per layer and selects two for each token; the name does not mean eight independent 7B models. The original paper describes approximately 47B total parameters and 13B active parameters during inference. Crusoe’s benchmark ran on 240 H100 GPUs.

The results slide gives the comparison in samples per second:

ConfigurationFine-tuning throughput
PXN disabled41.798 samples/sec
Crusoe Cloud rail optimized with PXN47.735 samples/sec

The slide reports a 14.2% throughput improvement for Mixtral 8x7B fine-tuning on 240 H100 GPUs with PXN versus PXN disabled.

Mixture of Experts Training Results slide for Mixtral 8x7B on 240 H100 GPUs, comparing 41.798 samples/sec with PXN disabled against 47.735 samples/sec with Crusoe Cloud Rail Optimized with PXN.Open full source frame
Mixtral fine-tuning results show a 14.2% throughput improvement with PXN.

Vakulenko rounds the result to 14% and connects the gain to training time and cost. Higher throughput means a fixed amount of training work can finish sooner if the rest of the run remains unchanged; a throughput percentage is not the same percentage reduction in elapsed time. The talk does not supply the dataset, precision, parallelism configuration, software versions, or repeated-run measurements needed to reproduce the result. Its substantive endpoint is nevertheless concrete: the topology change improved the measured model workload, bringing the network optimization back to the customer’s paid training run.

15:3515:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:35 · section reference included

Resources

Updates since the talk

Transcript download
Full transcript

Loading transcript…