AI Engineer World's Fair 2026
From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Loading slides…
Read the talk
From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Tarun Sunkaraneni walks through a multimodal training pipeline whose bottleneck moves from serial image loading to batch timing, tensor copies and finally a single node’s network card.
From a talk by Tarun Sunkaraneni
At a glance
Ideas worth remembering
A batch that needs more than 100 seconds of fetching and only 15–20 seconds of training is primarily a data-pipeline optimization problem.
Use concurrency that fits the work: asynchronous requests for I/O, threads where operations release the GIL, and processes for separate CPU execution.
Prefetching hides preparation behind training when the producer stays ahead, while adding queue startup and checkpoint-resumption complexity.
Passing object references lets training ranks retrieve large tensors directly, avoiding repeated copies through the coordinator.
Evaluate configuration at the intended scale: spread scheduling and zero-copy retrieval gave no small-scale benefit here, but together added 50% throughput at large scale.
Before the GPU does any math
An expensive GPU can spend most of its training run waiting for images. In Tarun Sunkaraneni’s experiment, the starting pipeline spends about 85% of its time acquiring data, leaving GPU utilization at roughly 15%. The first question is therefore practical: is the training loop actually blocked on GPU computation, or on the work needed to feed it?
Multimodal training adds substantial work before a forward pass. Images need loading, resizing and normalization; audio needs loading and cropping; video combines these kinds of preparation. A cluster may have many more CPUs than GPUs, yet still leave its GPUs hungry if those CPUs and the input pipeline cannot deliver prepared examples fast enough.
The talk uses a wait-time ratio to track that problem:
Wait-time ratio = (preparation time + transfer time) / (preparation time + transfer time + training time).
The numerator represents time spent preparing and transferring data; the denominator includes training as well. Once stages overlap, the useful distinction is between preparation happening in the background and preparation that still makes the trainer wait. Sunkaraneni introduces an overall improvement from 15% to 90% utilization, with data waiting below 10%; the intermediate results later in the talk describe specific stages of that progression.
The workload is vision-language fine-tuning with multi-turn conversations. Its input path has three jobs before GPU computation begins: load the data, transform it, and transport it. Keeping those jobs distinct makes it possible to see which change removes which delay.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A batch of 256 waits for one image after another
The dataset uses JSONL records with relative image links, with the data stored in S3. One example points to an image described as a blue bus. Follow that example through the pipeline: its image must arrive from object storage, become a decoded and normalized image array, and reach the training process alongside the text.
A data generator handles loading, transformation and transport. A coordinator partitions examples among Megatron data-parallel ranks—the training processes that receive their respective portions of the data, collate them, and perform forward and backward passes. Megatron’s weight optimization is treated as a working black box. The changes in this experiment happen upstream of it.
In the baseline, a request for 256 examples triggers serial image fetching and serial processing. The bus image waits its turn among the other examples before the whole batch can reach the trainer. Training takes 15–20 seconds, while fetching the data takes more than 100 seconds. The profile is dominated by S3 loads, followed by image loading through PIL. That imbalance explains why optimizing GPU kernels would leave the largest delay intact.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match concurrency to the work
Each sample has two broad costs: fetching its images, then decoding them, applying vision preprocessing and tokenizing its text. Samples are independent, so the bus image does not need another example’s result before it can be prepared. A worker pool can work on many examples concurrently instead of carrying out the entire batch one sample at a time.
The choice of concurrency depends on what consumes the time:
- Asynchronous I/O: Network requests spend time waiting for responses. Async programming lets other requests progress during those waits.
- Threads: CPU-heavy operations can benefit when their implementation releases Python’s Global Interpreter Lock, or GIL. Whether a particular tokenizer, image transform or dataframe operation releases it needs investigation.
- Processes: Separate processes can use multiple CPU cores and isolate workloads, at the cost of a heavier execution and communication setup.
The chosen combination is asyncio for fetching images and Ray actors for processing them. Async fetching overlaps remote waits; actors supply separate worker processes and APIs for communicating with them. This avoids building the worker communication machinery from scratch while giving image preparation multiple processes in which to run.
The reported fetch time falls to roughly 40 seconds per batch, and Sunkaraneni reports a 25% speedup. Those figures describe the experiment’s different observations; the fetch-time reduction alone should not be read as an equivalent end-to-end speedup. More importantly, the trainer still asks for a batch before the workers begin obtaining it. Concurrency shortens the wait, but leaves the request on the critical path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare the next batch while the current one trains
Prefetching changes when preparation starts. A background thread fills a queue ahead of the consumer, rather than waiting for the trainer’s next request. The producer must keep ahead of the trainer’s consumption rate. For the bus example, fetching and transformation now happen before its batch is requested; when the trainer needs it, the prepared example is already available.
What delay does the queue remove? The flow below separates the producer’s early work from the trainer’s later request. The queue creates a place for completed preparation to wait, allowing data work and training to overlap. In the reported run, fetching and processing contribute close to zero seconds of waiting per batch because they have already happened—not because image preparation has become free.
That timing change introduces two costs. The queue needs a cold-start period to fill, and checkpointing becomes more intricate because model state and the data loader’s progress must support a consistent resumption. Prefetching pays off after the producer gets ahead; it does not eliminate startup or the need to manage that extra progress.
Move preparation ahead of the request
Read the diagram as text
- Background producer. Fetch and process examples ahead of demand.
- Prefetch queue. Hold prepared examples until the trainer requests them.
- Trainer. Consume available data without waiting for its preparation.
- Background producer → Prefetch queue: Fill ahead of consumption.
- Prefetch queue → Trainer: Supply already-prepared examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stop sending the image array through the coordinator
Once fetching and transformation stop making the trainer wait, transport becomes conspicuous. Each prepared image array is megabytes per sample. Returning samples by value means pickling them on a worker, copying them to the coordinator, then serializing them again for the trainer. The bus image’s contents now take an expensive detour through a process that only needs to decide where the example belongs. Repeated copies can also push the data generator out of memory.
Ray’s object store changes what travels through the coordinator. Each sample carries an object reference; the driver holds that reference rather than the large tensor itself. Megatron’s data-parallel ranks retrieve the data from the store as needed, so the tensor no longer round-trips through the head node. This removes an unnecessary intermediate transfer while preserving the coordinator’s partitioning role.
Where do the large bytes go, and where does only a reference go? The diagram makes that distinction visible. The worker stores the prepared array; the coordinator handles its reference; the training rank retrieves the array directly. It avoids the coordinator detour, although retrieval still has a cost—a distinction that matters when the pipeline scales.
Using the store also removes an extraneous object-put operation introduced by the prefetcher. Communication costs fall, and the intermediate pipeline reaches a wait-time ratio of 20%, down from 85%. The reported utilization graph’s sawtooth pattern becomes much less pronounced: repeated periods of starvation have been reduced. Concurrency, early preparation and fewer copies each address a different part of that change.
Coordinate references, retrieve tensors directly
Read the diagram as text
- Processing worker. Prepare the image array.
- Ray object store. Keep the prepared tensor available for retrieval.
- Coordinator. Hold object references and assign data partitions.
- Megatron data-parallel rank. Retrieve the tensor when needed for training.
- Processing worker → Ray object store: Prepared tensor.
- Processing worker → Coordinator: Object reference.
- Coordinator → Megatron data-parallel rank: Assigned sample reference.
- Ray object store → Megatron data-parallel rank: Direct data retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
At production scale, locality overloads one network card
The same pipeline starts waiting more when scaled to a production run. The workload now has four times the data-parallel ranks, about 200 data streams, and four times the batch size. The generator and worker pool are colocated to make transfers fast, but concentrated processing also concentrates traffic. A node’s network interface card cannot keep up with the training pipeline’s demand.
Investigating framework behavior reveals that Ray’s scheduling in this setup prefers colocating workers. That preference can exhaust one machine before making useful use of another. Spreading workers across nodes distributes image processing and storage; the data-parallel ranks then retrieve those prepared objects lazily, when needed. The tradeoff has changed: keeping work close together is less valuable once the shared network card becomes the limiting resource.
Two settings make the difference at this scale:
- Spread scheduling: Place workers across multiple nodes so image processing and storage do not funnel through one overloaded machine.
- Zero-copy retrieval: Reduce copying when retrieving prepared data, complementing the earlier change from returning arrays by value to passing object references.
Neither setting helps in the small-scale experiment, but together they produce an additional 50% throughput at large scale. Their value depends on which resource is saturated, so testing only the small workload would have led to the wrong configuration decision for production. Sunkaraneni’s advice is to ablate settings across the scaling ladder: compare their effects at multiple workload sizes, including combinations whose benefits may appear together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Each fix changes what needs profiling next
The completed pipeline prepares independent examples concurrently, gets ahead of requests, passes references instead of repeatedly copying tensors, and distributes workers when concentrated retrieval traffic becomes a problem. Its improvement comes from changing the path to the model while leaving Megatron’s training work as the black box established at the start.
The ending’s practical lesson is to keep asking what the run is waiting on: CPU work, I/O, GPU computation, memory or retrieval. Solving one bottleneck exposes another. Parallelism and prefetching are useful starting moves, but framework semantics—how results are returned, where workers are placed, how objects are retrieved—determine whether those moves keep paying off. Profile the baseline, then profile the changed system at the scale that matters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…