[
  {
    "startMs": 13080,
    "text": "Hello, everyone. Thank you for joining me here. Um, yeah, today I'm gonna talk about Ray Actors, Vision Tokens, and the GIL. Um, engineering a multimodal SFT data pipeline that keeps the GPUs busy. So there's a lot of talk about, you know, how to optimize training paradigms. Um, but I wanna give a different perspective on the whole ecosystem of"
  },
  {
    "startMs": 43060,
    "text": "what it takes to get a model trained. Um, and so the question I'm gonna be asking here is, does your training loop really depend on the GPUs right now? Is it blocked on the GPUs? And I think we're gonna learn that-- We're gonna see in this experiment that the hardest part of optimizing multimodal train throughput is not necessarily kernel optimization. It is making sure that the D-- GPUs are constantly fed with data. So let's go over the overview of what this topic, what this talk will be"
  },
  {
    "startMs": 72980,
    "text": "about anyway. So we will establish a baseline multimodal training pipeline, um, and then we will encounter bottlenecks and then address them one at a time. So loosely, they are, one, concurrency. How fast can you process one image before feeding it to the model? Two, prefetching. How can we make sure that we have data on demand? Three, um, how can we minimize data copies across different processes? And four,"
  },
  {
    "startMs": 102900,
    "text": "fr-- uh, the additional framework optimizations, um, so just tweaking your bells and whistles. Um, yeah, so let's set up our premise. We all know that GPUs incur expensive costs and are generally want to maximize their utilization while we have them at hand. Multimodal processing is different than text-only, um, because there's a lot of CPU-intensive operations that have to happen before you actually"
  },
  {
    "startMs": 132680,
    "text": "feed the model, um, the data. So whether that is, um, you know, loading, resizing, or normalizing images using, like, a library like PIL. For audio, you'd do a lot of loading and cropping. You can use, like, NumPy arrays. And then videos are some sort of combination of both. Um, and then it's also true that usually your training clusters have a lot more CPUs than GPUs at hand, yet the GPU is what is starving in this pipeline."
  },
  {
    "startMs": 162880,
    "text": "So today we're gonna introduce this equation called, like, wait time ratio, which is generally, I formulate it as, like, the time it takes to prep your data and transfer it to the GPUs. Um, and then d-we divide all of that by the time it takes to do both of those and the train as well. So basically, the denominator is, like, your total pipeline. Um, and we're gonna see that during our run, we will go from a utilization of fifteen percent to utilization of ninety"
  },
  {
    "startMs": 192640,
    "text": "percent. So our baseline has, like, eighty-five percent of its-- spends eighty-five percent of its time, um, backed on data acquisition, whereas after we do these optimizations, less than ten. And so for the scope of this talk, GPUs are not the goal. It's making sure that the data is never the bottleneck is the goal. And so here's a little bit of background as to, like, what our stack will look like. Um, we kinda take, like, a"
  },
  {
    "startMs": 222640,
    "text": "Qwen3-VL base model, um, kind of try to train it on the paradigms of ChatVQA and, like, multi-turn VLM, uh, conversations. So to recap, before the GPUs do any math, um, we have to load the data, we have to transform the data, and then we have to, we have to transport it to the GPUs."
  },
  {
    "startMs": 248720,
    "text": "Okay, this is a, this is a overview of what our data actually looks like. So we will be using JSONL files, um, and then images are linked relatively, uh, to where the example lives. And then, uh, we're gonna assume that our data is just stored in S3. Um, in this example, for example, it seems like exa-- that image three three four seven one.jpeg seems to be a picture of, like, a blue bus. Um, yeah, so this is our system"
  },
  {
    "startMs": 278500,
    "text": "layout. On the left, we have a data generator, which is responsible for the data loading, data transforming, and tra-- data transporting. Um, and the-- that's, like, labeled as, like, fan-out to the Mega-Megatron data parallel ranks. Um, so the parallel ranks receive partitions of the data as decided by the coordinator. Um, it collates it, performs forward pass and backwards pass on this data. So Megatron takes care of optimizing the model weights across the DP ranks, and we're gonna assume"
  },
  {
    "startMs": 308500,
    "text": "that it's, like, a-- it's kinda-- it is kinda working as a black box. Um, so let's start with the naive baseline path. So for each example, we fetch the images from object store. Um, we kind of, like, we tokenize it, we resize it, rewrite-- resize the images, normalize them, and then feed them to the model. Um, yeah, and then when we do that, we can see that"
  },
  {
    "startMs": 338480,
    "text": "there's serial operations in the red, which is namely getting all the images one by one and then processing them one by one. So if your, if your trainer is asking for a batch size of, like, two fifty-six, then you have to serially process all these images. And then we can see that our train time is somewhere between 15 to 20 seconds. Uh, however, the data s- uh, the time spent to actually fetch the data is more than 100 seconds. So that's why we result in only 15%"
  },
  {
    "startMs": 367980,
    "text": "utilization of our GPUs. Um, and then you can even see a profiling flame graph that we spend most of the time doing S3 loads, and then you can see a long t- a little tail at the end about, like, loading the actual image on the PIL, uh, library. So okay, solution one, let's try to add some concurrency to this. Instead of those red singular p- uh, you know, serial processes, uh, let's try to parallelize it by kinda having, like, a worker pool. So when you request 256"
  },
  {
    "startMs": 397640,
    "text": "examples, maybe let's just process all of those at once. Um, yeah, so for example... for a sample, we have two costs. One is fetching the images, two is processing, so that's, like, decoding, vision pre-processing, and then tokenizing your text. Um, samples are very... are in-independent, so this just means it's, like, embarrassingly parallelizable. Um, and so yeah, you have a design decisions here. What forms of"
  },
  {
    "startMs": 427420,
    "text": "concurrency do you want? So there's asynchronous programming, uh, which is suitable for, like, I/O work, such as making SQL queries or, like, performing network requests. There's threads, uh, which you can use for, like, CPU-intensive, uh, workloads that relinquish the Python GIL, like, the Global Interpreter Lock. Um, so this... it's not straightforward what operations actually release a lock, so you have to do a little more of your investigation. But common uses are, like, using Hugging Face"
  },
  {
    "startMs": 457100,
    "text": "Transformers Tokenizers, using the PIL Transforms API, or, like, some Pandas operations. Processes, um, are more heavyweight, but also provide total isolation between your workloads. So you can leverage multiple cores on your machine as you finish your work. Um, yeah, doing a deep dive on the right solution would warrant its own talk, but, like, there's plenty of online resources that can help you decide for your use case. What's i-important for our solution is, is that we need to match our bottleneck,"
  },
  {
    "startMs": 487220,
    "text": "not simply going with a lever that's worked for us before. Um, so we'll choose asyncio for fetching images here, and then use Ray Actors as our processors, uh, as our process- multi-processing library. Um, Ray Actors are essentially the same as, like, general processes, but they offer us APIs to communicate across workers that just make it so that we don't have to reinvent the wheel. So yeah, your process is only as slow as your slowest link. Um, and we can see that our wait time ratio has gone"
  },
  {
    "startMs": 517000,
    "text": "down, um, on the second example, which is in the pink. Uh, you can see that the time spent f-fetching data per batch has gone basically, yeah, around, like, around, like, 40, 40 seconds. Um, so this is, this gives us a 25% speed up. However, you... yeah, that's, uh... as you can see, the... there's a huge gap that starts generating, especially as we're going through man-many steps. Um, so can we somehow limit our bottleneck"
  },
  {
    "startMs": 546980,
    "text": "of waiting on the trainer to request before provisioning our worker pool to fetch the data? So can we just, like, preemptively fetch our data? Um, so we came up with solution two, which was that we have a background thread that fills up a queue ahead of time, and it's always stays ahead of the consumer, which means that you... your producer is producing examples at a multiplier of your tra- of your consumer, and so the"
  },
  {
    "startMs": 576860,
    "text": "trainer is never bottlenecked on actually requesting the data. When it requests data, it's, it is available on demand. There's a caveat to prefetching. It makes your model and data loader checkpointing and resumption, like, a little intricate of a process, and it can also suffer from a cold start period in which your queue is actually getting filled up, uh, so it can be ready for training. Um, but a- if you work around those, those details, I think prefetching is a, is a great idea."
  },
  {
    "startMs": 607880,
    "text": "Um, yeah, so I think making sure your producer is always ahead of your consumer will be... will pay great dividends for you here. Um, unsurprisingly, we achieved close to zero seconds of data fetching and processing per batch because we've already fetched data and processed it way before the trainer even needs it. Um, and so we've effectively solved both the data load and the transform bottlenecks. So after these two mitigations, we thought we were done. Throughput up, GPUs"
  },
  {
    "startMs": 637900,
    "text": "fed. However, we observed that the data transport costs still linger. In fact, they've gone up. So every sample transfers a large amount of, like, image arrays across process boundaries multiple times. Every sample is pickled on the worker, copied to the coordinator, and then re-pickled onto the trainer. Per sample, per step, we have a memcpy that nobody really asked for. So what ha- how this manifests is that your data generator can often OOM,"
  },
  {
    "startMs": 668520,
    "text": "um, and so multimodal tensors are huge. They're megabytes per sample. Um, and so by default, every sample is returned by value, which means this, that these expensive arrays are copied over from process to process. So a fix for this, you can come up with a object store. Um, this is a nice convenience method of... that Ray provides you. Each sample carries a object reference, which is just a pointer to the data, um, and then the driver just holds the pointer. The"
  },
  {
    "startMs": 698480,
    "text": "tensor never round trips to the head node. Um, and the Megatron data parallel ranks, they can just directly call the Ray store to fetch the data as they need it."
  },
  {
    "startMs": 710840,
    "text": "So ev-with every choice of your parallelism and concurrency solution, you always have to think about the commu- added communication cost. So plan wisely around that. Okay. Yeah, and we were able to eliminate a extraneous, like, put object, uh, kinda call that our prefetcher solution had that we did not have to deal with by using the Ray store. And we can see that our, our communication costs go down, and the ratio of the time spent waiting also"
  },
  {
    "startMs": 740704,
    "text": "goes down. So to recap, we took a serial baseline, we added concurrency for data loading and processing, we added prefetching to stay ahead of training requests, and we added the Ray store to minimize data transfer. We w-went from a baseline wait ratio of 85% to 20%. And you can see that the sawtooth, uh, pattern that you see in the top graph is alleviated, uh, to a large extent, which means that our utilization of the GPUs has gone up."
  },
  {
    "startMs": 771664,
    "text": "So when we're scaling up, now that we have, like, a good multimodal training pipeline, let's try to scale it up to, like, a production run. While doing that, we observed our wait time ratio for the same pipeline went up when all we did was scale some numbers. So we just added four times the data parallel ranks. We added, like, 200 data streams, and then batch size is X by four. Um, so it turns out that since the data generator and the worker pool are co-located for"
  },
  {
    "startMs": 801484,
    "text": "fast data transfer,"
  },
  {
    "startMs": 805224,
    "text": "what happens is that there's a bottleneck on the network interface card of that, of that node. Uh, which means that while we're packing all of this working and processing into specific nodes, that their network throughput is not able to keep up with the demand of the training pipeline."
  },
  {
    "startMs": 825284,
    "text": "Um, so at this point, we did a little bit of digging into the frameworks that we are using, and we saw that by default, Ray schedules workers on the s-- it prefers to schedule workers on the same nodes and co-locate workers, which means that it's easy to bottleneck one machine before you move to another machine. So what we did was consider a possibility where we disaggregate this whole process where workers are spread across"
  },
  {
    "startMs": 855044,
    "text": "multiple nodes, um, so they can do their own image processing and storing, and in a lazy, just-in-time manner, are retrieved and used by our DP ranks. So yeah, to re-- And then so these are the two parameter, two f-flags that we kinda tuned. One is the zero-copy retrieval, and second is a spread scheduling mechanism. Um, neither of these actually helped in a small-scale experiment, but together they had, um, a"
  },
  {
    "startMs": 884524,
    "text": "synergistic effect in a large-scale experiment. Um, yeah, so s-config can lose... A config can do poorly at a small scale, but can do really good at a larger scale. So you should always think about how to ablate across your, your scaling ladders. And these were the two flags that ended up making a huge difference, uh, in how, how fast our data processing pipeline was. And so now this is the final pipeline that we've accomplished. A concurrent, eager,"
  },
  {
    "startMs": 914404,
    "text": "um, and a data pro-- uh, access pattern-aware system that shares the training across loads and enables that GPU stay warm while providing smoothness and stability in training. Overall, we achieved a 50% additional throughput at scale just from these two flags. It's the same knob, opposite verdict, set by the bottleneck. So these are the learnings we get from this talk, if nothing else. Um, GPUs are expensive, but sometimes the"
  },
  {
    "startMs": 944444,
    "text": "biggest battles happen outside of the GPU. So first ask yourself, are you CPU-bound or GPU-bound? Are you IO-bound or compute-bound? And then solving one bottleneck often exposes another. So first it was throughput for us, then it became wait time, then it became memory, then it became re-retrieval cost. Um, classic moves are not the end. You know, parallelism and prefetching matter, and I think a lot of content has, like, covered this. Uh, but so do the semantics of the"
  },
  {
    "startMs": 974264,
    "text": "frameworks that you're using and the, and the constraints that you're dealing with. And scale always changes the answer. Tuning parameters like zero-copy waste, um, and spreading workers become important only at larger scales, and they may not manifest as obvious improvements at smaller levels. S-overall, profiling your baseline helps you move the needle. Um, yeah, thank you everyone for attending the talk. Uh, if you have any questions, I'm happy to answer them."
  },
  {
    "startMs": 1004024,
    "text": "Thanks."
  }
]
