Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

Good afternoon, everybody. Um, I'm Sitanshu from CoreWeave. I'm gonna be talking about vertical, uh, mobility. It's, uh, quite a fancy topic, uh, the title that we came up with, but basically gonna be talking about the inference platform that we have at CoreWeave that we are building to serve, uh, small to big models and various different types of workloads. Uh, q- a quick intro about me. Um, I joined CoreWeave just about four months back, uh, leading all of inference over there.

And, um, before this, I was managing everything at AWS, uh, Annapurna Labs for, uh, training, and before that, inference and training at SambaNova. So quite a bit of experience in, uh, in this particular space. Um, what I will-- The way I'll be taking you through is explaining to you the consumption models that we have, and from that, how we have derived what the platform should look like, so that we do not need to keep changing the platform, and we keep making enhancements in the platform that we have for serving inference. And how and why the,

why performance plays such an important role over there. Uh, I think a little bit of this might be, uh, common with the previous topic that was discussed over here. Um, consumption models. So we have, uh, at large, two biggest consumption models. One is, uh, serverless, which is where the customers can come in, consumers can come in, uh, do not need to worry about managing the hardware themselves, do not need to worry about managing the clusters, orchestration, anything at all. There's API, there's UI. You come in,

you pay per token, uh, and you get your model served. Um, biggest thing over here is that, uh, the type of models, uh, that we serve in the catalog, that is the breadth of models that the customer will be able to go through. Uh, I'll talk about dedicated, and then I'll come back to serverless because there is one thing unique on the serverless side. Dedicated inference service that we provide is more for customers who want to know exactly what hardware they are gonna be, you know, using and running on. Uh,

but the model deployment also depends on them. So they use our service, they use our orchestration layers, the-- but the model deployment depends on them. Model performance also depends on them, as long as we provide in the platform the capability and the knobs to serve those features. Coming back to serverless, uh, one of the interesting pieces over here is, um, typically serverless models are very, uh, are noisy neighbor problems, where, uh, if let's say everyone is banging on the exact same model, then you might be

timing out quite a bit, depending on how much capacity I have behind it. So another feature that we have on the serverless side is what we are calling, uh, provisioned throughput. So as a customer, if you know your traffic profile and if you can let us know about that, we can carve it out specifically for you behind the scenes. You still do not need to worry about what hardware it is exactly running on, as long as your throughput, your SLAs are maintained. So that is another one on the serverless side, and that is still charged by, by the token.

But you know that you're not running into the noisy neighbor problem over there.

Um, let me take a quick stab at few different types of workloads, uh, workload shapes that, uh, we have, um, that we are seeing. And the ratio between these is, like, continuously changing, though agentic is, like, really high up there. Uh, agentic and chat, kind of very similar. Super high on the input sequence lengths, very low on the output sequence lengths typically. But the biggest difference between agentic and chat being the fact that the multi-turns in agentic are super low latency versus in chats,

because when you get the response as a user, you have to read the answer, and then you respond to it. So there are, there are differences over there. And that big difference ultimately converts into something, uh, related to, uh, the KV cache management. Uh, but these are-- these two are both real-time, and another real-time workload is, uh, your voice and videos, uh, which are steady streaming and super latency sensitive. On the agentic, a-agentic and chat side, um, largely the requirements are from

throughput point of view, not so much from latency. But real-time voice and videos are absolutely latency sensitive. Coming to batch, batch is where the SLAs are, like, super loose. They run into like way-- like, seconds and minutes. Sometimes for some customers, actually even in hours. They are like, "I'll just throw... Give me ten to twelve hours of, uh, workload capability, and, uh, I'll throw whatever I can. Process it whenever you can." Uh, these batch workloads, um,

the way they come into the picture over here in, um, deciding, um, uh, sorry, being the requirement for some of the design choices that we make in the stack. Imagine these, uh, four different types of workload shapes. Um, in the time dimension, you have to play the game of Tetris on how you can fit it in to utilize the underlying infrastructure the most.

I'll give a high level on how our stack is shaped right now, um, and I'll walk you through a bit of a request flow over here. So for both serverless and dedicated, if you look at the right-hand side of the screen, you'll see that on the platform side, we, uh-- you'll go to the control plane to, uh, to have your, uh, authorizations, your rate limitings, and your usage being tracked, et cetera, so that we, you can be billed accordingly. And, uh, observability, so that we can, uh, make sure that we

are not violating the SLAs that have been signed, right? Uh, underlying on the platform, I've shown at a super high level that we have these, uh, different, um, inference engines, vLLMs, SGLangs, and, uh, TensorRT-LLM, but there are qui-- there's quite a few, quite a bit of detail over here that, uh, I'll touch upon. Uh, and underlying that, what I'm trying to show over here in green is, um, various different pieces of hardware. So it's not that, um-- So the platform

needs to be capable enough of share-- of having the workload getting distributed across various different generations of these GPUs, uh, specifically NVIDIA GPUs that we use, right? So let's take up a-- Let's take a few examples, uh, over here. Um, let's say the request originates from the client side through apps or notebooks, any, any of those, or through the agents, right? It hits the gateway. Once it hits the gateway, then, uh, like I mentioned on the control plane, goes through authentication, et cetera,

et cetera, et cetera. But then, um, comes either the serverless or dedicated. So in the case of serverless, it'll be pay per tokens, so that the token usage would be monitored over here. Not the exact tokens, but just the token usage, because we maintain ZDR, zero data retention policies. Uh, it is multi-data-- Depending on the multitenancy or the provisioned, uh, if it is provisioned, then we know underlying for the router, it needs to go in and, uh, target the, uh, the explicit deployments for the provision

throughput, uh, customers. For the multi-tenant customers, there are separate deployments. Router over here, specifically, uh, the, the router is very important since, um, the router is responsible for making, uh, KV cache-aware routing choices. Why is it important? Because like I mentioned when, uh, we were discussing the workload, uh, profiles, uh, the agentic use cases are typically super heavy on the input sequence lengths, and bulk of the input sequence length, about eighty to

ninety percent, depending on which company it is, depending on the customers, eighty to ninety percent of it is the same for various different requests. So there is no point in going in and recomputing the prefill-- or redoing the prefill for that. Prefill is super compute-bound, very expensive. That's why as much as you can hit the cache, more you can save. Which is why if you look at the token pricing anywhere, uh, there's a specific price for input tokens, and there's a

way cheaper price for the cache input token. So caching becomes, like, really important over here. Underlying, uh, the, uh-- Underlying how you want to split the hardware is totally dependent on the choice in the platform, and we provide the capability to do either. Either do a prefill-decode disaggregation if the use case desires it, or do not do it, because prefill-de-decode disaggregation is not, uh, cheap for every type of use case. Uh, let's

take another request flow, uh, over here. Let's see if, uh, when it was a dedicated customer, then what will happen. Uh, dedicated customer, again, will go, go through the gateways that have been set up for them with proper isolations. Um, billing is not based on tokens. Billing is based on usage of per GPU per hour. Uh, it's a private gateway so that, uh, there is no noisy neighbor problem, no one else can get in. Uh, same router logic over here so that, uh, if there

are, uh, cache heavy-- if there are requests which are very similar, then it's, it hits the cache most. And depending on the deployment that the customer makes in their-- on their dedicated GPUs, uh, they can decide if they wanna do prefill-decode disaggregation or not. They can decide which, um, uh, which e-engine to use, vLLM or SGLang or TensorRT-LLM. And given the bulk of capacity that the customer has reserved, they can decide if they want to have just one deployment, uh, with the ability to scale, uh,

through the whole cluster, or they want to have multiple different models, multiple different deployments. Um, one thing that I do want to mention about the router over here, um, is the fact that heterogeneous capacity across different zones and regions is supported. It is a little-- It's quite a bit of a hard problem to load balance across that. So the priority order that we typically take is, uh, first KV cache locality, and then the least

loaded fallback. Um, that's that. Um, another request flow that I want to go over here, which, um, might be a little hard to see from the diagram, is I want to take the batch workflow. For the batch workflow, what we would actually do is the underlying capacity that the customer has, let's say the same dedicated inference customer, uh, during US daytime, they're running their real-time workloads, and from evening to night, they want to run batch workloads. The same

capacity, after time, can be scheduled to run the batch workloads. So we provide the capability in the API to tell when to scale up and when to scale down. And as per schedule, if we can sc-- if they tell us that we have to scale down, so we will scale down and open it up for batch processing through the night.

I think I've spoken quite a bit about, uh, optimizations on the KV cache side, but I do want to repeat a little bit because this is one of the most interesting pieces. Um, it uh-- If we can hit on the cache more, you can, uh, you will avoid the cost of prefill, which is the most expensive piece over here. Um, uh, reusing the KV cache across multiple different turns in your agentic workloads. Between turns also, there is a lot of, um, uh, similar prefill, uh, that comes

in, uh, in the input sequence length. Um, think about the chat workloads, which is where offloading KV cache also becomes extremely important because with cache-- with the, with the chat workloads, we have a lot of latency between different-- between multiple turns that we as users put in. But, uh, if we completely evict whatever we had in our particular conversation, then the next time we ask a question in that same chat, it's gonna take a little bit longer. Uh, so instead of actually completely evicting

and redoing the prefill again, what the, the techniques being used are, uh, maybe using, um, we are, we are using our own, but, uh, externally we know about LMCache and, uh, Mooncakes. Um, what we do is we will offload the KV cache to a high bandwidth storage so that we can store a lot of these prefills, uh, such that whenever the accompanying request comes for that particular conversation, it can be loaded in right away into the HBM.

Um, on the performance lever, I would-- I, I just want to mention a few performance levers. We've discussed the prefill-decode disagg, that is one. But quantization and speculative decoding are others, and how to carefully choose the parallelization, uh, degrees and the strategies, that is actually very important. Uh, two of the biggest levers that we have been working with are quantization to NVFF4 and SpecDec. Uh, we do provide capability where if the customer has their data set and they want us to

train speculators for their data sets, for better acceptance lengths, which will ultimately make the output throughput significantly higher, uh, we do have that as well, so. But that happens async. We, uh, we get the data async, we train the speculators async, and then we deploy the speculators into the customer deployments, uh, if that's what they wanted. Uh, you see three screenshots over here. I have posted them from, uh, the last one month, one month's worth of work that,

uh, some of us in my team have done. You can see we came quickly on top of the leaderboard on Kimi two point six, two point seven, and, um, those are, those are from Artificial Analysis. And going back to the session before this, can we trust that? That's why for GLM, I have the results from OpenRouter. So Artificial Analysis, when they run benchmarks, they are running very specific workloads. OpenRouter is actual user traffic. Uh, and you can see, uh, on the OpenRouter side, uh, Weights & Biases, so the

branding is different, but Weights & Biases is basically CoreWeave. We bought, uh, Weights & Biases, uh, about a year back. Uh, you can see the speed over here that we have from our deployment is pretty close to what Fireworks is providing as Fireworks Fast, right? But underlying techniques that we are using is what I want to emphasize the most over here, uh, for, for performance optimization. That becomes critical because ultimately what you want to serve to the customer, what we want to serve to the customer is, uh, price performance benefit.

Quick recap. Um, single platform is, uh, what I've been trying to emphasize, is what I've tried to show. Uh, two different consumption models, serverless and dedicated for customers. And within serverless, I described two different consumption models as well, pay-as-you-go and provisioned throughput, if you care about that. And ultimately compounding the gains, uh, through performance optimizations, uh, in the stack. That's all. Thank you, folks.