What's New in Inference Engineering — Philip Kiely, Baseten

Um, I am here to talk about what is new in inference engineering. So, hi, I'm Philip, and I'm here because I wrote a book. Uh, this is my third year at the AI Engineer World's Fair. This is my favorite conference in the entire world. It's the highlight of the calendar every single year. I really got my start as a speaker and as an engineer here in twenty twenty-four. I came back in twenty twenty-five and did a bunch of stuff. I'm here again. I love it here, and I'm very thankful to the organizers for always having me. Um,

I wrote this book called Inference Engineering. We published it three or four months ago, and I've just been overwhelmed by the response. Uh, we've done more than, uh-- I, yeah, I published it in, on February twenty-third. Um, at this point, we've done more than eleven thousand paper copies. We're coming up on thirty thousand digital copies and twenty-four million people around the world, or twenty-four million Twitter accounts, so we'll see how many people that actually is, um, have seen something out of Inference Engineering.

And with all of this, you know, great reception, there has been one question that people have been asking me: Why in the world would you do this? Like, why would you write a book about something that's changing so fast? Well, you know, I, I believe that a lot of the principles of inference engineering at this point have been pretty solidified, and there's a lot that we can, you know, learn and, and kind of repeat over generation and generation of model. But today, I'm here to talk about what's new in inference

engineering. This is the first public addendum of all new information since the book came out. We are gonna review the inference engineering principles a little bit, and then we're gonna talk about all the stuff that's happened since February twenty-third of twenty twenty-six in the inference world. We're gonna talk about what happened to TurboQuant, talk a little bit about KV compaction. We're gonna spend a lot of time on DFlash and some other new exciting things in speculative decoding. And then I'm going to do a little bit of

prognosticating, a little bit of forecasting of what I think is gonna happen in inference coming up here and what I'm excited about, you know, hopefully being able to talk about next time you guys see me up here. Cool. So let's get started. So, you know, one thing I've been identifying now out of tons and tons of conversations with people about inference is a handful of, of shared principles. And one of the big ones-- I was on this, uh, podcast the other di- ga- day with, uh, Saro. Uh, we were talking

about inference, and, you know, there's two types of inference engineering that have really emerged. There's local inference, where the overwhelming strategy is just get it working on whatever hardware you have by squishing the model with quantization, distillation, pruning, however you can. You know, splitting it across whatever GPUs you happen to have in your house. And first you get it working, and then you make it less dumb. You take away whatever, you know, catastrophic issues all of this

compression of the model has created, and you try and get it back to that baseline intelligence running at a batch size of one. And then there's, there's my world, which is the batch size N data center world, where it's get it working, you know, just day zero, get the build of vLLM up, get it working, and then make it less slow. Do stuff like KV-aware routing, speculation disaggregation. And, you know, with- within these two worlds, I think that we have a lot to learn from each other. I am in this talk going to be focused on

advances in data center-oriented inference engineering, 'cause that's what I know. But there's a lot of really cool stuff happening in the local world as well. So in the book, in Inference Engineering, I generally assume that the weights are a finished product. And I do think that the handoff from training to inference is an important one to keep in mind, and it's a good way of kind of delimiting the space. However, what I've found more and more recently is that many optimizations for

inference come from a dedicated training process. And so the lines between training and inference are getting blurrier and blurrier. And that's an interesting thing to keep in mind. We're seeing this cycle where you get faster inference, which gives you more data, which you use to train a better model, which gives you faster inference, which gives you more data, and you just kinda, like, keep doing that until you're super rich. Uh, so with, with training for inference, uh, we have a bunch of new techniques to talk about

across what I like to call the big three. So we're gonna talk about some news in quantization, some news in caching, specifically the, the KV cache mechanism, and some news in speculation, because these are-- You know, there's a lot of other stuff in the world of inference, including some stuff I'm gonna talk about at the end. But when it comes to the practical day-to-day of how do I make X model faster, usually these are the three techniques that people are reaching for.

So first thing, I publish a book. It's February. I'm feeling awesome about myself. I'm like, "Wow, everything you need to know about inference in one place." And then, uh, we, uh, we had some news in the quantization world. So just as a quick review, quantization is when we use a smaller, less precise number format in order to save ourselves on bandwidth, save ourselves on compute, make TTFT better, make TPS better. Uh, it's usually kind of

hardware specific, gives you cost savings, but potentially degrades model quality a little bit. And by the way, if you want to hear my whole rant about quantization, I did a talk at AI Engineer Miami last month about how quantization is, is not necessarily as evil as it sounds, and that there is many things you can do to preserve quality through that process. So I was feeling good about my treatment of quantization, and then twenty million people saw TurboQuant. And in fact,

it, like, made the memory stock macro dip for, for a minute just because everyone was like, "Oh, memory is gonna be so much more efficient now. Like, we don't need any more f-flash memory," uh, which, which was wrong. But anyway, it was this new quantization approach, um, that was popularized in March of this year that uses polar coordinates for quantization and allows you to quantize the KV cache down to four bits. And it was, like, super hot, and I was like, "Oh, man, like, there's this whole thing that, that I left out, and,

like, what, what is this gonna look like?" And so our, our team did a bunch of research on this. Um, this is, if you know, uh, shout out @waterloo_intone on Twitter, um, Ali from our model performance team. Um, I'm not sure if he's still an intone actually. Uh, but yeah, he is from Waterloo. Um, anyway, so he, he wrote this great piece about the math behind TurboQuant. And basically, the benefit you get out of TurboQuant is that you get to represent the KV cache with four bits

instead of eight bits. You save half the room and half the band-- and, and you get effectively double the bandwidth when you're moving KV cache around in your system memory. But the drawback is pretty big. Um, for TurboQuant, it turns out that you need to do additional computation in the forward pass to account for this during decode, and it cuts TPS by more than half. And that's just an unacceptable trade-off for a lot of the production use cases. So we, we took a good hard look at TurboQuant, but, but are not using it for, you know, any, any of

these real workloads. We're, we're still on the traditional NVFP4 quantization. That said, it actually is a great technique for the local inference folks. So if you are running a model, especially a long context language model on your local computer, on GPUs in your basement, um, you have a very limited amount of memory. That's the number one bottleneck. And so anything that can free up memory from KV cache and allow you to put those longer sequences on there is gonna be very valuable, and

the f-additional forward pass computation is gonna be, like, less of a, a drawback. Um, so still TurboQuant is a fantastic research paper, a really great technique that just ended up not being as applicable in the data center inference world as it might have first appeared. Um, you know, instead, we're focused on, you know, NVFP4 with a focus on quantizing the weights versus the KV cache. Um, you know, doing our best to find rough edges in the quantized weights, um,

make sure that we're not flattening out our probability distributions. For the KV cache itself, focusing instead on KV-aware routing, KV offloading, KV sharing, using, uh, you know, NCCL and using NVIDIA Dynamo and other tools in order to move the KV cache around the system and potentially offload to CPU, ordinary memory, et cetera, um, versus trying to use, uh, Tur-TurboQuant to, to compress it. Um, and then we're also focused on quantization across modalities.

Um, so thinking about how can we apply the benefits of NVFP4 not only to language models, but also to, you know, image and video models. Ali also wrote a lot of great stuff on Twitter about that, which you should check out. So that said, the KV cache is still very important, and let's talk about it. Let's talk about KV compaction. Again, quick review, KV cache, if you put in the same prompt with the same prefix, uh, you get to reuse the tokens that you calculated prefill on last time. That makes your whole system faster and more efficient.

Broadly, KV cache is lossless memory. There's only a couple sources of lossless memory when we think about our inference system. We have the content of the prompt, the context, you have the, the KV cache, and that's gonna scale linearly with the amount of data you pass in. And now if you're thinking about, you know, million token sequence lengths, uh, that actually gets pretty substantial. So a lot of people are thinking about how do you compress memory? How do you compress context? Agent harnesses will compress context. RAG,

search, all these techniques that we've been talking about for years are, are sort of compression of a larger context into something that you can give to a model. You can write to files. All of these things scale sublinearly with the amount of data that you have. But what if there was a middle load? What if there was a way where you could get quite a bit of compression in the data that you were remembering with near lossless information retention? So we, uh, you know, we have a lot of different ways that we can think of what to keep in

the cache. You know, recent compaction methods have shown that we can replace the cache with a much shorter one. We've got papers like attention matching and cartridges that have given really promising, uh, outcomes here with, uh, high compression ratios. But both of these are run at inference time. Again, one of the techniques I wanna talk about or one of the themes I wanna talk about is training for inference. So in this case, I want to introduce something called Still by the Baseten research team, where the synthesis on top of

the cache, where we're keeping a learned representation of the information rather than a, the information directly or a sort of deterministic subset of it, is amortized via training. So Charlie and Mudith from our post-training team did a fantastic chalk talk at COSA Compile recently. Um, it's up on YouTube. I would encourage you to take a look at it if you're interested in learning about KV compaction. Um, I do not unfortunately have the time or the genius to explain everything

up here. Um, but the, the basic mechanism is that Still is a perceival bottleneck, um, that takes a, uh, fixed set of learned query vectors, cross-attends it against the full KV cache, and produces a set of compact keys and values in a single forward pass. This creates a fast, differentiable compressed memory that the LLM can attend to as if it was real context. So if you're interested in KV compaction, definitely check out Charlie and Mudith's work. Um, it's been a really fantastic

thing to learn about. So that's two of the techniques. We've talked about quantization. We've talked about caching. The final one is speculation, and there's been a lot of change here. Um, as a review, uh, speculative decoding, we're going to use draft tokens, we're going to verify them during the forward pass, and we're going to use that to generate more than one token per forward pass. It helps a lot with tokens per second, and it is a fully lossless optimization, which is great because we don't have to worry about quality at

all. Now, in, in the sort of history of speculation, we started with speculative decoding. All of these are in the book. Um, you have SpecDec, where you use a small model from the same family to generate draft tokens. Turns out small models are, like, not great draft token generators. They're great small models. So we invented as an industry a bunch of new methods like Medusa, where maybe just you add decoder heads to the model, and then eventually Eagle 3, which was, hey, what if instead of taking a tiny model

from the same family, we actually train a billion-parameter model on the hidden states of the target model to generate draft tokens? And that actually worked really well. And so, you know, as of, of maybe February last year-- of, of February of this year, Eagle 3 was the best method in speculation. Now we got DFlash. DFlash is even better. So it's diffusion for speculation. Uh, DFlash creates a sequence of draft tokens instead of a single token. So the model is a

diffusion language model, which means it creates a whole sequence of tokens in the same way that a video or image generation creates a sequence of frames or a sequence of pixels and iterates over it rather than doing a autoregressive token generation. DFlash models might be two or four times slower to run, but they're gonna predict eight or 16 tokens at once, um, in that, in that window, while Eagle is only doing one at a time. So a single DFlash forward pass is faster than the entire Eagle draft

phase and predicts more tokens. These tokens are able to cross-attend to each other and generally create a higher acceptance rate because in speculation, acceptance rate is everything. So in the wild, we're seeing a more than three x improvement, uh, from DFlash. Um, this is measured with a single B200 Qwen3-8B, um, and we can see it versus Eagle. It's a substantial improvement in the tokens, uh, both the token acceptance rate and the

tokens per second. These DFlash models are trained with a attention mask for bidirectional drafting, um, so the target model is going to provide the context, um, and within each block, we're going to have a subset of clean tokens that are sampled. Uh, and the attention mask is going to enforce causal consistency, uh, but it is still going to allow for bidirectional attention, um, where in, you know, a traditional autoregressive model, you're only looking at the tokens in a single

direction. So that's why we're able to, you know, take advantage of this diffusion-based architecture. And then, you know, I thought I was done, and then a couple days ago, uh, DSpark came out. Now, DFlash we do have up and running in production. Uh, DSpark is, is new research, um, so this one I can basically only say, like, "Hey, it exists. It's cool. We're looking at it." Um, the difference versus DFlash, it still has that diffusion model, but it also pairs it with a sequential model, and the idea is

that we're going to improve acceptance rates, um, by having these two models work together rather than having just the iterative speculator, uh, just the diffusion speculator or just the autoregressive speculator. So DSpark, very exciting, um, but we don't have any production results with it yet to show. Um, hopefully, you know, we'll have those for next time. Um, what we do have production results though on is continuous speculator retraining. So this is-- we're back to DFlash here,

um, and this is the idea that, you know, speculative decoding is very dependent on the actual content of the prompts and responses that you're looking at in your system. And so if you are continuously retraining on those prompts and responses in your live system, you can see a twenty percent to even two x improvement in your token acceptance rates. This is actually, like, really hard to do. Uh, it takes a lot of storage, and you have to make sure that you have permission to use the data that you're processing in this way.

Uh, it takes a ton of compute, and you have to move all of this information around, and if you change the underlying model, you also have to change the speculator model. But when I look forward into the future, I do think that continuous speculation for very, very large scale systems is going to be a worthwhile optimization. So what is next in inference? Um, the following is like personal opinion and speculation and public information, and like, if I knew anything that was actually coming out, I wouldn't

be able to talk about it. Um, so, so this is just, like, what I think is gonna happen. You know, I've been through three hardware cycles, um, through the Ampere release, the Hopper release, the Blackwell release, um, and it always takes time for, you know, when these chips get shipped to when they get installed in data centers, when the entire software stack really is able to take advantage of the capabilities. But some things that I'm excited about are, you know, with, with Luban, it looks like

the NVFP4 performance is going to be fantastic. So the more we can, like, honestly borrow techniques from local inference and get a lot of confidence running models in this NVFP4 data format, the more we're gonna be able to take advantage of the awesome performance of, of the upcoming Luban systems. I think that, like, disaggregation and system-wide communication is going to be increasingly important. Um, we're seeing really excellent early gains from PD disaggregation and the ability to, you know, move information like KV

cache data around the system is going to be increasingly important. And then, like I said, the theme of training for inference is going to be something that continues to have a big impact in the industry moving forward. So thank you all so much for the talk, uh, for coming to the talk. Um, I'm on Twitter. I'm on LinkedIn. Um, and I'm giving out free books. Um, you can download a PDF at the QR code or come down with me to the Baseten booth to get your free copy of Inference Engineering. Uh, we've got a bunch there, maybe

enough for everyone. If not, we will have a courier bring some more from the office. Um, so yeah, I'll be downstairs at the Baseten booth. Thank you all so much and have a great day.