AI Engineer World's Fair 2026
Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell — Akamai
Slide chapters
3 slide and demo groups · Timestamps are approximate. 1 additional group was omitted from this pass.
Read the talk
Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell
A small model can draft several tokens for a larger model to verify together. Sheilah Kirui’s vLLM demonstration shows why the payoff depends on prediction accuracy, spare GPU memory, and how much of a request is spent generating the answer.
From a talk by Sheilah Kirui
At a glance
Ideas worth remembering
Speculative decoding uses a small model to draft several tokens and a target model to verify them in one pass. Rejection requires target-model correction.
The structured-output demonstration generated 1.6× faster with high acceptance; the high-temperature creative task had low acceptance.
A useful draft must be cheap, predict tokens the target accepts, and fit alongside the target and both KV caches. Shared tokenization simplifies the setup.
Spare VRAM, smaller batches, and substantial output generation make better candidates. Long-context requests with short answers leave less time for faster decoding to save.
The expensive part: writing one token at a time
An application using a large language model can spend hundreds of forward passes writing a single answer. Each new token depends on the tokens before it, so ordinary generation proceeds one token at a time. In this talk, Sheilah Kirui, a developer advocate at Akamai, starts with that latency problem and asks when speculative decoding is worth enabling.
The distinction between two inference phases explains where the opportunity sits:
- Prefill: The model processes the input prompt and builds a KV cache, which Kirui describes as its working memory for the rest of the request. This initial prompt processing happens once.
- Decode: The model writes the answer, repeatedly using the preceding tokens to generate the next one.
A large model such as the Llama 70-billion-parameter model makes those repeated generation passes expensive. Speculative decoding targets this second phase.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Draft cheaply, verify with the target model
A smaller draft model does some of the next-token guesswork. Its generation is still autoregressive: it also predicts tokens one after another. The advantage is that each of its passes is cheaper and faster. Instead of asking the large target model to generate every token individually, the system asks it to verify a group of proposed tokens in one forward pass.
Kirui gives a typical draft length of three to five tokens per cycle. The target model approves or rejects those proposals and recomputes rejected tokens. This preserves the target model’s role in determining the answer. The talk describes this as keeping the output the same, but does not specify the sampling and rejection procedure; that statement should not be read as a promise of identical text across separate stochastic runs.
What happens to a proposal after verification? The useful relationship is a branch: accepted tokens can contribute to the answer, while rejected proposals require the target model to supply corrections. High acceptance lets one expensive verification pass carry more useful generation forward. Frequent rejection reduces the value of the draft work.
Where draft work survives verification
Read the diagram as text
- Small draft model. Generates a proposed group, typically three to five tokens.
- Target-model verification. Checks the proposals in one forward pass.
- Accepted tokens. Draft predictions approved by the target contribute to generation.
- Target correction. The target recomputes rejected tokens.
- Small draft model → Target-model verification: Proposed tokens.
- Target-model verification → Accepted tokens: Approved.
- Target-model verification → Target correction: Rejected.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The second model needs memory—and must earn its cost
“It’s not free.” Hosting a draft model adds its weights to GPU memory, and both models need space for their KV caches. A small model limits the added weight cost, but it does not remove the cache requirement. Spare GPU capacity is therefore part of the decision before any speedup measurement.
High concurrency makes this tradeoff harder. If the GPU is already busy serving many requests, extra drafting work competes with those requests. Kirui treats that as a poor starting point for speculation and later identifies smaller batches as a better candidate. The same optimization can make sense on a lightly loaded deployment and offer less value when serving capacity is already occupied.
Draft selection balances two questions: can it predict what the target would accept, and can it do so cheaply enough? A fast draft that repeatedly gets rejected spends work without advancing much of the answer. A more accurate draft must still cost only a fraction of the target’s generation work. Accuracy and speed have to be assessed together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fitting both comparisons on one Blackwell GPU
The demonstration uses one NVIDIA Blackwell GPU for both a baseline server and a speculative-decoding server. GPU utilization has to be divided between them, so the model choices are constrained by what fits. Kirui reports about 16 GB of baseline model weights and 2.5 GB for the draft model, leaving room for KV caching. The configuration view answers a practical question: how are the two serving setups fitted onto the shared GPU?
The selection guidelines extend beyond fitting the demo:
- Model size: Kirui gives ten to fifty times smaller than the target as a typical draft-model guideline. That is a general recommendation, separate from the demo’s reported weight-memory figures.
- Tokenizer compatibility: The models should share a tokenizer, ideally within the same model family. Otherwise, translating between tokenizers introduces extra work.
- Cost and accuracy: The draft must remain inexpensive while predicting tokens the target is likely to accept.
The workload supplies the other half of that match. Coding, JSON, and SQL are presented as promising structured tasks because their next tokens can be easier for a small model to predict. Poetry and brainstorming allow more varied continuations. The expectation is therefore task-dependent acceptance, which the demonstration tests by switching between structured output and creative writing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structured output: high acceptance and 1.6× faster generation
The application offers two task tabs and runs the baseline and speculative setups at the same time. Its stated purpose is to compare acceptance rates across task types. The first structured-output run stalls. Kirui considers reconnecting and restarting the vLLM servers, checks that the setup is running, then gets the application responding and repeats the run. No cause for the interruption is established.
After recovery, the speculative setup on the right generates 1.6× faster. Its high acceptance rate is the key explanatory metric:
The numerator counts draft work that survives verification. When that fraction is high, the target can validate more useful tokens per pass, making the cheap drafting work worthwhile.
The results also show more generated tokens per second with speculation enabled. The side-by-side display answers two connected questions: how much faster did this task generate, and how much draft work did the target accept? Together, generation speed and acceptance explain this successful case. The 1.6× figure belongs to the demonstrated structured-output run on the shared GPU; it does not establish the same gain for other workloads or isolated production servers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Creative writing: more variety, fewer accepted guesses
Switching to the creative task changes the important result: acceptance is low. The temperature is set high, and the task permits more variety in the next tokens. The creative-task result view answers whether the small model’s guesses still survive under those conditions. Kirui attributes the lower acceptance to the greater variety of possible continuations.
The verification mechanism still applies, but less draft work is useful. Rejected proposals require target-model correction, so a lower acceptance rate weakens the reason to pay for drafting. No numerical creative-task acceptance rate or speedup is stated. The supported comparison is that this case has much lower acceptance than the structured-output example, rather than a measured claim that creative writing always becomes slower.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Long prompts leave less generation time to save
Context length adds a separate limitation. Speculative decoding accelerates answer generation; it does not accelerate reading the prompt. Retrieval-augmented generation and document analysis can supply a large amount of input, requiring substantial prompt processing and KV-cache construction before the answer begins.
A long prompt followed by a short answer leaves little decode work to optimize. Even a faster generation phase affects only a small part of that request’s total time. This is why the ratio of input to output matters alongside acceptance: a draft can predict well and still offer a limited end-to-end gain when most of the request is spent processing context.
The closing checklist asks what the application actually does:
- Prediction: Is the output structured enough for the draft to achieve high acceptance, or is it a creative task with varied continuations?
- Memory: Is there enough VRAM for the additional model and both KV caches?
- Serving load: Are batches small and capacity available, or is the GPU already handling high concurrency?
- Time spent decoding: Does the request generate enough output for faster decoding to materially affect completion time?
These conditions identify promising experiments. The structured demo then shows what to inspect: acceptance alongside generation speed and tokens per second.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Starting with vLLM—and recognizing other approaches
In the Q&A, Kirui recommends vLLM, the serving engine used for this demonstration, as a place to start implementing speculative decoding. The two-model setup is one approach. She also names n-gram, Medusa, and EAGLE alternatives, without explaining their internal mechanisms or benchmarking them. The demonstration’s conclusions therefore apply to the setup shown, rather than a comparison across all speculative methods.
The practical invitation at the end is to replicate the experiment using Akamai Developers resources. Kirui’s focus there is inference optimization; other contributors cover AI agents, managed Kubernetes, and Akamai Functions. For this experiment, the useful question remains concrete: does a cheap draft produce enough accepted tokens to repay its memory and generation cost under the application’s real task, context length, and serving load?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
1 slide groups · 2 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 1 additional group was omitted from this pass.
- 0:12
My name is Sheilah. I'm a DA at Akamai, and we're here today to discuss, um, the topic speculative decoding and how you can, um, figure out whether it's worth enabling for your workloads.
- 0:27
Okay, I'm a little anxious.
- 0:36
Okay, um.
- 0:39
So typically, when you ask LLM a question, there are usually two phases that happen under the hood. The first is the model would read your prompt, process all the input tokens, and then build what is known as a KV cache, um, which is essentially the model's working, um, memory, um, for the rest of the request, and this happens only once. Um, and then you have the decoding phase, which is, uh, when the model starts to actually write the answer. Um, so because every token depends on the previous tokens, um, this process usually happens one at a time.
- 1:09
Um, and so imagine you're deploying an application that uses a big model like Llama seventy billion model. Um, typically, like this would require like even up to hundreds, like forward passes on the model, and this is a very expensive process, right? Um, and it can cause a lot of latency for your user applications. Which, um, brings a question of like, what if you didn't have to use your bigger model to handle the token generation? What if you used a smaller
- 1:39
model to, um, do some of that guesswork to speculate on a couple tokens, um, which is still an auto-regressive process, but the whole point is because it's smaller, there's-- it's a faster process, so it would be a cheaper process. Um, the output is still the same because, um, the target model, the main model that has the accuracy still has to verify the tokens, and it does this in one forward pass. Um, so that's the idea behind speculative decoding. Um,
- 2:09
I'll just leave this here for a second. So as I said, you have a small model, um, which generates token. Typically, you can set it between three to five for each cycle. The target model would handle the predictions, um, like be able to scale, like basically approve or reject it. Um, and then for all the-- for the tokens that are rejected, the target model would,
- 2:33
um, recompute the, the token for that. Okay. So, um, this process, it's great. It can save you a lot in terms of inference generation, but it's not free. Um, because you're hosting now two models, you have to account for the memory required to host that second model. Not that much since it's a smaller model, but you also have to allocate extra space for, um, the KV cache for both models. Um, yeah, so typically, you would consider running
- 3:03
speculative decoding when you have, um, extra space in GPU. Um, for example, if your workload is high concurrency, your pro-- your GPUs are probably already busy trying to run all the requests. So it wouldn't make sense to implement speculative decoding. But if you have extra GPU space, it's worth considering, but I will talk about how to evaluate whether it's worth. Um, and the first question is, um, choosing the draft model. Um,
- 3:34
yeah. So typically, when you're ch-- de-deciding whether it's worth enabling, um, can the draft model predict as accurate as the target model? And then the other question is whether, um, it's as fast. So you would typically choose a model that's definitely like a fraction of the cost. Like, um, yeah, so these are the two main, um, decisions that you have to balance, right? Um,
- 3:58
I'm just gonna leave this here. Um, so for my situation, um, I will explain why I chose these two models. For my demo setup, when I was running this, I have access to one GPU. I was running on a Blackwell GPU, and I wanted to run both the baseline and the speculative decoding setup in one. Um, so to be able to do this, I had to choose a model that was small enough to be able to fit because I have to split the GPU utilization between the both servers. Um, so this is what that look like. For instance, you can-- the
The Demo Environment
The demo splits one RTX 6000 Blackwell between baseline and speculative vLLM servers. A Streamlit dashboard controls both, and the pipeline leads to Prometheus and Grafana for monitoring.
Text in image
The Demo Environment HARDWARE ARCHITECTURE 1× NVIDIA RTX 6000 Blackwell Workstation layout loaded with 96 GB GDDR7 VRAM. SERVING PIPELINE Streamlit Dashboard (Control) ↓ Baseline vLLM (8B) | Speculative vLLM (8B + 1B) ↓ Prometheus Engine → Grafana UI VRAM MEMORY BUDGET RESOURCE | BASELINE | SPECULATIVE Target (8B) Weights | 16 GB | 16 GB Draft (1B) Weights | — | 2.5 GB Remaining KV Cache | ~21 GB | ~28 GB Memory footprints are carefully balanced inside individual docker containers on the Blackwell GPU. SECTION 6: DEPLOY SETUP 06 // 11 Interface text: SECTION 5: MATHEMATICAL BALANCE 05 // 11 vLLM Inference Optimization demo2 - Dashboards - Grafan[illegible] The Speculation Tax: Practic[illegible] Between the Core and the Bu[illegible] Ask Gemini File /Users/skirui/Downloads/between_the_core_and_the_bus.html LinkedIn Gmail Google Calendar Google Drive ChatGPT Google Gemini Discord Fidelity NetBenefits Login | Charles Sc... akamai developers... Transformers in Pr... All Bookmarks
- 4:28
baseline alone, um, takes about sixteen gigabytes of weight, and then the draft model is two point five, so that leaves a huge amount and space for KV caching. Um, and then when you're thinking about choosing the model, some of the things that you would usually consider is, uh, first of all, like I said, it has to be smaller, typically ten to fifty times smaller than your target model. Um, it has to have the same tokenizer,
- 4:59
um, ideally from the same model family. Otherwise, you know, you would have to like manually translate between the different tokenizers. Um, what's the other?
- 5:12
Yeah. Those are the main things. I think I'm forgetting one. Um, oh yeah, and like I said, the cost and accuracy. Uh.
- 5:24
Okay, so...
- 5:29
Oops.
- 5:36
So when I was running this, there's three questions. Um, speculative decoding, there's some workloads that would benefit from it versus others that wouldn't. Um, typically because the goal here is to get a smaller model that can predict almost as accurately as the target model would. It helps with some use cases when you have a workload that is more, like highly structured, like coding use cases. Um, what's another one? Like writing
- 6:06
JSON or SQL prompts, um, it would probably like benefit more versus if you were using it for a highly creative use case, like writing poetry or brainstorming or anything like that. Yeah, so.
- 6:21
I think I have...
- 6:26
This is a quick application. I'm not even sure if it's gonna work, but, um. So here I do have for the first demo, I just have, um, two tabs here. Um, like I said, this is just to focus on looking at how the acceptance rate performs depending on the type of task that you have. Um, so the first tab here is a structured output. Um, when I click this, it will run both at the same time.
- 6:53
Whoa.
- 6:57
Oh, God.
- 7:04
I'm just waiting.
- 7:24
Technical difficulties. I'm not sure why my demo is not running. Um, let's see.
- 7:34
Oh, do I have to reconnect? Um, just give me one second. I think I might have to restart the...
- 8:03
Fuck. Damn it. Looks like I need to restart the vLLM servers and my setup. Give me just a few minutes. Is this on? Um.
- 8:14
Um, LSOF minus I thousand. Okay, so it's running. Okay, that's running. Oh, okay. It's working now.
- 8:53
Oh, um.
- 9:00
Should I-- Yeah, I think I, I need... Oh, it's showing up. Okay. There we go. Um, so yeah, I was saying this is a demo. So when I run this... I'll redo it.
- 9:17
So you can see, um, as expected with the speculative on the right side, um, it was able to generate much, much faster, um, one point six times faster. Um, the key thing to note here is how high the acceptance rate is, which is the rate of the tokens that were accepted by the-- the number of tokens that were accepted over the total generated by the draft model. Um, another metric here, um, you can see with the
- 9:46
speculative enabled, um, you-- we are able to generate a lot more tokens per second, um, which improves the throughput. And then if I run it for the other case.
- 10:09
So here it's another use case.
- 10:14
Um, as expected, same thing. Um, but in this case, I guess the main metric to see here is the low acceptance rate. And like I said, this is because this is a more, um-- There's more variety in how the next tokens that would be generated because first of all, the temperature is set high. It's a creative use case. Um, yeah. Um, what else? I think that's pretty much for this, um,
- 10:43
case. And then the other situation where...
- 10:49
Oops.
- 10:52
Oh, yeah. This is what I was-- I should have put this up here, um, just to explain when it would benefit. And then the other use case would be context length. Um, so as I mentioned earlier, there's two parts of generating tokens. Um, with speculative decoding, it's supposed to accelerate the generating part, not the part where the model reads the prompt. So typically, if you have a use case where you need to provide the model with a lot of context, like for RAG applications or, um, analyzing lots of documents,
- 11:22
um, with those situations, um, the model will spend a lot more time with the portion of like building the KV caching. Um, especially if you're using it to do like a quick-- where the, the generated tokens are much less than the context, like the input. Um, yeah. So ba-- I think the main point here is just how to think through like what is your application doing. Um, is it highly structured? Do you have enough VRAM
- 11:53
space to even consider it? Are you doing more creative scenarios? Are you running a lot of concurrency, um, or not? Like smaller batch sizes would benefit from this. Um,
- 12:07
yeah.
- 12:11
Yeah, I'm gonna leave this slide here. This just basically summarizes, uh, what I just went through, and if anyone has questions, I'll take one or two questions now.
- 12:27
Ooh, I can't hear you. Oh, yeah. Um, the question was, are there particular tools that I recommend for speculative decoding? So I implemented this with vLLM, um, the serving engine. Um, they have really good documentation on how to get started with it. This is the simpler-- one of the simple versions of speculative
- 12:57
decoding. There's a couple others like the n-gram, Medusa, EAGLE that's more, like, the architecture beneath is, like, more advanced. Um, but for getting started, yeah, I would say you can't go wrong with just reading up. I-- When I was doing this, there's, there's not much content out there on this, but I did find a few, like, blogs. Uh, when you res-- when you type in speculative decoding, um, one of the first blogs that will come up by General Compute talks about more in detail. Um, there's also, like, two research
- 13:27
papers that were published, uh, by Google and what's the other company? Yeah. It-- But that's more, like, goes, like, more in-depth. Yeah. Um, but I'm also trying to maybe, like, make more content around this, like, maybe, like, videos. It, it was kinda hard to go through this. I w-- I'm a little all over the place, but I would definitely love to maybe, like, publish a blog or make snippet videos, um, yeah, on, like, YouTube or somewhere. But you can definitely find the resources.
- 14:03
Okay, yeah. All right, I should have covered that. Um, but I don't have the-- I should have put a QR code in.
- 14:12
Okay. Oh, um, as I said, I-I'm from Akamai, and if you're interested in, like, learning how to replicate this, you can find it on our website, our GitHub, GitHub, um, Akamai Developers.
Akamai Developers GitHub Overview
The speaker points to Akamai Developers’ GitHub for resources to replicate the demo, alongside work on inference optimization, AI agents, managed Kubernetes and Akamai Functions.
Text in image
akamai developers The Official Akamai Developers GitHub Account Follow README.md Akamai Developers Welcome to the place for builders who push the edge of what's possible. If you're exploring what comes next in AI, LLMs, cloud-native systems, serverless architectures, and WebAssembly, you're in the right community. At Akamai Developers, we share practical code, reference implementations, and experiments that help developers move from idea to production faster. What We’re Building Around • AI & LLMs: Intelligent applications, retrieval workflows, and real-world AI patterns. • Cloud: Scalable, resilient systems designed for modern global applications. • Serverless: Event-driven development with less operational overhead. • WebAssembly (Wasm): High-performance, portable runtimes at the edge and beyond. Why This Community Exists Technology moves fast. This organization is our home for open collaboration, where developers can learn, test ideas, and build the future together. Get Started • Explore our repositories: akamai-developers/repositories View as: Public You are viewing the README and pinned repositories as a public user. People Top languages Python Rust Go Astro HCL Most used topics spin webassembly akamai-functions akamai gpu Manage Interface text: github.com/akamai-developers akamai-developers Overview Repositories 63 Projects Packages Teams 1 People 15 Type / to search vLLM Inference Optimization demo2 - Dashboards - Grafa… The Speculation Tax: Practic… Between the Core and the Bu… akamai developers Ask Gemini LinkedIn Gmail Google Calendar Google Drive ChatGPT Google Gemini Discord Fidelity NetBenefits Login | Charles Sc… akamai developers… Transformers in Pr… All Bookmarks
14:28Demo milestone
- 14:29
Yeah. Um, yeah, so we have a, a lot of, like, different resources on here. Um, I'm focusing on infr-inference optimizations. Uh, we have other people that are focusing on building AI agents, um, and how to leverage our, like, managed Kubernetes service, how to use Akamai functions. So if you have time, please go check out our GitHub pages, and also we have a Discord channel that we're trying to build up now. Um, so it would be great to chat with some of you on the side. Thank you.
Akamai Developers Resources
The speaker points to Akamai Developers’ GitHub resources for inference optimization, AI agents, managed Kubernetes, and Akamai Functions. The page also shows RAG and GPU LLM quickstarts.
Text in image
At Akamai Developers, we share practical code, reference implementations, and experiments that help developers move from idea to production faster. What We’re Building Around • AI & LLMs: Intelligent applications, retrieval workflows, and real-world AI patterns. • Cloud: Scalable, resilient systems designed for modern global applications. • Serverless: Event-driven development with less operational overhead. • WebAssembly (Wasm): High-performance, portable runtimes at the edge and beyond. Why This Community Exists Technology moves fast. This organization is our home for open collaboration, where developers can learn, test ideas, and build the future together. Get Started • Explore our repositories: akamai-developers/repositories • Watch and star projects you care about • Open issues and share ideas • Contribute improvements through pull requests • Create an Akamai Cloud account here (we have a $300 credit just for you) Build With Us Whether you’re prototyping a new AI workflow, deploying globally distributed services, or experimenting with Wasm at the edge, we’d love to see what you create. Let’s push the edge of technology together. Top languages Python, Rust, Go, Astro, HCL Most used topics spin, webassembly, akamai-functions, akamai, gpu Pinned rag-langgraph-k8s-quickstart · Public Airline policies evidence-based Q&A assistant built with RAG with Python and LangGraph running on Linode Kubernetes Engine in Akamai Cloud HTML · 34 stars · 11 forks kuberay-gpu-llm-quickstart · Public A sample LLM deployed via KubeRay on Kubernetes running on GPUs on Akamai Cloud (coding assistant based on Qwen) 19 stars · 6 forks Interface text: github.com/akamai-developers vLLM Inference Optimization Akamai developers Ask Gemini LinkedIn, Gmail, Google Calendar, Google Drive, ChatGPT, Google Gemini, Discord, Fidelity NetBenefits, All Bookmarks Manage
14:34Demo milestone