AI Engineer World's Fair 2025
How fast are LLM inference engines anyway?
Loading slides…
Read the talk
How fast are LLM inference engines anyway?
Open models make self-hosting plausible, but useful speed depends on workload, precision and latency targets. Charles Frye walks through how to measure those tradeoffs.
From a talk by Charles Frye
Before you start: Familiarity with language-model tokens, GPU serving and the difference between latency and throughput will help you follow the benchmark comparisons.
When does running your own model make sense?
How fast can you run an open language model—and is it capable enough to run your application? For years, building useful AI applications largely meant calling proprietary APIs. Access to weights offered the freedom to modify and experiment, but that freedom mattered less when the available models could not do the job. The Llama, Qwen and DeepSeek families changed the practical question: many applications that previously required proprietary models became feasible with open weights.
Serving those models efficiently takes more than wrapping a transformer in torch.nn.Module. KV caching is only the beginning; paged attention, multi-token prediction and speculative decoding make the implementation increasingly specialized. Engines such as vLLM, SGLang and TensorRT-LLM provide that serving machinery without requiring every application team to build it.
Together, better models and better engines broaden the case for self-hosting. Air gaps, government requirements and a strong commitment to open source were already reasons to run your own models. Frye's opening claim is that self-hosting now makes practical sense for a wider set of applications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Applications need sufficient capability
In a 2023 presentation, Frye proposed a conditional argument: if an application's capability requirements saturate, open models can catch up and eventually dominate that use case. Operating systems, databases and programming languages provide the analogy. Once a collaborative project clears the capability threshold users actually need, its other properties can become decisive. The application need not use the most capable system available; it needs one capable enough for its task. Frye treats the subsequent progress of open models as support for that prediction.
The serving ecosystem also changed. Looking back at his earlier list of inference libraries, Frye characterizes TGI as having fallen away while vLLM remained relevant. That is his retrospective assessment of the landscape, not a claim that every library on the old list had formally shut down.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn repeated performance questions into a database
The operational questions came from people building code completion in editors, enriching databases with large backfill jobs, or serving models they had trained themselves. Cost or model customization could make a hosted model provider unattractive, leading them toward generic infrastructure such as Modal. But choosing infrastructure immediately raised a more specific question: how fast would an eight-billion-parameter Llama model run on SGLang with 128 input tokens and 1,024 output tokens? A model name alone could not answer it.
Producing a trustworthy number first required getting packages working together and installing suitable engine versions. Frye reports that answering a configuration question went from a couple of days to one or two hours, then roughly 15–20 minutes with benchmarking software. Once the same questions kept recurring, the next optimization was to compute the answers before anyone asked.
The resulting benchmark covered roughly ten models, three engines—vLLM, SGLang and TensorRT-LLM—and about ten context lengths. Publishing those results turned repeated investigations into a lookup problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the results as configurations, not engine rankings
The LLM Engineer's Almanac Engine Advisor makes the results browsable. Alongside the interface are the benchmarking methodology, open-source benchmark code in Stopwatch, and LLM Engines: An Executive Summary. Frye describes speculative decoding, multi-token prediction and quantization as topics for future additions to the Almanac. The live walkthrough begins with an audience request for Qwen3.
Selecting any engine does not mean every engine has a result. For the tested eight-bit Qwen3 configuration, Frye reports problems with SGLang and believes only vLLM results are available. This is a gap in that experiment's configuration coverage, not a permanent statement about SGLang's Qwen3 support. The full search space spans many combinations, and some are missing because they did not work or because a working setup was not yet clear. Contributions can fill those gaps.
These are initially out-of-the-box results, not fully tuned performance ceilings. Optimizing every configuration would be a substantial separate project. Frye particularly invites optimized TensorRT-LLM implementations because it exposes many tuning controls. A result therefore describes a model, engine and configuration together; it does not isolate an engine's ultimate capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a latency target before reading throughput
The first filter is time to first token, or TTFT: how long the user waits before output begins. Frye starts with a one-second service-level objective, then tightens it to 300 milliseconds for a more interactive experience. His reference to the Doherty threshold is informal; the useful engineering step is choosing an explicit latency target for the application.
With the first-token filter set to 300 milliseconds, Frye reports about one request per second for the selected Qwen3 mixture-of-experts model on vLLM, using 128 input tokens and 1,024 output tokens. That throughput belongs to the selected workload and filter, rather than being a general speed rating for Qwen3.
The interface also supplies a launch snippet. In the demonstration, the command uses uvx modal run and requires a configured Modal token. After saving the supplied snippet as serve.py, the command has this form:
bash
uvx modal run serve.py
Use the generated configuration that accompanies the result; the command alone does not specify the model or server. Frye estimates roughly five minutes to load the weights and start the server. That startup time is separate from the first-token latency of requests to the running server.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model size does not determine serving speed
The next audience selection is Gemma 3 27B. Frye first relaxes the first-token filter, which had been too restrictive for the displayed results. This configuration uses BF16, the easiest numerical format to get working in the initial experiments; he recalls subsequently getting eight-bit variants working too. The comparison has therefore changed both model and precision, as well as the latency filter.
Frye describes Gemma as roughly ten times smaller in total weights than the selected Qwen3 model, with roughly similar active parameter counts and approximately comparable throughput at 128 input and 1,024 output tokens. A mixture-of-experts model's total parameter count and its active parameter count describe different serving burdens. Here, the differing precision and filters also prevent treating the result as a controlled test of model size.
The displayed Gemma results show a pronounced gap between vLLM and SGLang. Frye interprets this, along with the Qwen and Llama results, as evidence that optimization effort varies across model and engine combinations. That is a plausible explanation for the observed differences, not a measured attribution of which team's work caused them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reverse the workload and watch throughput change
Keeping Gemma selected, Frye starts with a short prompt and a long response. At 128 input tokens and 1,024 output tokens, the displayed Gemma configuration delivers about one request per second with roughly 400 milliseconds to first token. He then reverses the lengths: 1,024 input tokens and 128 output tokens. The total token count stays the same, but most of the work moves from generation into prompt processing.
He loosely calls the first shape reasoning-like and the second RAG-like. These are workload analogies, not tests of reasoning quality or retrieval quality. The distinction that matters to the engine is decode versus prefill:
- Decode: Output tokens are generated autoregressively. Each new step depends on earlier generated tokens.
- Prefill: Input tokens are already known, allowing the engine to process their positions in parallel.
Moving tokens from output to input therefore changes the available parallelism even when the combined length is unchanged.
The displayed Gemma comparison is:
| Workload | Input tokens | Output tokens | Approximate throughput |
|---|---|---|---|
| Decode-heavy | 128 | 1,024 | 1 request/s |
| Prefill-heavy | 1,024 | 128 | 4 requests/s |
For this Gemma workload swap, Frye reports roughly four times the throughput with vLLM. Prefill can use large matrix–matrix multiplications, loading weights and reusing them across more arithmetic. This favors the GPU's compute capacity relative to the cost of moving memory or communicating. The gain belongs to this particular workload comparison; it is not a universal multiplier for inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Lower precision could widen the gap
The Gemma example uses BF16. Frye briefly questions its Tensor Core support, then immediately corrects himself: BF16 does use Tensor Cores. Lower-precision formats can offer greater arithmetic throughput on hardware that supports them, but support depends on the GPU generation—FP8 on Hopper and FP4 on Blackwell are distinct cases.
Frye invokes quadratic scaling with bit width to explain why shorter numbers can make multiplication cheaper. That arithmetic intuition should not be read as a quadratic law for hardware throughput or end-to-end serving speed. He predicts that an FP4 experiment on Blackwell could show an even larger separation between these workload shapes, but no such benchmark is demonstrated here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Try useful context before adding generated reasoning
An application cannot always choose its input/output balance: users bring their own queries, and some tasks inherently require long answers. But when the product goal is better quality under a latency constraint, the benchmark suggests an option worth testing. Consider adding useful context before immediately asking the model to generate more reasoning. Parallel prompt processing may make that route easier to fit within the latency budget. The benchmark does not establish that context always substitutes for reasoning or improves quality.
Frye also observes nearly unchanged time to first token after increasing the input from 128 to 1,024 tokens. Those lengths represent an eightfold increase in input, despite his informal description of it as tenfold. The observation is useful because extra context did not impose a proportionate first-token penalty in this example; it is not a guarantee that arbitrary amounts of context are free.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Take the measurements back to your own workload
The interface is a starting point for further investigation. Frye offers access to downloadable raw results and points to the open-source code for running benchmarks independently. The methodology explains how to interpret the measurements, while the executive summary provides material for discussing open-model serving with leadership. These serve different needs: inspect the data, reproduce the experiment, or explain why the infrastructure decision matters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the throughput ceiling without confusing it with latency
The closing question asks how far request throughput can rise beyond the displayed result before latency bends upward. Frye first clarifies the unit of deployment: the displayed Gemma example measures one replica on one H100. That identifies the scope of this example, not the earlier Qwen3 configuration. Higher aggregate throughput eventually requires more replicas. His hypothetical 400-QPS target illustrates a reason to scale out, not a demonstrated fleet result.
To establish an upper throughput bound, the benchmark submits roughly 1,000 requests together, waits for all of them to finish, and divides the completed request count by elapsed seconds:
This exposes substantial parallelism to the engine. It estimates saturation throughput on the assumption that the engine schedules the available work effectively; it does not establish that requests at that load have acceptable latency.
The remaining procedure separates capacity from responsiveness:
- Use the burst result as a capacity estimate. Sustained arrivals above the server's service rate cause queues to grow, so latency rises rather than throughput increasing indefinitely.
- Measure the low-load endpoint. Send one request, wait for its response, then send the next. This estimates the minimum-latency behavior of the running server, not its maximum throughput.
- Sweep request rates between the endpoints. Measure the latency achieved at each load to obtain the operating points shown in the Advisor.
The useful deployment target is therefore the request rate a replica can sustain while meeting the application's latency objective. Once that per-replica operating point is understood, scaling out addresses the larger traffic requirement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore inference configurations by model, workload and latency objective.
Explains throughput bounds, latency measurements and request-rate sweeps for individual serving replicas.
Open-source tools for provisioning and benchmarking vLLM, SGLang and TensorRT-LLM on Modal.
Discusses self-hosting decisions, engine selection and the limitations of default benchmark configurations.
Further reading
Technical background on H100 Tensor Cores and supported numerical formats.
- Introducing NVFP4Article
Explains Blackwell's NVFP4 format, scaling factors and accuracy considerations.
Updates since the talk
- TGI maintenance status and migration guidanceDocumentation
Official documentation describing TGI's maintenance mode and migration options.
Full transcript
Loading transcript…