AI Engineer World's Fair 2025

Hacking the Inference Pareto Frontier

Loading slides…

Read the talk

Hacking the Inference Pareto Frontier

Inference efficiency depends on matching the model, cache, scheduler, and GPU fleet to the application’s quality, latency, and cost requirements.

From a talk by Kyle Kranen

Before you start: Basic familiarity with autoregressive language models and GPU inference is helpful; prefill, decode, and KV caching are explained in the article.

What makes an LLM deployable?

Can your model complete the application’s task well enough, fast enough, and cheaply enough? Choosing a capable model answers only part of that question. The model and the system serving it must meet the application’s constraints together. Kyle Kranen approaches this problem from operating large inference systems: he describes previously leading NVIDIA’s largest inference deployment, with a quarterly cloud bill in the multiple tens of millions of dollars. His subsequent work on NVIDIA Dynamo targets data-center-scale inference, using techniques such as disaggregation to meet better service-level agreements or reduce costs at an existing SLA.

Three requirements determine whether a deployment is useful:

  • Quality: Can the model and surrounding application complete the target task with sufficient accuracy?
  • Latency: Does the response arrive quickly enough for the user—or for a safety constraint, as in robotics?
  • Cost: Is the expense per request low enough to support the application’s margins?

These are joint requirements. Excellent answers that arrive too late can fail the product just as surely as fast answers that are wrong.

Three green rows ask whether an LLM can complete a target task, complete it fast enough, and complete it cheaply enough.Open full source frame
Can you apply an LLM? Quality, latency, and cost.

A Pareto frontier describes the best achievable tradeoffs. The talk projects the problem onto two dimensions: tokens per second per GPU as an efficiency proxy, and tokens per second per user as a responsiveness proxy. Moving up and right improves both. Although the initial explanation briefly calls the GPU measure requests per second, the later graph explicitly uses tokens per second per GPU.

The deployment does not need every point on that frontier. It needs one operating point: the required quality and latency, delivered at the lowest feasible cost.

0:150:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Choose the operating point from the application

Consider three applications with very different tolerances:

ApplicationDominant requirementAccepted tradeoff
Personalized cancer curesQuality of the resultPotentially substantial time and expense
IDE tab completionImmediate responsivenessA tight latency budget
Asynchronous code commitsQuality and costMore time to finish the work

In Kranen’s hypothetical cancer-cure example, spending millions of dollars to prove a single successful cure could be justified by its value. Cursor-style tab completion has a different contract: the next line or tokens must appear quickly enough to feel immediate. An asynchronous coding agent working alongside the user, such as the Agent mode example in the talk, has more room to spend time producing a good result. Whether the user is waiting in the loop changes the optimization target.

Slide titled “Consider the Application” with pill, code window, and stacked message icons above the three application labels.Open full source frame
Three application examples: personal cancer cures, tab completion, and async code commits.
3:103:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:10 · section reference included

Combine techniques that move different constraints

The familiar optimization tools affect different parts of the problem:

  • Quantization can accelerate inference and reduce memory requirements, enabling larger batches and lower serving costs.
  • Retrieval-augmented generation adds retrieval work and context, generally increasing latency and cost in exchange for better answers.
  • Reasoning spends additional generated tokens on thinking, increasing work to improve quality.
  • Parallel configuration changes how the model runs across devices, altering speed and cost. Kranen also mentions possible quality effects for what he calls non-haloed context parallelism, without developing that variant’s mechanism.

These tools interact. Adding retrieval may produce the quality an application needs while making it too slow; quantizing the model afterward may recover some of that latency. The useful unit of optimization is the combined system. The remaining techniques exploit three properties of that system: scale, structure, and dynamism.

4:354:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:35 · section reference included

Separate prefill and decode, then balance them

Autoregressive generation repeatedly attends to the sequence produced so far. A KV cache retains each token’s key and value vectors so subsequent steps can reuse them instead of regenerating those vectors for the entire sequence.

This creates two distinct phases. Prefill processes the prompt and populates the cache. Decode produces output tokens and extends that cache. In an aggregated deployment, both phases run on the same set of GPUs. Disaggregation places them on separate workers and GPU sets so each phase can receive a different allocation of resources.

The phases have different bottlenecks:

PhaseResource pressureAllocation opportunity
PrefillPrimarily compute-boundSize workers to saturate compute
DecodeCan be memory-boundUse a different GPU count and batch size

Using DeepSeek-R1 as an example, Kranen describes compute saturating relatively early during prefill: fewer GPUs can handle smaller prefill batches, while more GPUs support much larger decode batches. Separating the phases permits this heterogeneous allocation.

It also removes a scheduling conflict. When one request needs prefill and another is already decoding on the same machine, the scheduler must arbitrate between admitting new work and advancing existing output. In-flight batching and chunk piggybacking help manage that contention, but retain the cost of scheduling both phases together. Separate workers simplify that decision.

Kranen reports that, for his Llama 70B comparison using the same total of 16 H100 GPUs, disaggregation achieves up to 2× tokens per second per GPU at a fixed latency. The graph plots GPU token throughput vertically and per-user token throughput horizontally. At that operating point, twice the throughput per GPU suggests roughly half the GPU serving cost per token; it does not establish a halving of total application expenditure. The talk does not specify the exact model variant, workload, concurrency, software version, or latency definition needed to reproduce that point.

The gain depends on where the workload sits. Short prompts require little prefill, leaving less contention to remove. At the high-throughput, high-latency extreme and the low-throughput, low-latency extreme, aggregated execution can converge with or outperform disaggregation. The middle of the curve is often more attractive for interactive applications; Kranen places many of these applications around 20–200 tokens per second per user.

Even there, the worker ratio matters. Too many decode workers leave GPUs waiting for prefill to supply work. Too many prefill workers feed decode faster than it can keep up, increasing load and queue depth. Each phase’s parallel configuration changes its capacity, so tuning the ratio also means exploring a broad—and expensive—configuration space.

6:206:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:20 · section reference included

Route for both cache reuse and available capacity

Disaggregation requires moving KV state between machines. Once state has a location, requests acquire an affinity for particular workers: a worker may already hold a reusable prefix from earlier requests. That cache can reside in GPU memory, host memory, or external storage.

Random routing ignores this affinity. Routing exclusively to the worker with the largest matching prefix has the opposite problem: popular prefixes can concentrate traffic on an already overloaded machine. Kranen corrects the labels and orientation of the routing illustrations as he walks through them; the underlying distinction is between random assignment, cache-only assignment, and a decision that considers both reuse and load.

A cache hit is valuable only if waiting for it does not erase the saved work. The routing decision should balance the prefix work already completed against the worker’s existing load.

A small Python example expresses that tradeoff using estimated queue delay and remaining prefill time. These teaching estimates put both terms in milliseconds; this is a scoring example, not Dynamo’s routing implementation.

python

from dataclasses import dataclass

@dataclass(frozen=True)
class Worker:
    name: str
    cached_prefix_tokens: int
    queue_ms: float
    prefill_ms_per_token: float


def estimated_start_ms(worker: Worker, prompt_tokens: int) -> float:
    reused = min(prompt_tokens, worker.cached_prefix_tokens)
    remaining = prompt_tokens - reused
    return worker.queue_ms + remaining * worker.prefill_ms_per_token


workers = [
    Worker("a", cached_prefix_tokens=800, queue_ms=100, prefill_ms_per_token=0.1),
    Worker("b", cached_prefix_tokens=500, queue_ms=10, prefill_ms_per_token=0.1),
]
prompt_tokens = 1000
selected = min(workers, key=lambda w: estimated_start_ms(w, prompt_tokens))
print(selected.name)

Worker a has the better prefix match, but its score is 120 ms. Worker b has more prefill left to do, yet scores 60 ms because its queue is shorter. The selection favors b; it does not execute or dispatch an inference request.

Scale can increase the opportunity for reuse. Kranen argues that a larger fleet can hold more previously computed KV state, increasing cache hits and reducing prefill work. That benefit depends on requests actually sharing reusable prefixes and the system retaining and finding them. Routing aims to improve responsiveness and cost efficiency while preserving quality: it reuses work for the same model computation.

11:0811:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:08 · section reference included

Use repeated reasoning as scheduling information

Agents introduce structure: moderately predictable patterns across successive and concurrent requests. Inference-time scaling is one example. Re-querying a model to reconsider an answer or reason further creates additional opportunities to improve the result.

The displayed comparison uses an 8B model in green, a 49B model in yellow, and a 235B model in red. Kranen reports that approximately three or four re-queries bring the 8B model near the 49B model’s quality and the 49B model near the 235B model’s quality. The specific model identities, task, quality metric, and prompting or answer-selection procedure are not established here, so the comparison illustrates a possible operating-point shift rather than a recipe for substituting arbitrary models.

Chart with green, yellow, and red model series, different marker shapes for inference time scaling rounds, dotted curves, and explanatory annotations.Open full source frame
Inference time scaling chart comparing accuracy and cost across three model sizes.

There are two ways to evaluate the extra calls. Holding the model fixed, repeated queries spend more time and money to improve quality. Holding the required quality fixed, repeated calls to a smaller model may still be cheaper—and potentially faster—than one call to a larger model. Kranen’s comparison makes the latter case: additional inference work can reduce the overall expense if it enables a sufficiently smaller model.

The repeated-query structure also gives the serving system information it can use. In a reasoning example from NATURAL PLAN, Kranen compares runtime across concurrency levels. This example has short input sequence lengths and long output sequence lengths, so disaggregation alone offers only a small benefit: there is relatively little prefill work to separate.

The next changes target the workflow itself:

  1. Move re-query control into the router. A follow-up request no longer has to return to the external client before coming back into the serving system.
  2. Make the router and LLM scheduler aware of repetition. They can schedule with knowledge that the current request belongs to a sequence of related calls.

The first change removes external round trips. The second produces the additional improvement Kranen identifies as the red-to-green curve transition. His broader conclusion is that, at fixed quality, model selection plus structured scheduling can lower latency and raise throughput. The recording does not provide enough implementation or curve detail to quantify those scheduling gains.

13:1113:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:11 · section reference included

Keep useful KV state through a tool-call pause

Workflow structure can also reveal when cached state will be needed again. Consider Kranen’s separate tool-call example: an LLM invokes a tool that takes about 30 seconds. During that pause, other requests consume GPU high-bandwidth memory, and the idle request’s KV cache may be evicted. The 30 seconds describe this example, not a universal cache timeout.

If the serving system knows the request will resume, it can preserve the expensive prefill work instead of letting eviction force recomputation.

The proposed lifecycle is:

  1. Prefill the prompt, then decode the output that initiates the tool call.
  2. Offload the reusable KV state to host memory while the tool runs.
  3. Restore the state to GPU memory near the expected completion time.
  4. Resume the LLM call with the tool result, reusing the prior context and processing the added context.

This requires knowing more than which cache entry was recently used. The workflow supplies a prediction about future use. Repeated reasoning and tool calls both expose that information. The intended benefit is faster, cheaper continuation with unchanged quality; Kranen explicitly corrects his initial suggestion that KV manipulation improves quality.

16:1316:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:13 · section reference included

Specialize the fleet and rebalance it as demand changes

Dynamism begins with accepting that one worker configuration may not suit every request. Input sequence length (ISL) and output sequence length (OSL) determine how much prefill and decode work a request creates. Their joint distribution can guide a mixed fleet:

Sequence-length regimeIllustrative configuration
Short input, long outputAggregated workers with higher tensor parallelism
Middle input-length rangeDisaggregated workers
Long contextDisaggregation with context parallelism

These regions are model-dependent examples, not fixed boundaries. Specialization aims to improve speed and cost without changing quality because it changes resource allocation rather than the model’s executed mathematics.

The distribution itself changes after deployment. Suppose the initial configuration fits the input/output histograms of App A and App B. A change in the user mix shifts demand toward longer sequences. In the illustrated shift, both input and output lengths grow, but input length grows more. That creates relatively more demand for prefill capacity, invalidating the previous prefill/decode balance.

Input sequence length versus output sequence length diagram with App A at upper left and App B farther right, highlighted by a red arrow pointing up and right.Open full source frame
App B shifts toward longer input and output sequences.

Prefill and decode capacity must adapt independently as the workload changes. Real-time autoscaling across the two worker types keeps one phase from starving or overwhelming the other. Dynamic load balancing is therefore essential to sustaining the speed and cost benefits of disaggregation, not merely an additional optimization after configuration is finished.

The implementation handoff is Dynamo, the open-source project introduced at the start. Kranen closes by inviting attendees to a San Francisco meetup the following Thursday, from 5–8 PM, for more detail on implementing these techniques. For a deployment, the ongoing work is equally concrete: keep matching worker specialization and phase capacity to the requests users actually send.

17:4418:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:44 · section reference included

Resources

From the talk

  • Kranen's coauthored introduction to Dynamo's planner, routing, cache management, data transfer, and launch benchmarks.

  • Original benchmark design and experiments on planning with supplied tool outputs.

Updates since the talk

Transcript download
Full transcript

Loading transcript…