Contents
  1. From request to generated tokens
    1. The serving contract
      1. From acceptance to release
      2. Three operational quantities
    2. Prefill and decode
      1. The first-token boundary
  2. State and progress
    1. The KV cache
      1. A cache-payload estimate
      2. Keep the memory ledger separated
    2. Latency has boundaries
      1. Progress and completion metrics
  3. Sharing a server
    1. Continuous batching
      1. Three batching policies
    2. Scheduling and admission
      1. Allocating delay
      2. Policy dimensions
  4. Selecting and delivering output
    1. Sampling, stopping, and streaming
      1. Selection is not system determinism
      2. Ways generation ends
    2. Speculative decoding
      1. Accept, reject, correct
      2. Conditions governing benefit
  5. Development and capacity
    1. Serving developments
    2. Precision and resources
      1. Cache precision
      2. A comparison ledger
    3. The capacity envelope
      1. A consistency relation
      2. Workload objectives
  6. Measurement and decisions
    1. Benchmark the workload
      1. Open-loop and closed-loop load
    2. Locate the bottleneck
      1. Controlled sweeps
      2. Warm execution is not startup
    3. Routing and reasoning budgets
      1. Responsibility boundary
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

Inference

Inference engineering turns trained model weights into a service that can handle real requests. It includes loading and placing the model, processing prompts, generating tokens, managing attention state and scheduling competing work. The goal is useful response time and throughput within hardware and cost limits. This chapter follows the serving path so batching, caching, quantization and speculative decoding can be understood as parts of a complete system rather than isolated speed claims.

Part I — From request to generated tokens

The serving contract

A serving request is not merely the user's visible text. It is the complete model-visible sequence produced by a model-specific chat template, together with generation controls and an allowance for new output. System instructions, message history, retrieved passages, tool definitions, role markers, and special tokens may all contribute to the input. The application should count the complete serialized request and reserve room for generation before admission. A context limit constrains the combined sequence; cache capacity does not enlarge that architectural limit.

From acceptance to release

The request then passes through distinct responsibilities. Admission control decides whether bounded resources and policy permit activation. An admitted request may wait in a queue before model work begins. Execution produces zero or more incremental outputs until an end token, stop match, output limit, cancellation, or error supplies a disposition. Completion initiates cleanup, but it does not prove that already-launched accelerator work stopped instantly or that every allocation became reusable at the same moment. One documented vLLM lifecycle, for example, can defer block release while a GPU step or remote cache transfer remains in flight.

Three operational quantities

QuantityMeaningRequired boundary
LatencyElapsed time for one named intervalStart event, end event, and observation location
ThroughputCompleted work per unit timeWork unit, completion rule, and measurement duration
CapacityWorkload sustainably served under stated constraintsArrival pattern, request mix, latency objectives, and completion requirements

These quantities answer different questions. High accelerator utilization says that a resource was busy, not that useful requests completed within their objectives. Likewise, an HTTP success status describes a transport or API boundary; it does not establish answer quality. Production inference also needs readiness, lifecycle management, observability, recovery, and controls around the model engine itself. The chronology traces distinct serving constraints: latency-aware batching, benchmark contracts, iteration scheduling, speculative execution and KV organization. These mechanisms can coexist.

Different constraints, complementary serving mechanisms

  1. March 2017ClipperAdapt batch size to online prediction’s latency objective.Sources & context

    Contributors: Berkeley team

    What changed: Published at NSDI 2017. Per-model controllers adjusted maximum batch size to amortize overhead without ignoring interactive latency. Clipper addressed ordinary prediction calls, not token-by-token generation.

  2. November 2019 preprint; ISCA 2020MLPerf InferenceCompare performance under explicit workload, quality, and latency contracts.Sources & context

    Contributors: Reddi and 46 coauthors

    What changed: The initial server scenario measured sustainable throughput under independent arrivals and a tail-latency constraint. The offline scenario supplied all inputs together. Their distinction explains why bulk throughput does not establish interactive capacity.

  3. July 2022OrcaReturn scheduling control after each generation iteration.Sources & context

    Contributors: Seoul National University and FriendliAI researchers

    What changed: Published at OSDI 2022. Completed requests could leave and new requests enter between iterations, addressing wasted work and delayed admission when outputs have unequal lengths.

  4. November 2022 preprint; ICML 2023Fast Inference from Transformers via Speculative DecodingTrade draft and verification work for fewer serial target steps.Sources & context

    Contributors: Google

    What changed: The procedure preserves the target output distribution while using draft proposals and parallel target scoring. Benefit depends on accepted advancement, draft cost, and wasted verification work—not acceptance rate alone.

  5. September 2023 preprintPagedAttentionOrganize KV state in noncontiguous blocks allocated as needed.Sources & context

    Contributors: Woosuk Kwon and colleagues

    What changed: Block-based organization reduces reservation waste and fragmentation and permits compatible state sharing. It addresses memory organization, separately from the serial-execution constraint targeted by speculative decoding.

Five contributions address distinct constraints. Dates distinguish preprints and conference publication; ordinal spacing is not to scale. The sequence does not imply influence or universal replacement.

Prefill and decode

An autoregressive language model maps its current prefix to vocabulary scores called logits. A selection rule chooses one token, appends it to the prefix, and changes the conditional distribution used for the following choice. The full model mechanism lives in Producing an unknown continuation; the serving consequence is that known input positions and unknown output positions create different execution shapes.

The first-token boundary

Prefill processes the complete supplied prefix and establishes contextual state for its positions. The logits at the final prompt position support selection of the first output token. Because all prompt tokens are already known, an implementation can process substantial work across their positions together while causal masking prevents a position from reading future content. Decode begins after that first selection. Each iteration processes the newest selected token with retained state, produces the next logits, selects another token, and repeats. The next iteration cannot know its input token until the preceding selection finishes.

A selected token gets its KV state in the next step

Symbolic prompt P₁–P₃; ordinary full-history cache. Columns follow computation dependencies from left to right.

At this boundaryPrefill → select O₁Process O₁ → select O₂Process O₂ → select O₃
Known input processedP₁ P₂ P₃, together under causal maskingO₁, selected in the prior columnO₂, selected in the prior column
Retained numerical KV readNo prior prompt cache in this exampleKV(P₁ P₂ P₃)KV(P₁ P₂ P₃ O₁)
New numerical KV writtenKV(P₁ P₂ P₃)KV(O₁)KV(O₂)
Logits used to selectFinal prompt position P₃ → O₁Processed O₁ position → O₂Processed O₂ position → O₃
Cache after selection3 positions: P₁ P₂ P₃4 positions: P₁ P₂ P₃ O₁5 positions: P₁ P₂ P₃ O₁ O₂
Selected, not yet processedO₁ has no KV yetO₂ has no KV yetO₃ has no KV yet
The cache retains processed positions, one output token behind the latest selection. Causal masking permits parallel prompt processing while preventing each position from reading future tokens. Retained KV is reused rather than recomputed.

Prompt length therefore primarily changes prefill work and the amount of state created before generation. Output length primarily changes the number of decode iterations and how long each active sequence retains state. Prefill often exposes matrix-shaped parallel work, whereas low-batch decode often performs less arithmetic per byte moved. These are tendencies, not universal bottleneck labels: batching, architecture, kernels, cache representation, hardware, and sequence lengths can move either phase into another regime. The correct diagnosis comes from profiling the intended workload.

Part II — State and progress

The KV cache

Attention separates addressing from transmitted content through keys and values. During prefill, each transformer layer computes key and value representations for the supplied positions. A KV cache retains the positions selected by the model architecture and cache policy. In an ordinary full-history cache, that means every processed position; sliding-window or chunked layers may retain only a bounded subset. During decode, the new query can attend to retained keys and combine the corresponding values without recomputing their old projections, then append state for the newly processed position. The cache stores numerical representations derived from tokens—not text, an answer cache, or durable model memory.

A cache-payload estimate

MKV=L S HKV D 2 B M_{KV}=L\,S\,H_{KV}\,D\,2\,B Here LL is the number of layers, SS the retained positions, HKVH_{KV} the number of key-value heads, DD each head's dimension, and BB bytes per stored element. The factor two accounts for keys and values. Multiply again by active sequences when each owns separate state. This is tensor payload, not total device allocation: block rounding, metadata, temporary activations, communication buffers, graph storage, and headroom remain separate.

The number of KV heads is architectural. Conventional multi-head attention may retain a key and value head for every query head. Grouped-query attention shares each key-value pair among a group of query heads, while multi-query attention uses one key and one value head for all query heads. That can reduce cache storage and bandwidth, but changing the head arrangement changes the model architecture; it is not a lossless allocator setting applied after deployment.

Keep the memory ledger separated

CategoryLifetimeWhy it matters
Model weightsUsually residentMust fit before request state; precision affects bytes and supported arithmetic
KV stateRequest or reusable-prefix lifetimeGrows with retained positions and active sequences
Activations and workspaceTransient peaksCan reduce the cache budget even though allocations are temporary
Runtime and graph storageEngine-dependentMay be reserved separately from weights and KV payload
Free or fragmented capacityChanges over timeDetermines whether another request or block can be admitted

PagedAttention, introduced by Woosuk Kwon and colleagues in a September 2023 preprint, organizes KV state in fixed-size noncontiguous blocks allocated as needed. This reduces reservation waste and fragmentation and permits compatible sharing. Prefix caching extends reuse across requests, but only for exactly compatible preceding state. A practical cache identity can include parent-block identity, exact token IDs, adapter identity, multimodal hashes, and an isolation salt. Similar meaning or an identical suffix is insufficient. Reuse saves prefill work; it does not accelerate the new output tokens. In a documented vLLM implementation, cache blocks carry reference counts: releasing one request decreases ownership, but a block referenced elsewhere cannot be overwritten. A zero-reference cached block may remain reusable until eviction. Other engines may use different ownership and eviction contracts.

Conceptual reconstruction of documented vLLM block ownership. Table entries reference allocations rather than moving tokens. Reuse requires compatible exact tokens, preceding state and engine identifiers; it is not semantic matching. Zero-reference cached content can remain until eviction. Weights and KV are selected allocations, not the complete memory budget.

Latency has boundaries

“Latency” is incomplete without a start event, end event, and observation location. Queue time ends when execution starts. Prefill duration covers prompt processing at the engine. Server time to first token reaches the server's first token event, while a client-observed time to first token (TTFT) commonly ends when the client receives the first nonempty content response. Network, proxy, decoding, and buffering can separate those events. Observability's elapsed-time framework develops this boundary discipline for complete systems.

Progress and completion metrics

MetricTypical observationImportant qualification
Queue timeAdmission or arrival to engine startRequires a defined admission boundary
TTFTRequest send to first nonempty content receivedIncludes client-visible path; server TTFT may differ
ITLIntervals between successive responses or chunksA chunk can contain multiple tokens
TPOTPost-first-token generation time divided across later output tokensSome tools label a related average as ITL
Request latencyRequest start to final content responseDoes not necessarily include later cleanup
Output-token throughputGenerated tokens divided by benchmark durationTokenizer, completion accounting, and workload must match

A generated token, a decoded text fragment, a transport chunk, and a rendered UI update are different events. A streamer may buffer token IDs until they form printable text, and transport infrastructure may combine fragments. Incremental delivery can make an interface responsive before the full request completes, but it need not reduce model execution time. A delivered prefix also does not prove eventual successful completion.

One request, several latency intervals

Example timings

Server work, client visibility, completion, and cleanup use different start and end events.

Client request lifetime0–100 msDuration 100 ms
Queue residence0–18 msDuration 18 msWithin Client request lifetime
Prefill18–43 msDuration 25 msWithin Client request lifetime
First-content transit43–51 msDuration 8 msWithin Client request lifetime
Decode execution43–94 msDuration 51 msWithin Client request lifetime
Incremental delivery51–100 msDuration 49 msWithin Client request lifetime
Deferred cleanup94–108 msDuration 14 ms
Example milliseconds for one request; arrival coincides with client send at 0. First selection is at 43ms, while client TTFT ends at first nonempty content at 51ms. Decode (43–94) overlaps delivery (51–100). Cleanup (94–108) overlaps final delivery and finishes after client completion. Do not sum overlapping spans.
Read the diagram as text
  • Client request lifetime. Send through final client content. 0 to 100 ms; duration 100 ms.
  • Queue residence. Arrival until engine work begins. 0 to 18 ms; duration 18 ms. Parent: Client request lifetime.
  • Prefill. Prompt processing before first token selection. 18 to 43 ms; duration 25 ms. Parent: Client request lifetime.
  • First-content transit. Server selection and delivery to first nonempty client content. 43 to 51 ms; duration 8 ms. Parent: Client request lifetime.
  • Decode execution. Repeated generation after the first selection. 43 to 94 ms; duration 51 ms. Parent: Client request lifetime.
  • Incremental delivery. Decoded fragments transported while generation continues. 51 to 100 ms; duration 49 ms. Parent: Client request lifetime.
  • Deferred cleanup. Resource ownership is released after terminal state permits it. 94 to 108 ms; duration 14 ms.

Report distributions rather than only means. The 99th percentile is the value below which 99 percent of the specified observations fall; it is not an average. Partitioning by prompt length, output length, and request class often explains more than one aggregate percentile. Do not add independently computed p99 values for queue, prefill, network, and decode and call the sum an end-to-end p99: those percentiles may come from different requests.

Part III — Sharing a server

Continuous batching

A batch executes work from several requests together. Larger parallel operations can improve hardware use and amortize launch, scheduling, and weight-access overhead. But waiting to form the batch adds latency, and every active sequence consumes state. The useful batch is therefore constrained by the application's response budget and memory—not simply by the engine's configured maximum. Clipper, published at NSDI in March 2017 by a Berkeley team, adapted per-model batch limits to prediction-latency objectives. Its grouped ordinary prediction calls amortized overhead; it did not schedule autoregressive tokens.

Three batching policies

PolicyAdmission pointCharacteristic tradeoff
Fixed batchBefore the batch startsSimple, but unequal completion lengths can leave lanes idle
Dynamic batchAfter a short formation windowCan increase batch size by purchasing extra waiting time
Continuous or iteration-level batchBetween generation iterationsCompleted requests leave and new requests enter while others continue

Orca, published at OSDI in July 2022 by Seoul National University and FriendliAI researchers, returned scheduling control after each transformer iteration. Completed requests could leave and new work enter. Continuous batching is especially useful because generated outputs have unequal and initially unknown lengths. When one request ends, the scheduler can replace it without waiting for every other sequence in the batch. This reduces idle lanes, but it does not eliminate interference: new prefills compete with ongoing decodes, larger active sets require more KV state, and each request's token cadence can slow even as total device token throughput rises. Distinguish the configured sequence limit, the currently active sequences, and the prompt or decode tokens actually scheduled in an iteration.

Scheduling and admission

Admission control and scheduling solve different problems. Admission limits how much work can enter the bounded service; the scheduler chooses which admitted work runs next. If offered work persistently exceeds service capacity, rearranging the queue cannot make every objective feasible. The service must reject, defer, shed, degrade, or add capacity.

Allocating delay

A long prefill can delay token production for existing requests. Conversely, always prioritizing decode can make new prompt-heavy requests wait indefinitely for their first token. Chunked prefill divides a long prompt into bounded pieces that the scheduler can interleave with decode. Smaller chunks reduce individual stalls but introduce more scheduling boundaries, fixed overhead, and repeated reads of state from earlier prompt chunks. The best token budget is workload- and hardware-dependent.

Smaller prompt chunks divide delay—and add work

A and B are already decoding at time 0 and need 5 and 2 more steps. At 2ms, C arrives with a 12-token prompt and D with a 2-token prompt. Each new request produces one token at prefill completion, then needs 2 decode steps. Three active slots; D replaces B at the next iteration boundary. Memory is sufficient.

QuantityUnchunked (12)Selected (4)
C arrival → first selection10ms15.2ms
Largest A/B engine selection gap9ms5ms
Finish all work23ms26.2ms
Iterations / overhead5 / 5ms7 / 7ms
Prompt tokens / decode steps14 / 1114 / 11
Earlier prompt positions reread012
Budget 4: C first-selection delay 15.2ms; maximum continuing-request gap 5ms; finish at 26.2ms.
Fixed teaching workload and cost model

One iteration costs 1ms scheduling overhead + 0.5ms per prompt token + 1ms per decode step + 0.1ms per earlier prompt position read when continuing a prefill. Prompt requests share the selected prompt budget in arrival order (C before D for their tied arrival); every active decoding request gets one step. The total batch-token limit is fixed at 15. Admission and completion changes occur only between iterations. Prior A/B prefill is outside this window. No transport costs, memory pressure or failures are modeled.

Unchunked baseline

IterationTime (ms)ABCD
10–3D1D1——
23–12D1D1 · doneP12—
312–16D1—D1P2
416–20D1—D1 · doneD1
520–23D1 · done——D1 · done

Prompt budget 4 / iteration

IterationTime (ms)ABCD
10–3D1D1——
23–8D1D1 · doneP4—
38–12.4D1—P4Waiting
412.4–17.2D1—P4Waiting
517.2–21.2D1 · done—D1P2
621.2–24.2——D1 · doneD1
724.2–26.2———D1 · done

P = prompt tokens processed; D1 = one decode step; done = leaves before the next iteration. Gaps are engine selections, not measured client ITL.

These timings follow a fixed teaching model, not measured hardware. Smaller chunks shorten individual interruptions to ongoing decodes, but repeated prompt-state reads and more scheduling boundaries can delay the new request and increase total modeled work.

Policy dimensions

DecisionBenefit soughtFailure to test
First-come-first-servedSimple orderingA long request blocks short later work
Priority classesProtect important trafficLower classes starve
Aging or SLO-relative urgencyBound starvation across classesSustained overload still violates objectives
PreemptionLet urgent work run soonerRetained KV uses memory; swapping or recomputation costs work
CancellationStop unwanted lifecycle workPropagation and resource release may be delayed
Backpressure or admission limitsBound queued and active workRejection or deferral becomes caller-visible

Prefix-aware routing creates another tension. Sending a request to a worker with reusable state can avoid prefill, but routing only by cache match can overload that worker and turn saved computation into queueing. A useful policy considers both compatible reusable state and present load. In large deployments, separating prefill and decode workers can reduce phase interference, but transferring KV state adds communication whose cost depends on its size, placement, and bandwidth.

Part IV — Selecting and delivering output

Sampling, stopping, and streaming

Each decode step begins with logits. Greedy decoding chooses a highest-scoring token. Sampling converts processed scores into a probability distribution and draws from it. Temperature changes concentration; lower values emphasize larger logits while higher values flatten the distribution. Top-k keeps a fixed number of high-probability candidates. Top-p, or nucleus sampling, keeps the smallest high-probability set whose cumulative mass reaches a threshold, then renormalizes. These settings alter token selection, not whether the answer is true.

Selection is not system determinism

Enabled controls also have an implementation order. One documented Transformers version applies temperature, then top-k, then top-p before softmax and sampling; custom processors or other engines may differ. Greedy execution instead takes an argmax. A random seed constrains random-number generation, but portable reproducibility can additionally require the same engine version, hardware, scheduling behavior, and supported batch-invariant execution. Temperature zero does not validate reasoning or guarantee identical outputs across changed environments.

Ways generation ends

After selecting a token, the server appends it and checks termination. An end-of-sequence token is model-visible. A stop string is matched against decoded text and may require holding back a suffix so the stop marker is not leaked. A maximum-output limit ends generation because a budget was exhausted, not because the task necessarily completed. Structured or machine-consumed responses must also handle refusal and incomplete output rather than parsing every partial prefix as a finished record.

Streaming exposes completed fragments while generation continues. It can reduce perceived waiting without changing the number of model steps. Backpressure at the transport layer prevents an unbounded application buffer, but pausing a network writer does not automatically pause accelerator generation; that requires explicit propagation into the serving scheduler.

Speculative decoding

Ordinary autoregressive generation pays for one serial target-model step per advancement. Speculative decoding uses a cheaper draft process to propose several future tokens, then asks the target model to score their conditional distributions together. The target—not the draft—determines what is committed. This can exchange extra arithmetic for fewer serial target invocations, which is useful when target execution has spare parallel capacity but repeated state or weight access dominates elapsed time. Google's Fast Inference from Transformers via Speculative Decoding appeared as a November 2022 preprint and at ICML 2023. Its analysis separates accepted advancement from draft cost and wasted verification work.

Accept, reject, correct

In distribution-preserving speculative sampling, proposals are examined left to right. Proposal xix_i, drawn from draft distribution qiq_i, is accepted with probability min⁡(1,pi(xi)/qi(xi))\min(1,p_i(x_i)/q_i(x_i)), where pip_i is the target distribution for that position. The accepted prefix is retained. At the first rejection, that token and later proposals are discarded; a replacement is drawn from the normalized positive part of pi−qip_i-q_i. If every proposal passes, the target supplies an additional token. This procedure preserves the target sampling distribution within numerical precision; it does not guarantee identical samples or correct answers.

Unused proposals must not leak into output or persistent target state. In Transformers v4.50.0, assisted generation supplies only the valid continuation to the streamer and crops target KV to the total committed sequence length minus one. The last selected token has not yet been processed; after a rejection, this can be the replacement token. Logical cropping need not release allocator reservations immediately. Useful advancement must justify both draft and verification cost.

Commit only the accepted prefix and its correction

At each position, accept xᵢ with probability min(1, pᵢ(xᵢ)/qᵢ(xᵢ)). Examine proposals left to right and stop at the first rejection.

Main branchPosition 1Position 2Position 3Position 4
Draft proposalsx₁ from q₁x₂ from q₂x₃ from q₃x₄ from q₄
Target conditional scoringp₁(· | prefix)p₂(· | prefix,x₁)p₃(· | prefix,x₁,x₂)p₄(· | prefix,x₁,x₂,x₃)
Acceptance✓ Accept x₁✓ Accept x₂× Reject x₃— Not tested; discard x₄
Committed continuationx₁x₂r: replacementNone
Valid tokens supplied to streamerx₁x₂rNone
Retained new target KVKV(x₁)KV(x₂)No KV(r) yet; drop KV(x₃)Drop KV(x₄)

Replacement r ∼ normalize(max(p₃ − q₃, 0)), not an ordinary draw from p₃.

If all four proposals pass: commit x₁, x₂, x₃, x₄, t, with t sampled from the target distribution after x₁–x₄. Supply those five tokens to the streamer. Retain target KV through x₄; newly selected t has no KV yet.
The sampling algorithm preserves the target distribution; it does not verify truth or reproduce identical seeded samples. The KV row follows Transformers v4.50.0: retain the processed committed prefix, excluding the newly selected last token. Supplying valid tokens to a streamer does not guarantee immediate client delivery or allocator reclamation.

Conditions governing benefit

Tune speculative decoding for the deployed target, draft process, request content, and concurrency regime. Sweep draft length and related settings across relevant batch sizes, then compare elapsed time and accepted advancement—not acceptance rate alone. A configuration that helps one workload can lose its benefit when drafting or verification competes with other requests.

Part V — Development and capacity

Serving developments

A serving design combines choices with different effects. Batch formation controls which work runs together; iteration scheduling controls when membership changes; state organization controls what occupies memory. No single setting resolves all three constraints.

Saved computation can become additional waiting. A larger batch or a useful prefix-cache hit matters only if the complete request still meets its delivery objective.

More retained state can avoid repeated work while reducing room for other sequences. Recomputation can therefore remain useful for short contexts or severe memory pressure.

Speculation changes how many tokens advance per target invocation. It still competes for arithmetic, memory and scheduling time with other requests, so a single-request benefit need not survive concurrent load.

Evaluate the combined policy under a matching workload and completion contract. Preserve the distinction between engine work, client-visible progress and safe resource reuse when interpreting a change.

Precision and resources

Numerical precision is a serving resource choice applied separately to weights, activations, and KV state. Lower-bit weight storage can reduce resident model bytes, but storage precision, computation precision, and device placement are independent. Some methods preserve higher precision for outliers; offloaded weights may use a wider format. Halving weight bytes therefore does not halve total serving memory or latency.

Cache precision

KV quantization targets a different allocation. KIVI, a February 2024 preprint later accepted at ICML 2024, used asymmetric two-bit quantization with a full-precision recent window. Its reported single-A100 experiment showed how reduced KV storage could permit larger batches and greater throughput, while quality varied by model, task, bit width, group size, and residual-window length. Fewer stored bytes can also add conversion overhead, so capacity gains do not independently establish latency gains.

Realized speed requires compatible arithmetic and kernels. In PyTorch's reported 2023 GPT-fast experiment, compilation and a static KV cache first exposed weight-loading bandwidth as the next constraint; supported int8 weight storage then improved the reported batch-one generation rate further. The point is bottleneck migration, not the historical numbers: changing coordination can expose bandwidth, and changing bytes can expose another limit. The numerical mechanisms of calibration, scales, outliers, and quantization-aware training belong in Quantization.

A comparison ledger

FieldWhy it must be reported
Weight format and bytesShows resident-model change
KV format and bytes per retained positionShows request-state change
Activation and workspace peakPrevents hidden capacity changes
Supported hardware and kernelsEstablishes whether lower precision accelerates execution
Latency and sustainable capacityMeasures operational consequence
Quality and completion behaviorDetects altered outputs rather than assuming equivalence

The capacity envelope

Capacity is an operating region, not a hardware label. Resident weights and runtime allocations consume memory before requests arrive. Each prompt adds prefill work and initial KV state. Each generated token adds another decode iteration and usually another retained position. Concurrency can improve reuse and batching while increasing contention and state. Arrival rate adds queueing when offered work approaches what the complete service can finish.

A consistency relation

L=λW L=\lambda W Little's Law relates the long-run average number of requests in a consistently bounded system, LL, to arrival rate λ\lambda and average residence time WW. If residence includes queueing and execution, occupancy must count requests across that same boundary. The relation is an accounting check under stability and finite averages; it neither predicts p99 latency nor rescues an overloaded queue.

Near saturation, completed throughput may flatten while queue residence and tail latency rise. Output tokens per second, requests per second, per-request token pace, and concurrent sequences therefore cannot substitute for one another. Prompt-heavy work emphasizes first-token delay and prompt processing; generation-heavy work spends more iterations retaining state; interactive service constrains onset and cadence; offline work can accept waiting to maximize completed volume.

Workload objectives

WorkloadPrimary concernCommon constraint to test
Interactive chatTTFT and output cadenceBatch formation, prompt interference, network delivery
Long generationCompletion time and state residencyDecode service rate and KV growth
Prompt-heavy processingFirst-token latency and prefill capacityPrompt length, chunking, and prefix reuse
Offline batchCompleted volume and total job durationLargest efficient batch under memory limits
Streaming voiceUseful response onset and concurrent streamsEntire audio pipeline, not text token rate alone

More memory can raise the number or length of active sequences without increasing a compute-limited token service rate. Conversely, faster arithmetic may leave capacity unchanged when KV storage or memory bandwidth is limiting. Admission policy must therefore reserve against both state and work. During severe overload, explicit rejection or a clearly exposed degraded mode is safer than silently accepting requests that cannot meet their contract.

Choose capacity within both latency objectives

Completed throughput (requests/s)06120714Offered requests / secondL2: 2; stable; qualifiesL4: 4; stable; qualifiesL6: 6; stable; qualifiesL8: 8; stable; excludedL10: 9; unstable; excludedL12: 11; errored; excludedp99 client TTFT (ms)080016000714Offered requests / secondL2: 80; stable; qualifiesL4: 140; stable; qualifiesL6: 260; stable; qualifiesL8: 520; stable; excludedL10: 1000; unstable; excludedL12: 1400; errored; excludedp95 request TPOT (ms/token)0501000714Offered requests / secondL2: 12; stable; qualifiesL4: 20; stable; qualifiesL6: 32; stable; qualifiesL8: 48; stable; excludedL10: 68; unstable; excludedL12: 90; errored; excluded

Filled circle: qualifies; larger circle: selected. Empty circle: misses an objective. Cross: invalid run. Dashed: chosen objective. L14 has unavailable measurements and is not plotted.

L6 selected: 6 completed requests/s at offered rate 6/s.
Offered /sSuccessful / 60sCompleted /sp99 TTFTp95 TPOTRun status / errorsDecision
2120280ms12ms/tokenstable / 0Qualifies
42404140ms20ms/tokenstable / 0Qualifies
63606260ms32ms/tokenstable / 0Selected
84808520ms48ms/tokenstable / 0Misses objective
1054091000ms68ms/tokenunstable / 0Invalid run
12660111400ms90ms/tokenerrored / 60Invalid run
14UnavailableUnavailableUnavailableUnavailableincomplete / unknownInvalid run
Dataset and measurement contract

Fictional sweep of one unchanged configuration and workload: 128-token prompts, 32-token outputs, periodic independent arrivals. Discard 30s warmup, then observe 60s. Throughput counts successful final-content completions inside that window divided by 60s. Every counted output passes the fixed task-quality check; errors are separate. p99 TTFT is over completed requests, from client send to first nonempty content. For each completed request, TPOT = (final-content time − first-content time) / 31; p95 is taken across those per-request averages. Valid runs require complete logs, zero errors and no positive queue-length trend over the final 30s. L10 is unstable, L12 has 60 errors, L14 has incomplete logs. Tied throughput selects the lower offered rate, then lexical run ID.

This fictional load sweep selects among fixed workload-specific operating points. Adjusting objectives changes qualification, never the measurements. Marginal latency percentiles cannot determine compliant requests per second or real hardware capacity.

Part VI — Measurement and decisions

Benchmark the workload

MLPerf Inference, introduced by Reddi and collaborators in a November 2019 preprint and presented at ISCA 2020, distinguished independently arriving server traffic under tail-latency limits from offline bulk throughput, with required output-quality targets. A serving benchmark is defined by its workload and execution contract, not just a model and accelerator. Record the exact model revision, tokenizer, chat template and complete serialization; hardware and topology; engine and runtime versions; weight and KV formats; scheduler, batch-token limits, and cache policy; sampling and stopping controls; prompt and output distributions; warmup and readiness; arrival process or concurrency; duration; observation boundaries; errors; completion accounting; and quality checks. Use completed-work timing and sweep the important workload range.

Open-loop and closed-loop load

Load modelIssuance ruleQuestion answeredFailure to expose
Open loopArrivals occur independently of prior completionsCan the service sustain an offered arrival rate under latency constraints?Delayed issuance and unfinished work must remain visible
Closed loopA client issues new work after earlier work completes while holding concurrencyWhat throughput and latency occur with this many active clients?A saturated load generator can silently cap offered demand

Separate unloaded single-request latency, bulk throughput, and the load sweep between them. Fixed input and output lengths can isolate execution behavior but do not represent production unless the real distribution is similarly fixed. Continuing beyond natural stop tokens controls length for a benchmark while changing ordinary completion behavior; disclose it. Likewise, raw single-batch engine throughput is not a serving result with in-flight batching, queueing, and transport.

The client and network are part of an endpoint measurement. Record client location, connection behavior, request-generation capacity, target and achieved arrival rates, and incomplete work. A benchmark client that cannot feed the server can make a fast deployment appear slow; a distant or buffered transport can dominate caller-visible TTFT even when model execution is fast. Match output length, completion status, and quality so shorter, refused, or failed responses are not counted as optimizations.

Locate the bottleneck

Diagnosis begins with a symptom conditioned on workload. Align queue depth, request lengths, batch composition, KV occupancy, preemptions, prefill and decode timing, accelerator activity, host gaps, transfers, client observations, cancellations, errors, and completion outcomes. A high utilization percentage does not identify whether compute units, memory bandwidth, host coordination, or another resource limits useful work.

Controlled sweeps

Change one variableObservationSupported hypothesis to investigate
Increase prompt lengthTTFT and prefill time rise; TPOT stablePrompt processing is consequential
Increase output lengthCompletion and KV residency riseDecode duration or cache growth matters
Increase offered rateQueue rises; completed throughput flattensService is near sustainable capacity
Increase active sequencesAggregate throughput rises but TPOT worsensBatching helps throughput while adding interference
Reduce cache budgetPreemption or admission failures increaseKV capacity constrains concurrency
Change precisionBytes fall but latency does notKernel support or another bottleneck dominates
Change speculative settingsAcceptance rises without elapsed-time gainDraft or verification overhead dominates

Expect bottleneck migration: relieving one constraint can make another resource determine useful work. Reducing resident weight bytes may permit more active sequences until KV capacity, scheduling, compute, or host coordination dominates. An external cache tier can add capacity yet stall accelerators if its reads and writes cannot keep pace. State what changed, what remained fixed, which resource signal moved, and whether the completion and quality contract still held.

Warm execution is not startup

Cold starts form another path. New capacity may require resource acquisition, container startup, transfer of model weights through host memory into accelerator memory, compilation or graph capture, and warmup before readiness. A system can have excellent warm-request latency yet fail to absorb a burst because replicas become useful too slowly. Measure startup stages separately from steady-state inference and test actual scaling events rather than inferring them from loaded-instance throughput.

Routing and reasoning budgets

Three adjacent responsibilities change the workload in different ways. A model-routing or gateway policy chooses a model or provider, escalation path, and failure fallback. A reasoning and test-time-compute policy chooses how many samples, search steps, verifications, or retries to request in pursuit of a better solution. Inference serving executes and measures every invocation those policies create.

Responsibility boundary

ResponsibilityDecisionOutputEvaluate by
RoutingWhere should this invocation run?Selected model, provider, or fallbackPolicy quality, compatibility, reliability, and total cost
Reasoning policyHow much solution work should be requested?Samples, search, verification, and stopping decisionsTask quality under a stated compute and latency budget
Inference servingHow should each invocation execute?Tokens, stream events, completion disposition, and resource usageLatency, throughput, capacity, correctness, and operational reliability

The boundaries compose but should not be collapsed. Re-querying a smaller model may be preferable to one larger-model call for a particular quality target, yet those repeated calls still consume requests, tokens, cache, and concurrency. Moving a predictable re-query into a router can remove client round trips and expose scheduling opportunities, but the quality gain belongs to the reasoning policy and the execution gain belongs to serving. Evaluate the resulting combined workload at matched task quality rather than describing additional computation as free.

Open questions

  1. Can generative-serving systems expose provider-neutral, observable state transitions for cancellation and resource reuse? A useful contract would let conformance tests distinguish a terminal request from accelerator work or allocations that remain in flight.

  2. How should schedulers optimize jointly for TTFT, token cadence, fairness, energy, and completed task quality under changing request distributions? Present policies usually prioritize only a subset, while sustained overload can make the objectives mutually infeasible. Progress would require workload-replay studies with explicit admission and degradation semantics, not only average throughput gains.

  3. When does KV compression improve end-to-end capacity after accounting for conversion work, quality changes, allocator behavior, and the batch sizes it enables? Results vary across models, tasks, formats, and recent-window policies. Progress would be reproducible, matched-quality load sweeps that report payload bytes, actual allocations, preemption, TTFT, TPOT, and completed throughput.

  4. Can speculative decoding remain beneficial under high continuous-batch pressure, where draft work and additional state compete with unrelated requests? Single-request acceptance rates do not answer this fleet-level question. Progress would connect accepted advancement to queueing, memory, fairness, and goodput across real prompt and output distributions.

  5. What bounded reproducibility contract is practical across engine upgrades, kernel changes, hardware revisions, and different batch compositions? Seeds control only part of execution, while exact outputs may not be necessary for every application. Progress would define useful levels—from distributional equivalence to stable state transitions—and publish the conditions each engine can enforce.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

173 matching talks

TalkSpeakerEventYear
Stephen Hood, Justine TunneyAI Engineer World's Fair 20242024
Ishan AnandAI Engineer World's Fair 20242024
Charles FryeAI Engineer Summit 20232023
Ishan AnandAI Engineer World's Fair 20252025
Daniel HanAI Engineer World's Fair 20242024
Kevin HouAI Engineer World's Fair 20242024
AI Engineering 101

Transcript reviewed

Noah HeinAI Engineer Summit 20232023
Aman KhanAI Engineer World's Fair 20252025
Sarah ChiengAI Engineer Europe 20262026
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Natalie SerrinoAI Engineer Code 20252025
Philip Kiely, Pankaj GuptaAI Engineer World's Fair 20242024
Allen PikeAI Engineer World's Fair 20262026
Jerry LiuAI Engineer Summit 20232023
Erik MeijerAI Engineer World's Fair 20262026
Elizabeth Fuentes LeoneAI Engineer World's Fair 20262026
Nupur SharmaAI Engineer Europe 20262026
Ben BurtenshawAI Engineer Europe 20262026
Compression at the Edge

Transcript reviewed

Chris Alexiuk, Daniel Han, Asma Beevi, Merve Noyan, Parth SareenAI Engineer World's Fair 20262026
Dylan PatelAI Engineer World's Fair 20242024
Nathan LambertAI Engineer World's Fair 20252025
Zhou YuAI Engineer Summit 20252025
Aakanksha ChowdheryAI Engineer World's Fair 20252025
Vivek MuppallaAI Engineer World's Fair 20262026
Alex CheemaAI Engineer Europe 20262026
Rémi LoufAI Engineer World's Fair 20242024
Luis Romero-SevillaAI Engineer World's Fair 20262026
Sai Krishna RallabandiAI Engineer World's Fair 20262026
Val Bercovici, Callan FoxAI Engineer Code 20252025
Tisha Chawla, Susheem KoulAI Engineer World's Fair 20262026
Julián Duque, Anush DSouzaAI Engineer World's Fair 20252025
Cassidy HardinAI Engineer Europe 20262026
Philipp KrennAI Engineer World's Fair 20252025
Amir HaghighatAI Engineer World's Fair 20252025
Lin Qiao, Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Dan Fu, Olive SongAI Engineer World's Fair 20262026
Rishabh BhargavaAI Engineer Europe 20262026
Keegan McCallumAI Engineer World's Fair 20252025
Samuel HumeauAI Engineer Europe 20262026
Filip MakraduliAI Engineer World's Fair 20252025
Walden, Carter, Tanay, Alex Atallah, NavAI Engineer World's Fair 20262026
Sahil Yadav, Hariharan GanesanAI Engineer World's Fair 20252025
Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Daniel Kim, Daria SobolevaAI Engineer World's Fair 20252025
Steve KorshakovAI Engineer World's Fair 20262026
Daniel HanAI Engineer World's Fair 20252025
Neil Dwyer, Jack DwyerAI Engineer World's Fair 20252025
Arjun Desai, Rohit TalluriAI Engineer World's Fair 20252025
Daniel HanAI Engineer World's Fair 20262026
Arek BoruckiAI Engineer World's Fair 20262026
Joseph NelsonAI Engineer Summit 20232023
Hamed Firooz, Maziar SanjabiAI Engineer World's Fair 20252025
Shelby HeineckeAI Engineer World's Fair 20242024
A Song of Types and Agents

Metadata candidate

Roberto StagiAI Engineer World's Fair 20262026
Sharmila Chokalingam, ShubhiAI Engineer World's Fair 20242024
Will Hang, Cathy ZhouAI Engineer Code 20252025
Kevin HouAI Engineer Summit 20252025
AGI: The Path Forward

Metadata candidate

Eiso Kant, Jason WarnerAI Engineer Code 20252025
Charles FryeAI Engineer Summit 20232023
Vibhor KumarAI Engineer World's Fair 20242024
Grace IsfordAI Engineer Summit 20252025
Sunny MadraAI Engineer World's Fair 20242024
Samuel DentonAI Engineer World's Fair 20262026
Varun Badrinath Krishna, Petro Junior Milan, Rachelle MatternAI Engineer World's Fair 20242024
Raj NavakotiAI Engineer Europe 20262026
Soumya Gupta, Jai ChopraAI Engineer World's Fair 20262026
Building Cursor Composer

Metadata candidate

Lee RobinsonAI Engineer Code 20252025
Eno ReyesAI Engineer World's Fair 20242024
Dan MasonAI Engineer World's Fair 20252025
Codex, Behind the Harness

Metadata candidate

Dominik KundelAI Engineer World's Fair 20262026
Jedrick Kosinski, ComfyAnonymousAI Engineer World's Fair 20252025
Yusuf OlokobaAI Engineer Code 20252025
Santosh RadhaAI Engineer World's Fair 20242024
Rhythm Garg, Linden LiAI Engineer Code 20252025
Ending AI Slop

Metadata candidate

Thais Castello BrancoAI Engineer World's Fair 20262026
Sayash KapoorAI Engineer Summit 20252025
Maxime LabonneAI Engineer Europe 20262026
Maxime LabonneAI Engineer World's Fair 20242024
Emma Ning, Tsavo KnottAI Engineer World's Fair 20252025
Omri Bruchim, Tomer AstAI Engineer World's Fair 20262026
Rachel Lee Nabors (RL Nabors)AI Engineer World's Fair 20262026
Alex AtallahAI Engineer World's Fair 20252025
Mike BursellAI Engineer World's Fair 20252025
Mithun HunsurAI Engineer Summit 20232023
Vasant KearneyAI Engineer World's Fair 20262026
How Deep Research Works

Metadata candidate

Mukund Sridhar, Aarush SelvanAI Engineer Summit 20252025
Vinoo GaneshAI Engineer World's Fair 20262026
Kwindla Hultman KramerAI Engineer World's Fair 20242024
Paul GilbertAI Engineer Summit 20252025
Kyle CorbittAI Engineer World's Fair 20252025
Hypermode Launch

Metadata candidate

Kevin Van GundyAI Engineer World's Fair 20242024
Nicolas SchlaepferAI Engineer World's Fair 20242024
Jake NationsAI Engineer Code 20252025
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
AI Engineer Summit 20252025
Stefano FiorucciAI Engineer Europe 20262026
Rachelle Mattern, Petro Milan, Varun KrishnaAI Engineer World's Fair 20242024
Sally Ann O'MalleyAI Engineer Europe 20262026
Carter Abdallah, Vincent Weisser, Lucas Atkins, Chris AlexiukAI Engineer World's Fair 20262026
Joe FiotiAI Engineer World's Fair 20252025
Kelvin MaAI Engineer World's Fair 20252025
Matthias LoiblAI Engineer World's Fair 20252025
Pietro ZulloAI Engineer World's Fair 20262026
Liad Yosef, Ido SalomonAI Engineer Europe 20262026
Stefania DrugaAI Engineer World's Fair 20262026
Will BrownAI Engineer World's Fair 20262026
Alvaro MoralesAI Engineer World's Fair 20252025
Cedric VidalAI Engineer World's Fair 20242024
Ahmed MenshawyAI Engineer World's Fair 20242024
Frank CoyleAI Engineer World's Fair 20262026
Saoud RizwanAI Engineer World's Fair 20262026
Lech KalinowskiAI Engineer World's Fair 20262026
OpenLLMetry is all you need

Metadata candidate

Nir GazitAI Engineer Summit 20252025
Christopher HarrisonAI Engineer World's Fair 20252025
Yuval Belfer, Niv GranotAI Engineer World's Fair 20252025
Tengyu MaAI Engineer World's Fair 20252025
Idan GazitAI Engineer World's Fair 20262026
Adrien GrondinAI Engineer Europe 20262026
Scaling Compute on Context

Metadata candidate

Jack MorrisAI Engineer World's Fair 20262026
Alessandro CappelliAI Engineer Europe 20262026
Scaling to Long Horizons

Metadata candidate

Ross Taylor, Chengxi TaylorAI Engineer World's Fair 20262026
Merve NoyanAI Engineer Europe 20262026
Sarah GuoAI Engineer World's Fair 20252025
Nader Khalil, Alex Cheema, Matthew Berman, Ahmad Osman, Joseph NelsonAI Engineer World's Fair 20262026
Thiyagarajan MaruthavananAI Engineer World's Fair 20262026
Taylor Jordan SmithAI Engineer World's Fair 20252025
Rob CheungAI Engineer World's Fair 20242024
Devansh TandonAI Engineer World's Fair 20252025
Patrick DeboisAI Engineer World's Fair 20252025
Darius EmraniAI Engineer World's Fair 20252025
Travis FrisingerAI Engineer World's Fair 20252025
Beyang LiuAI Engineer World's Fair 20252025
Travis Bartley, Myungjong Kim, Byungjoong, JaehanAI Engineer World's Fair 20252025
Justin SchroederAI Engineer World's Fair 20262026
Kyle CorbittAI Engineer World's Fair 20242024
Dylan PatelAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer World's Fair 20242024
Gorkem YurtsevenAI Engineer World's Fair 20252025
Jonathan MortensenAI Engineer World's Fair 20252025
Maxime Rivest, Isaac MillerAI Engineer World's Fair 20262026
Thinking Deeper in Gemini

Metadata candidate

Jack RaeAI Engineer World's Fair 20252025
Sangwu LeeAI Engineer World's Fair 20262026
Eric AllamAI Engineer World's Fair 20252025
Chris LattnerAI Engineer World's Fair 20242024
Hassan El MghariAI Engineer World's Fair 20252025
Matt DaileyAI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20252025
Benjamin CowenAI Engineer Europe 20262026
Cormac BrickAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer Europe 20262026
Ziv IlanAI Engineer Europe 20262026
Ramana Siddanth EmaniAI Engineer World's Fair 20262026
Yuchen Fama, Ashish KamraAI Engineer World's Fair 20262026
Sitanshu GuptaAI Engineer World's Fair 20262026
Abdul DakkakAI Engineer World's Fair 20262026
Bogdan GazaAI Engineer Code 2025Unknown
Nishant Gupta, Naman AhujaAI Engineer World's Fair 20262026
Du’An Lightfoot, Khaja OmerAI Engineer World's Fair 20262026
Asaf Gardin, Yuval BelferAI Engineer World's Fair 20262026
Eugene CheahEvent unknownUnknown
Harshul Jain, Tanmay SahEvent unknownUnknown
Chris VelaEvent unknownUnknown
David AbitbolEvent unknownUnknown
Oliver KennonEvent unknownUnknown
Daniel KimAI Engineer World's Fair 20262026
Yunmo KooAI Engineer World's Fair 20262026
Branislav BrzakAI Engineer NYC 2026 Online Talks2026
Sheilah KiruiAI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20262026
Byung-Gon (Gon) ChunAI Engineer World's Fair 20262026
Tarun SunkaraneniAI Engineer World's Fair 20262026
Qianru Lao, Lu ZhangEvent unknownUnknown
Philip KielyAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
46 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
132 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Chat templates — Transformers

    https://huggingface.co/docs/transformers/chat_templating

  2. vLLM scheduler: admission, preemption and abort cleanup

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/core/sched/scheduler.py

  3. Metrics design — vLLM

    https://docs.vllm.ai/en/stable/design/metrics/

  4. The Rise of Open Models in the Enterprise

    https://www.youtube.com/watch?v=3WV1vT0B0cg&t=697

  5. Language Models are Unsupervised Multitask Learners

    https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

  6. Improving Language Understanding by Generative Pre-Training

    https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

  7. Caching — Transformers v4.50.0

    https://huggingface.co/docs/transformers/v4.50.0/en/cache_explanation

  8. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=378

  9. Compute & System Design for Next Generation Frontier Models

    https://www.youtube.com/watch?v=gFyBdBm0AGo&t=96

  10. GPU Performance Background User's Guide — NVIDIA

    https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html

  11. Attention Pooling by Similarity — Dive into Deep Learning

    https://d2l.ai/chapter_attention-mechanisms-and-transformers/attention-pooling.html

  12. Cache strategies — Transformers

    https://huggingface.co/docs/transformers/main/en/kv_cache

  13. Transformers: cache allocation versus retained history

    https://huggingface.co/docs/transformers/en/kv_cache

  14. Mastering LLM Techniques: Inference Optimization — NVIDIA

    https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/

  15. vLLM memory profiling: allocations and transient peaks

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/utils/mem_utils.py

  16. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    https://aclanthology.org/2023.emnlp-main.298.pdf

  17. vLLM GPU worker: profiling available KV capacity

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/worker/gpu_worker.py

  18. Automatic prefix caching — vLLM

    https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/

  19. Automatic Prefix Caching — vLLM v0.14.1

    https://docs.vllm.ai/en/v0.14.1/design/prefix_caching/

  20. vLLM block pool: reference counts and reusable cache blocks

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/core/block_pool.py

  21. Efficient Memory Management for Large Language Model Serving with PagedAttention

    https://arxiv.org/html/2309.06180v1

  22. AIPerf Metrics Reference

    https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference

  23. GenAI-Perf: latency and throughput measurement

    https://github.com/triton-inference-server/perf_analyzer/tree/main/genai-perf

  24. Utilities for Generation: TextStreamer and TextIteratorStreamer

    https://huggingface.co/docs/transformers/internal/generation_utils

  25. Module ngx_http_proxy_module — nginx

    https://nginx.org/en/docs/http/ngx_http_proxy_module.html

  26. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1251

  27. Batchers — NVIDIA Triton Inference Server

    https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/batcher.html

  28. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=5143

  29. Clipper: A Low-Latency Online Prediction Serving System

    https://www.usenix.org/system/files/conference/nsdi17/nsdi17-crankshaw.pdf

  30. Continuous batching — Transformers

    https://huggingface.co/docs/transformers/main/en/continuous_batching

  31. Orca: A Distributed Serving System for Transformer-Based Generative Models

    https://www.usenix.org/system/files/osdi22-yu.pdf

  32. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1492

  33. Compute & System Design for Next Generation Frontier Models

    https://www.youtube.com/watch?v=gFyBdBm0AGo&t=277

  34. Google SRE: Handling Overload

    https://sre.google/sre-book/handling-overload/

  35. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    https://www.usenix.org/system/files/osdi24-zhong-yinmin.pdf

  36. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

    https://www.usenix.org/system/files/osdi24-agrawal.pdf

  37. Optimization and tuning — vLLM

    https://docs.vllm.ai/en/stable/configuration/optimization/

  38. Fast Distributed Inference Serving for Large Language Models

    https://arxiv.org/pdf/2305.05920

  39. Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

    https://www.youtube.com/watch?v=EY4O9M6AsWI&t=649

  40. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=668

  41. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    https://www.usenix.org/system/files/osdi24-zhong-yinmin.pdf

  42. Generation — Transformers

    https://huggingface.co/docs/transformers/main_classes/text_generation

  43. The Curious Case of Neural Text Degeneration

    https://arxiv.org/html/1904.09751v2

  44. Hugging Face: autoregressive decoding and temperature sampling

    https://huggingface.co/blog/how-to-generate

  45. Transformers v4.50.0 GenerationMixin implementation

    https://raw.githubusercontent.com/huggingface/transformers/v4.50.0/src/transformers/generation/utils.py

  46. Reproducibility — vLLM

    https://docs.vllm.ai/en/stable/usage/reproducibility/

  47. Your Agent Failed in Prod. Good Luck Reproducing It.

    https://www.youtube.com/watch?v=Lc8zRh9muoY&t=165

  48. vLLM incremental detokenizer implementation

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/engine/detokenizer.py

  49. Structured model outputs: supported contracts and incomplete responses

    https://developers.openai.com/api/docs/guides/structured-outputs

  50. Backpressuring in Streams — Node.js

    https://nodejs.org/learn/modules/backpressuring-in-streams

  51. Fast Inference from Transformers via Speculative Decoding

    https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf

  52. Accelerating Large Language Model Decoding with Speculative Sampling

    https://arxiv.org/pdf/2302.01318

  53. Transformers v4.50.0 assisted-generation implementation

    https://raw.githubusercontent.com/huggingface/transformers/v4.50.0/src/transformers/generation/utils.py

  54. Introduction to LLM serving with SGLang

    https://www.youtube.com/watch?v=Ahtaha9fEM0&t=1624

  55. How do I benchmark an LLM engine? — Modal

    https://modal.com/llm-almanac/how-to-benchmark

  56. Bitsandbytes — Transformers

    https://huggingface.co/docs/transformers/quantization/bitsandbytes

  57. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

    https://arxiv.org/abs/2402.02750

  58. Accelerating Generative AI with PyTorch II: GPT, Fast

    https://pytorch.org/blog/accelerating-generative-ai-2/

  59. A Proof for the Queuing Formula: L = λW — John D. C. Little

    https://fisherp.scripts.mit.edu/wordpress/wp-content/uploads/2015/11/ContentServer.pdf

  60. Notes on Little's Law — Karl Sigman

    https://www.columbia.edu/~ks20/4106-18-Fall/Notes-LL.pdf

  61. High-performance LLM inference — Modal

    https://modal.com/docs/guide/high-performance-llm-inference

  62. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=4438

  63. Optimizing inference for voice models in production

    https://www.youtube.com/watch?v=gmTHs5T_YAE&t=211

  64. Model Configuration — NVIDIA Triton Inference Server

    https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html

  65. MLPerf Inference Benchmark

    https://arxiv.org/html/1911.02549v2

  66. MLPerf LoadGen: independent arrivals and explicit overload termination

    https://raw.githubusercontent.com/mlcommons/inference/master/loadgen/issue_query_controller.cc

  67. MLPerf LoadGen: arrival distribution and completion latency

    https://raw.githubusercontent.com/mlcommons/inference/master/loadgen/loadgen.cc

  68. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=5346

  69. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=4921

  70. Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA

    https://www.youtube.com/watch?v=NTBX-wxUhHs&t=985

  71. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=4514

  72. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=812

  73. A Taxonomy for Next-Generation Reasoning Models

    https://www.youtube.com/watch?v=jQcsVk0KWiQ&t=276

  74. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=920

  75. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=4576

  76. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1111