Contents
  1. Purpose and development
    1. What quantization changes
    2. Turning points
  2. Numerical representation
    1. Range and resolution
    2. Scales, rounding and clipping
    3. Which tensors change precision
    4. Sharing a scale
    5. Outliers and consequential error
  3. Preparing a quantized model
    1. Calibration and scale timing
      1. When parameters are chosen
    2. Preserving useful computation
      1. Output reconstruction
      2. Activation-informed protection
      3. Equivalent rescaling
    3. Training for quantized execution
  4. Storage and execution
    1. From nominal bits to bytes
    2. What the runtime computes
      1. Packed-weight execution
      2. Integer execution
    3. Capacity is not speed
  5. Validation and operating choices
    1. Diagnose numerical changes
    2. Measure retained task quality
    3. Choose and maintain a configuration
  6. Check understanding
  7. Open questions
  8. Selected talks
  9. References
  10. Talk library
← All topics

Quantization

Quantization represents model values with fewer bits so models can use less memory and, with suitable kernels, less costly computation. The choice is not simply “four-bit or eight-bit”: weights, activations and the KV cache can use different formats, scales and exceptions. Current deployment recipes combine these choices to fit a model and workload to particular hardware. This chapter explains the numerical approximations, calibration and execution choices, then shows how to measure their effect on memory, speed and useful behavior.

Purpose and development

What quantization changes

Quantization reduces the numerical distinctions a model can represent so its values can occupy fewer bits. Applied to weights, the numerical parameters learned during training, it can shrink a model without removing layers. Inference evaluates the model on an input using those weights, so approximating them can change the output. This tradeoff between weight storage and retained behavior motivates the quantization discussion in A Practical Guide to Efficient AI.

Several benefits are possible, and they are different claims. A smaller artifact needs less storage. Compact resident weights can leave memory for other work. Reading fewer bytes can reduce data movement. Supported low-precision arithmetic can increase computation throughput. Yet a runtime may reconstruct weights before multiplying them, and an application's memory includes more than weights. The inference resource model explains these separate allocations.

Quantization also differs from distillation, which trains a receiving model using another model's outputs as supervision. Distillation changes what a student learns; quantization specifies how numerical values are represented. The methods can be combined, but their mechanisms and costs differ. See Distillation.

A useful quantization decision therefore names three things: the resource that needs saving, the behavior that must remain acceptable, and the execution path that will deliver the saving. Choosing a bit width comes after establishing those requirements.

Turning points

Quantization grew from the problem of representing continuously varying signals with finitely many codes. Neural networks added another concern: numerical approximation could affect both a learned computation and the process that learns it. Later deployment systems had to make those representations executable on ordinary hardware, then scale conversion to increasingly large models.

DevelopmentDateContribution
Quantizing for Minimum Distortion — Joel Max, MIT Lincoln LaboratoryMarch 1960Explained how a signal's distribution should influence placement of a fixed number of reconstruction levels.
BinaryConnect — Courbariaux, Bengio and David2015Used binary weights during propagation while retaining higher-precision state for learning.
Integer-arithmetic-only inference — Benoit Jacob and colleagues, GoogleDecember 2017 preprintConnected simulated quantization during training to practical integer execution on mobile hardware.
GPTQ — Elias Frantar and colleaguesOctober 2022 preprintMade accurate one-shot conversion tractable for large pretrained transformers.
OCP Microscaling Formats — multi-company specificationSeptember 2023, version 1.0Standardized shared-scale numerical blocks while leaving their physical memory layout unspecified.

These developments address complementary problems: choosing numerical levels, learning with coarse values, converting trained models and executing the result. They coexist because solving one problem does not settle the others. Understanding the numerical representation is the first step toward choosing among them.

Numerical representation

Range and resolution

Bit width counts the bits in an encoding. Range describes the values it can represent; resolution describes the spacing between neighboring values. Fewer bits allow fewer encodings, but the format determines where those encodings are spent. With a fixed scale, integer codes reconstruct to evenly spaced values. Floating point combines a sign, an exponent that selects a magnitude scale, and a significand that distinguishes values within that scale. Its spacing generally grows with magnitude: it can cover a wide range without maintaining the same absolute resolution throughout.

FormatMeaningImportant distinction
FP3232-bit single-precision floating pointA higher-precision reference format.
FP1616-bit half-precision floating pointLess range and precision than FP32.
BF1616-bit Brain Floating PointA wider exponent field but fewer fraction bits than FP16.
INT8Eight-bit integer codes with a numerical mappingLiteRT uses −127…127 for symmetric weights, but −128…127 for activations.
INT4Four-bit integer codes with a numerical mappingThe documented TensorRT path uses −8…7.

Even “FP8” leaves choices unresolved. FP8 Formats for Deep Learning specifies E4M3 with four exponent and three fraction bits, and E5M2 with five exponent and two fraction bits, each plus a sign bit. Their maximum finite magnitudes are 448 and 57,344 respectively. More exponent range comes at the expense of fraction precision; equal width does not mean equal representable values. External scaling can also change how model values fit these encodings.

Storage, multiplication and accumulation precision are separate fields in the numerical contract, not consequences of a filename. An accumulator sums products and may be wider than its operands.

Scales, rounding and clipping

An affine quantizer uses a positive scale to set reconstructed spacing and an integer zero point to identify the code for numerical zero. Encoding rounds to a code and clips it to legal bounds. Dequantization reconstructs that code's numerical value; it cannot recover distinctions discarded during encoding.

q=clip⁡ ⁣(round⁡(x/s)+z, qmin⁡,qmax⁡),x^=s(q−z).q=\operatorname{clip}\!\left(\operatorname{round}(x/s)+z,\ q_{\min},q_{\max}\right),\qquad \hat{x}=s(q-z). Here xx is the original value, qq its code, s>0s>0 the scale, zz the zero point, and x^\hat{x} the reconstruction. Clipping saturates at the code bounds. Use nearest-even rounding here: an exact tie selects the even integer.

For a small example, use codes −2…2, scale 0.5 and zero point 0. The five reconstruction levels span −1…1. This is a teaching grid, not a named storage format.

Many inputs share one reconstruction

Example

Horizontal intervals merge inputs; filled endpoints show which level owns each exact nearest-even tie.

Rounding intervals and code clipping

Scale 0.5; zero point 0; codes −2…2. Open circles exclude an endpoint; filled circles include it. Exact ties −1.25, −0.75, −0.25, 0.25, 0.75 and 1.25 reconstruct to −1, −1, 0, 0, 1 and 1. Clipping changes the rounded code only when x < −1.25 or x > 1.25; equality is excluded. Worked encodings (original → rounded code → stored code → reconstruction): 0 → 0 → 0 → 0; 0.3 → 1 → 1 → 0.5; 1.4 → 3 → 2 → 1.

-1.6-0.800.81.6-1.6-0.800.81.6Original value x (dimensionless)Reconstructed value (dimensionless)Identity: no reconstruction error−1 for x ≤ −0.75−0.5 for −0.75 < x < −0.250 for −0.25 ≤ x ≤ 0.250.5 for 0.25 < x < 0.751 for x ≥ 0.75Clipping changes code: x < −1.25Clipping changes code: x > 1.25Open endpoints: tie belongs to the neighboring levelFilled endpoints: even code owns the exact tieWorked reconstructions−1−0.500.510 → 00.3 → 0.51.4 → 1−1.251.25
  • 1. Identity: no reconstruction error
  • 2. −1 for x ≤ −0.75
  • 3. −0.5 for −0.75 < x < −0.25
  • 4. 0 for −0.25 ≤ x ≤ 0.25
  • 5. 0.5 for 0.25 < x < 0.75
  • 6. 1 for x ≥ 0.75
  • 7. Clipping changes code: x < −1.25
  • 8. Clipping changes code: x > 1.25
  • 9. Open endpoints: tie belongs to the neighboring level
  • 10. Filled endpoints: even code owns the exact tie
  • 11. Worked reconstructions
Read coordinates and regions as data

X: -1.6–1.6 dimensionless; Y: -1.6–1.6 dimensionless, increasing up. Equal scale on both axes.

Identity: no reconstruction error (polyline)

(-1.5, -1.5); (1.5, 1.5)

−1 for x ≤ −0.75 (polyline)

(-1.5, -1); (-0.75, -1)

−0.5 for −0.75 < x < −0.25 (polyline)

(-0.75, -0.5); (-0.25, -0.5)

0 for −0.25 ≤ x ≤ 0.25 (polyline)

(-0.25, 0); (0.25, 0)

0.5 for 0.25 < x < 0.75 (polyline)

(0.25, 0.5); (0.75, 0.5)

1 for x ≥ 0.75 (polyline)

(0.75, 1); (1.5, 1)

Clipping changes code: x < −1.25 (polyline)

(-1.5, -1); (-1.25, -1)

Clipping changes code: x > 1.25 (polyline)

(1.25, 1); (1.5, 1)

Open endpoints: tie belongs to the neighboring level (points, open: excluded)

(-0.75, -0.5); (-0.25, -0.5); (0.25, 0.5); (0.75, 0.5)

Filled endpoints: even code owns the exact tie (points, closed: included)

(-1.25, -1); (-0.75, -1); (-0.25, 0); (0.25, 0); (0.75, 1); (1.25, 1)

Worked reconstructions (points)

(0, 0); (0.3, 0.5); (1.4, 1)

−1: (-1, -1.12)

−0.5: (-0.5, -0.62)

0: (-0.1, 0.13)

0.5: (0.65, 0.38)

1: (1, 0.88)

0 → 0: (0.05, -0.23)

0.3 → 0.5: (0.1, 0.77)

1.4 → 1: (1.5, 1.25)

−1.25: (-1.25, -1.35)

1.25: (1.25, 1.52)

Horizontal intervals map inputs to five reconstruction levels in [−1, 1]. Orange segments mark where clipping changes the rounded code: strictly beyond ±1.25. The filled blue threshold points remain legal without clipping.

Code bounds constrain the stored integer; reconstruction bounds describe its decoded values. These boundaries act at different steps of the mapping.

A symmetric mapping centers zero at code zero. An asymmetric mapping can shift the represented interval using a nonzero zero point; it gains no additional codes. LiteRT uses symmetric weights but allows asymmetric activations. Exact zero remains useful: zero-valued padding should not become an unintended nonzero input.

The equally spaced levels of an affine mapping are a design choice. In his 1960 squared-error analysis, Joel Max placed decision boundaries midway between neighboring reconstruction levels and each level at the probability-weighted mean of inputs in its interval. That lets level placement reflect the input distribution; uniform spacing is not generally the minimum-distortion choice. Minimizing this numerical distortion still does not establish neural-network task quality. Rounding can also affect learning. A 1991 Carnegie Mellon report, describing experiments from fall 1990, found that discarding small weight updates could stall training. Rescaling and probabilistic rounding helped in its tested networks. The latter sometimes preserves an update's effect instead of always rounding it away.

Which tensors change precision

A tensor is a multidimensional numerical array. Weights are reusable parameter tensors. Activations are intermediate values computed from the current input—not the functions, such as ReLU, that transform them. A linear layer forms weighted combinations of incoming values; its inputs, weights and outputs can have different representations. Transformer arrays and parameters provides the model context.

Weight-only quantization compresses weights while leaving incoming activations at higher precision. W4A16 denotes four-bit weights and sixteen-bit activations; W8A8 denotes eight-bit weights and activations. Neither label specifies integer versus floating-point encodings, scales, grouping, accumulator width or exceptions. A documented INT4 weight-only path reconstructs weights before higher-precision multiplication.

The key-value cache, or KV cache, retains attention representations derived from processed tokens so later steps can reuse them. It is neither a weight tensor nor stored text. Its lifetime and precision policy are separate choices. The TensorRT-LLM workshop demonstrates independently configured weight and cache precision. Quantizing one does not establish what happened to the other.

Sharing a scale

Granularity specifies which entries share quantization parameters. For a weight matrix written with outputs in rows and inputs in columns, each row produces one output channel. Per-tensor quantization shares one scale across the matrix; per-output-channel quantization gives each row its own. Block or group quantization partitions it further. Per-token activation scaling instead groups the features at one token position. Record the axis and group size: transposing a matrix changes what a row means.

Consider the matrix with rows [1, 3] and [4, 12]. Use three-bit storage with symmetric codes −3…3, zero point zero and nearest-even rounding; one encoding is unused. Compare one global scale of 4 with separate row scales of 1 and 4.

Change the sharing boundary

Example

Separating unequal row ranges improves reconstruction without changing the entries or code width.

One scale: 4

Entries are labeled original → reconstructed.

0.30.91.52.12.70.10.751.42.052.7Input column (index)Output row (index)Shared scale 4Matrix entries1 → 03 → 44 → 412 → 12
  • 1. Shared scale 4
  • 2. Matrix entries
Read coordinates and regions as data

X: 0.3–2.7 index; Y: 0.1–2.7 index, increasing down. Equal scale on both axes.

Shared scale 4 (polygon)

(0.6, 0.6); (2.4, 0.6); (2.4, 2.4); (0.6, 2.4)

Matrix entries (points)

(1, 1); (2, 1); (1, 2); (2, 2)

1 → 0: (1, 1.23)

3 → 4: (2, 1.23)

4 → 4: (1, 2.23)

12 → 12: (2, 2.23)

Two scales: 1 and 4

Only the sharing boundary and scales change.

0.30.91.52.12.70.10.751.42.052.7Input column (index)Output row (index)Row 1: scale 1Row 2: scale 4Matrix entries1 → 13 → 34 → 412 → 12
  • 1. Row 1: scale 1
  • 2. Row 2: scale 4
  • 3. Matrix entries
Read coordinates and regions as data

X: 0.3–2.7 index; Y: 0.1–2.7 index, increasing down. Equal scale on both axes.

Row 1: scale 1 (polygon)

(0.6, 0.6); (2.4, 0.6); (2.4, 1.4); (0.6, 1.4)

Row 2: scale 4 (polygon)

(0.6, 1.6); (2.4, 1.6); (2.4, 2.4); (0.6, 2.4)

Matrix entries (points)

(1, 1); (2, 1); (1, 2); (2, 2)

1 → 1: (1, 1.23)

3 → 3: (2, 1.23)

4 → 4: (1, 2.23)

12 → 12: (2, 2.23)

The same matrix uses codes −3…3. One global scale gives total absolute error 2; two row scales give zero error. Regions mark shared scales, not memory size.

Smaller groups can resolve unequal ranges, but add metadata and constrain execution. Scales along an output dimension can sometimes be applied after a dot product; scales varying inside its summed dimension require different handling. SmoothQuant explicitly designs its scaling axes around its efficient integer matrix-multiplication path. The finest grouping is not automatically the best supported configuration.

Outliers and consequential error

An outlier is unusually large relative to the values sharing its scale. Accommodating it expands the range and spreads a fixed number of levels farther apart. Clipping it permits finer central resolution but distorts the extreme. Neither choice is intrinsically safer: numerical rarity is not functional unimportance.

For example, with codes −3…3 and zero point zero, [1, 2, 12] at scale 4 reconstructs as [0, 0, 12] under nearest-even rounding. At scale 1 it becomes [1, 2, 3]. The smaller values improve, but the last value loses 9. This arithmetic cannot decide which representation preserves the model's answer; subsequent operations determine how those perturbations matter.

LLM.int8() addressed systematic large activation features observed in its 2022 transformer experiments. Selected outlier dimensions use FP16 computation; the rest use INT8 with wider accumulation, followed by combination. This mixed-precision response preserves selected computation instead of clipping it. Its outlier thresholds are experimental choices, not universal constants.

Finer grouping, higher-precision exceptions and range redistribution solve different aspects of the problem. Their effects can interact: layers that appear safe when quantized separately may fail in combination. The Compression at the Edge discussion highlights this combinatorial difficulty. Diagnose locally, then test the combined model rather than adding isolated sensitivity results.

Preparing a quantized model

Calibration and scale timing

Post-training quantization, or PTQ, derives a quantized representation from an already-trained model without ordinary task retraining. It can still involve substantial preparation. Some conversions use weight statistics alone; others observe model inputs to choose ranges or improve reconstruction. Calibration data means representative inputs used for those choices. Labels are often unnecessary. This differs from probability calibration, which concerns whether predicted probabilities match outcome frequencies.

A range observer collects numerical statistics. Min–max calibration covers observed extrema; histogram, percentile or error-based choices can sacrifice tails for better resolution elsewhere. Different observers can produce different mappings from the same tensors. More calibration examples do not repair systematically wrong coverage.

Preserve deployment preprocessing and cover relevant tasks, languages and sequence lengths. Calibrating Beyond English compared calibration languages at fixed token budgets for grouped four-bit GPTQ and AWQ. Language-matched or multilingual data often helped, but benefits differed by model, quantizer and downstream task. The practical lesson is to evaluate coverage choices, not assume one universal mixture.

When parameters are chosen

Scale policySource of parametersConsequence
StaticPreparation-time calibrationReuses fixed parameters; unexpected ranges can saturate.
DynamicCurrent inference valuesAdapts to current ranges; adds range calculation and conversion work.

Dynamic scaling does not update learned weights. It is also independent of bit width and grouping: a per-token scale may be calculated at runtime, while weight scales remain fixed. Some formats combine offline global scales with runtime local scales. Conversely, practitioners sometimes call selective layer precision “dynamic quantization.” Specify the actual behavior rather than relying on that label.

Keep preparation inputs, development cases used to select configurations, and final assessment cases distinct. Repeatedly adjusting a quantizer after inspecting test results makes those cases development data. Independent assessment explains the boundary.

Preserving useful computation

Independent rounding minimizes each weight's local error without considering its use. Better conversion can instead preserve layer outputs, protect weights associated with important input activity, or redistribute difficult ranges between equivalent operands. These approaches differ in what they optimize and what their runtime must execute.

Output reconstruction

GPTQ targets layer-output reconstruction: keeping the layer's outputs close to their original values on observed inputs. It rounds weights and adjusts remaining weights to compensate. Second-order information describes how this reconstruction error curves as weights change, helping choose the corrections. GPTQ derives it from calibration inputs. Its original method quantizes weights, not activations. Processing blocks sequentially lets later blocks receive outputs from already-quantized predecessors.

min⁡W^  ∥WX−W^X∥F2.\min_{\hat W}\;\lVert WX-\hat W X\rVert_F^2. Here columns of XX contain calibration inputs, WW is the original weight matrix, and W^\hat W is its quantized reconstruction. The squared Frobenius norm sums squared output differences. This objective weights errors by observed use; it remains a surrogate for final task quality.

Activation-informed protection

Activation-aware Weight Quantization, or AWQ, appeared as a June 2023 preprint. Activation magnitudes help identify consequential weight channels. AWQ scales weights and inversely scales inputs, searching scaling choices against layer-output differences on calibration inputs and selecting weight clipping. It protects channels without storing them separately in FP16. Excessive scaling can enlarge a group's range and harm neighboring weights. “Activation-aware” describes information used during preparation; the method remains weight-only. Avoiding backpropagation does not mean ignoring reconstruction error.

Equivalent rescaling

Range redistribution also has a pre-transformer history. Qualcomm's 2019 data-free quantization work addressed deployment without customer training data. For compatible ReLU networks, paired cross-layer scaling preserved the unquantized computation while balancing weight ranges; biases also needed corresponding treatment. The equivalence does not apply to arbitrary nonlinearities.

SmoothQuant, from Guangxuan Xiao, Ji Lin and colleagues at MIT and NVIDIA, appeared in November 2022. It sought hardware-friendly INT8 execution without a separate higher-precision outlier path.

XW=(XD−1)(DW),D=diag⁡(s).XW=(XD^{-1})(DW),\qquad D=\operatorname{diag}(s). Here XX has inputs in rows, WW maps input features to outputs, and DD is a diagonal matrix of positive feature scales. Dividing activation features and multiplying corresponding weight rows preserves the product before quantization.

Calibration chooses how much range difficulty moves from activations to weights. Compatible preceding operations absorb scaling offline. The quantized operands now incur different errors: algebraic equality before quantization is not equality afterward.

Pair activation columns with weight rows

Feature k is a vertical strip in X and a horizontal strip in W. Reciprocal scales cancel in their product before quantization.

Before rescaling: XW = Y

X has M input rows and K feature columns; W has K feature rows and N output columns. The highlighted column k of X pairs with the highlighted row k of W in their product. Ellipses indicate omitted entries; the grid is a symbolic window, not a numerical 3×3 example.

0123402.3754.757.1259.5Symbolic column positions (layout)Matrix arrangement (layout)Activation feature k: column kWeight feature k: row kX: symbolic matrix gridW: symbolic matrix gridX · M×K — feature column k⋯X[1,k]⋯⋯X[i,k]⋯⋯X[M,k]⋯W · K×N — feature row k⋮⋮⋮W[k,1]W[k,j]W[k,N]⋮⋮⋮Product Y · M×N
  • 1. Activation feature k: column k
  • 2. Weight feature k: row k
  • 3. X: symbolic matrix grid
  • 4. W: symbolic matrix grid
Read coordinates and regions as data

X: 0–4 layout; Y: 0–9.5 layout, increasing down. Axes scaled independently; screen angles and distances are not comparable.

Activation feature k: column k (polygon)

(1.5, 1); (2.5, 1); (2.5, 4); (1.5, 4)

Weight feature k: row k (polygon)

(0.5, 6.3); (3.5, 6.3); (3.5, 7.3); (0.5, 7.3)

X: symbolic matrix grid (polyline)

(0.5, 1); (3.5, 1); (3.5, 4); (0.5, 4); (0.5, 1); (1.5, 1); (1.5, 4); (2.5, 4); (2.5, 1); (3.5, 1); (3.5, 2); (0.5, 2); (0.5, 3); (3.5, 3)

W: symbolic matrix grid (polyline)

(0.5, 5.3); (3.5, 5.3); (3.5, 8.3); (0.5, 8.3); (0.5, 5.3); (1.5, 5.3); (1.5, 8.3); (2.5, 8.3); (2.5, 5.3); (3.5, 5.3); (3.5, 6.3); (0.5, 6.3); (0.5, 7.3); (3.5, 7.3)

X · M×K — feature column k: (2, 0.7)

⋯: (1, 1.85)

X[1,k]: (2, 1.85)

⋯: (3, 1.85)

⋯: (1, 2.85)

X[i,k]: (2, 2.85)

⋯: (3, 2.85)

⋯: (1, 3.85)

X[M,k]: (2, 3.85)

⋯: (3, 3.85)

W · K×N — feature row k: (2, 5)

⋮: (1, 6.15)

⋮: (2, 6.15)

⋮: (3, 6.15)

W[k,1]: (1, 7.15)

W[k,j]: (2, 7.15)

W[k,N]: (3, 7.15)

⋮: (1, 8.15)

⋮: (2, 8.15)

⋮: (3, 8.15)

Product Y · M×N: (2, 9.2)

After rescaling: (XD⁻¹)(DW) = Y

Every activation column divides by its positive diagonal scale and the matching weight row multiplies by that scale. The highlighted feature k uses sₖ > 0: X[:,k]/sₖ and sₖW[k,:]. D acts on the shared K dimension. The product remains Y before quantization.

0123402.3754.757.1259.5Symbolic column positions (layout)Matrix arrangement (layout)Activation feature k: column kWeight feature k: row kX: symbolic matrix gridW: symbolic matrix gridXD⁻¹ · M×K — column k ÷ sₖ⋯X[1,k]/sₖ⋯⋯X[i,k]/sₖ⋯⋯X[M,k]/sₖ⋯DW · K×N — row k × sₖ⋮⋮⋮sₖW[k,1]sₖW[k,j]sₖW[k,N]⋮⋮⋮Product Y · M×N
  • 1. Activation feature k: column k
  • 2. Weight feature k: row k
  • 3. X: symbolic matrix grid
  • 4. W: symbolic matrix grid
Read coordinates and regions as data

X: 0–4 layout; Y: 0–9.5 layout, increasing down. Axes scaled independently; screen angles and distances are not comparable.

Activation feature k: column k (polygon)

(1.5, 1); (2.5, 1); (2.5, 4); (1.5, 4)

Weight feature k: row k (polygon)

(0.5, 6.3); (3.5, 6.3); (3.5, 7.3); (0.5, 7.3)

X: symbolic matrix grid (polyline)

(0.5, 1); (3.5, 1); (3.5, 4); (0.5, 4); (0.5, 1); (1.5, 1); (1.5, 4); (2.5, 4); (2.5, 1); (3.5, 1); (3.5, 2); (0.5, 2); (0.5, 3); (3.5, 3)

W: symbolic matrix grid (polyline)

(0.5, 5.3); (3.5, 5.3); (3.5, 8.3); (0.5, 8.3); (0.5, 5.3); (1.5, 5.3); (1.5, 8.3); (2.5, 8.3); (2.5, 5.3); (3.5, 5.3); (3.5, 6.3); (0.5, 6.3); (0.5, 7.3); (3.5, 7.3)

XD⁻¹ · M×K — column k ÷ sₖ: (2, 0.7)

⋯: (1, 1.85)

X[1,k]/sₖ: (2, 1.85)

⋯: (3, 1.85)

⋯: (1, 2.85)

X[i,k]/sₖ: (2, 2.85)

⋯: (3, 2.85)

⋯: (1, 3.85)

X[M,k]/sₖ: (2, 3.85)

⋯: (3, 3.85)

DW · K×N — row k × sₖ: (2, 5)

⋮: (1, 6.15)

⋮: (2, 6.15)

⋮: (3, 6.15)

sₖW[k,1]: (1, 7.15)

sₖW[k,j]: (2, 7.15)

sₖW[k,N]: (3, 7.15)

⋮: (1, 8.15)

⋮: (2, 8.15)

⋮: (3, 8.15)

Product Y · M×N: (2, 9.2)

Here inputs occupy rows, unlike the earlier output-row weight convention. D is diagonal with positive entries; sₖ > 0 scales the matched feature k. XW = (XD⁻¹)(DW) is mathematical equality before quantization, not a promise of bitwise floating-point equality. Quantizing the transformed operands is a separate step: their new rounding errors need not preserve Y.

Training for quantized execution

Quantization-aware training, or QAT, adjusts parameters while exposing forward computation to quantization effects. Fake quantization rounds and clamps values, then reconstructs them in floating-point tensors. It simulates representational loss without creating the final packed artifact.

Ordinary rounding has zero derivative between boundaries and is nondifferentiable at them. A straight-through estimator substitutes an approximate gradient so training can continue. This is not the true derivative of quantization. Higher-precision parameters can accumulate small updates even while forward values occupy a coarse grid. See parameter updates for the underlying optimization process.

Retain updates between rounding boundaries

Training updates higher-precision parameters, while forward computation consumes their coarse reconstructions.

Retained W_tHigher-precision parameter state preserves small updates between training steps.
Fake quantizationRound, clamp and reconstruct using the training quantization mapping.
Reconstructed forward valuesFloating-point tensors restricted to the reconstruction levels; distinct from retained parameter state.
Forward computation → lossThe model uses reconstructed values to compute the training loss.
STE backward approximationA straight-through estimator substitutes a gradient through the fake-quantization boundary. It is not the true derivative of rounding.
Optimizer updateCombines the approximate gradient with retained higher-precision parameters.
Retained W_(t+1)Updated higher-precision state supplies the next training step. Forward reconstructions do not overwrite it.
Small updates can persist in W even when its forward reconstruction stays unchanged. Dashed arrows carry the approximate backward gradient; solid arrows carry forward values and retained state.
Read the diagram as text
  • Retained W_t. Higher-precision parameter state preserves small updates between training steps.
  • Fake quantization. Round, clamp and reconstruct using the training quantization mapping.
  • Reconstructed forward values. Floating-point tensors restricted to the reconstruction levels; distinct from retained parameter state.
  • Forward computation → loss. The model uses reconstructed values to compute the training loss.
  • STE backward approximation. A straight-through estimator substitutes a gradient through the fake-quantization boundary. It is not the true derivative of rounding.
  • Optimizer update. Combines the approximate gradient with retained higher-precision parameters.
  • Retained W_(t+1). Updated higher-precision state supplies the next training step. Forward reconstructions do not overwrite it.
  • Retained W_t → Fake quantization: Forward: parameters.
  • Fake quantization → Reconstructed forward values: Forward: reconstruction.
  • Reconstructed forward values → Forward computation → loss: Forward: model evaluation.
  • Forward computation → loss → STE backward approximation: Backward: loss gradient.
  • STE backward approximation → Optimizer update: Backward: approximate gradient.
  • Retained W_t → Optimizer update: Retained update state.
  • Optimizer update → Retained W_(t+1): Updated parameters.

BinaryConnect separated binary propagation weights from higher-precision update state. Binarized Neural Networks, by Courbariaux, Hubara, Soudry, El-Yaniv and Bengio in 2016, extended binary representations to activations. Its substituted gradient passed information within a bounded interval and canceled it outside. Binary dot products could use XNOR and population counts, but neither every operation nor all training state became binary.

TorchAO's QAT workflow separates preparation, training and conversion. The deployed mapping must match the trained simulation closely enough to test. TensorFlow's QAT guide explicitly distinguishes supported deployment configurations from experimental simulations without a supported export path.

Adaptation also needs suitable data and compute. Quantization-aware distillation adds teacher supervision to recovery training; the edge-compression panel reports that mismatched recovery data can damage reasoning models with capabilities acquired through multiple training stages. QAT is therefore an additional intervention to justify and evaluate, not a guaranteed repair after aggressive PTQ.

Storage and execution

From nominal bits to bytes

Bit packing places multiple low-bit codes into larger storage units. For NN values with bb-bit payloads, payload storage is approximately Nb/8Nb/8 bytes. One billion sixteen-bit values require 2 GB; four-bit values require 0.5 GB, using decimal units. Neither estimate includes scales, zero points, padding, container information or higher-precision exceptions.

Effective bits per value includes representation overhead under a stated accounting boundary. Two four-bit floating-point schemes illustrate why this matters. Both use E2M1 elements—one sign, two exponent and one fraction bit—but their shared scales differ.

RepresentationBlock fieldsBits per valueAdditional boundary
NVFP416 × 4-bit elements + one 8-bit E4M3 scale4 + 8/16 = 4.5Also has an FP32 tensor scale.
MXFP4, OCP MX v1.032 × 4-bit elements + one 8-bit E8M0 scale4 + 8/32 = 4.25Specification does not fix physical packing.

These are encoded-field calculations, not device-memory measurements. Artifact size describes a serialized object; resident weight memory describes loaded parameters; total runtime memory includes other allocations; peak memory is the maximum over a specified interval. Loading, repacking or conversion may create temporary representations. Use the inference memory ledger to keep these boundaries separate.

A concrete example comes from MobileAIBench. Its four-bit TinyLlama 1B artifact was reported as 0.6 GB on disk, while a HotpotQA run in its iPhone 14 application used 3.34 GiB RAM. The units and boundaries differ. The RAM figure is application usage, not isolated weights or a demonstrated startup peak; the comparison does not measure a speedup over higher precision.

What the runtime computes

A kernel implements a numerical operation; on a GPU it executes device-side parallel work. The kernel chapter explains that execution model. Here the central distinction is between compact storage followed by reconstruction and actual low-precision multiplication.

Packed-weight execution

Weight-only execution can reconstruct packed weights as they are consumed, avoiding a complete expanded weight buffer. Marlin rearranges weights and scales offline and schedules dequantization alongside matrix computation. A separate conversion that materializes expanded weights has different traffic and memory costs. Fusion describes where work occurs, not a new numerical format.

Integer execution

Integer execution centers codes by subtracting their zero points, then sums their products. Multiplying by both operand scales restores numerical units. Requantization encodes the result in the next operation's quantized mapping, when required.

A=∑i(qxi−zx)(qwi−zw),y^=sxswA,A=\sum_i(q_{x_i}-z_x)(q_{w_i}-z_w),\qquad \hat y=s_xs_wA, qy=clip⁡ ⁣(round⁡ ⁣(sxswsyA)+zy).q_y=\operatorname{clip}\!\left(\operatorname{round}\!\left(\frac{s_xs_w}{s_y}A\right)+z_y\right). The subscripts identify input, weight and output mappings. AA is the accumulator; clipping uses the output code bounds. Bias is omitted here.

Centered operands [2, −1] and [3, 4] give A=6−4=2A=6-4=2. Scales 0.5 and 0.25 yield y^=0.25\hat y=0.25; output scale 0.125 and zero point 0 yield code 2 without clipping.

Wider arithmetic is part of the contract. LiteRT's INT8 convolution uses INT32 bias with scale equal to input scale times the corresponding weight scale. Its zero-centered weights also eliminate a runtime-dependent zero-point correction term. Other operators impose additional restrictions, such as matching quantization parameters across concatenated tensors.

Conceptual memory and kernel boundaries, not a literal chip layout. The fused path uses tile working storage instead of a full expanded weight buffer; its accumulator type remains kernel-specific. The integer route rescales with the input, weight and output mappings. Bias is omitted, and the alternatives need not produce equal outputs.

The artifact, converter, runtime and device must agree on layouts and operations. LiteRT conversion may retain floating-point operators, whereas requiring integer-only operators exposes unsupported conversion. TensorRT can fail to build an unsupported low-precision path. Verify the selected implementation rather than treating successful loading as proof of acceleration. Implementation-layer choices belong to this execution decision.

Capacity is not speed

Memory capacity limits what can remain allocated. Memory bandwidth limits how quickly bytes can move. Arithmetic throughput limits the rate of computation. Quantization can relieve one without relieving the others. Smaller weights may make a model fit or leave room for more simultaneous sequences while conversion work or unchanged operations still determine latency.

Prefill processes known prompt positions; decode generates continuations incrementally. At small batches, decode may repeatedly load weights for relatively little arithmetic. Processing more positions together can reuse those weights. The relevant quantity is arithmetic intensity, work per byte crossing a memory boundary. Recall prefill and decode and arithmetic-versus-traffic bounds; neither phase has one universal bottleneck.

Current deployment recipes make precision a choice across several parts of the model. NVIDIA Model Optimizer separates the model-body scheme from the KV-cache scheme and provides both weight-only and weight-and-activation formats. A four-bit checkpoint therefore does not tell you which operations run at four bits or how much memory a long conversation will use. Read the precision map and confirm runtime support before predicting a benefit.

Three precision choices in current recipes—not a performance ranking.
Recipe familyWhat changesWhat to check
Weight-only W4A16Four-bit weights with 16-bit activationsPacked layout, dequantization and supported kernels
NVFP4 W4A4Four-bit weights and activations in selected layersCalibration, layer exceptions and target hardware support
Separate KV-cache precisionStored attention keys and valuesLong-context memory use and task quality

Kernels can change that balance. Marlin's published A10 comparison used group size 128 and a large, favorably partitioned matrix. It reported retaining substantial weight-traffic benefits across larger batches where competing four-bit kernels lost relative speed. These are kernel-level results under stated shapes, not endpoint latency or quality-qualified serving throughput.

Measure first-token delay, token cadence, complete-request latency and throughput separately at relevant lengths and concurrency. Keep startup measurements separate from warm execution. A compact file does not establish loading time or peak startup memory, and a faster loaded model does not establish how quickly a new replica becomes ready.

Validation and operating choices

Diagnose numerical changes

A difference from the higher-precision model is not automatically a bug: quantization deliberately changes representable values. First establish whether the deployed path implements the intended quantized computation. Then measure the approximation it introduces. Finally assess the resulting behavior.

ComparisonReferenceSupported conclusion
Implementation agreementSame mapping, grouping and arithmetic assumptionsThe deployed operation agrees within justified tolerances.
Numerical approximationHigher-precision tensors or layer outputsWhere and how much numerical values changed.
Task behaviorMatched cases with independent outcome checksWhether required work remains successful.

Use the supported-domain testing approach, including rounding boundaries, saturation, grouping axes and supported shapes. A tolerance is an allowance, not an explanation. For finite values, a common check accepts ∣a−e∣≤atol+rtol∣e∣|a-e|\leq\mathrm{atol}+\mathrm{rtol}|e|, where aa is actual and ee expected. Absolute tolerance controls behavior near zero; library defaults do not establish an application's error budget.

Compare intermediate outputs and try controlled higher-precision exceptions to locate a consequential change. Recheck the complete configuration because exceptions and quantized layers interact. Execution optimizations themselves can alter numerics. TensorRT-LLM documents a Hopper FP8 fusion that combines matrix multiplication with the following activation function, SwiGLU. This implementation discards a scaling factor and can slightly reduce accuracy. An unchanged “FP8” label therefore does not guarantee unchanged behavior.

Measure retained task quality

A paired comparison evaluates baseline and candidate on the same cases. Keep prompts, preprocessing, generation settings and assessment policy fixed when isolating quantization. Repeat variable runs and preserve their uncertainty rather than treating one changed completion as a stable effect. Matched-work comparisons explains the design.

Perplexity measures how much probability a language model assigns to the tokens in an evaluation text. It exponentiates the average negative log probability assigned to each observed token; assigning those tokens higher probabilities lowers perplexity. This measures token prediction, not complete task success. Accuracy is Not All You Need also examines answer flips between correct and incorrect in either direction. Gains and regressions can cancel in aggregate accuracy. Flips omit changes between two incorrect answers, and matching the baseline does not make an answer correct.

In this illustrative set of 100 paired cases, both versions score 80%, but ten previous successes fail.
Baseline outcomeCandidate correctCandidate incorrect
Correct7010 regressions
Incorrect10 gains10

A task slice is a meaningful subset, such as long inputs, a language or a critical action type. Cache precision can matter especially at longer lengths. In Evaluating Quantized Large Language Models, the LongEval task tests retrieval of a requested value from key-value pairs in a long input. The following Vicuna-7B results use 16K-token inputs and the study's grouped asymmetric weight and cache quantization protocol.

Weight precisionKV precisionReported score
FP16 baselineFP16 baseline57.80%
8-bit8-bit56.40%
8-bit4-bit37.00%

Holding eight-bit weights fixed, changing the cache to four bits reduced the score by 19.40 percentage points. This is a quality result, not a speed measurement, and its magnitude need not transfer to another model. It demonstrates why short-context or weight-only checks can miss a consequential regression.

Finally, run the intended application harness—the software that executes complete tasks and records outcomes. Benchmark retention can coexist with altered multi-step behavior. The edge-compression panel describes both benchmark and harness testing, including retaining higher precision when smaller models became less usable. On devices, also examine latency, hardware use and battery drain alongside task and safety behavior.

Choose and maintain a configuration

Start with required behavior and the limiting resource. Filter out representations the target stack cannot execute. Try PTQ, representative calibration and selective precision before committing to adaptation whose data and compute costs may exceed the saving. QAT becomes a candidate when recovery is valuable and its deployment path is supported—not simply because it is more elaborate.

A configuration is Pareto-dominated if another candidate is at least as good on every compared objective and strictly better on at least one. The Pareto frontier contains candidates for which no such alternative exists among those examined. Compare measurements from the same operating conditions and account for uncertainty. A lower-memory candidate with worse latency may remain useful; one that violates required quality is infeasible regardless of its speed. Hacking the Inference Pareto Frontier frames this around application-specific operating requirements.

For each candidate, maintain separate measurements rather than combining unrelated published results into a ranking.
Decision fieldRecordAcceptance basis
QualityPaired outcomes, critical regressions and slicesRequired task behavior
MemoryResident and peak measurements at named boundariesActual deployment ceiling
LatencyFirst-token, token cadence and completion distributionsInteractive or batch response requirement
ThroughputCompleted work under stated load and qualityRequired sustainable service rate
Preparation and maintenanceConversion, adaptation, evaluation and debugging effortBenefit sufficient to justify ongoing work

Record the exact model revision; numerical encodings; scale and zero-point types; grouping axes and sizes; scale timing; calibration provenance; higher-precision exceptions; cache policy; and converter, runtime, compiler and device versions. Preserve workload and assessment records too. Evaluation manifests and serving benchmark contracts supply the broader reproducibility framework. An unavailable measurement remains unavailable, not zero.

Retest the affected claims when model weights, calibration coverage, precision policy, runtime kernels or workload change. Language coverage can alter conversion quality; a fusion can alter arithmetic; batching can alter performance. The justified operating choice is the complete measured configuration, not its nominal bit width. Broader lifetime economics belong in AI Cost and Performance Engineering.

Open questions

  1. Predicting behavioral sensitivity remains difficult because numerical errors interact across layers and tasks. Progress would make precision allocation cheaper while predicting critical regressions on independent workloads, not merely matching reconstruction scores.

  2. Calibration must cover changing languages, tasks and input lengths without becoming prohibitively expensive. Progress would identify compact preparation sets whose benefits survive distribution changes and downstream assessment.

  3. Recovery training needs data that preserves capabilities acquired through multiple training stages. Progress would provide reproducible ways to recover quantized behavior without sacrificing specialist abilities or requiring inaccessible original training mixtures.

  4. Portable low-precision deployment requires agreement beyond numerical encodings: packing, operation support and conversion behavior still vary. Progress would connect shared format specifications to verifiable execution contracts across backends.

Follow the curated reading path through the speakers and demonstrations behind this entry.

46 min

AI Engineer World's Fair 2026 · 2026

Compression at the Edge

Chris Alexiuk · Daniel Han · Asma Beevi · Merve Noyan · Parth Sareen

Cited in this entry

Develops practical sensitivity, selective precision, recovery-data difficulties and application-harness validation beyond headline benchmark scores.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

29 matching talks

TalkSpeakerEventYear
Nan JiangAI Engineer World's Fair 20262026
Marah Abdin, Robert McHardyAI Engineer World's Fair 20262026
Ziv IlanAI Engineer Europe 20262026
Daniel HanAI Engineer World's Fair 20262026
Natalie SerrinoAI Engineer Code 20252025
Cormac BrickAI Engineer Europe 20262026
Building AI For All

Transcript reviewed

Amjad Masad, Michele CatastaAI Engineer Summit 20232023
Charles FryeAI Engineer World's Fair 20252025
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Bertrand CharpentierAI Engineer Europe 20262026
Charles FryeAI Engineer World's Fair 20252025
Vibhu SapraAI Engineer World's Fair 20252025
Ishan AnandAI Engineer World's Fair 20252025
Hamed Firooz, Maziar SanjabiAI Engineer World's Fair 20252025
Maxime LabonneAI Engineer World's Fair 20242024
Alex CheemaAI Engineer Europe 20262026
Philip Kiely, Yineng ZhangAI Engineer World's Fair 20252025
Stephen Hood, Justine TunneyAI Engineer World's Fair 20242024
Philip KielyAI Engineer World's Fair 20252025
Tengyu MaAI Engineer World's Fair 20252025
Daniel HanAI Engineer World's Fair 20252025
Cormac BrickAI Engineer World's Fair 20262026
Hiral Shah, Sean SodhaEvent unknownUnknown
Du’An Lightfoot, Khaja OmerAI Engineer World's Fair 20262026
Dylan Bristot, Sujee ManiyamAI Engineer Code 2025Unknown
Morgan SuoEvent unknownUnknown
Harshul Jain, Tanmay SahEvent unknownUnknown
David AbitbolEvent unknownUnknown
Paige BaileyAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
18 processed in full · 4 in the curated path
Automated source review
Passed
Metadata candidates
15 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. ONNX QuantizeLinear: quantization granularity

    https://onnx.ai/onnx/operators/onnx__QuantizeLinear.html

  2. Learning with Limited Numerical Precision Using the Cascade-Correlation Algorithm

    https://citeseerx.ist.psu.edu/document?doi=ed7eaefb93994bfee42309bbd7cda541c08696ba&repid=rep1&type=pdf

  3. Deep Learning with Limited Numerical Precision

    https://proceedings.mlr.press/v37/gupta15.pdf

  4. A Practical Guide to Efficient AI

    https://www.youtube.com/watch?v=Yyg_BoeB2LU&t=751

  5. TensorRT 10.x: Working with Quantized Types

    https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/work-quantized-types.html

  6. MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

    https://arxiv.org/html/2406.10290v1

  7. Distilling the Knowledge in a Neural Network

    https://arxiv.org/html/1503.02531

  8. Quantizing for Minimum Distortion

    https://web.stanford.edu/class/ee398a/handouts/papers/Max%20-%20Quantizing%20for%20Min%20Distortion.pdf

  9. BinaryConnect: Training Deep Neural Networks with binary weights during propagations

    https://papers.neurips.cc/paper_files/paper/2015/file/3e15cc11f979ed25912dff5b0669f2cd-Paper.pdf

  10. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

    https://arxiv.org/html/1712.05877

  11. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — original preprint

    https://arxiv.org/html/2210.17323v1

  12. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

    https://arxiv.org/html/1712.05877

  13. OCP Microscaling Formats (MX) Specification, Version 1.0

    https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf

  14. FP8 Formats for Deep Learning

    https://arxiv.org/html/2209.05433

  15. NVIDIA TensorRT: Accuracy Considerations

    https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html

  16. LiteRT 8-bit quantization specification

    https://developers.google.cn/edge/litert/conversion/tensorflow/quantization/quantization_spec

  17. gemmlowp: Building a quantization paradigm from first principles

    https://github.com/google/gemmlowp/blob/master/doc/quantization.md?plain=1

  18. Attention Is All You Need

    https://arxiv.org/html/1706.03762v7

  19. LiteRT: Post-training quantization

    https://developers.google.cn/edge/litert/conversion/tensorflow/quantization/post_training_quantization

  20. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

    https://proceedings.mlr.press/v202/xiao23c/xiao23c.pdf

  21. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=400

  22. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=3411

  23. Practical Quantization in PyTorch

    https://pytorch.org/blog/quantization-in-practice/

  24. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    https://arxiv.org/html/2208.07339

  25. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=211

  26. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=836

  27. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=1779

  28. Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLMs

    https://aclanthology.org/2026.eacl-long.223.pdf

  29. Quantize ONNX models — ONNX Runtime

    https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html

  30. Special topics in Kernels, RL, Reward Hacking in Agents

    https://www.youtube.com/watch?v=uIiA6DquRiE&t=1689

  31. Cross-validation and held-out evaluation

    https://scikit-learn.org/stable/modules/cross_validation.html

  32. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    https://arxiv.org/html/2210.17323

  33. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration

    https://arxiv.org/html/2306.00978v5

  34. Data-Free Quantization Through Weight Equalization and Bias Correction

    https://arxiv.org/html/1906.04721

  35. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

    https://arxiv.org/html/2211.10438

  36. Quantization-Aware Training for Large Language Models with PyTorch

    https://pytorch.org/blog/quantization-aware-training/

  37. Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or -1

    https://arxiv.org/html/1602.02830

  38. Quantization aware training comprehensive guide

    https://www.tensorflow.org/model_optimization/guide/quantization/training_comprehensive_guide

  39. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=1887

  40. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference

    https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/

  41. vLLM GPU worker: profiling available KV capacity

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/worker/gpu_worker.py

  42. Marlin: FP16×INT4 inference kernel

    https://github.com/IST-DASLab/marlin

  43. What every AI engineer needs to know about GPUs

    https://www.youtube.com/watch?v=y-UGrYbJsJk&t=306

  44. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=4576

  45. What every AI engineer needs to know about GPUs

    https://www.youtube.com/watch?v=y-UGrYbJsJk&t=867

  46. Optimization and tuning — vLLM

    https://docs.vllm.ai/en/stable/configuration/optimization/

  47. NVIDIA Model Optimizer: PTQ recipes and schemes

    https://github.com/NVIDIA/Model-Optimizer/blob/main/modelopt_recipes/ptq.md

  48. Metrics design — vLLM

    https://docs.vllm.ai/en/stable/design/metrics/

  49. PyTorch: torch.testing

    https://docs.pytorch.org/docs/main/testing.html

  50. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=896

  51. FP8 Quantization — TensorRT-LLM

    https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/fp8-quantization.html

  52. Accuracy is Not All You Need

    https://arxiv.org/html/2407.09141

  53. Evaluating Quantized Large Language Models

    https://arxiv.org/html/2402.18158

  54. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=2029

  55. A Practical Guide to Efficient AI

    https://www.youtube.com/watch?v=Yyg_BoeB2LU&t=978

  56. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=181

  57. 20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI

    https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=867

  58. Taking Reinforcement Learning Cross Datacenter

    https://www.youtube.com/watch?v=maRzp4kImJ4&t=510

  59. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=1004

  60. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=3520

  61. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=3652

  62. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=2993