AI Engineer World's Fair 2026

Compression at the Edge

Loading slides…

Read the talk

Compression at the Edge

Making a model fit is only the beginning: selective precision, architecture-aware evaluation and runtime compression determine whether local intelligence remains useful.

From a talk by Chris Alexiuk, Daniel Han, Asma Beevi, Merve Noyan and Parth Sareen

Before you start: Familiarity with model weights, inference, attention and basic floating-point precision will help; the article explains mixed precision, block scaling and KL divergence as they arise.

How much intelligence fits on your computer?

How do you make a large language model small enough to run on your own computer without losing the capability that made you want it? That question connects the perspectives on this panel: moderator Chris Alexiuk, who introduces himself as a NVIDIA product research engineer working on Nemotron; Daniel Han from Unsloth; Asma Beevi, who builds NVIDIA Model Optimizer; Merve Noyan from Hugging Face; and Parth Sareen from Ollama.

Title slide reading “Compression at the Edge,” with five speaker portraits, names, roles and organization logos.Open full source frame
Compression at the Edge panel title and speaker lineup.

For Sareen, quantization is what makes larger models viable on relatively small machines. Noyan adds the constraint: compression has costs, whether in latency or quality, even as it brings models such as Gemma 4 and Qwen 3.6 onto personal hardware. Beevi frames the objective as more intelligence for the same cost. Her FP32-to-FP4 example gives an eightfold reduction in nominal bits per value, before accounting for scales and other storage overhead. Preserving nearly the same capability is the engineering goal, not a consequence of that arithmetic.

Han makes the bargain concrete with GLM 5.2: he describes shrinking a 1.5 TB model to 250 GB. He calls this an 86% reduction, although those rounded sizes imply about 83.3% using decimal units. The mechanism is mixed precision: put most layers at one or two bits, while retaining important layers at 16 bits. Han reports recovering 76% of accuracy with this approach, without specifying the benchmark or the recovery baseline. The essential observation is that storage reduction and capability loss need not move together: precision can be spent where it matters most.

0:000:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:21 · section reference included

From open reasoning to local agents

The hardware in question includes consumer RTX cards and Macs. For Han, the turning point was DeepSeek-R1: a strong open reasoning model whose size made local execution difficult. Keeping selected layers at higher precision produced Unsloth’s dynamic 1.58-bit quantization, which worked better than the team initially expected. Subsequent releases—including Qwen 3.5, Qwen 3.6, Gemma 4 and Nemotron—kept improving capability while renewing the memory problem. Alexiuk describes a continuing race between making models bigger and making them small enough to use.

Beevi arrived through computer vision and pruning. Removing parts of a model often cost quality and required fine-tuning to recover it. When LLMs became central, quantization offered a comparatively straightforward way to compress them with limited accuracy loss. Sparsity remained useful, but in her experience the trade-off was less favorable. Work on NVFP4 and mixed precision reinforced the idea that the design of the numeric representation could determine how much compression a model tolerates.

Noyan’s first milestone was QLoRA: fine-tuning on a free-tier Colab T4 became possible, even if painfully slow. The surrounding stack—TRL, bitsandbytes and PEFT—also enables more advanced training methods such as GRPO, with colocation and careful VRAM management. After the GGML team joined Hugging Face, she moved further into the llama.cpp ecosystem. Her second milestone was running OpenClaw and Hermes Agent on long-horizon tasks with quantized Qwen models. She describes Qwen 3.5 doing substantial coding work, including repairing its own harness.

Sareen’s entry point was simpler: as a student with little money, he wanted to run AI on his own computer for free. Ollama made a local model useful enough to complete tasks. His recollection combines 2023 with Llama 3, whose release was in April 2024, so the precise date and model cannot both be right. The experience itself is clear: having built models before, he knew how difficult running them could be. Getting one working on a personal machine changed what was accessible without a hosted API or rented GPU.

4:414:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:41 · section reference included

Precision belongs where errors matter

A smaller checkpoint is not automatically a useful checkpoint. Han contrasts selective compression with randomly deleting weights or rounding them without regard to the architecture—interventions that can destroy model behavior. His starting heuristic is that layers have unequal sensitivity: in a model with several dozen layers, the first and last layers may deserve more protection than the middle. That is a starting point for measurement, not permission to discard all middle layers.

Han offers a training-saturation hypothesis for why this room exists. A trillion-parameter model trained on tens of trillions of tokens may still have weights whose available capacity is underused; he speculates that much longer training could reduce that room. He also points to near-zero weights as candidates for removal. These observations concern different interventions: setting weights to zero changes sparsity, while representing them with fewer bits changes precision. His broad claim that most weights need not remain should therefore not be read as a measured pruning budget for every model.

The practical search begins with sensitivity tests:

  1. Quantize one layer and measure the effect on accuracy.
  2. Repeat for other layers.
  3. Test combinations, because individually tolerable changes can interact.
  4. Inspect within tensors when layer-level decisions are too coarse.

Even choosing pairs or triples from 32 layers creates many candidates; searching individual tensor elements expands the problem further. Precision allocation is a constrained optimization problem, not merely a file conversion.

The extreme case is an exceptional scalar. Han invokes The Super Weight in Large Language Models to explain why a single value can require protection. His shorthand is a 20% capability loss from quantizing one number, but the paper’s supported lesson is more specific: exceptional weights can have disproportionate influence, and pruning such a scalar can severely damage generation. Pruning and quantization are different interventions; the useful implication is to identify sensitive values rather than assume all weights tolerate the same error.

11:3611:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:36 · section reference included

Benchmark the allocation, then examine the format

Beevi’s answer to how quality is verified begins with extensive benchmarking. The Model Optimizer team publishes quantized checkpoints on Hugging Face and checks them against the original model. Beevi states a target of less than 1% overall accuracy degradation for FP4 across the AA benchmarks. The panel does not specify the exact suite, aggregation method or whether that percentage is relative or absolute; it is a stated target rather than a result established for every checkpoint. Simple, strong quantization strategies help the team release models faster.

Her precision allocation distinguishes sensitive attention components from more tolerant expert weights:

ComponentPrecision approach described
Linear-attention projectionsProtect at higher precision
KV and QKV layersMay retain FP8 or BF16
MoE layersDefault to FP4

Gradient-based sensitivity analysis feeds an optimization solver, while benchmarking checks whether the resulting allocation preserves quality. The search can automate choices, but it does not eliminate the evaluation work.

What makes NVFP4 more than simply shorter numbers is microblock scaling. A group of 16 four-bit values shares one additional FP8 scale. The values encode a small local range, and the shared scale adjusts that range for the block. The full format also includes a tensor-level FP32 scale. Consequently, four-bit payloads do not imply exactly four stored bits per weight once scaling overhead is included.

Beevi credits Tim Dettmers and bitsandbytes with an earlier related block-scaling approach and describes NVIDIA’s design as similar but different. Her explanation emphasizes that the scale representation and grouping matter to accuracy: two formats called four-bit can behave differently because they distribute representational error differently.

14:4614:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:46 · section reference included

A benchmark score does not describe the whole deployment

Compression also matters when an organization can afford more than a consumer laptop. Its deployment options range from employee computers to internally operated clusters. Sareen argues that evaluating the actual harness is part of choosing a quantization: a model can score acceptably yet behave poorly when placed in a coding workflow such as Claude Code. Benchmarks help narrow the search; running the intended workflow checks whether model behavior remains useful. Local execution then becomes an option for individual employees as well as centralized infrastructure.

Noyan considers a quantization arena for comparing behavior, then raises a limitation of preference leaderboards. She refers to work by Singh and colleagues, likely The Leaderboard Illusion, which examines selective disclosure, private variant testing and unequal access to evaluation data. An arena can add evidence about preferences, but its ranking process also needs scrutiny.

Many business tasks do not require the most capable frontier model. Compression can improve the economics of serving more concurrent requests, and distillation can produce a much smaller model specialized for a narrow task. Noyan describes companies distilling midsize models into small rerankers for tasks that do not need general text generation. She reports savings in the millions, without a named case study, time period or cost breakdown. Her broader point is that in-house deployment, concurrency and specialization remain underused options outside communities already immersed in open models.

17:5118:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:51 · section reference included

A large quantized model or a small native model?

Alexiuk points to a REAP-compressed GLM running at the event and asks why one should choose it when smaller models already exist. He names Nemotron-3-Nano and small Gemma and Qwen variants as alternatives. The decision is not just whether a model fits: it is how much capability and speed a given memory budget buys.

Han cites an unnamed paper and compares a 35-billion-parameter BF16 model with a 120-billion-parameter four-bit model. He says the larger quantized model performs better in the experiments he has in mind. The exact experiment is not identified, but the weight-storage arithmetic explains why the comparison is plausible:

Model optionBits per weightApproximate weight payload
35B parameters1670 GB
120B parameters460 GB

These decimal sizes exclude scales, metadata and runtime memory. They are comparable budgets, not identical footprints, and this comparison alone does not establish better training-cost efficiency or superiority on every task.

Continued growth would make compression more necessary. Han imagines today’s terabyte-scale checkpoints becoming tens of terabytes, eventually exceeding even high-memory workstations such as DGX Station. Smaller models still have a role: a system can route planning to a large model and execution to a smaller one.

Speed can reverse the choice. Han illustrates the trade-off with 5–10 tokens per second for a compressed large model without enough GPU power, versus 200 tokens per second for a small model. No matched hardware or workload is supplied for those examples. The deployment decision therefore includes latency, throughput and available GPU resources alongside capability and weight storage.

21:4121:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:41 · section reference included

Local demand changes what gets built

For Ollama, compression is central to making personal models viable, including the Qwen-based agent workflows discussed earlier. Sareen also points to techniques originating inside model labs: quantization-aware training, or QAT, and the MXFP4 format used by GPT-OSS. Demand for personal language models gives labs a reason to design training and releases around local constraints. Large compressed models and small full-precision models remain complementary choices with different trade-offs.

The supporting ecosystem matters as much as an individual checkpoint. Alexiuk recalls the familiar image of an enormous dependency stack held up by bitsandbytes; Noyan highlights Unsloth’s pipelines for producing and verifying quantizations. These tools turn numeric techniques into artifacts people can actually download and use.

Noyan’s work on SmolVLM addressed a gap below LLaVA and larger vision-language models. She recalls experiments around a billion parameters and at 500M and 256M: the smallest was not useful enough in her assessment, while the 500M model was somewhat capable and could run on an iPhone. The anecdote shows why simply minimizing parameter count is not the whole objective. A model still needs enough capability to justify running it.

Quantization offers another route to that balance. Noyan recounts a Qwen 3.6 checkpoint poll in which users asked for a midsize model they could quantize rather than a smaller native model. As the starting models improve, their compressed versions can become both more capable and smaller than earlier generations. Community demand consequently influences which checkpoints model builders prioritize.

24:2124:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:21 · section reference included

Creating the checkpoint is the easy part

Alexiuk asks about the engineering behind Nemotron’s NVFP4 releases. Beevi separates post-training quantization, or PTQ, from methods that require further training. PTQ starts with a released BF16 checkpoint and converts it into a lower-precision representation. Beevi reports that quantizing large GLM or trillion-scale models to FP4 takes a couple of hours on the team’s Blackwell nodes. The more time-consuming work starts afterward: evaluating the checkpoint and matching the original model card.

In her experience, PTQ usually works for medium and large models—roughly 20B–30B or more in dense size—with selective quantization. Sparse MoE components can often be compressed aggressively. Automatic sensitivity analysis and a knapsack solver can assign formats under a precision budget, while heuristics protect known-sensitive components.

Below roughly 20B, Beevi says quantization-aware distillation may be needed to recover accuracy. This introduces a data problem. Modern reasoning models can pass through multiple training stages, reinforcement learning and distillation from different teachers specializing in coding or reasoning. Even access to the original Nemotron dataset does not automatically provide the right recovery-training mixture. In her experience, using the wrong data for QAD commonly damages the model rather than restoring it.

When Alexiuk concludes that this sounds difficult, Beevi corrects the impression: around the 30B scale, PTQ should usually work out of the box. The difficult cases should not discourage trying existing tools. Model Optimizer’s PTQ and AutoQuantize examples and Unsloth provide practical entry points; the training-based recovery path is a separate, more demanding step.

28:5429:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

28:54 · section reference included

New architectures invalidate old shortcuts

Supporting a new release now involves more than changing the layer count of a familiar Llama-like architecture. Hybrid and linear attention introduce different computation and state. Sareen divides Ollama’s work into two tracks: implement the architecture correctly, then find a quantization that behaves well. Early collaboration with a lab can mean supporting several variants before release.

Ollama’s default quantization is consequently a product decision backed by testing in different harnesses. For some small models, the team leaves the default at higher precision because aggressive quantization produces a worse experience. The smallest artifact is not necessarily the best default.

The panel contrasts this diversity with the older Transformer++ shorthand: a Transformer with RMS normalization and additional refinements. Current variations include activation functions, linear attention, different mixes of sliding-window and global attention, and even changes to normalization epsilon such as 1e-5 versus 1e-6. Each may have a reason grounded in the model builder’s ablations, but each also adds work for inference and compression implementations.

Han describes a particularly consequential failure: quantized linear-attention layers can look healthy in initial tests, then produce gibberish on long-context benchmarks or during real use. In contrast, some MoE layers in his experience tolerate precision as low as one bit; some components may even be removable. A quantization policy must be tested against the architecture and the intended context length. A short successful response cannot validate a model intended for long-running agent work. Han welcomes architectural diversity despite the extra work, because having everyone build the same model would constrain progress.

32:3032:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:30 · section reference included

The next compression targets are runtime state and computation

Asked to look six to eighteen months ahead, Sareen expects model labs to take a larger role in compression, including more QAT, while community projects continue pushing post-release techniques. Noyan connects the next stage to privacy: sensitive personal data creates demand for useful models on devices rather than only on servers. She expects work in llama.cpp to extend that reach, including to phones where earlier attempts were not yet satisfactory.

Beevi broadens the discussion beyond stored weights. FP4 can also accelerate matrix multiplication when hardware executes four-bit math. She speculates that weight quantization may be approaching a favorable limit in its quality-versus-size trade-off, with less room left to reduce precision. Other parts of inference still offer opportunities:

  • KV cache: Reduce the memory retained for previously processed tokens. She describes eight-bit use as common and says four-bit cache quantization preserves quality more readily in large models than in small ones.
  • Cache compaction: Combine a smaller retained state with quantization to support longer-horizon reasoning.
  • Sparsity: Reduce computation by exploiting zeros or inactive values, although accuracy degradation has limited adoption compared with quantization.

These techniques address different resources, so their benefits can combine.

She points prospectively to Rubin’s dynamic activation sparsity as an opportunity to improve attention computation. At the architecture level, DeepSeek’s multi-head latent attention, or MLA, sparse attention, indexed attention and changes that skip parts of attention processing expand the design space further. The future she describes combines numeric formats, runtime-state compression and architectural changes to make inference cheaper.

Han’s desired outcome is frontier-capable intelligence on phones, GPU laptops and other personal hardware. Local execution would let users fine-tune and customize models without depending entirely on a model lab. But ownership also has a physical constraint: an assistant that drains the device’s battery is not a satisfactory local product. Capability, control and energy efficiency have to arrive together.

37:2737:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

37:27 · section reference included

How do you find a trustworthy modified model?

The audience’s closing question brings the discussion back to selection. Given a heavily modified model, where can someone find the original benchmarks rerun after every change, or a matrix showing which variant works best for a particular task? Alexiuk says there is no comprehensive resource at the time of the panel. Han says Unsloth publishes benchmarks for some dynamic quantizations, while noting that accuracy evaluation requires decisions about sampling, repeated trials and averaging.

Han proposes another measurement: KL divergence between the original BF16 model and the quantized model on calibration data. Pass the same inputs through both models, collect output logits, convert them to probability distributions, and compare those distributions. For a token position with original distribution P and quantized distribution Q, one direction of the comparison is:

DKL(P∥Q)=∑v∈VP(v)log⁡P(v)Q(v)D_{\mathrm{KL}}(P\parallel Q) = \sum_{v\in\mathcal{V}} P(v) \log\frac{P(v)}{Q(v)}

Here, 𝒱 is the vocabulary. The objective is to reduce checkpoint size while keeping the quantized output distribution close to the original. KL divergence is directional, so the baseline and direction should remain explicit.

The calculation can be expressed directly in Python. This function takes matching vocabulary logits for one token position and uses log-softmax for numerical stability:

python

import math


def log_softmax(logits: list[float]) -> list[float]:
    peak = max(logits)
    shifted = [value - peak for value in logits]
    normalizer = math.log(sum(math.exp(value) for value in shifted))
    return [value - normalizer for value in shifted]


def output_kl(
    bf16_logits: list[float],
    quantized_logits: list[float],
) -> float:
    if not bf16_logits or len(bf16_logits) != len(quantized_logits):
        raise ValueError("Expected matching, nonempty vocabulary logits")
    if not all(math.isfinite(x) for x in bf16_logits + quantized_logits):
        raise ValueError("Expected finite logits")

    log_p = log_softmax(bf16_logits)
    log_q = log_softmax(quantized_logits)
    return sum(
        math.exp(p) * (p - q)
        for p, q in zip(log_p, log_q)
    )

Apply the comparison at matching token positions under the same input prefixes, then aggregate over the chosen calibration data. Using natural logarithms expresses divergence in nats. A smaller result means closer output distributions on those inputs, which gives a more detailed view of quantization’s effect than a single aggregate accuracy score.

Alexiuk closes by recommending Accuracy is Not All You Need. Its central warning fits the panel’s recurring concern: a compressed model can keep similar aggregate accuracy while changing which individual answers it gets right. Distributional comparison adds evidence about preservation; it does not replace checking the tasks, harnesses and context lengths for which the model is being chosen.

43:2243:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

43:22 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…