AI Engineer Europe 2026
You Might Not Need 50 Diffusion Steps
Loading slides…
Read the talk
You Might Not Need 50 Diffusion Steps
Making diffusion generation usable means reducing latency without losing quality. Quantization, caching and step distillation attack different parts of that problem—and can be combined.
From a talk by Ziv Ilan
Before you start: Basic familiarity with diffusion denoising, model inference and GPU memory will help; no prior experience with distillation is required.
From good images to usable latency
Why does generating a good image still take too long for an interactive application? A diffusion model repeatedly denoises an image or video rather than emitting one token at a time. Ilan describes typical generation as taking 20–50 denoising steps. That repeated computation is the starting point for the latency problem.
Models such as FLUX.2, LTX 2.1 and Google’s Nano Banana have made image and video generation useful across more applications. But a developer or enterprise needs more than an impressive output: generation must be fast, scalable and supported by dependable serving infrastructure. Ziv Ilan, who introduces himself as working with frontier model builders in NVIDIA’s Paris AI Labs team, frames diffusion serving as less mature than the autoregressive LLM ecosystem around vLLM. Some of that ecosystem’s optimization ideas can transfer, but they need to fit diffusion’s computation pattern.
The target is real-time image and video generation for robotics world models, computer games and content creation. Latency matters both before the first image appears and while producing high-quality 720p or 1080p content. Three techniques address different parts of that cost: quantization, caching and distillation. They are a selected toolkit, not an exhaustive list of diffusion optimizations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make each computation cheaper
Quantization is the simplest entry point in this presentation, followed by caching and then distillation. That teaching order differs from the order Ilan describes for deploying a fully optimized model: distill first, then quantize, then apply caching.
| Approach | When precision is reduced | Practical consideration |
|---|---|---|
| Post-training quantization (PTQ) | After training | Simpler adoption; preserving generation quality can be difficult |
| Quantization-aware training (QAT) | Accounted for during training | Requires a training process that accommodates reduced precision |
Diffusion introduces a particular constraint: image and video quality must survive the precision reduction. Ilan also describes these models as attention-heavy, limiting the gains from quantization relative to LLMs and VLMs. Even so, lower precision is an accessible way to exploit features of Blackwell and newer hardware.
The FLUX.2 work with Black Forest Labs illustrates a second choice: static versus dynamic scaling. Static scaling establishes ranges ahead of execution; dynamic scaling computes ranges as needed so they can adapt to the values being processed. NVFP4 has multiple scaling levels: per-tensor scaling may be static or dynamic, while per-block scaling is computed dynamically for blocks of 16 elements. The important distinction is when a scale is determined, not whether every model parameter receives a fixed range upfront.
There are two adoption paths. TensorRT-LLM VisualGen provides an open-source implementation to work from. Partner pre-quantized checkpoints on Hugging Face offer a more direct loading path when subsequent fine-tuning or LoRA adapters are unnecessary. Quantization reduces memory requirements, potentially bringing a model within reach of consumer or lower-end data-center GPUs, and can improve inference performance as well. The slide pairs BF16 and NVFP4 images with these two ways of running quantized FLUX.2.
Attention remains an optimization target in its own right. Ilan points to recent FP4-attention research from Hao Lab as an example of work aimed at this expensive part of diffusion models. Improving the rest of the model’s arithmetic does not eliminate the need to address attention cost and quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse work that has barely changed
Autoregressive serving makes extensive use of the KV cache, preserving attention keys and values from previously processed tokens. Diffusion does not append one token on every denoising step, so that reuse pattern does not transfer directly. Its opportunity is different: successive denoising steps can contain sufficiently similar computation that some work need not be repeated.
TeaCache provides a concrete example. It estimates changes in model outputs using timestep-modulated inputs as a proxy, then uses that estimate to decide whether expensive computation can be reused. It does not first calculate the complete output merely to discover that calculating it was unnecessary. Ilan presents this as a broad reuse decision over the pixel or latent space, before contrasting it with more granular, chunk-based approaches.
For chunk-based reuse, consider the room in which he is speaking. Much of the audience sits still, watching the screen, while the presenter moves. The audience region is a candidate for reuse; the presenter’s region needs recomputation. A threshold controls how much change is acceptable before cached work must be refreshed. This classroom analogy explains selective reuse, rather than the specific internal algorithm of TeaCache.
A small Python function makes that decision boundary explicit. Given estimated changes for the audience and presenter chunks, it returns a proposed action for each; it does not perform a denoising step.
python
def plan_chunk_updates(change_estimates, threshold):
return {
chunk: "reuse" if change < threshold else "recompute"
for chunk, change in change_estimates.items()
}
change_estimates = {"audience": 0.01, "presenter": 0.20}
plan = plan_chunk_updates(change_estimates, threshold=0.05)
# {"audience": "reuse", "presenter": "recompute"}
These teaching values show why the same threshold can preserve one chunk’s cached computation while requesting fresh computation for another.
The threshold is a quality decision as well as a performance decision. Reusing too aggressively can significantly damage the generated image. Evaluate the resulting content, not just the saved computation. In the talk’s VisualGen interface, caching is enabled with a flag and configured with a threshold; the current beta documentation exposes cache_config, including cache_backend and teacache_thresh. Ilan also names vLLM Omni and SGLang Diffusion as serving options with caching support.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce steps instead of parameters
Distillation often brings to mind DeepSeek and the transfer of capabilities from a large teacher into a smaller student. The diffusion objective here is different: keep the parameter count, but reduce the number of denoising steps. The student learns to produce a good image or video with fewer repeated evaluations of the model.
Ilan gives the example of reducing 50 denoising steps to four, eight or sometimes one while aiming to preserve quality. This is not simply a request to stop an ordinary sampler early: the student is trained for the shorter generation process.
Ilan cites potential performance improvements of 10×–200×, without specifying a benchmark baseline, hardware configuration or quality metric. The objective is to make high-quality real-time generation possible, and he reports that a GTC demonstration in San Jose used two distillation techniques to achieve real-time generation. Streaming output would make these models usable in a broader range of interactive applications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn the teacher’s path or its output distribution
Both main distillation families use a teacher and a student, but they differ in what the student must reproduce.
| Family | What the student learns | Constraint on the path |
|---|---|---|
| Trajectory-based | The teacher’s denoising trajectory | Follow the teacher’s progression |
| Distribution-based | The teacher’s output distribution | May take a different route |
Matching a distribution does not require retracing the teacher’s exact denoising path. It gives the student freedom to learn a shorter route to the desired output distribution.
Ilan describes distribution-based approaches as more common and higher quality at the time of the talk. The two families can also be combined: he cites a recent FastVideo release for a hybrid approach that preserved quality while making training more stable.
That training requirement explains why distillation comes last in the presentation. It is a post-training process whose success depends on suitable data and reliable convergence. It requires additional compute, time and expertise, with recipes still evolving through research. Both open-source and closed-source model builders are applying these methods, but selecting a recipe remains more demanding than enabling an inference optimization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Organize distillation at model scale
NVIDIA FastGen is infrastructure for this work, not another distillation algorithm. Its open-source repository brings multiple techniques into a framework for experimenting with and scaling post-training. Ilan cites video diffusion models with 20B, 30B and 40B parameters, and anticipates models reaching hundreds of billions. At those scales, sharding work across GPUs becomes part of the training problem. FastGen structures that machinery so practitioners can concentrate on output quality and the recipe they want to use.
Training data is optional in the illustrated workflow. Open-source data can provide a useful starting point, but a specialized target distribution changes the decision: if the application’s data differs substantially from general-purpose material, Ilan recommends using the application’s own data for fine-tuning. The purpose is to make the optimized model work for the intended distribution, not merely for a generic demonstration.
The benefit can be lower compute requirements as well as lower latency. Ilan reports near-real-time or real-time video generation at GTC on one Blackwell B200 GPU, depending on output quality. The talk does not specify resolution, frame rate or a quality evaluation for that result. This is an inference demonstration; it does not establish the hardware required to train the distilled model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Diffusion within frames, autoregression across them
Further opportunities may come from architectures that combine diffusion with autoregressive generation. Ilan mentions Transfusion and autoregressive diffusion as developing directions, then describes a frame-by-frame mechanism: diffusion generates a frame, and successive frames are generated autoregressively. That structure could bring more autoregressive optimization techniques into image and video generation, although he characterizes the area as still research-driven.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose the optimizations incrementally
These optimizations can be combined. Distillation reduces the number of steps; quantization reduces the cost and memory footprint of computation; caching avoids some repeated work. Multi-GPU execution and context parallelism offer additional options. Choosing one does not require abandoning the others.
For incremental experimentation, Ilan proposes a practical sequence:
- Start with quantization. If it meets the application’s needs, stop there.
- If more improvement is needed, consider multi-GPU execution, context parallelism and caching.
- Take on distillation when the remaining latency target justifies the more demanding post-training work.
This sequence starts with the easiest intervention to assess. It differs from ordering the stages of a complete optimization pipeline, where distillation can precede quantization and caching. Ilan presents distillation as the most impactful option, but also the one requiring the greatest investment.
The available open-source tooling includes support for the Wan, FLUX.2 and LTX2 model families, with more support being added. The invitation is to try these implementations and contribute improvements, helping diffusion serving approach the maturity already available in the LLM and vLLM ecosystem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What hardware and data does distillation need?
An audience question brings the discussion back to access: if GB200 systems are difficult to obtain, what compute is needed for fine-tuning or distillation, and how large should the dataset be? GB200 is not a requirement. Ilan names Hopper/H100, B200 and B300 as possible hardware choices. Distillation requires less compute than pre-training, but it still needs substantial resources; his answer does not promise that an arbitrary model can be distilled on a single instance.
Model size changes the requirement. Ilan points to smaller video generation models with 2B or 4B parameters as requiring less compute. A single-GPU inference result should therefore not be used as a training budget: the size of the model and the chosen post-training process still determine the resources needed.
On data, the first requirement is knowing how to evaluate the result. Compare what a general-purpose dataset produces against what data specific to the intended application produces. Ilan reports that general demonstrations work without a special dataset, while a domain such as protein generation calls for more specialized data. He gives no numerical dataset-size requirement. The actionable starting point is an evaluation that can reveal whether general data is sufficient for the application—or whether its target distribution requires a different training set.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current beta guide to supported diffusion models, quantization, caching and multi-GPU inference.
Training-free diffusion caching implementations that estimate when transformer computation can be reused.
Open-source infrastructure for experimenting with diffusion distillation methods and few-step generation.
Video generation framework with inference optimizations, fine-tuning, distributed training and distillation workflows.
Further reading
Explains NVFP4 scaling, TeaCache and combined inference optimizations for FLUX.2, with code examples and benchmark conditions.
Hao AI Lab's study of quantization-aware training for preserving quality with FP4 attention.
Introduces trajectory-based, distribution-based and causal distillation, including a Wan2.1-14B training example.
Updates since the talk
May 2026 release of an LTX-2 video generation and editing application, with deployment instructions and streaming architecture.
Full transcript
Loading transcript…