AI Engineer Europe 2026

Building Generative Image & Video Models at Scale

Loading slides…

Read the talk

Building Generative Image & Video Models at Scale

Training a visual generator means choosing what to compress, how to denoise, where to spend sampling compute, and which controls let people shape the result.

From a talk by Sander Dieleman

Before you start: Basic familiarity with neural networks, tensors and conditional generation is helpful; the article develops the diffusion and sampling intuition from first principles.

What goes into a visual generator?

What does it take to train a model that generates images and video at scale? The neural network is only one part of the answer. Sander Dieleman introduces himself as a research scientist who has spent more than a decade at Google DeepMind, working on the generative media team behind models such as Veo and Nano Banana. His starting point is diffusion: the prevailing approach to audiovisual generation in this account, alongside the autoregressive approach that dominates language modeling.

The engineering decisions run from data curation and representation through modeling, architecture and distributed training, then into sampling, distillation and control. Representation determines what the network must learn; sampling determines how its predictions become a finished output; control determines whether that output is useful. Dieleman’s blog develops the intuition behind these generative mechanisms.

Slide with eight labeled photographs arranged in two rows, illustrating the talk’s eight topics.Open full source frame
Eight topics: data, representation, modelling, architecture, training, sampling, distillation and control.

Improving the data can be a better investment than tweaking the model or optimizer. That priority runs against a familiar research incentive: use the same fixed dataset as everyone else so results remain comparable. Building a capable generator requires inspecting and improving the data itself. Dieleman describes curation as still underrated, while leaving specific practices undisclosed because they are part of what differentiates the models and are sparsely documented in publications.

0:150:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Make each training example tractable

Images arrive as two-dimensional pixel grids; video adds time to that grid. Early diffusion models operated directly on pixels, but increasing resolution and duration quickly makes that representation expensive. Dieleman’s example is a single 30-second, 1080p video at 30 frames per second: several gigabytes of pixel data for one training item. The exact memory footprint depends on the numeric precision used, before accounting for training overhead.

Existing image and video codecs solve a different compression problem. JPEG and conventional video formats prioritize compact storage, even when their representations obscure the structure that a generative model needs. Instead, train a compressor whose output remains convenient to model. An autoencoder learns to reconstruct its input after forcing it through a compact latent bottleneck.

The resulting pipeline has three distinct operations:

  1. Train the autoencoder. An encoder compresses an image or video into latents, and a decoder learns to reconstruct it.
  2. Train the generator. Encode training examples, then learn a generative model over those latent representations. Either autoregression or diffusion can occupy this stage; Dieleman reports that diffusion generally has an edge for audiovisual quality at a given parameter budget.
  3. Generate and decode. Sample a new latent representation, then pass it through the previously trained decoder to obtain pixels.

The generator therefore learns a distribution over compressed examples, while the decoder handles the return to the visible image or video.

Three-row diagram showing encoder–decoder reconstruction, generator training on latents, and sampling through a generator and decoder.Open full source frame
Two training stages and a sampling pipeline connect encoders, latents, an iterative generator and a decoder.
3:594:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:59 · section reference included

Compress detail while preserving the grid

Consider a 256×256 RGB image encoded into a 32×32 latent grid. Dieleman tentatively associates these dimensions with the original Stable Diffusion; they match its eightfold spatial downsampling. The important distinction is between spatial resolution and total tensor size: latent channels also consume memory. Extra channels carry information that simply resizing the image would discard, including some high-frequency detail. Compression remains lossy, but the representation is much smaller.

Video offers additional redundancy along the time dimension. Dieleman says learned compression can often reduce video tensor sizes by roughly two orders of magnitude. That is an attributed general observation rather than a ratio for a specified codec or latent configuration; its practical consequence is the difference between fitting a training example in memory and being unable to process it.

These learned compressors deliberately preserve more obvious structure than a codec such as H.265. The latent representation still has a grid topology, only at a coarser scale, so the neural network can exploit spatial relationships. A visualization from EQ-VAE makes this visible: principal components across latent channels are mapped to RGB. Animals remain recognizable in the resulting images. Much of the compression affects local texture and fine structure, while the broad arrangement of the scene remains available to the generator.

6:537:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:53 · section reference included

Why one denoising prediction is blurry

Once the representation is tractable, the next decision is how to generate it. Autoregression turns the data into a sequence and predicts successive elements. That ordering is natural for language, but a two-dimensional image requires an imposed traversal order. Diffusion instead defines a corruption process: gradually add noise, then learn a denoiser that predicts how to reverse the corruption. Both approaches build an output iteratively.

The bunny example shows what corruption removes first. With a little Gaussian noise, the whiskers become difficult to distinguish, but the silhouette still identifies the animal. More noise eventually erases the global structure as well. This ordering—fine detail disappearing before broad structure—will matter when the process runs backward.

To explain the reverse process, Dieleman reduces image space to a two-dimensional diagram. Actual images and videos are high-dimensional objects, so this is a geometric aid rather than a literal picture of their space. Let x₀ be a clean sample and xₜ its noisy version at corruption time t. During generation, the sampler begins with noise and eventually reaches an intermediate state resembling xₜ. The denoiser then predicts a clean image from that observation.

The prediction is ambiguous because corruption destroyed information. Many clean originals could explain the same noisy observation. A denoiser that averages those possibilities produces blur: the backward-process slide contrasts a noisy cat, a blurry prediction and a sharp original. The prediction supplies a direction toward plausible images, not a finished image to accept immediately. The surrounding region of possibilities is the useful interpretation of that blurry estimate.

Slide titled “Diffusion: backward process” showing a noisy cat at upper right, a blurry cat prediction at left connected by a dotted line, and a sharp original cat below.Open full source frame
The backward diffusion diagram contrasts a noisy cat image, a blurry prediction, and the clean original.
9:399:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:39 · section reference included

Take a small step, then predict again

Jumping all the way to the predicted clean image would preserve the blur. Instead, move a small distance toward it and ask the denoiser again. This resembles neural-network optimization: an update direction is useful locally, and an excessively large step can carry you beyond the region where it applies. The difference is the space being traversed. An optimizer changes model parameters; a diffusion sampler changes pixels or latent values.

Many sampling algorithms then add a little fresh noise, less than the amount removed by denoising. This is new randomness, not the exact noise just removed. The denoiser makes imperfect predictions, and repeatedly feeding those imperfections back can accumulate error. Fresh noise obscures some of those mistakes and can make the trajectory more robust. Other samplers omit this step.

At the next state, xₜ₋₁, the image is slightly less noisy. The denoiser now has more information, so its prediction changes and the region of plausible originals shrinks. Repeating the process progressively resolves the ambiguity until a particular sample emerges. The sampler succeeds by repeatedly revising local predictions rather than trusting its first estimate.

13:4914:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:49 · section reference included

Why diffusion generates from coarse to fine

Fourier analysis explains why this procedure fits visual data so well. Starting with four ImageNet images, Dieleman takes a two-dimensional Fourier transform of each. The result is complex-valued and can be separated into magnitude and phase. Setting phase aside for this analysis, radial averaging summarizes each two-dimensional magnitude spectrum as a one-dimensional curve.

On log-log axes, those curves look approximately straight: a signature of a power-law relationship between frequency and spectral magnitude. Natural images typically have stronger low-frequency components and weaker high-frequency components; Dieleman extends the same broad observation to video. Gaussian noise, by contrast, has a flat spectrum in expectation. When added to an image, it leaves the stronger image frequencies visible while drowning out weaker ones.

Increasing the noise level therefore obscures the highest frequencies first, then progressively lower frequencies, until no image structure remains distinguishable. Reversing that process establishes low-frequency structure before adding high-frequency detail. Dieleman calls this spectral autoregression: an approximate analogy, not a hard autoregressive ordering. It lets the generator establish the scene’s broad semantics before filling in its details, and it lets training emphasize the noise levels—and therefore scales—that matter most perceptually.

16:2216:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:22 · section reference included

Choose an architecture for space and time

The denoiser’s input-output contract is straightforward: receive a noisy image and predict its clean counterpart. U-Nets, originally developed for segmentation, were a natural fit because they support tasks that preserve spatial dimensions, including image restoration. Early Stable Diffusion models used this architecture. Transformers also work, with one important difference from a typical language model: denoising permits fully bidirectional attention, so there is no need for a causal mask. Adopting transformers also brings the scaling knowledge developed for language models.

For video, the relationship between autoregression and diffusion admits several choices:

ApproachGeneration order
Fully autoregressiveFlatten height × width × time; generate token by token
Joint diffusionNoise and denoise the full spatiotemporal volume
HybridGenerate frames in temporal order; use diffusion within each frame

Dieleman describes joint diffusion as the common approach among modern video generators. The hybrid becomes especially useful for real-time generation, where new frames must follow events as they unfold. He names Genie as an example of this compromise.

Three cubes with width, height and time axes show small blue blocks, blue temporal slices and a fully blue volume beneath an autoregression-to-diffusion arrow.Open full source frame
Autoregression, hybrid generation and diffusion illustrated across video dimensions.

Training these networks eventually requires more than splitting a batch across accelerators. Data parallelism distributes examples across chips; model parallelism distributes the model itself. Dieleman says these visual models still tend to be smaller than contemporary LLMs, but they are large enough that sharding and communication matter. He names JAX and pjit as tools for expressing sharding and letting the compiler reduce inter-chip communication. Current JAX documentation deprecates pjit in favor of jax.jit; the underlying objective remains efficient distributed execution, without a guarantee of globally optimal placement.

19:5920:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:59 · section reference included

Amplify the effect of the prompt

Sampling introduces choices independent of the denoiser architecture. A deterministic sampler associates an initial noise sample with a fixed output, which is useful for distillation. A stochastic sampler introduces fresh randomness along the way and can resist accumulated prediction errors. Guidance adds another choice: trading diversity for quality. In Dieleman’s account, increasing guidance makes samples more alike while improving their individual quality, helping diffusion models perform well without reaching LLM parameter counts. Guidance also applies to autoregressive models, though he finds it especially effective for diffusion.

The mechanism starts with two predictions for the same noisy input. First, predict without the text prompt. Then predict with it. A prompt describing a rabbit restricts which originals could explain the observation, so the conditioned prediction is typically less blurry. The useful quantity is the difference between those predictions. Amplifying that difference yields a new denoising direction that emphasizes what the prompt contributes.

Writing the unconditional prediction as u, the conditional prediction as c, and the guidance scale as s, the combination is:

Δ=c−ux^0,guided=u+sΔ\begin{aligned} \Delta &= c-u \\ \hat{x}_{0,\mathrm{guided}} &= u+s\Delta \end{aligned}

In Python with JAX, the tensor operation is compact:

python

import jax.numpy as jnp


def guided_prediction(
    unconditional: jnp.ndarray,
    conditional: jnp.ndarray,
    scale: float,
) -> jnp.ndarray:
    delta = conditional - unconditional
    return unconditional + scale * delta

Both inputs must be predictions for the same noisy sample and noise level, expressed in the same prediction parameterization. With this convention, scale=1 returns the conditional prediction; larger values extrapolate beyond it. The sampler uses the combined prediction for its next update, while the rest of the procedure—including optional fresh noise—continues as before. The described guidance procedure requires two model evaluations per sampling step.

The guided-versus-unguided examples come from OpenAI’s 2021 GLIDE paper, an early large-scale pixel-space diffusion model. In the displayed comparisons, guidance reduces diversity while making individual images better match their prompts. Dieleman characterizes guidance as effectively standard in modern image generation and suggests that users would be surprised by the quality loss if it were removed. For a user seeking one useful image, the improvement can matter more than preserving the full diversity of possible outputs.

23:2923:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:29 · section reference included

Distill the trajectory into fewer steps

Here, distillation primarily means reducing the number of network passes needed to generate a sample, rather than shrinking the network. Sampling traces a nonlinear trajectory through pixel or latent space. At each point, the denoiser supplies a local direction—a tangent telling the sampler where to move next. With a deterministic sampler, a particular initial noise sample follows a fixed path toward a particular output.

Once that path is available, a different learning target becomes possible: predict where it ends. Consistency models provide a way to distill that endpoint prediction from a diffusion model, potentially replacing many local updates with a direct jump. This is one of several approaches to reducing sampling steps.

A single jump asks a great deal of the network. Dieleman illustrates the difficulty as asking one pass to do work previously spread over roughly fifty passes. A compromise is to learn endpoint predictions over shorter intervals of the trajectory, retaining several generation steps—three in his example. Those counts illustrate the quality-versus-compute choice; they are not a measured speedup. Smaller intervals give the network less work to compress into each prediction.

28:0128:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

28:01 · section reference included

Give users controls beyond language

A generator is useful only to the extent that people can influence what it produces. Text prompts are a convenient interface for describing a scene, but they are awkward for some precise requests. To place yourself in a generated video, supplying a photo or short video is more direct than describing your appearance in exhaustive detail. Video also calls for explicit controls over camera motion, event speed and event timing. These signals need representations suited to the thing being controlled.

Conditioning signals slide lists text prompts, reference-based style, subject and scene, and video controls beside a large camera photograph.Open full source frame
Conditioning signals include text, reference images and controls for camera motion, speed and event timing.

The training question is when to introduce those signals. Most pre-training examples may lack specialized camera or timing annotations. One option is to pre-train with text conditioning, then introduce additional controls during post-training. That stage can also incorporate human preferences through reinforcement learning or direct preference optimization. The broad visual distribution is learned first; later training shapes how users steer it and which outputs the system favors.

30:0230:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:02 · section reference included

Why models acquire a style—and where guidance helps

The audience questions begin with a visible consequence of the quality-diversity trade-off: does guidance explain why a model such as Nano Banana has a recognizable style? Dieleman attributes part of that effect to guidance, but suspects post-training plays the larger role. Pre-training models a broad image distribution; post-training selects an opinionated portion of it.

Guidance can also produce more specific artifacts. Dieleman identifies excessive saturation as a common sign that the guidance scale is too high. Because the conditional–unconditional difference is amplified at different noise levels, the same scale can have different effects over the trajectory. He describes work that avoids guidance at the beginning and end while emphasizing it in the middle. A guidance schedule therefore offers another way to manage the trade-off, beyond choosing a single scale for every step.

For text diffusion, a prompt can likewise serve as a conditioning signal, making the same guidance principle applicable. How well it works is a separate question. Dieleman’s intuition is that guidance benefits from a semantic gap between the conditioning signal and the output: a text description supplies high-level meaning while an image contains detailed visual realization. A segmentation mask operates closer to the spatial structure of the pixels and, in his experience, benefits less from guidance. He offers the same intuition for why guidance is less common in language generation, including autoregressive models.

32:1532:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:15 · section reference included

Framework choices and the limits of more sampling

Asked whether JAX is better suited than PyTorch to sharding, Dieleman qualifies the comparison: he has little hands-on PyTorch experience after more than a decade at Google. He connects JAX’s design to making TPU use convenient, particularly across multiple chips with fast interconnects. PyTorch also supports parallelism and sharding; these capabilities are not exclusive to JAX. The explanation concerns design priorities and his experience, not a comparative performance measurement.

Does taking more denoising steps improve the result? Up to a point. Finite sampling updates approximate a nonlinear trajectory with a piecewise-linear path. If those updates deviate too far from the underlying trajectory, sample quality suffers. More, smaller steps can reduce that approximation error, but the benefit has a limit. Another route is to learn straighter trajectories from the outset: rectified flow and Reflow aim to make accurate generation possible with coarser, fewer updates.

35:2735:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:27 · section reference included

Insert the right signal, choose the right sampler

A question about camera control asks whether a simulator could provide known camera parameters for training. Dieleman does not confirm such a workflow: he cannot disclose internal practices and is not directly involved in the capabilities team doing that work. The general design problem is to find a semantically meaningful representation for the control, then choose how to introduce it into the transformer.

Two concrete interfaces are available:

  • Additional tokens: include conditioning information as extra tokens the model can attend to.
  • Broadcast conditioning: make the signal available to every token. The diffusion noise level is typically supplied this way, rather than represented by one extra token.

The signal’s role should determine its interface. A global property such as the current noise level needs to inform processing throughout the representation. Dieleman mentions another conditioning approach without specifying it.

The final question returns to fresh-noise injection: can generation be deterministic without it? The same denoiser can support stochastic or fully deterministic sampling. Deterministic trajectories are especially useful when moving between the data and noise distributions in both directions, and when training a student to reproduce a teacher’s trajectory. The consistency-distillation approach discussed here uses a deterministic teacher; that requirement does not extend to every way of training a consistency model. Dieleman closes by pointing to a longer version of the talk on YouTube. The practical distinction remains that training a denoiser leaves several generation procedures available, each with different consequences for robustness, reversibility and inference cost.

37:2537:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

37:25 · section reference included

Resources

From the talk

  • EQ-VAEPaper8:55

    Research on equivariance regularization for latent representations used in generative image modeling.

  • Original implementation, model configuration, checkpoint information and sampling instructions for Stable Diffusion v1.

  • GLIDEPaper27:49

    Early large-scale text-guided diffusion research comparing guidance methods for image generation and editing.

  • Introduces direct noise-to-data generation with one-step or multistep sampling, using either distillation or standalone training.

  • Explains rectified flow and repeated rectification for straighter generation trajectories and fewer numerical sampling steps.

Updates since the talk

  • JAX API changesDocumentation

    Official release history documenting the transition from pjit to jax.jit and changes to sharding interfaces.

Transcript download
Full transcript

Loading transcript…