Contents
  1. Purpose and development
    1. From intent to usable media
    2. Must-know turning points
  2. How samples are produced
    1. Signals, latents and media codes
    2. Decoding, competition and sequence prediction
    3. Learning to generate from noise
    4. Prediction fields and sampling rules
  3. Directing image generation
    1. Conditions, references and guidance
    2. From spatial latents to image detail
    3. Source-guided changes and spatial control
    4. Masks and preservation boundaries
  4. Generating sound
    1. From sound representations to speech
    2. Pronunciation, delivery and local speech edits
    3. Music as events, recordings and parts
    4. Generating acoustic events
  5. Motion, identity and continuity
    1. Generating across frames
    2. Identity across variations
    3. Continuity through change and repair
    4. Sound and picture on shared time
  6. Creative production and assessment
    1. Keep the work editable
    2. Assess the intended result
    3. What media scores actually measure
  7. Origins and permitted use
    1. Provenance through production
    2. Permission for the actual use
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

Generative media

Generative media lets creators specify desired content without constructing every pixel, sound sample or frame. The engineering challenge is making that freedom controllable: choosing what may vary, preserving what has already been accepted, and checking the finished result. Understanding the representations and generation methods makes those decisions easier—and explains why a convincing sample can still be the wrong deliverable.

Purpose and development

From intent to usable media

Generative media uses learned models to produce or modify images, sound and moving images. It supports creating alternatives, transforming existing material and completing missing regions. These operations reduce the need to construct every detail manually, but leave creative direction, selection and finishing as distinct responsibilities.

A creative brief states the purpose, intended audience, required content, permitted variation and delivery constraints. Agreeing on scope and revision expectations makes exploration manageable. An exploratory product concept may welcome an invented shape; a depiction of an existing product may require its geometry and lettering to remain unchanged. The same attractive image can satisfy the first brief and fail the second. Adobe's brief-writing guidance connects these decisions to deliverables, budgets and review.

Keep three judgments separate. Perceptual plausibility concerns whether something looks or sounds convincing. Reference fidelity concerns whether it preserves specified source details. Factual truth concerns whether the depicted event or asserted fact is real. A realistic generated scene need not document an event, and even a valid record of its creation does not make the scene true.

Conditional generation uses supplied information to guide an output: text can describe a scene, an image can establish a subject, and audio can guide a voice. Multimodal operation contracts introduce that distinction. Here the next task is to understand what those inputs influence—and which requirements need enforcement outside the generator.

Must-know turning points

Media synthesis predates learned generators. Programmed synthesis specified how signals should be constructed; example-based methods reused patterns from supplied material. Learned methods shifted more of that construction into fitted parameters. These traditions coexist because explicit control and learned completion address different needs.

Selected developments across sound and image synthesis.
DevelopmentContribution
1963 — Mathews's programmable instrumentsAt Bell Laboratories, M. V. Mathews described instruments that computed sound from a separately specified musical score.
1999 — Example-based texture synthesisEfros and Leung grew textures by sampling pixels whose example neighborhoods matched the surroundings already synthesized.
2014 — Generative Adversarial NetsGoodfellow and colleagues trained a generator using an adversary that distinguished generated samples from training data.
September 2016 — WaveNetDeepMind announced learned waveform-sample prediction, with improved naturalness in its tested speech comparisons but expensive sequential generation.
2020 — DDPMHo and colleagues developed effective image generation through learned, repeated denoising.
2022 publication — Latent Diffusion ModelsRombach and colleagues moved denoising into compressed image representations, trading retained detail against computation.
2022 — Video Diffusion ModelsHo and colleagues combined spatial and temporal interactions when denoising frame blocks.

The progression is not a ranking. A programmable instrument exposes exact parameters; a texture synthesizer works from a particular example; a learned generator can supply variation that neither explicitly specifies. The important change is where structure comes from and how much control remains available to the creator.

How samples are produced

Signals, latents and media codes

A representation is numerical data that preserves distinctions needed by a later operation. An encoder transforms source material into that representation; a decoder converts a compatible representation into an output. A latent representation is an intermediate learned representation rather than the visible or audible signal itself. Shared representation principles explain the broader idea.

The object being generated determines the final reconstruction step.
Generated objectHow it becomes mediaConsequence
Signal valuesPixel values form an image; waveform samples form audio.Generation works directly at the signal's granularity.
Continuous latentsA learned decoder maps arrays of real-valued features to pixels or acoustic features.Compression reduces representation size but may discard fine distinctions.
Discrete media codesInteger indices select vectors from a learned codebook; a decoder reconstructs media from those vectors.Codes represent learned visual or acoustic patterns, not necessarily words or named objects.

Quantization chooses entries from a finite codebook instead of retaining arbitrary continuous vectors. EnCodec, for example, compresses waveform segments into vectors and uses successive codebooks to represent residual error. More codebooks carry more information at greater bitrate. Image latents can instead retain a coarser spatial grid. In the ComfyUI workshop's historical example, a 512×512×3 image maps to a 64×64×4 latent. That is a tensor-size comparison, not a measured runtime speedup.

A representation useful for matching meanings is not automatically a reversible encoding. Decoding requires a model trained to interpret that space, and its output can be approximate. Linus Lee's embedding experiments used purpose-built decoders and adapters to reconstruct text or generate images from embeddings. They do not imply that arbitrary embedding APIs preserve recoverable originals.

Decoding, competition and sequence prediction

A generation method is the procedure that produces a sample using learned parameters and available inputs. Training changes parameters; generation ordinarily holds them fixed while changing the candidate. Fitting is not inference develops this boundary.

A variational autoencoder, or VAE, learns to reconstruct examples through latent samples while making those samples compatible with a chosen prior—a distribution from which new latent values can be drawn. Kingma and Welling's 2013 Auto-Encoding Variational Bayes trains an encoder to describe a distribution of latent values for each example and a decoder to reconstruct data from them. Generation draws from the prior and decodes, without first encoding an existing image. Balancing reconstruction with the prior makes this sampling possible, but does not make the representation lossless or its concepts independently controllable.

A generative adversarial network, or GAN, trains two learners. The generator maps noise to candidate data; the discriminator learns to distinguish candidates from training examples. Their competing objectives supply the learning signal. After training, the generator produces samples without consulting the discriminator. That discriminator is neither a publication reviewer nor a certificate of correctness. Balancing the two learners is a training challenge; the ideal distribution-matching result assumes capacities and optimization conditions that practical systems need not satisfy.

Autoregressive generation predicts each element from preceding elements. Van den Oord and colleagues' 2016 PixelCNN uses masked convolutions to prevent access to later pixel values. Known training images allow many predictions to be computed together, but sampling remains sequential: a newly sampled value becomes context for the next prediction. Esser and colleagues' Taming Transformers, first posted in 2020, applies sequence prediction to compressed image-code indices and then decodes the completed grid. Compression shortens the sequence while introducing a reconstruction bottleneck.

These methods differ in training signals, sample dependencies and representations—not simply output quality. Direct decoding avoids a long sequence of sampling decisions, while autoregression makes earlier sampled content explicit context. The next family takes another approach: repeatedly revise an entire noisy state.

Learning to generate from noise

Diffusion learns generation through transitions between noise levels. Sohl-Dickstein and colleagues' 2015 formulation gradually destroyed data structure and learned a reverse chain. Many manageable transitions offered a way to model a complex distribution without specifying it in one difficult step.

In Denoising Diffusion Probabilistic Models (DDPM), training combines images with known Gaussian noise: random values drawn from a bell-shaped distribution. Across examples and noise levels, the network learns to predict the added noise from the corrupted image.

xt=αˉt x0+1−αˉt ϵx_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon Here x0x_0 is the training image, ϵ\epsilon is sampled Gaussian noise, and αˉt\bar\alpha_t is the scheduled retained-signal fraction at level tt. Training reduces squared error between known ϵ\epsilon and predicted ϵθ(xt,t)\epsilon_\theta(x_t,t); θ\theta denotes learned weights.

Sampling freezes the weights and starts with independent noise. Each noise prediction guides an update toward a less noisy state. Repeated updates build a new sample; they do not require an identified training image.

Learning and sampling start differently

The sampling path has no supplied training image.

Training image
Noise and level
Noisy example
Fit noise predictor
Independent noise
Reverse updates
Generated sample
Training changes weights; sampling changes a candidate using frozen weights. Only learned parameters cross between these paths; sampling starts from independent noise.
Read the diagram as text
  • Training image.
  • Noise and level.
  • Noisy example.
  • Fit noise predictor.
  • Independent noise.
  • Reverse updates.
  • Generated sample.
  • Training image → Noisy example: Data.
  • Noise and level → Noisy example: Corruption.
  • Noisy example → Fit noise predictor: Input.
  • Noise and level → Fit noise predictor: Noise target; level input.
  • Fit noise predictor → Reverse updates: Learned parameters; held fixed.
  • Independent noise → Reverse updates: Initial state.
  • Reverse updates → Generated sample: Result.

This separation does not rule out memorization. Extraction research recovered near-copies of particular training images from studied diffusion systems through repeated generation and matching. Such reproduction is different from a new depiction that merely shares a person or category. The reported counts depend on the attack budget and matching procedure, not a population-wide copying rate.

Prediction fields and sampling rules

The learned prediction and its application are separate choices. A noise schedule specifies corruption levels. A sampler specifies how predictions update the current state. A sampling step performs an update. Song, Meng and Ermon's DDIM, introduced in 2020, reused a trained diffusion predictor with a different update rule and fewer selected noise levels. Setting its added-noise term to zero gives a deterministic trajectory from the initial noise; fewer evaluations still trade computation against result quality.

A score describes how log probability changes around a candidate, providing a direction toward more probable states—not an aesthetic rating. Estimating this direction is difficult where training examples provide little coverage. Song and Ermon's 2019 method learns scores after adding several levels of noise; sampling follows those directions with random perturbations, progressing toward lower noise. Yang Song and colleagues' 2020–2021 continuous-time framework connected this approach with diffusion. It supports both random sampling trajectories and deterministic probability-flow trajectories that share distributions at corresponding times under exact scores and continuous integration. Their individual paths differ, and finite numerical implementations only approximate that result.

Flow matching, introduced by Lipman and colleagues in 2022, learns a velocity field: a direction and rate of movement at each state along a chosen probability path. Training regresses these directions using tractable conditional paths. Generation follows the learned field with a numerical solver. The framework includes diffusion-related paths but is not restricted to reversing one particular corruption process. Training without simulating the full flow does not mean generation takes zero or one step.

Three ways to reduce generation work have different consequences.
ChangeWhat changes
Use fewer sampler updatesThe sample traverses fewer numerical updates; approximation and perceptual quality can change.
Reuse intermediate featuresMethods such as DeepCache reduce computation within updates; stale features can alter colors or shapes.
Train a cheaper approximationDistillation changes learned behavior to approximate a more expensive process and requires training work.

The last option belongs to Distillation. For creative work, compare completed outputs under a specified configuration: greater numerical effort is not itself evidence of better composition, identity or usefulness.

Directing image generation

Conditions, references and guidance

Conditioning supplies information that changes the generator's output distribution. A reference image is existing visual material used to guide selected properties. Text describes intent; a subject reference supplies appearance; a style reference supplies visual treatment; a spatial map supplies arrangement. A reference can carry several of these at once. IP-Adapter's image features, for example, contain both content and style rather than an identity-only signal.

Cross-attention lets the states being updated obtain information from separate conditioning features. Other designs concatenate conditioning channels with the generated representation. These are information paths, not constraint solvers. Multimodal interaction explains their shared role; prompting explains instructions supplied without weight updates.

Ho and Salimans's 2022 classifier-free guidance trains conditional and unconditional prediction by sometimes dropping the condition. Sampling combines the two predictions to emphasize their difference.

ϵ^=ϵu+s(ϵc−ϵu)\hat\epsilon=\epsilon_u+s(\epsilon_c-\epsilon_u) Here ϵu\epsilon_u and ϵc\epsilon_c are unconditional and conditional noise predictions at the same state and noise level; ss is the guidance scale in this convention. At s=1s=1, the expression equals the conditional prediction. Above one, it extrapolates in the conditional direction. Stronger guidance changes the sampled distribution and can narrow diversity; it does not enforce the prompt.
Controls that should not be conflated.
ControlRoleNot a guarantee of
Guidance scaleChanges conditional steering; some implementations use negative conditioning in the comparison branch.Forbidden content being absent.
Source-corruption strengthChanges how much source information is removed before regeneration.Preservation of a particular object or pixel.
Random seedInitializes a pseudorandom sequence.Semantic identity or identical results across changed execution environments.

Stronger steering can also damage appearance. Google's 2022 Imagen paper reports that high guidance improved text alignment while producing oversaturation: predicted clean pixels could leave their training range, and repeated updates compounded the mismatch. Its dynamic-thresholding remedy applies to that pixel-space formulation, not automatically to every latent model.

From spatial latents to image detail

Image synthesis separates the representation, prediction network, sampling rule and decoder. Spatial latents retain a grid at reduced resolution, often with additional feature channels. This keeps useful neighborhood structure while reducing the size of the object repeatedly processed. The decoder converts the final latent into pixels; its compression and reconstruction limits remain part of the result.

A U-Net is a multiscale prediction network that combines coarse and fine features. A diffusion transformer instead predicts from latent patches using transformer blocks. Peebles and Xie's 2022 DiT changed the prediction architecture, not diffusion's iterative sampling principle. Smaller patches create more tokens and more computation even without increasing parameter count. Using a transformer therefore does not imply autoregressive generation.

The sampler repeatedly updates a spatial latent using network predictions. The decoder converts the final latent into pixels; changing the prediction architecture does not remove the sampling loop.

Generative super-resolution adds higher-resolution content conditioned on a lower-resolution image. In cascaded diffusion, each stage learns plausible additional detail. Training only on clean low-resolution conditions can leave a mismatch when inference supplies imperfect generated ones; conditioning augmentation introduces noise or blur during training to improve robustness. Excessive augmentation also removes useful guidance. A sharper output cannot certify hidden lettering or repair an incorrect coarse composition merely by adding detail.

Inspect composition, typography and small details separately. Increasing resolution changes the available canvas, not the truth of its contents. The distinction between plausible reconstruction and observed content matters especially when generated detail could be mistaken for recovered evidence.

Source-guided changes and spatial control

Image-to-image transformation generates a result using an existing image as a guide. Meng and colleagues' 2021 SDEdit deliberately adds noise to that guide, then regenerates it with a pretrained reverse process. More corruption permits greater change by removing more source information; less corruption preserves more structure but can retain unwanted strokes or artifacts. This is a realism–faithfulness tradeoff, not an object-preservation guarantee.

Inversion pursues a different goal: finding a starting representation and trajectory that reconstruct a reference under a specified model and sampler. Null-text Inversion shows why a deterministic sampler is not automatically an exact inverse. Finite-step inversion errors accumulate, and guidance can amplify them. Good reconstruction under the original instruction also does not establish preservation after the instruction changes.

Brooks, Holynski and Efros's InstructPix2Pix, introduced in 2022, learns from image pairs and editing instructions. Separate image and instruction guidance strengths trade preservation against modification. A request can be local in meaning without being local in pixels: changing a skyline may appropriately change its reflection. The paper also documents unwanted widespread changes and difficulty isolating objects.

Structural conditions specify arrangement more directly. Edges describe visible boundaries; depth describes distance structure; pose maps locate body landmarks. Zhang and colleagues' 2023 ControlNet learns a branch that injects spatial features into a frozen generator. The branch requires training, but a new pose map is an inference input—not a new subject fine-tune. Learned feature injection encourages the requested arrangement without enforcing exact geometry or identity. A pose map and an editable-region mask therefore solve different problems.

Masks and preservation boundaries

Inpainting fills a selected region; outpainting extends the canvas beyond supplied content. A mask identifies regions according to an explicit convention. RePaint, introduced in 2022, combines correctly noised known pixels with generated unknown regions during sampling. It revisits noise levels to reconcile the boundary. Enlarging the canvas and marking its surroundings unknown extends the same formulation, but does not guarantee every outpainting implementation preserves its input.

Compositing combines visual elements into a result. For a bounded replacement, retain the source on one layer and place generated content on another, with a mask controlling visibility. This application of layer masks keeps the original independently accessible. It does not establish equality of the final composite: opacity, adjustments, transformations and blended boundaries still affect rendering.

If pixels must remain exact, specify the decoded source array, channel order, bit depth, color handling and coordinate mapping. Copy protected pixels rather than asking a model to reproduce them. Avoid unintended resizing and lossy export, then compare the protected region after decoding the exported file. Decoded-pixel equality differs from identical file bytes or matching appearance in different viewers.

Illustrative pseudocode

Python-like pseudocode
# Arrays already share coordinates, channels and color space.
# protected is True wherever source pixels must remain exact.
composite = replacement.copy()
composite[protected] = source[protected]
export_losslessly(composite, destination)
exported = decode_with_same_contract(destination)
assert array_equal(exported[protected], source[protected])

Use generation for content that must be invented; use explicit rendering for known lettering, layout or protected pixels. Production editors already combine learned and conventional stages. Google's 2021 Magic Eraser system, described in Magic Editor Under the Hood, joined detection, segmentation, inpainting and device rendering into one interaction.

Generating sound

From sound representations to speech

A waveform is a sequence of sound-signal measurements over time. Sample rate counts measurements per second; frequency counts oscillations per second. A spectrogram represents changing frequency content across short time windows; a mel-spectrogram uses a perceptually motivated frequency scale. These representations let a speech system predict acoustic features before producing sound. A magnitude-only spectrogram omits phase—the position within each oscillation—so it cannot directly reconstruct the exact waveform. A vocoder, in the synthesis systems discussed here, produces waveform audio from acoustic features, supplying detail those features do not specify.

Text-to-speech, or TTS, must connect written content to acoustic realization. Earlier systems made much of that structure explicit. Dennis Klatt's 1980 software synthesizer used controlled sound sources and digital resonators representing vocal-tract acoustics. Time-varying parameters provided a flexible instrument for speech-perception experiments. Learned synthesis changes how those acoustic decisions are obtained rather than eliminating them.

Different systems learn different boundaries between text and sound.
SystemPredicted representationWaveform production
WaveNet, DeepMind, 2016The next waveform-sample distribution, conditioned on earlier samples and other inputs.Sample-by-sample generation.
Tacotron, Wang and colleagues, 2017Successive mel-spectrogram frames from characters, learning alignment through attention.A postprocessor predicts a linear-frequency magnitude spectrogram; Griffin–Lim reconstructs audio.
Tacotron 2, Shen and colleagues, 2017–2018A predicted mel-spectrogram.A modified WaveNet neural vocoder supplies waveform detail.
Voxtral TTS, Mistral, 2026Autoregressive audio-frame progression with semantic codes and flow-generated acoustic values.A codec decodes the generated representation into audio.

A neural audio codec learns compression and reconstruction rather than assigning one code to each word. Its decoder must reconstruct sound from the information retained in the codes. Voxtral illustrates how autoregression and flow matching can coexist: one establishes progression across frames, while the other generates acoustic values within a frame. Neither a frame nor a codec token is necessarily a waveform sample. Conversational streaming and turn-taking are developed in Voice and Real-Time AI.

Across frames and within a frame

Across frames and within a frameThe current autoregressive decoder state supplies a semantic head and an iterative acoustic flow transformer. Acoustic output is quantized before feedback to the next frame. Generated semantic and acoustic representation enters the audio codec decoder.Text + reference audioconditioning tokensAR decoder · frame kcurrent hidden stateAR decoder · frame k+1next frame progressionExpanded computation for one frame kCurrent hidden stateSemantic classifierSemantic codeAcoustic flowwithin-frame iterationQuantize acousticcontinuous values → codescontinuous valuesRepeated flow evaluationsQuantized feedbackGenerated representationsemantic + acoustic informationAudio codec decoder → waveform

Outer autoregression advances codec frames. Inner flow evaluations construct acoustic values within the current frame; neither step is a waveform sample or a fixed latency budget. This is the reported Voxtral TTS architecture, not a universal speech pipeline.

Voxtral TTS, reported March 2026: sequence prediction advances through audio frames; flow matching supplies acoustic values within a frame. Quantized acoustic information supports subsequent progression, and the codec reconstructs waveform audio.

Pronunciation, delivery and local speech edits

Speech content and performance require different controls. Normalization determines how written forms such as numbers are spoken. Pronunciation determines their speech sounds; a phoneme is a sound unit that distinguishes words in a language. Prosody includes pitch patterns, timing, emphasis, rate and pauses. The Speech Synthesis Markup Language specification separates these roles, while treating many controls as processor-dependent guidance rather than absolute specifications.

For a specialized acronym, changing a pronunciation dictionary can be more direct than repeatedly asking for a different voice. The ElevenLabs speech workshop describes phoneme-based dictionary entries for this purpose. Such a control affects the spoken output, not recognition of incoming speech; available alphabets and model support still belong to the particular interface.

Duration makes the text-to-time relationship explicit. Yi Ren and colleagues' 2020 FastSpeech 2 expands phoneme representations according to predicted durations and generates acoustic frames in parallel. Pitch and energy provide additional conditions. Training obtains durations from alignment and acoustic variables from recordings; ablations found that removing pitch or energy conditioning reduced listener-rated quality. One sentence can therefore have several valid spoken realizations even when its words are fixed.

Local speech editing must decide whether timing may change along with the words. Meta's 2023 Voicebox fills missing speech representations using text and surrounding audio. For word replacement, it retains the durations of unchanged speech sounds and predicts durations for the new ones. Its alignment-preserving style transfer instead keeps the existing mapping between text and acoustic frames. The first operation can change how long the passage lasts; the second preserves that timing structure while changing delivery. A longer replacement can disrupt later synchronization, and preserving acoustic features does not prove sample-exact preservation after waveform decoding.

Music as events, recordings and parts

Music can be generated as instructions or as recorded sound. Mathews's programmable instruments separated a score from the computation of its waveform. MIDI likewise represents performance instructions—notes, timing, instruments and relative volumes—not recorded samples. Changing those instructions changes what the receiving instrument performs; the audible result depends on that renderer.

Musical requirements span different time scales. A beat supplies a recurring pulse; melody is a sequence of pitches; harmony concerns pitches sounding together. Timbre is the sound character that distinguishes sources even when pitch, loudness and duration match. It depends on frequency content and its evolution, especially the onset. Changing an instrument's timbre is therefore not simply changing its pitch.

Meta's 2023 MusicGen generates several streams of audio-codec indices with an autoregressive transformer. Their interleaving determines which predictions can use which earlier codes; prediction steps are not playback samples. Text supplies descriptive conditioning. Melody conditioning uses a chromagram, pitch-class activity over time, simplified to emphasize dominant pitches and discourage copying the reference recording. This guides a musical relationship rather than guaranteeing exact score reproduction or full-song structure.

Stems are separately represented musical parts, such as an instrument performance or synthesizer layer, that combine into a mix. StemGen, introduced in 2023, learns to generate a complementary part conditioned on a mixture of other parts. This supports adding or refining a layer instead of regenerating the entire recording. Retaining stems preserves a direct revision boundary that a single mixed waveform does not expose.

Choose the retained artifact according to the expected revision.
ArtifactDirectly available change
Symbolic performanceEdit notes, timing or instrument instructions, then render again.
Separate stemsChange one retained part or its contribution to the mix.
One rendered mixtureEdit the combined recording; changing an embedded part requires additional processing or regeneration.

Generating acoustic events

Sound effects are organized around events and acoustic context rather than words or musical form. A mechanical impact needs a recognizable source, an onset, a decay and an appropriate surrounding space. Several plausible sounds may fit the description while differing in duration, intensity or background noise.

Liu and colleagues' 2023 AudioLDM applies latent diffusion to audio. It generates continuous latents, decodes a mel-spectrogram and uses a vocoder to produce a waveform; mel is a perceptually motivated frequency scale. Diffusion training uses audio embeddings from a pretrained language–audio alignment model, while sampling can use text embeddings. That reduces dependence on paired descriptions during diffusion training, but the alignment model still supplies cross-modal knowledge. The learned generator also supports audio inpainting and style transfer.

Review acoustic plausibility separately from meaning. The ElevenLabs Text to Bark demonstration used genuinely synthesized barking, but the claimed translation into a dog's language was an April Fool's premise. A convincing sound does not establish the semantics attributed to it. For production effects, listen for the intended event and acoustic character; synchronization with a visible event is another check.

Motion, identity and continuity

Generating across frames

A frame is an image associated with a time; a clip is a temporal interval; a shot is a continuous view between cuts. Video generation must organize change across those images. Repeated low-latency image edits can make an interface feel continuous, as the FLUX demonstration shows, without establishing the temporal coherence expected of video.

Joint denoising updates a block of frames together, allowing information at one time to influence predictions at another. In Video Diffusion Models, spatial operations combine features within each frame, while temporal attention combines features across frames at corresponding grid locations. Alternating these operations lets information travel across both space and time; it does not explicitly track objects or enforce correct motion. Gupta and colleagues' 2023 W.A.L.T. instead works with continuous latents compressed in space and time, using spatial and spatiotemporal attention windows. Compression reduces the representation, temporal interaction connects its states, and conditioning on preceding frames supplies context for continuation.

Google's VideoPoet, first submitted in 2023, instead generates ordered discrete video codes. Each new prediction uses conditioning and preceding codes; a decoder reconstructs pixels. Its longer-generation example predicts successive one-second segments from the preceding generated second. That is bounded recent context, not access to an unlimited history. Playback frame rate, temporal compression and generation context therefore describe different quantities.

Spatial, temporal and causal context

Spatial, temporal and causal contextVDM alternates spatial operations within a frame with temporal operations at matching grid coordinates. VideoPoet predicts the next code from selected recent context, current prefix and conditioning; unavailable older and future codes have no dependency arrows.VDM: two operations on a joint frame blockFrame 1ABCFrame 2ABCSolid: spatialDashed: temporalMatching feature-grid coordinates, not tracked object identities.VideoPoet: bounded context before a next-code predictionOlderunavailablep₁retainedp₂retainedc₁currentc₂currentNextpredictFutureunavailableSelected recent segment contextTask conditioningCode slots are schematic; they do not assert a code count, frame rate or unlimited history.
Video Diffusion Models alternate spatial operations within frames and temporal operations at corresponding feature-grid sites. VideoPoet predicts codes from available preceding codes and selected recent segment context. These are information dependencies, not elapsed time, a quality ranking or an illustration of W.A.L.T.’s windowed architecture.

Text-to-video supplies a description; image-to-video anchors generation to an image; video-to-video supplies an existing sequence for transformation. First and last frames constrain endpoints, but do not uniquely determine the intervening motion. Interpolation fills between known endpoints; continuation extends beyond available content. In the Veo workshop, reusing a static-image prompt omitted what should happen next, including who should speak. A motion instruction needs temporal content, not just another scene description.

Camera conditioning specifies viewpoint changes rather than each object's motion. The 2024 CameraCtrl paper converts camera geometry, position and rotation into ray representations that condition generation. This is more explicit than prose such as “move the camera right,” but remains learned conditioning—not a guarantee of exact reconstruction or physically valid motion.

Identity across variations

Identity consistency preserves distinguishing attributes of a particular subject while allowing intended changes in pose, setting, lighting or delivery. A category such as “brown dog” does not identify one dog; a shared illustration style does not make two characters the same. A useful test varies the requested circumstances instead of rewarding copies of one reference composition.

Reference-time conditioning and personalization differ. Ye and colleagues' 2023 IP-Adapter encodes an image and supplies its features through a separate attention path. Once trained, the adapter accepts new references without a new subject-specific training run. Multiple views can supply additional appearance information, but the image features still combine content and style.

Ruiz and colleagues' 2022 DreamBooth adapts a generator to subject examples and a distinctive identifier. Prior-preservation training helps retain class meaning and diversity. Documented failures include changed subject features, copied settings and reduced variation. Low-rank adaptation, or LoRA, instead names a compact way to learn parameter updates; it is not an identity guarantee. Post-training explains parameter adaptation.

Voice references pose the same separation between identity and permitted variation. Google's 2022 AudioLM combines semantic tokens for longer-range structure with acoustic tokens for speaker and recording properties. Conditioning on a short real-audio prefix supported voice and recording continuity in its evaluated continuations, including unseen speakers. This is bounded continuation evidence, not unlimited identity preservation or permission to reproduce someone's voice.

Continuity through change and repair

Temporal consistency means coherent evolution over time, not maximum similarity. Flicker, jerky motion, identity drift and broken scene state are different failures. Object permanence concerns an object's continued existence when temporarily hidden. A clip can look smooth yet return a different object after an obstruction. Conversely, a static clip can score well on consistency while failing the requested action—a problem explicitly recognized by VBench.

Continuation reuses context. In Video Diffusion Models, accepted clean frames remain fixed targets while noisy versions participate internally in sampling new frames. Reusing those frames does not mean regenerating and blending the accepted output.

Generated history can itself be imperfect. The March 2026 Helios preprint retains the first frame as an anchor and corrupts historical frames during training to improve tolerance of color shifts, noise and reduced resolution. Its ablations report worse drift without these components. The lesson is to model the conditions a continuation actually receives, not to assume that earlier generated frames are pristine.

Temporal repair can reuse visible information from other frames. Zhou and colleagues' 2023 ProPainter completes optical flow—estimated correspondence between frame locations—then propagates available content into masked regions and refines it. Incorrect correspondence can misalign content, and remaining holes still require synthesis. Computer Vision's treatment of apparent motion explains why correspondence is an estimate.

Cross-frame repair can reuse content visible elsewhere when estimated correspondence passes its checks. Masked regions still lacking evidence require synthesis and refinement. This fictional frame pair illustrates the mechanism, not a measured model result.

Audio boundaries need similarly specific repairs. A crossfade lowers one clip's level while raising another's, smoothing their handoff. It cannot fix mismatched words, rhythm, speaker identity or room acoustics. Continuation context addresses those broader relationships differently. Neither smooth sound nor smooth video establishes a validated model of environmental dynamics; that stronger claim belongs to World Models.

Sound and picture on shared time

Audiovisual synchronization connects sound and visible action on a shared time basis. It is separate from semantic correspondence and source attribution. A door-slam sound can be appropriate but late, or synchronized with the wrong visible door. Shared coordinates and timestamps establish the address system for those relationships.

Foley supplies sound effects matched to visible actions. The 2024 FoleyCrafter system separates semantic conditioning—what should sound—from temporal control—when it should sound. Dubbing supplies replacement speech for existing picture; lip synchronization concerns correspondence between that speech and visible mouth movement. These require more than independently acceptable tracks.

Prajwal and colleagues' 2020 Wav2Lip uses a frozen synchronization expert alongside a visual-quality discriminator. The expert compares short lower-face sequences with audio, learning from aligned and shifted pairs. Whole-face reconstruction can underweight mouth errors because the mouth is a small region. Attractive faces therefore do not establish speech–lip correspondence. Conversely, Google's native-audio video interface produces audiovisual output in one workflow, but that capability does not certify every result's alignment.

A duration-changing speech repair can invalidate accepted timing. Moving downstream tracks together preserves selected timeline relationships, as Premiere's Sync Lock illustrates; it does not generate new matching mouth motion. If shot duration and visuals are fixed, the replacement must fit those constraints or the task requires renegotiation. Listen and watch the final combination, including edit boundaries, rather than approving tracks separately.

A longer speech take changes dependencies

Original speech occupies 2–5 s. Each duration describes a hypothetical replacement take starting at 2 s; this tool neither stretches audio nor generates media. Only selected downstream clips V-B, V-C and C-Q may move. V-A and music M stay fixed.

Duration delta: +1 s. Replacement ends at 6 s.
1 s beyond the original speech/picture slot. Retained picture has a 1 s gap before V-B; no new frames were created.
Speech–mouth event offset: +0.5 s. Moving later clips does not repair this internal mismatch.
Replacement speech, selected downstream shifts and unresolved lip correspondenceRetained source clip identities and durations remain unchanged. The selected policy changes only placement of downstream video and cue clips. Mouth motion in V-A stays fixed while the replacement speech event follows its hypothetical take.024681012Retained videoSpeech takeSound cueMusicMouth event in V-ASpeech event in RV-AV-BV-CC-QMR-4s: 2–6s3.5s unchanged4sTimeline time (seconds)
Retained source IDSelected to move?StartEndSource duration
V-ANo0 s5 s5 s
V-BYes6 s9 s3 s
V-CYes9 s11 s2 s
C-QYes7 s7.5 s0.5 s
MNo0 s12 s12 s

The marked mouth event remains at 3.5 s. For this timing fixture, each replacement take’s corresponding speech event is at its midpoint; no mouth motion is synthesized.

4 s replacement (delta +1 s)V-B / V-CCue C-QInternal event offset
Fixed picture5–8 / 8–10 s6–6.5 s+0.5 s, unresolved
Move selected clips6–9 / 9–11 s7–7.5 s+0.5 s, unresolved
Fictional timeline arithmetic, not measured synchronization quality. Replacement takes have different durations; moving selected downstream clips preserves their source identities and durations but neither fills picture gaps nor regenerates mouth motion.

Creative production and assessment

Keep the work editable

Exploration, selection, bounded revision and finishing need different controls. Exploration seeks useful alternatives. Selection commits to particular artifacts. Revision changes specified parts while protecting accepted decisions. Finishing combines and exports the actual deliverable. A storyboard makes planned shots, action, camera direction, dialogue and timing inspectable before generating finished sequences.

Whole-output generation reduces explicit handoffs but may leave fewer independently replaceable parts. Staged production retains shots, speech, music and graphic layers. Inspect starting images before animating them and individual clips before assembly; this localizes repair. The video-evaluation talk describes this pattern for catching character drift early. Its benefit is qualitative, not a measured universal saving.

Retain approved files and identify their versions. Frame.io version stacks illustrate reviewable revisions, though stack position alone is not an immutable approval identifier. A generation manifest can connect each artifact to references, masks, model versions, prompt processing, sampler settings, dimensions, precision, software and seed. This is an application record, not a universal standard. A seed alone cannot recreate an approved file across changed software, hardware or random-draw sequences.

Explicit composition keeps known timing and layout editable. Remotion, for example, defines a composition through dimensions, frame rate, frame count and a component rendered at a specified frame. Such structure complements generated footage. The duration-edit example shows why changing a speech take must update selected timing while preserving source identities. Apply explicit intent and bounded correction: a speech change should trigger checks of dependent timing, not erase unrelated source assets.

Measure creative outcomes separately from elapsed effort. In a randomized field experiment, 120 working designers used Midjourney V6.1 during ideation, implementation, both or neither while making an agricultural-product advertisement. AI-assisted ideation improved panel-rated novelty, relevance and complexity but took longer. Implementation-only use increased time and revisions without statistically significant gains on those creativity measures. This bounded result distinguishes idea quality from completion speed; revisions are not a candidate-rejection rate.

Assess the intended result

Evaluation assesses work against an intended purpose. A rubric makes the criteria explicit. Creative variation does not prevent evaluation: several outputs can meet required content while differing in appeal. Separate mandatory requirements, aesthetic preference, perceptual defects, editability and permitted use. Human-reference design explains the assessment procedure.

Literal correctness is not a complete definition of artistic success. An intentionally impossible object may be essential to an artwork, while the same distortion would fail a catalog image. Perceptual Evaluations: Evals for Aesthetics develops this distinction and the importance of whose preferences the evaluator represents. Define the target audience rather than treating one person's taste as a universal quality scale.

Match each criterion to the observation it needs.
CriterionInspection
Composition and written contentInspect the final image at its intended display size; check required objects, relationships and lettering.
Requested edit and preservationCompare the source and result; test exact protected regions when the contract requires equality.
Speech content and performanceListen against the script, pronunciation requirements, pauses and delivery intent.
Musical coherenceListen for the intended parts and development, not only clean recording quality.
Identity and temporal behaviorCompare references and watch full-speed motion, including occlusion, cuts and continuation boundaries.
Audiovisual correspondenceWatch and listen together; inspect the source and timing of speech and effects.
EditabilityPerform a bounded change and inspect what else it disturbs.

Evaluate a candidate set for useful alternatives, a selected asset for its assigned role, and the finished export for the whole brief. Include required captions or other accessibility features in that final review. A good intermediate clip can fail after assembly; approval belongs to the artifact actually delivered.

What media scores actually measure

Automatic scores are useful when their inputs and comparison units match the claim. Distribution similarity, reference similarity and instruction adherence are different properties. A score can improve while a mandatory detail remains wrong.

Common measurements and their interpretation boundaries.
MeasurementInputs and comparisonWhat it does not establish
Fréchet Inception Distance, FIDFits Gaussian means and covariances to image features from real and generated collections; lower distance means closer fitted statistics.Correctness of one image, prompt compliance or equality of the complete distributions.
Learned Perceptual Image Patch Similarity, LPIPSCompares paired images using weighted differences between normalized network features.Equal pixels or semantic correctness of either image.
CLIPScoreMeasures image–caption embedding compatibility; originally evaluated captions for supplied images.Every count, spatial relation, identity or editing requirement being satisfied.
Fréchet Audio Distance, FADCompares fitted audio-feature distributions with a background collection; it does not require a paired clean recording for each clip.Reference-free evaluation in the sense of needing no background distribution, or correct full-song structure.
Mean opinion score, MOSAverages numerical listener ratings under a specified protocol.An intrinsic, universally comparable property of an audio file.

Statistical and perceptual limitations are separate. Chong and Forsyth's FID analysis finds generator-dependent finite-sample bias, so equal sample counts do not remove every ranking distortion. An unsuitable feature space remains unsuitable even if its statistical estimate improves. For music, Gui and colleagues show that FAD interpretation changes with embeddings and reference collections. Their per-song adaptation helps inspect outliers, but short-excerpt listening judgments do not establish full-song creative usefulness.

Video features can underweight motion defects. Ge and colleagues' 2024 FVD content-bias study compared consistent frame distortions with frame-varying distortions that introduced additional temporal damage. The conventional Fréchet Video Distance implementation responded insufficiently to that damage in their experiments. VBench therefore offers a useful complementary principle: assess dynamics separately from consistency, since an unmoving clip can look stable. Neither approach alone establishes physical correctness.

Keep generation conditions, selection effort and assessment procedure visible. Comparing one unselected sample with another system's best of many changes the claim. Use matched-work comparisons and an explicit attempt policy, then retain criterion-specific inspection alongside aggregate scores.

Origins and permitted use

Provenance through production

Provenance records origins, responsible actors and transformations. An internal generation manifest helps a team reconstruct its work; recipient-verifiable provenance requires additional mechanisms. General lineage principles explain why an assertion about history is different from proof of correctness.

The Coalition for Content Provenance and Authenticity, or C2PA, specifies Content Credentials: signed assertions bound to an asset. Validation checks that binding and recorded assertions, with trust depending on the signer. It does not prove that a depicted event occurred, that all history was recorded or that inputs were licensed. Credentials can be missing or removed; their absence is not proof of fabrication.

The version 2.2 specification represents ingredients and actions such as creation, editing, placement and transcoding. Included ingredient histories can be validated recursively. An ordinary action list does not automatically assert completeness: the schema separately permits that claim. A composite can therefore have valid recorded ingredients while leaving another ingredient's prior history unavailable.

Assets and their recorded histories

Example

An asset is distinct from the manifest asserting its history.

Generated asset A
Manifest A: creation
Edited asset B
Manifest B: editing
Additional ingredientEarlier history unavailable.
Composite export C
Manifest C: composition
This conceptual credential chain records creation, editing and composition; it is not a literal serialization or a tested editor export. The additional ingredient has no supplied earlier history; that gap does not invalidate every recorded assertion or establish fabrication.
Read the diagram as text
  • Generated asset A.
  • Manifest A: creation.
  • Edited asset B.
  • Manifest B: editing.
  • Additional ingredient. Earlier history unavailable.
  • Composite export C.
  • Manifest C: composition.
  • Manifest A: creation → Generated asset A: Asset binding.
  • Manifest B: editing → Edited asset B: Asset binding.
  • Manifest C: composition → Composite export C: Asset binding.
  • Manifest C: composition → Manifest B: editing: Ingredient history reference.
  • Manifest B: editing → Manifest A: creation: Ingredient history reference.
  • Manifest C: composition → Additional ingredient: Recorded ingredient reference.

Watermarks embed signals in content; statistical detectors estimate synthetic origin from content features. NIST's synthetic-content report distinguishes both from provenance metadata. Watermarks face removal and coverage limits; detectors make false-positive and false-negative errors. None alone establishes publication authority. Keep sensitive references and identities in access-controlled records where possible instead of exposing the entire production history publicly.

Permission for the actual use

Possession, technical access, a license and a person's consent answer different questions. Rights clearance means establishing the authority needed for the intended material, operation and distribution—not merely finding an asset that can be uploaded. Layered permission provides the general framework. The following matrix is a review structure, not a universal legal determination.

Record the proposed use and who is responsible for resolving it.
Material or activityAuthority to establishResponsible review
Training or adaptation examplesWhether acquisition and the proposed learning use are permitted, including applicable confidentiality and supplier terms.Dataset or model owner with the relevant rights reviewer.
Uploaded referencesWhether this recipient may receive the material and use it for the stated operation and secondary purposes.Source owner and service-path owner.
Recognizable person or voicePermission for the particular creation and distribution; a copyright license may not clear third-party personal rights.The accountable producer and appropriate consent or legal reviewer.
Musical composition and recordingWhich permissions apply to the underlying music and lyrics versus the particular recording.Music-rights reviewer for the actual sources and intended use.
Model/service use and output distributionThe applicable model and service terms, destination, audience, disclosure obligations and unresolved input rights.Service owner and publication owner.

A license can cover one layer without covering another. Creative Commons explains that its licenses do not clear third parties' publicity, privacy or personality rights. Attribution and a notice of modification do not resolve those interests. The U.S. Copyright Office similarly distinguishes a musical composition and lyrics from a particular sound recording; they are commonly owned and licensed separately. Performer and likeness interests may add further review questions depending on the use and jurisdiction.

Confidential references require review of the actual provider path, including access, retention and secondary-use terms—not an assumption based on a provider's name. Supplier-path governance develops that responsibility. Keeping inference internal also leaves obligations concerning acquired weights, data and dependencies.

Output copyrightability is another separate issue. The U.S. Copyright Office's January 2025 report treats human authorship case by case: human-determined expressive elements may qualify, while prompts alone were unlikely to suffice at the technological stage examined. That analysis does not clear references or likeness rights. Record human contributions and the actual permissions relied upon. A usable deliverable combines creative success, controlled revision and authority for its intended audience; none can substitute for the others.

Open questions

  1. Independent creative controls remain difficult because subject, style and setting can share representations. Progress would preserve distinguishing attributes under controlled changes without merely copying reference views.

  2. Long-range continuity must survive imperfect generated history while permitting meaningful change. Better systems would retain scene state through occlusion and revisitation, and sustain musical development, without relying on static similarity as success.

  3. Creative evaluation needs measurements that remain useful across different intentions and audiences. Progress would connect criterion-specific automated checks to independent human judgments and successful revisions, rather than improving aggregate appearance scores alone.

Follow the curated reading path through the speakers and demonstrations behind this entry.

23 min

AI Engineer World's Fair 2026 · 2026

Evaling Video Slop

Maor Bril

Cited in this entry

Develops temporal and audiovisual review beyond still-image quality, with checks at starting frames, individual clips and final assembly.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

86 matching talks

TalkSpeakerEventYear
Gorkem YurtsevenAI Engineer World's Fair 20252025
Ziv IlanAI Engineer Europe 20262026
Keegan McCallumAI Engineer World's Fair 20262026
Stephen BatifolAI Engineer Europe 20262026
Ekaterina DeynekaAI Engineer World's Fair 20262026
Linus LeeAI Engineer Summit 20232023
Nick HeinerAI Engineer World's Fair 20262026
Brendan O'DonoghueAI Engineer Europe 20262026
Robotics: why now?

Transcript reviewed

Quan Vuong, Jost Tobias SpringenbergAI Engineer World's Fair 20252025
Raia HadsellAI Engineer Europe 20262026
Chad Bailey, Brian JohnsonAI Engineer World's Fair 20252025
Ending AI Slop

Transcript reviewed

Thais Castello BrancoAI Engineer World's Fair 20262026
Don't be data poor

Transcript reviewed

Anuj IravaneAI Engineer World's Fair 20262026
Doug GuthrieAI Engineer World's Fair 20252025
Ronan McGovernAI Engineer World's Fair 20252025
Ahmed AhresAI Engineer World's Fair 20262026
Paige BaileyAI Engineer Europe 20262026
Sangwu LeeAI Engineer World's Fair 20262026
Veo 3 for developers

Cited in this entry

Paige BaileyAI Engineer World's Fair 20252025
HTML Is All Agents Need

Cited in this entry

James RussoAI Engineer World's Fair 20262026
MuhtesemAI Engineer Summit 20252025
Joseph NelsonAI Engineer Summit 20232023
Guillaume VernadeAI Engineer Europe 20262026
Sidney PrimasAI Engineer World's Fair 20262026
The Weekend AI Engineer

Transcript reviewed

Hassan El MghariAI Engineer Summit 20232023
Kat Kampf, Ammaar ReshiAI Engineer Code 20252025
Daniel ChalefAI Engineer World's Fair 20262026
Philip KielyAI Engineer World's Fair 20252025
Kyle KranenAI Engineer World's Fair 20252025
Thor Schaeff, PaulAI Engineer World's Fair 20252025
Rachna SrivastavaAI Engineer World's Fair 20252025
Charles FryeAI Engineer Summit 20232023
Siddharth AhujaAI Engineer World's Fair 20252025
Logan KilpatrickAI Engineer World's Fair 20252025
Dax RaadAI Engineer Code 20252025
Patrick LöberAI Engineer Europe 20262026
Alex DuffyAI Engineer World's Fair 20252025
Angus J. McLeanAI Engineer Europe 20262026
Sunny MadraAI Engineer World's Fair 20242024
Paige Bailey, Guillaume Vernade, Ian BallantyneAI Engineer Europe 20262026
Cat Wu, Thariq Shihipar, Simon WillisonAI Engineer World's Fair 20262026
Karina NguyenAI Engineer Summit 20252025
Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Defying Gravity

Metadata candidate

Kevin HouAI Engineer Code 20252025
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
AI Engineer Summit 20252025
Randall HuntAI Engineer World's Fair 20252025
Proactive Agents

Metadata candidate

Kath KorevecAI Engineer Code 20252025
See, Hear, Speak, Draw

Metadata candidate

Logan Kilpatrick, Simón FishmanAI Engineer Summit 20232023
Lars GrammelAI Engineer Summit 20232023
Rob CheungAI Engineer World's Fair 20242024
Barr YaronAI Engineer World's Fair 20262026
Travis FrisingerAI Engineer World's Fair 20252025
Diego Rodriguez, Eugene, Jonas Bauer, Shijia Liao, David Vorick, Alex AtallahAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer Summit 20252025
Todd FisherAI Engineer World's Fair 20262026
Eugene CheahAI Engineer Summit 20252025
Nicholas Kang, Michael AaronAI Engineer Europe 20262026
Phlo YoungAI Engineer World's Fair 20242024
Louis-François Bouchard, Paul Iusztin, Samridhi VaidAI Engineer Europe 20262026
Building a Chess Coach

Metadata candidate

Anant Dole, Asbjørn SteinskogAI Engineer Europe 20262026
Soumya Gupta, Jai ChopraAI Engineer World's Fair 20262026
Zeke SikelianosAI Engineer World's Fair 20252025
Keegan McCallumAI Engineer World's Fair 20252025
Mounir MouawadAI Engineer World's Fair 20252025
Git push, get an AI API.

Metadata candidate

Ryan Fox-TylerAI Engineer World's Fair 20242024
Preetika Bhateja, Daniel BumpAI Engineer World's Fair 20262026
Imagination Engineering

Metadata candidate

Eve BouffardAI Engineer World's Fair 20262026
Justin ReockAI Engineer Code 20252025
Shafik Quoraishee, Joanne SongAI Engineer World's Fair 20262026
Sharif ShameemAI Engineer World's Fair 20252025
Stephan SteinfurtAI Engineer Europe 20262026
Vikash Agrawal, Linda HavivAI Engineer World's Fair 20252025
Tara AgyemangAI Engineer Europe 20262026
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
Arturo NunezAI Engineer World's Fair 20262026
Danielle An, David HoeAI Engineer Europe 20262026
Jeff SchomayAI Engineer Summit 20232023
Vincent WendyEvent unknownUnknown
Abdul DakkakAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20262026
CJ Avilla, Harrison StallEvent unknownUnknown
Dumitru Erhan, Shane Gu, Nicole Brichtova, swyxAI Engineer World's Fair 20262026
Thor SchaeffAI Engineer World's Fair 20262026
Dean QuiñanolaAI Engineer World's Fair 20262026
Thais Castello BrancoEvent unknownUnknown

References

Coverage and source review
Processed transcripts
38 processed in full · 6 in the curated path
Automated source review
Passed
Metadata candidates
54 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. The State of Generative Media Today

    https://www.youtube.com/watch?v=P370D8Kmlkw&t=284

  2. High-Resolution Image Synthesis with Latent Diffusion Models

    https://arxiv.org/html/2112.10752

  3. Adobe: Create a concrete brief to share out in the discovery phase

    https://www.adobe.com/learn/illustrator/in-app/create-a-concrete-brief-to-share-out-in-the-discovery-phase

  4. C2PA and Content Credentials Explainer, version 2.2

    https://spec.c2pa.org/specifications/specifications/2.2/explainer/Explainer.html

  5. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

    https://arxiv.org/html/2208.12242

  6. The Digital Computer as a Musical Instrument

    https://www.digitalmusicacademy.ru/sites/default/files/content/mathews-1963.pdf

  7. Texture Synthesis by Non-parametric Sampling

    https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/papers/efros-iccv99.pdf

  8. Generative Adversarial Nets

    https://arxiv.org/html/1406.2661

  9. WaveNet: A Generative Model for Raw Audio

    https://deepmind.google/blog/wavenet-a-generative-model-for-raw-audio/

  10. Denoising Diffusion Probabilistic Models

    https://arxiv.org/html/2006.11239

  11. Video Diffusion Models: Space-time factorized U-Net

    https://arxiv.org/html/2204.03458

  12. Pixel Recurrent Neural Networks

    https://jmlr.csail.mit.edu/proceedings/papers/v48/oord16.pdf

  13. Building Generative Image & Video Models at Scale

    https://www.youtube.com/watch?v=xOP1PM8fwnk&t=313

  14. Taming Transformers for High-Resolution Image Synthesis

    https://arxiv.org/html/2012.09841

  15. High Fidelity Neural Audio Compression

    https://arxiv.org/html/2210.13438

  16. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

    https://www.youtube.com/watch?v=_FKeSzM9fPc&t=1029

  17. The Hidden Life of Embeddings

    https://www.youtube.com/watch?v=YvobVu1l7GI&t=916

  18. The Hidden Life of Embeddings

    https://www.youtube.com/watch?v=YvobVu1l7GI&t=952

  19. The Hidden Life of Embeddings

    https://www.youtube.com/watch?v=YvobVu1l7GI&t=663

  20. Auto-Encoding Variational Bayes

    https://arxiv.org/html/1312.6114

  21. Deep Unsupervised Learning using Nonequilibrium Thermodynamics

    https://proceedings.mlr.press/v37/sohl-dickstein15.pdf

  22. Extracting Training Data from Diffusion Models

    https://arxiv.org/html/2301.13188

  23. Denoising Diffusion Implicit Models

    https://arxiv.org/html/2010.02502

  24. Generative Modeling by Estimating Gradients of the Data Distribution

    https://papers.nips.cc/paper/9361-generative-modeling-by-estimating-gradients-of-the-data-distribution.pdf

  25. Score-Based Generative Modeling through Stochastic Differential Equations

    https://arxiv.org/html/2011.13456

  26. Flow Matching for Generative Modeling

    https://arxiv.org/html/2210.02747

  27. DeepCache: Accelerating Diffusion Models for Free

    https://arxiv.org/html/2312.00858

  28. You Might Not Need 50 Diffusion Steps

    https://www.youtube.com/watch?v=gHs5ZiY80PM&t=741

  29. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

    https://arxiv.org/html/2308.06721

  30. Adding Conditional Control to Text-to-Image Diffusion Models

    https://arxiv.org/html/2302.05543

  31. Classifier-Free Diffusion Guidance

    https://arxiv.org/html/2207.12598

  32. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

    https://www.youtube.com/watch?v=_FKeSzM9fPc&t=891

  33. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

    https://arxiv.org/html/2108.01073

  34. Diffusers Reproducibility

    https://huggingface.co/docs/diffusers/using-diffusers/reusing_seeds

  35. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

    https://arxiv.org/html/2205.11487

  36. Scalable Diffusion Models with Transformers

    https://arxiv.org/html/2212.09748

  37. Cascaded Diffusion Models for High Fidelity Image Generation

    https://arxiv.org/html/2106.15282

  38. Null-text Inversion for Editing Real Images using Guided Diffusion Models

    https://arxiv.org/html/2211.09794

  39. InstructPix2Pix: Learning to Follow Image Editing Instructions

    https://arxiv.org/html/2211.09800

  40. RePaint: Inpainting using Denoising Diffusion Probabilistic Models

    https://arxiv.org/html/2201.09865

  41. Photoshop: Add Layer Masks

    https://helpx.adobe.com/photoshop/desktop/create-masks/layer-masks/add-layer-masks.html

  42. Photoshop: Export Settings and Export Location Preferences

    https://helpx.adobe.com/uk/photoshop/desktop/save-and-export/export-files-to-different-formats/export-settings-and-export-location-preferences.html

  43. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

    https://www.youtube.com/watch?v=C13jiFWNuo8&t=452

  44. The Sampling Theorem

    https://dspguide.com/ch3/2.htm

  45. SciPy Signal Processing: Short-Time Fourier Transform

    https://docs.scipy.org/doc/scipy/tutorial/signal.html

  46. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions

    https://arxiv.org/html/1712.05884

  47. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

    https://proceedings.mlr.press/v202/liu23f/liu23f.pdf

  48. Software for a Cascade/Parallel Formant Synthesizer

    https://www.fon.hum.uva.nl/david/ma_ssp/doc/Klatt-1980-JAS000971.pdf

  49. Tacotron: Towards End-to-End Speech Synthesis

    https://www.isca-archive.org/interspeech_2017/wang17n_interspeech.pdf

  50. Voxtral TTS

    https://arxiv.org/html/2603.25551v1

  51. Speech Synthesis Markup Language (SSML) Version 1.1

    https://www.w3.org/TR/speech-synthesis11/

  52. Building Conversational AI Agents - Thor Schaeff, ElevenLabs

    https://www.youtube.com/watch?v=MPtCBaZn84A&t=3514

  53. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

    https://arxiv.org/html/2006.04558

  54. Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

    https://arxiv.org/html/2306.15687

  55. About MIDI—Part 1: Overview

    https://midi.org/about-midi-part-1overview

  56. ASA Standards Terminology: Timbre

    https://asastandards.org/terms/timbre/

  57. Simple and Controllable Music Generation

    https://arxiv.org/html/2306.05284

  58. StemGen: A music generation model that listens

    https://arxiv.org/pdf/2312.08723

  59. FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

    https://arxiv.org/html/2407.01494

  60. Building Conversational AI Agents - Thor Schaeff, ElevenLabs

    https://www.youtube.com/watch?v=MPtCBaZn84A&t=383

  61. Adobe: What Is a Storyboard?

    https://www.adobe.com/in/products/firefly/discover/what-is-a-storyboard.html

  62. Black Forest Labs: FLUX, Open Research, and the Future of Visual AI

    https://www.youtube.com/watch?v=x8Yb4RidLgM&t=1106

  63. Photorealistic Video Generation with Diffusion Models

    https://arxiv.org/html/2312.06662

  64. VideoPoet: A Large Language Model for Zero-Shot Video Generation

    https://arxiv.org/html/2312.14125

  65. Veo 3 for developers

    https://www.youtube.com/watch?v=hlcAZ2lX_ZI&t=244

  66. Let's go Bananas with GenMedia

    https://www.youtube.com/watch?v=BcWFc3H7Khg&t=2090

  67. ProPainter: Improving Propagation and Transformer for Video Inpainting

    https://arxiv.org/html/2309.03897v1

  68. CameraCtrl: Enabling Camera Control for Video Diffusion Models

    https://arxiv.org/html/2404.02101

  69. Black Forest Labs: FLUX, Open Research, and the Future of Visual AI

    https://www.youtube.com/watch?v=x8Yb4RidLgM&t=296

  70. LoRA: Low-Rank Adaptation of Large Language Models

    https://arxiv.org/html/2106.09685

  71. AudioLM: a Language Modeling Approach to Audio Generation

    https://arxiv.org/html/2209.03143

  72. VBench: Comprehensive Benchmark Suite for Video Generative Models

    https://arxiv.org/html/2311.17982

  73. The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

    https://www.youtube.com/watch?v=5dCAmSDOAjI&t=757

  74. Video Diffusion Models

    https://arxiv.org/html/2204.03458

  75. Helios: Real Real-Time Long Video Generation Model

    https://arxiv.org/html/2603.04379v1

  76. Adobe Premiere: Audio Crossfade Transitions

    https://helpx.adobe.com/premiere/desktop/add-audio-effects/apply-audio-transitions/audio-crossfade-transitions.html

  77. Evaling Video Slop

    https://www.youtube.com/watch?v=b_PmGocP4rc&t=963

  78. A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

    https://arxiv.org/html/2008.10010

  79. Google AI for Developers: Video generation in the Gemini API

    https://ai.google.dev/gemini-api/docs/video

  80. Adobe Premiere: Sync Lock to Prevent Changes

    https://helpx.adobe.com/premiere/desktop/edit-projects/change-clip-sequence/sync-lock-to-prevent-changes.html

  81. HTML Is All Agents Need

    https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=654

  82. Evaling Video Slop

    https://www.youtube.com/watch?v=b_PmGocP4rc&t=372

  83. Veo 3 for developers

    https://www.youtube.com/watch?v=hlcAZ2lX_ZI&t=1107

  84. Frame.io: Version Stacking

    https://help.frame.io/en/articles/9101068-version-stacking

  85. Remotion: The fundamentals

    https://www.remotion.dev/docs/the-fundamentals

  86. The Double-Edged Roles of Generative AI in the Creative Process: Experiments on Design Work

    https://pubsonline.informs.org/doi/full/10.1287/isre.2024.0937

  87. Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

    https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=462

  88. Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

    https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=750

  89. Building in the Gemini Era – Kat Kampf & Ammaar Reshi, Google DeepMind

    https://www.youtube.com/watch?v=fgkXEIbZpGc&t=194

  90. Adapting Fréchet Audio Distance for Generative Music Evaluation

    https://arxiv.org/html/2311.01616

  91. Evaling Video Slop

    https://www.youtube.com/watch?v=b_PmGocP4rc&t=133

  92. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

    https://arxiv.org/pdf/1706.08500

  93. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

    https://arxiv.org/html/1801.03924

  94. CLIPScore: A Reference-free Evaluation Metric for Image Captioning

    https://arxiv.org/html/2104.08718

  95. Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms

    https://arxiv.org/html/1812.08466

  96. ITU-T P.800: Methods for Subjective Determination of Transmission Quality

    https://www.itu.int/rec/dologin_pub.asp?id=T-REC-P.800-199608-I!!PDF-E&lang=e&type=items

  97. Effectively Unbiased FID and Inception Score and Where to Find Them

    https://arxiv.org/html/1911.07023

  98. On the Content Bias in Fréchet Video Distance

    https://arxiv.org/html/2404.12391

  99. Content Credentials: C2PA Technical Specification, Version 2.2

    https://spec.c2pa.org/specifications/specifications/2.2/specs/C2PA_Specification.html

  100. NIST AI 100-4: Reducing Risks Posed by Synthetic Content

    https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-4.pdf

  101. NIST Generative AI Profile: Third-Party Risk

    https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

  102. Creative Commons: Frequently Asked Questions

    https://creativecommons.org/faq/

  103. U.S. Copyright Office: Musical Works, Sound Recordings & Copyright

    https://www.copyright.gov/music-modernization/sound-recordings-vs-musical-works.pdf

  104. U.S. Copyright Office: Copyright and Artificial Intelligence, Part 2—Copyrightability

    https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-2-Copyrightability-Report.pdf

  105. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

    https://www.youtube.com/watch?v=_FKeSzM9fPc&t=1919

  106. Building Generative Image & Video Models at Scale

    https://www.youtube.com/watch?v=xOP1PM8fwnk&t=1245

  107. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

    https://www.youtube.com/watch?v=_FKeSzM9fPc&t=1762

  108. Veo 3 for developers

    https://www.youtube.com/watch?v=hlcAZ2lX_ZI&t=140

  109. Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

    https://www.youtube.com/watch?v=3jGAU2sbAyY&t=862

  110. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

    https://www.youtube.com/watch?v=C13jiFWNuo8&t=287