Contents
  1. Task and representation
    1. A task, not a collection of file types
      1. Signal, transport, and use
    2. Different signals become different representations
    3. Must-know turning points
      1. Selected developments
  2. Correspondence and interaction
    1. Learning what belongs together
    2. Where signals first interact
    3. Keep the coordinates that make the claim true
  3. Capabilities and limits
    1. Operations enabled by cross-modal structure
      1. Operation contracts
    2. Reasoning must depend on the relationship
  4. Meaning-preserving boundaries
    1. Return claims with their evidence coordinates
    2. Absence and delay are distinct states
    3. Disagreement should remain visible
      1. Conflict handling
  5. Architecture and evidence
    1. Choose the boundary you can operate
      1. Operational tradeoffs
    2. One result contains several claims
    3. Test dependence, absence, and conflict
      1. A compact test matrix
    4. Measure the claim and localize the failure
      1. Claims and measurements
  6. Check understanding
  7. Open questions
  8. Selected talks
  9. References
  10. Talk library
← All topics

Multimodal AI

A multimodal system is valuable when a task depends on information distributed across different kinds of observation. A photographed menu contains visible words, layout, and food descriptions; a spoken question adds a user's immediate intent. Combining them can answer a dietary question that neither a generic transcript nor an unexamined image label answers alone. But accepting several file types is not enough. The system must preserve which observations describe which entities, when and where they occurred, what was unavailable, and how each output is supported. This chapter follows those obligations from the task contract through representations, alignment, fusion, interfaces, architecture, and evaluation.

Part I — Task and representation

A task, not a collection of file types

For engineering purposes, a modality is a kind of observed or generated signal whose measurement process and structure matter to the task. Text records symbols in an order; an image measures light over a spatial grid; audio measures a changing signal over time. This is an operational boundary, not a universal scientific classification. A modality is distinct from its encoding, its place inside a container, and the API field used to transport it. A video file, for example, may contain separate image, audio, and subtitle streams, while a downstream program may select only one of them.

A multimodal task contract begins with the decision or output, then names the observations it may use, the entities or events those observations concern, their timing, and the consequences of error. Additional signals can contribute complementary facts, corroborate the same fact, or offer another interaction method. They can also be irrelevant. In a photographed handwritten menu, pixels preserve writing, language, and layout; a native vision-language model can use those together when translating or answering a question. In infrastructure monitoring, visible damage may justify reporting damaged poles, but it does not by itself prove whether electricity remains energized. The output contract must preserve that narrower claim.

Signal, transport, and use

The layers answer different questions. None proves that the others were used.
LayerExampleQuestion to ask
Observed signalMicrophone waveformWhat physical or authored phenomenon was measured?
EncodingCompressed audio packetsHow was the signal represented, and what could be lost?
Container streamAudio track in a video fileWhich streams exist independently?
API representationA URI and MIME typeWhat did the request make available?
Model-visible inputDecoded and sampled audio featuresWhat did the model actually consume?

Different signals become different representations

An encoder transforms an observation into a task-facing numerical representation. The representation need not reproduce every input detail; it must keep the distinctions required by the permitted downstream operation. Text encoders operate over ordered tokens. Vision encoders may retain features associated with patches and spatial positions. Audio encoders operate over time-indexed features, while video representations add relationships across frames. The broader representation contract is developed in What the encoding must preserve; Patches and spatial interaction explains the visual case in depth.

Compression, sampling, and pooling decide what later computation can recover. Latent image models deliberately compress pixels into a smaller grid before generation, reducing computation while risking loss of fine detail. Video APIs may sample frames rather than processing every source frame; rapid actions between samples can disappear. Likewise, reducing visual tokens can hurt tasks that depend on small text or chart details even when broad image understanding changes less. A common tensor width merely makes arrays structurally compatible. It does not place independently trained encoders in a common semantic coordinate system.

Must-know turning points

Multimodal systems grew from several continuing lines of work rather than one replacement sequence. Statistical methods related paired measurements; interactive systems combined signals that supplied different parts of a command; audiovisual recognition explored feature-level and decision-level combination; and conditional sequence models turned visual representations into language. Later systems scaled paired representation learning and inserted visual information into generative language models. These traditions coexist because correspondence, fusion, and generation solve different problems.

Selected developments

Dates identify the cited publications or demonstrations, not universal invention dates.
DateDevelopmentWhat changed
1936Canonical correlationHotelling related paired sets of measurements through correlated linear combinations.
1980Put-That-ThereSpeech named an operation while pointing supplied an object or destination.
2003 accountAudiovisual speech fusionFeature fusion, decision fusion, and asynchronous integration addressed noise and timing differences.
2014Show and TellA visual encoder conditioned a language decoder that generated descriptions.
2015Show, Attend and Tell · VQAXu and colleagues made caption generation consult different image locations at each word. Agrawal and colleagues defined question-conditioned image answering and supplied the VQA dataset.
2021CLIPLarge-scale contrastive training made separately encoded images and text comparable for retrieval and transfer.
2022–2023Flamingo and LLaVAVisual representations entered generative language processing through cross-attention or a learned connector.

Part II — Correspondence and interaction

Learning what belongs together

Alignment establishes a supported correspondence across signals. Its resolution matters. Global alignment may connect an entire photograph with a caption. Region alignment associates a phrase with a location. Temporal alignment associates events in separate streams. Sequence alignment relates ordered elements. Paired data and training objectives determine which relationships receive pressure: contrastive training separates matched from mismatched pairs; caption generation predicts a description from an image; reconstruction asks one representation to preserve information about another signal.

Alignment is learned, not obtained by giving vectors equal width. CLIP trained separate image and text encoders so paired items scored above mismatches, enabling text-conditioned retrieval and open-set comparisons. Paint.wtf used this mechanism to rank arbitrary drawings against prompts by cosine similarity. That score represented CLIP's learned interpretation, not a human verdict about artistic quality or precise compositional compliance. Alignment across modalities develops the shared-space mechanism.

Whole-item pairs can leave fine relationships unresolved. A caption such as “a dog on a lawn” may fit images in which the dog faces toward or away from the camera. If captions rarely distinguish orientation, an image-caption objective receives little pressure to preserve it. Similarly, a brief bark supplies a useful audiovisual synchronization cue, while uniform engine noise may reveal no unique temporal offset. Alignment therefore exists at a particular resolution; broad semantic relatedness does not silently provide entity grounding or event timing.

Where signals first interact

Fusion combines information from modalities for a prediction or generated output. In early fusion, features are combined before substantial task processing. Intermediate fusion allows learned interactions inside the model. Late fusion combines independently produced scores or decisions. Hybrid designs retain both shared and separate paths. These terms are useful only when the implementation states what is being combined: some literature calls concatenating separately learned features “late fusion,” while other taxonomies reserve that name for final decisions.

Hybrid designs combine shared computation with independent paths, adding coordination and evaluation work. The location of fusion changes information access, not quality by definition.

Cross-attention is one intermediate mechanism: states being updated supply queries, while another representation supplies keys and values. It creates an information path but does not guarantee that every source detail influences the output. Three architecture families explains the mechanism. Original LLaVA instead illustrates a learned connector that maps visual features into a language model's embedding width; see Visual features in language responses. Architectural capacity alone is weak evidence of interaction: additive approximations preserved much of the predictive performance of several tested multimodal classifiers even when the original models could represent richer interactions.

Follow the first point where acoustic and visual information meets. Late fusion can combine scores but cannot recover feature interactions already discarded. The architectures and score bars are illustrative; access permits useful interaction without proving it occurs.

Keep the coordinates that make the claim true

A cross-modal claim needs an address system. Regions locate visual evidence; timestamps locate sounds and video events; track or speaker identities connect observations over time; and sampling intervals state what could have been observed. Co-presence is not synchronization. Transport standards distinguish stream identity from timing relationships, and two received tracks need not share a defined synchronization basis. A model can receive both while relating the wrong moments.

Consider the question, “Which indicator was lit when the second beep sounded?” Solving it requires detecting two beeps, preserving their order, locating the second interval, identifying the visible indicator at that time, and linking the observations. A pooled audio summary saying “two beeps occurred” and an image summary saying “green and amber indicators appear” preserve the topic but erase the required relationship. Frames, sampling and temporal order and Answers grounded in video develop the visual timing mechanics.

A brief sound must align with a visible state

Example timings

A topic-level summary can retain both beeps and both indicators while losing the particular relationship required by the question.

Captured interval0–6 secondsDuration 6 seconds
Audio stream0–6 secondsDuration 6 secondsWithin Captured interval
First beep1–1.4 secondsDuration 0.4 secondsWithin Audio stream
Second beep4–4.4 secondsDuration 0.4 secondsWithin Audio stream
Video stream0–6 secondsDuration 6 secondsWithin Captured interval
Green indicator illuminated0.5–2.5 secondsDuration 2 secondsWithin Video stream
Amber indicator illuminated3.5–5.5 secondsDuration 2 secondsWithin Video stream
With an established common timebase, the second beep overlaps the amber indicator’s illuminated interval. Overlap does not show that the indicator caused the sound; gaps mark no illustrated illuminated-indicator observation.
Read the diagram as text
  • Captured interval. The common analysis interval. 0 to 6 seconds; duration 6 seconds.
  • Audio stream. Time-indexed audio observations. 0 to 6 seconds; duration 6 seconds. Parent: Captured interval.
  • First beep. A brief detected audio event. 1 to 1.4 seconds; duration 0.4 seconds. Parent: Audio stream.
  • Second beep. The event named by the question. 4 to 4.4 seconds; duration 0.4 seconds. Parent: Audio stream.
  • Video stream. Time-indexed visual observations. 0 to 6 seconds; duration 6 seconds. Parent: Captured interval.
  • Green indicator illuminated. Tracked visible state before the change. 0.5 to 2.5 seconds; duration 2 seconds. Parent: Video stream.
  • Amber indicator illuminated. Tracked visible state overlapping the second beep. 3.5 to 5.5 seconds; duration 2 seconds. Parent: Video stream.

Part III — Capabilities and limits

Operations enabled by cross-modal structure

Multimodal capability is clearer when named as an operation with an input-output contract. A cross-modal transformation maps an observation in one modality to an output in another, as image captioning maps pixels to text or speech recognition maps audio to text. Such a transformation uses one observed modality; it does not by itself integrate evidence from multiple observed modalities. Retrieval ranks items in one modality using a query from another. Grounding returns a region or interval, not merely a sentence. Question answering produces a task-specific answer over related observations. Conditional generation uses one signal to guide another output. One model may support several operations, but success at one does not certify the rest.

Operation contracts

Each operation requires a different relationship and preserves a different result.
OperationInputsOutputRequired relationship
Cross-modal retrievalQuery plus mixed-media collectionRanked candidatesComparable global or local representations
ClassificationComplementary observationsLabel or scoreFusion of task-relevant evidence
Captioning or transcriptionImage or audioText sequenceContent-to-sequence conditioning
GroundingMedia plus phrase or questionRegion or intervalEntity or temporal alignment
Image to structureDocument imageStructured token sequenceVisual content and layout to fields
Conditional generationPrompt plus media conditionNew media or textConditioning representation available during generation

Reasoning must depend on the relationship

Cross-modal reasoning produces a conclusion whose support requires a relationship among facts from different modalities. Perception identifies observations. Association says items tend to occur together. Grounding connects a claim to a referent. Reasoning composes grounded facts. A music question such as which visible clarinet sounded first requires source association and temporal ordering; identifying clarinets and recognizing clarinet audio independently is insufficient.

Test dependence with controlled interventions. Compare the matched pair with each modality alone and with a deliberately mismatched but individually plausible pair. VQA v2 held a question fixed while finding a similar image that changed its answer. Winoground held words nearly fixed while changing their order and intended image relation. These designs make a relationship necessary in ways that an aggregate score does not. If swapping videos leaves answers unchanged, the system may rely on language priors, leaked labels, or the audio stream alone.

Change the pairing, change the required answer

Example

Holding the question and audio fixed while exchanging visual states changes the reference answer. An unchanged prediction would need investigation.

1 / 2 · Original pairing

With video A, the required answer is amber. Both alternatives remain listed for comparison; the hypothetical shortcut happens to agree with this reference.

Retained this stepFixed questionWhich indicator was lit when the second beep sounded?
Retained this stepFixed audioFirst beep: 1–1.4 seconds. Second beep: 4–4.4 seconds. Neither interval changes.
Retained this stepVideo AGreen illuminated: 0.5–2.5 seconds. Amber illuminated: 3.5–5.5 seconds.
Retained this stepA reference: amberThe second beep, 4–4.4 seconds, falls inside the amber interval.
Retained this stepHypothetical shortcut: amberAlways answers amber. It agrees with reference A and disagrees with reference B without using the changed video relationship.
Switch between two video assignments on an established common timebase. Reference answers follow from the specified intervals; the constant-answer shortcut is hypothetical, not a model measurement.
Read the diagram as text
  • Fixed question. Which indicator was lit when the second beep sounded?
  • Fixed audio. First beep: 1–1.4 seconds. Second beep: 4–4.4 seconds. Neither interval changes.
  • Video A. Green illuminated: 0.5–2.5 seconds. Amber illuminated: 3.5–5.5 seconds.
  • A reference: amber. The second beep, 4–4.4 seconds, falls inside the amber interval.
  • Video B. Amber illuminated: 0.5–2.5 seconds. Green illuminated: 3.5–5.5 seconds. Only the indicator labels are exchanged.
  • B reference: green. The second beep, 4–4.4 seconds, now falls inside the green interval.
  • Hypothetical shortcut: amber. Always answers amber. It agrees with reference A and disagrees with reference B without using the changed video relationship.
  • Fixed audio → A reference: amber: second-beep interval.
  • Video A → A reference: amber: illuminated state.
  • Fixed audio → B reference: green: same second-beep interval.
  • Video B → B reference: green: exchanged state.
  1. Original pairing. With video A, the required answer is amber. Both alternatives remain listed for comparison; the hypothetical shortcut happens to agree with this reference. Active: Fixed question, Fixed audio, Video A, A reference: amber, Hypothetical shortcut: amber. New: None.
  2. Exchange only the visual states. With video B, the required answer becomes green. The constant amber answer now fails. Collect real model predictions separately and reassess reference answers whenever replacement inputs change. Active: Fixed question, Fixed audio, Video B, B reference: green, Hypothetical shortcut: amber. New: None.

Fluent explanations, attention maps, and architecture diagrams do not prove integration. MCR-BENCH held audio fixed while changing accompanying textual descriptions; tested audio-language models often changed correct audio-based answers when contradictory text was introduced. The result demonstrates modality dominance under those conditions, not a universal rule that text dominates audio. The diagnostic lesson is general: vary one claimed source of evidence while preserving the others, then observe whether the decision changes as the task requires.

Part IV — Meaning-preserving boundaries

Return claims with their evidence coordinates

A request contract should identify every source and representation the model may use: stable source identity, media type, selected streams, ordering, preprocessing, coordinate mappings, timestamps, and readiness. A response contract should distinguish generated content from grounded claims and attach each claim to its supporting region, interval, entity, or text span. Completion, unsupported, and unavailable states belong beside the content rather than being inferred from missing fields. This applies the general principle in Specify what an operation promises.

Representation state matters because coordinates refer to a particular version. A box on a resized crop cannot be applied directly to the original image unless the transform is preserved. The Web Annotation model separates a source from a selector that identifies a segment and a state that identifies the intended representation. Provenance must also survive synthesis and change: when graph entities merge, their source links should be retained; when a fact is invalidated, the new evidence and time of change should remain inspectable.

Map the model’s box back to its source

Example

A box belongs to a specific representation. Reversing resize and crop coordinates locates the corresponding region in the original source version.

Original source: panel-image, version 1

Image: 1000×600 pixels. Forward preprocessing selects crop (200,100,400,200), then doubles both dimensions. Claim reference: source panel-image; version 1; selector (250,140,100,60).

025050075010000150300450600Horizontal position (pixels)Vertical position (pixels)Original imageCrop: (200,100,400,200)Source selector: (250,140,100,60)Crop origin (200,100)Mapped box: 100×60
  • 1. Original image
  • 2. Crop: (200,100,400,200)
  • 3. Source selector: (250,140,100,60)
Read coordinates and regions as data

X: 0–1000 pixels; Y: 0–600 pixels, increasing down. Equal scale on both axes.

Original image (polygon)

(0, 0); (1000, 0); (1000, 600); (0, 600)

Crop: (200,100,400,200) (polygon)

(200, 100); (600, 100); (600, 300); (200, 300)

Source selector: (250,140,100,60) (polygon)

(250, 140); (350, 140); (350, 200); (250, 200)

Crop origin (200,100): (200, 80)

Mapped box: 100×60: (250, 225)

Model input: resized crop

Input: 800×400 pixels, produced by scaling the 400×200 crop by 2. Model box: (100,80,200,120). Inverse mapping gives source position (250,140) and size 100×60.

025050075010000150300450600Horizontal position (pixels)Vertical position (pixels)Resized crop: 800×400Model box: (100,80,200,120)Box origin (100,80)Model box: 200×120Map back: halve coordinates, then add crop origin
  • 1. Resized crop: 800×400
  • 2. Model box: (100,80,200,120)
Read coordinates and regions as data

X: 0–1000 pixels; Y: 0–600 pixels, increasing down. Equal scale on both axes.

Resized crop: 800×400 (polygon)

(0, 0); (800, 0); (800, 400); (0, 400)

Model box: (100,80,200,120) (polygon)

(100, 80); (300, 80); (300, 200); (100, 200)

Box origin (100,80): (100, 60)

Model box: 200×120: (100, 225)

Map back: halve coordinates, then add crop origin: (0, 455)

Worked crop-and-resize example. Coordinates use a top-left origin and boxes use (x, y, width, height), in pixels. Map back with x = 200 + x_input/2 and y = 100 + y_input/2; halve width and height. A correct selector locates support but does not establish the claim’s truth.

Valid JSON is useful because downstream software can inspect these obligations, but structure is not truth. A monitoring system may return a well-formed claim that electricity is unavailable after seeing fallen poles; the image supports visible damage, while service status may require telemetry. Structure is not domain correctness explains the separate semantic check.

Absence and delay are distinct states

Confirmed absence is an observation: a complete inspected interval contains no alarm. Unavailable means the required observation was not obtained. Unsupported format, failed preprocessing, corrupted input, late arrival, and intentional omission are different again. Collapsing these states turns “we could not inspect the audio” into “no alarm sounded.” Optional-input contracts and modality masks make availability explicit before fusion.

Training with randomly withheld modalities can improve robustness for tested missingness patterns, but it cannot recreate an unobserved event. Cross-modal reconstruction predicts one signal from another; the prediction is an inference, not a recovered measurement. At runtime, an application may wait, return a narrower result, abstain, or revise after a late observation. A revision should preserve the earlier result’s availability history. The right policy depends on timing and consequences. Under overload, explicit reduced completeness or an error is preferable to silently presenting degraded evidence as a normal result.

Disagreement should remain visible

Modalities can disagree because they describe different moments or entities, because a sensor or transcription is wrong, because metadata is stale, because the situation is ambiguous, or because content was manipulated. Fusion should not erase these explanations. Reliability weighting is meaningful only for a defined task and condition: audiovisual speech systems, for example, can weight separate decisions according to stream reliability, but this does not create a universal priority among audio and vision.

Conflict handling

An honest response preserves the disagreement and the evidence needed to resolve it.
Observed patternPossible causesPermitted response
Streams agreeIndependent support or shared errorReport the scoped claim; retain both sources
Metadata is older than mediaStale record or changed stateReport the time difference; prefer neither silently
Text contradicts audioBad description, transcription error, or manipulationInspect task authority and report conflict
Entities do not matchAssociation failure or source mix-upReject the correspondence; do not fuse claims
Untrusted media issues instructionsIndirect prompt injectionTreat content as data; enforce action authority outside the model

Human perception itself shows that synchronized conflicting signals can alter interpretation: the 1976 McGurk and MacDonald experiment paired one spoken syllable with different lip movements, producing a changed percept for many participants. Machine systems need not behave identically, but the example makes the engineering issue concrete: integration can create a third interpretation rather than choosing one independent label. In consequential systems, retain source-specific observations and defer when the resolution rule lacks support. Missing information and uncertain interpretations separates absent evidence, ambiguity, and model failure.

Part V — Architecture and evidence

Choose the boundary you can operate

A unified model accepts several representations and permits direct internal interaction. This can avoid information bottlenecks created by explicit handoffs, but it may hide which stage failed. A modular pipeline uses specialized services—for example speech recognition, retrieval, a language model, and speech synthesis—with inspectable interfaces between them. Those boundaries aid replacement and diagnosis but can discard tone, layout, or timing and add latency. A cascade runs cheaper or narrower stages first, escalating to richer fusion or human review only under explicit conditions. Real-time turn-taking and interruption add timing requirements beyond treating audio as another input; Voice and Real-Time AI develops those interaction mechanics.

Choose according to the task's required interactions, not architectural fashion. Ask where raw media crosses a privacy boundary, what can run locally, which intermediate artifacts remain inspectable, how late or missing streams degrade the result, and where uncertainty triggers review. EMMA illustrates a unified research architecture in driving: camera video, routing commands, and vehicle history condition text-encoded future waypoints and related outputs. Its published limitations include few input frames, missing LiDAR and radar, expensive computation, and a need for closed-loop simulation; it is not evidence that unified models universally replace modular driving systems.

Operational tradeoffs

Compare candidates on the same task and output contract.
ArchitectureInformation pathOperational advantagePrimary burden
UnifiedModalities interact inside one modelDirect joint conditioning and simple invocationOpaque failure boundaries and broad data exposure
ModularSpecialized outputs cross explicit interfacesIndependent validation, replacement, and degradationHandoff loss, coordination, and accumulated latency
Conditional cascadeNarrow stages gate richer fusion or reviewSpend and scrutiny can follow case difficultyRouting changes coverage and can reject useful cases
Human-reviewedPeople decide unresolved consequential casesDomain judgment at a named decision pointCapacity, consistency, delay, and incomplete feedback

One result contains several claims

An end-to-end answer contains a ladder of separate claims. Did each modality-specific component perceive its input correctly? Were observations associated with the right entity and time? Was the answer grounded in the right region or interval? Did the conclusion require and correctly use the cross-modal relationship? Did the interface preserve the support? Did the result improve the actual workflow? A correct transcript attached to the wrong speaker can pass speech recognition while failing correspondence; a recognized object can be the wrong referent; a plausible summary can alter a date or causal relation.

Each claim needs its own check and oracle. TVQA+ separately assessed answer correctness, temporal localization, and object grounding. Voice-agent evaluation likewise needs audio-specific checks in addition to transcript and control-flow evaluation. An end score can tell you that something failed, but layered traces and component checks help distinguish routing, arguments, model execution, and final task completion. General case specification belongs in Specify the case and its outcome; oracle selection belongs in Choose checks that match the requirement.

Test dependence, absence, and conflict

A multimodal evaluation case states the available modalities, their source relationships and times, preprocessing, permitted resources, stopping conditions, and acceptable outcomes. Representative cases estimate behavior for the intended task mix. Challenge cases deliberately concentrate mismatched pairs, missing or corrupted streams, timing offsets, contradictory descriptions, and distribution shifts. Their frequencies diagnose mechanisms; without deployment weights, they do not estimate production prevalence.

A compact test matrix

Keep representative estimation separate from diagnostic coverage.
ConditionQuestion answeredExpected use
Matched complete inputsCan the system perform the intended task?Representative and regression testing
Each modality aloneWhat can one signal already support?Unimodal baseline
One modality missing or corruptedDoes supported scope shrink honestly?Robustness diagnosis
Relationship-breaking mismatchDoes the answer depend on the pairing?Cross-modal dependence test
Contradictory observationsCan the system preserve and handle conflict?Dominance and deferral test
Shifted capture or domainDo learned correspondences transfer?Distribution-shift assessment

Measure the claim and localize the failure

Measurements must follow the output. Retrieval needs ranking and coverage measures. Labels need task-specific correctness. Regions and intervals need declared overlap or tolerance rules. Generated video needs temporal, identity, physical, and audiovisual criteria in addition to frame-level prompt similarity. Deferral needs both coverage and error among accepted cases. Operations need latency and review burden, while the product needs its real workflow outcome. Per-modality and slice results should remain visible beside aggregates.

Claims and measurements

The denominator and oracle are part of every result.
ClaimMeasurementRequired qualification
Cross-modal retrievalRecall, precision, or ranking metricDefine relevant items, corpus, query mix, and cutoff
Spatial or temporal groundingRegion or interval overlapDeclare coordinate system, tolerance, and eligible references
Generated-media fidelitySeparate frame, temporal, identity, physical, and audio criteriaFrame similarity does not establish a coherent story
Automatic acceptanceSelective risk and coverageA threshold changes both error exposure and review volume
Human usefulnessExplicit rubric and workflow outcomeGive reviewers source access; measure agreement and burden

Failure investigation replays preserved inputs and intermediates, then changes one plausible cause at a time. Re-run the audio recognizer to test perception; replace a caption while holding the image fixed to test dominance; inspect timestamp mappings to test synchronization; supply verified component outputs to isolate fusion or reasoning; re-score preserved output with an independent oracle to test the grader. The same wrong answer can originate at any of these boundaries. Human assessment is appropriate when meaning or usefulness lacks a complete automatic oracle, but reviewers need explicit criteria, relevant source media, agreement analysis, and adjudication. Test the failure explanation develops the general method.

Open questions

  1. How should an asynchronous application decide whether to wait for a late modality, complete with narrower evidence, revise a prior answer, or reject stale observations? The answer depends on event timing, reversibility, and consequence, but no general policy covers all tasks. Progress would look like explicit timing contracts and evaluations that measure both delay and errors caused by premature completion.

  2. How can systems calibrate claim-specific uncertainty when several modalities differ in reliability or contradict one another? A single confidence score can hide whether uncertainty comes from perception, association, timing, or reasoning. Progress would include held-out conflict cases, source-conditioned reliability estimates, and risk-coverage results for the actual deferral policy.

  3. How can fine-grained alignment be learned without requiring exhaustive region, entity, and timestamp annotation? Whole-item pairs scale but often leave object and event correspondences unresolved; detailed annotation is expensive and sometimes ambiguous. Progress would preserve broad data scale while demonstrating grounded transfer on controlled local and temporal relationships.

  4. How should unified and modular architectures be compared fairly? Existing examples differ in sensors, data, component quality, latency accounting, output contracts, and deployment constraints. Progress would require same-task comparisons that preserve identical evidence and acceptance rules while measuring handoff loss, interaction quality, privacy crossings, latency, failure isolation, and review burden.

Follow the curated reading path through the speakers and demonstrations behind this entry.

37 min

AI Engineer World's Fair 2025 · 2025

Building Multimodal AI Agents (From Scratch)

Apoorva Joshi

Transcript reviewed

A practical treatment of multimodal retrieval, including screenshot-based document representations, the CLIP-style modality gap, and the limits of extending one retrieval representation to unrelated signals such as time series.

Watch talk
23 min

AI Engineer World's Fair 2026 · 2026

Evaling Video Slop

Maor Bril

Cited in this entry

Develops the evaluation problem for generated video: frame metrics are insufficient for temporal story, identity, motion, and audiovisual quality, so human calibration and specialized judges serve distinct roles.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

36 matching talks

TalkSpeakerEventYear
Frank LiuAI Engineer World's Fair 20252025
Preetika Bhateja, Daniel BumpAI Engineer World's Fair 20262026
Aparna Dhinkaran, Aparna DhinakaranAI Engineer Summit 20252025
Mukuntha Narayanan, Han WangAI Engineer World's Fair 20252025
Cassidy HardinAI Engineer Europe 20262026
Suman DebnathAI Engineer World's Fair 20252025
Ishan AnandAI Engineer World's Fair 20242024
Joseph NelsonAI Engineer Summit 20232023
Peter RobicheauxAI Engineer World's Fair 20252025
Linus LeeAI Engineer Summit 20232023
Travis Bartley, Myungjong Kim, Byungjoong, JaehanAI Engineer World's Fair 20252025
Vikhyat KorrapatiAI Engineer World's Fair 20242024
Kevin HouAI Engineer World's Fair 20252025
Cedric VidalAI Engineer World's Fair 20242024
Isaac RobinsonAI Engineer Europe 20262026
Sander DielemanAI Engineer Europe 20262026
Arjun Desai, Rohit TalluriAI Engineer World's Fair 20252025
Sangwu LeeAI Engineer World's Fair 20262026
Keegan McCallumAI Engineer World's Fair 20252025
Paige Bailey, Guillaume Vernade, Ian BallantyneAI Engineer Europe 20262026
Daniel ChalefAI Engineer World's Fair 20262026
Vinoo GaneshAI Engineer World's Fair 20262026
Doug GuthrieAI Engineer World's Fair 20252025
Jeremy Silva, Chris HernandezAI Engineer World's Fair 20252025
Randall HuntAI Engineer World's Fair 20252025
Akram BaharloueiAI Engineer World's Fair 20262026
Laurie VossAI Engineer Europe 20262026
Kuba RogutAI Engineer Europe 20262026
Armen AghajanyanAI Engineer World's Fair 20262026
Thomas Wolf, Olive SongEvent unknownUnknown
Pamela FoxAI Engineer World's Fair 20262026
Ivan LeoAI Engineer World's Fair 20262026
Nidhi Kaushik VyasEvent unknownUnknown
Valeria Wu Fon, Tom OuyangEvent unknownUnknown
Dumitru Erhan, Shane Gu, Nicole Brichtova, swyxAI Engineer World's Fair 20262026
Tarun SunkaraneniAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
32 processed in full · 4 in the curated path
Automated source review
Passed
Metadata candidates
8 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. FFmpeg Documentation

    https://www.ffmpeg.org/ffmpeg.html

  2. Multimodal Machine Learning: A Survey and Taxonomy

    https://arxiv.org/html/1705.09406

  3. Multi-model, multimodal, and multi-agent innovations in Azure AI

    https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=393

  4. Multi-model, multimodal, and multi-agent innovations in Azure AI

    https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=585

  5. Hello GPT-4o

    https://openai.com/index/hello-gpt-4o/

  6. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

    https://arxiv.org/html/2312.17172

  7. Gemma 4 Deep Dive — Cassidy Hardin, Google DeepMind

    https://www.youtube.com/watch?v=_A367W_qvc8&t=796

  8. Building Generative Image & Video Models at Scale

    https://www.youtube.com/watch?v=xOP1PM8fwnk&t=313

  9. Video understanding — Gemini API

    https://ai.google.dev/gemini-api/docs/video-understanding?hl=en

  10. LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?

    https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/

  11. Deep Learning, Chapter 14: Autoencoders

    https://www.deeplearningbook.org/contents/autoencoders.html

  12. Relations Between Two Sets of Variates

    https://gwern.net/doc/statistics/probability/1936-hotelling.pdf

  13. “Put-That-There”: Voice and Gesture at the Graphics Interface

    https://www.media.mit.edu/speech/papers/1980/bolt_SIGGRAPH80_put-that-there.pdf

  14. Joint Audio-Visual Speech Processing for Recognition and Enhancement

    https://www.isca-archive.org/avsp_2003/potamianos03_avsp.pdf

  15. Show and Tell: A Neural Image Caption Generator

    https://arxiv.org/abs/1411.4555

  16. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

    https://arxiv.org/html/1502.03044v3

  17. VQA: Visual Question Answering

    https://arxiv.org/abs/1505.00468

  18. Learning Transferable Visual Models From Natural Language Supervision

    https://arxiv.org/html/2103.00020

  19. Flamingo: a Visual Language Model for Few-Shot Learning

    https://arxiv.org/html/2204.14198

  20. Visual Instruction Tuning

    https://arxiv.org/html/2304.08485

  21. CoCa: Contrastive Captioners are Image-Text Foundation Models

    https://arxiv.org/html/2205.01917

  22. Multimodal Deep Learning

    https://ai.stanford.edu/~jngiam/papers/NgiamKhoslaKimNamLeeNg2011.pdf

  23. 120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson

    https://www.youtube.com/watch?v=OimPoLxioYg&t=55

  24. Vision AI in 2025 — Peter Robicheaux, Roboflow

    https://www.youtube.com/watch?v=IQc05eCvNYE&t=282

  25. Audio-Visual Synchronisation in the Wild

    https://www.robots.ox.ac.uk/~vedaldi/assets/pubs/chen21audio-visual.pdf

  26. CLIPScore: A Reference-free Evaluation Metric for Image Captioning

    https://arxiv.org/html/2104.08718

  27. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

    https://arxiv.org/html/2107.07502

  28. Does My Multimodal Model Learn Cross-Modal Interactions? It’s Harder to Tell Than You Might Think!

    https://aclanthology.org/2020.emnlp-main.62.pdf

  29. Attention Is All You Need

    https://arxiv.org/html/1706.03762

  30. RFC 8834: Media Transport and Use of RTP in WebRTC

    https://www.rfc-editor.org/rfc/rfc8834.html

  31. Media Fragments URI 1.0 (basic)

    https://www.w3.org/TR/media-frags/

  32. Learning to Answer Questions in Dynamic Audio-Visual Scenarios

    https://arxiv.org/html/2203.14072

  33. TVQA+: Spatio-Temporal Grounding for Video Question Answering

    https://aclanthology.org/2020.acl-main.730.pdf

  34. OCR-free Document Understanding Transformer

    https://arxiv.org/pdf/2111.15664

  35. VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response

    https://www.youtube.com/watch?v=hwCmfThIiS4&t=584

  36. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

    https://arxiv.org/html/1612.00837

  37. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

    https://arxiv.org/html/2204.03162

  38. Shifting the Baseline: Single Modality Performance on Visual Navigation & QA

    https://arxiv.org/pdf/1811.00613

  39. When Audio and Text Disagree: Benchmarking Text Bias in Large Audio-Language Models under Cross-Modal Inconsistencies

    https://aclanthology.org/anthology-files/anthology-files/pdf/emnlp/2025.emnlp-main.246.pdf

  40. Generating content — Gemini API

    https://ai.google.dev/api/generate-content

  41. Web Annotation Data Model

    https://www.w3.org/TR/annotation-model/

  42. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    https://www.youtube.com/watch?v=H7puB0RwJMM&t=316

  43. ModDrop: adaptive multi-modal gesture recognition

    https://arxiv.org/html/1501.00102

  44. Google SRE: Handling Overload

    https://sre.google/sre-book/handling-overload/

  45. Hearing lips and seeing voices

    https://www.nature.com/articles/264746a0

  46. OWASP LLM01:2025 Prompt Injection

    https://genai.owasp.org/llmrisk/llm01-prompt-injection/

  47. Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

    https://www.youtube.com/watch?v=knH3fmGAteQ&t=193

  48. Improving Pinterest Search Relevance Using Large Language Models

    https://arxiv.org/html/2410.17152v1

  49. EMMA: End-to-End Multimodal Model for Autonomous Driving

    https://arxiv.org/html/2410.23262v1

  50. Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

    https://www.youtube.com/watch?v=iS9YFW28XyM&t=529

  51. The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

    https://www.youtube.com/watch?v=1__V4KTv_Gw&t=251

  52. Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

    https://aclanthology.org/2021.naacl-main.383.pdf

  53. Ensure AI Agents Work: Evaluation Frameworks for Scaling Success

    https://www.youtube.com/watch?v=OC04sP_QgTI&t=725

  54. Ensure AI Agents Work: Evaluation Frameworks for Scaling Success

    https://www.youtube.com/watch?v=OC04sP_QgTI&t=855

  55. How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

    https://www.youtube.com/watch?v=xyL2Ltkh-SA&t=352

  56. Evaling Video Slop

    https://www.youtube.com/watch?v=b_PmGocP4rc&t=133

  57. Selective Classification for Deep Neural Networks

    https://papers.nips.cc/paper_files/paper/2017/file/4a8423d5e91fda00bb7e46540e2b0cf1-Paper.pdf

  58. Best practices for the human evaluation of automatically generated text

    https://aclanthology.org/W19-8643.pdf

  59. On Calibration of Modern Neural Networks

    https://proceedings.mlr.press/v70/guo17a/guo17a.pdf

  60. Ship Real Agents: Hands-On Evals for Agentic Applications

    https://www.youtube.com/watch?v=Xfl50508LZM&t=74

  61. Evaling Video Slop

    https://www.youtube.com/watch?v=b_PmGocP4rc&t=301

  62. Robust Speech Recognition via Large-Scale Weak Supervision

    https://arxiv.org/html/2212.04356