AI Engineer World's Fair 2026

Video Has No Memory. Here's How We Built One.

James Le20:27

Loading slides…

Read the talk

Video Has No Memory. Here’s How We Built One.

A video archive preserves footage, but useful memory requires durable links between moments, entities and evidence. James Le walks through the architecture and three Jockey demonstrations.

From a talk by James Le

Before you start: Familiarity with embeddings, retrieval-augmented generation and APIs will help; no video-model implementation experience is required.

Recorded footage is not system memory

Video already preserves the past: recordings, training material, incidents, creative work and history. Why would it need a memory layer? Because keeping footage does not give a system a durable understanding of what happened inside it—or a way to connect that understanding to the next recording. That is the opening problem in James Le’s talk.

Video is not a bag of frames. A stack of images with an attached transcript can approximate video for some tasks, but it discards continuity. Meaning depends on space, time, modalities and sequence. A better representation is a spatiotemporal volume containing visual information, speech, sound, motion, OCR, camera changes, scene transitions, metadata and time.

Slide with three columns illustrating video complexity, footage scale, and holistic understanding using a horse sequence, timelines, and a space-time diagram.Open full source frame
Video as a spatiotemporal volume, rather than a bag of frames.

The engineering problem is preserving relationships within that volume so an application can traverse them later. Entertainment, sports and short-form video archives can contain petabytes of footage. Finding a relevant moment is already difficult; preserving meaning across millions of moments is a deeper storage and retrieval problem. Le introduces TwelveLabs, which he describes as a Series B startup, in that context.

0:260:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:26 · section reference included

What the surrounding stack loses

Language models are powerful reasoning interfaces, and they increasingly accept multiple modalities. The limitation Le identifies is in the supporting stack: how it supplies context, retains memory and makes video available for reasoning.

  • Wrong context: Sampling frames, extracting a transcript and putting everything into a prompt can lose the spatiotemporal relationships that define an event.
  • Wrong memory: RAG, vector search and larger context windows are useful, but continuity may require connecting today’s scene to another file, episode, camera angle, season or year.
  • Wrong reasoning: A text-first representation does not automatically preserve motion and causality, or maintain a persistent record of who appeared, what happened and what changed.

A memory layer must therefore determine what the application can traverse, how observations connect and how they can be retrieved later.

Five properties make those requirements concrete:

  • Temporal: The same expression, product shot or physical action can mean different things depending on what happens before and after it.
  • Multimodal: A transcript can miss a logo; a frame can miss a spoken claim.
  • Dense: A few minutes may contain dozens of shots, people, objects, actions and location claims. Some seconds are decisive, while others add noise.
  • Ambiguous: People reappear under different lighting and angles, brands are partially visible, locations are implied, and concepts emerge over time.
  • Evidence-sensitive: Enterprise workflows need to trace an answer back to the moment that supports it.

The stored representation must preserve temporal spans, multimodal evidence and continuity together.

2:102:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:10 · section reference included

From temporal chunks to reasoning

The TwelveLabs stack begins with semantic chunks: meaningful temporal units rather than isolated frames. Marengo, its multimodal embedding encoder, represents those video spans as vectors. Above that, a spatiotemporal context store preserves moments, entities and metadata in a structure available for reasoning.

Pegasus, the video-context-aware language model, supplies the reasoning layer for summaries, metadata, synthesis and comparison. The models are exposed through APIs so developers can use them as infrastructure. The division of responsibility matters: embeddings represent content, the store retains reusable structure, and the language model reasons over video context.

5:055:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:05 · section reference included

Search finds candidates; memory preserves continuity

Search recovers relevant moments from a large library. Those moments are candidates, however, rather than a persistent account of the collection. Questions that span a corpus require preserved entities and timeline evidence, not just another retrieval call.

CapabilitySearchMemory
RequestFind something similarExplain what the collection knows
Returned unitTime-bounded momentStructured knowledge and timelines
Further useInspect a candidate clipCompose explanations and downstream outputs

The change is in the unit of output: from a retrieved clip to knowledge that remains useful across questions.

Slide titled “Search Finds Moments. Memory Preserves Meaning.” with clip-search diagrams and side-by-side examples of search and memory outputs.Open full source frame
Search returns moments; memory preserves structured knowledge, relationships, and composable output.

That shift has two scaling dimensions. Across time, the goal is to reason over years of footage without reprocessing the archive for every request. Reusable representations can support multi-hop timelines, episodic recall and follow-up questions; lower latency and cost are intended benefits, with no measurements supplied here. Across sources, the system must combine perspectives from camera angles, livestreams, creators, broadcasts, body cameras, store cameras and event feeds while maintaining a current understanding. Both dimensions require a representation the application can traverse.

5:515:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:51 · section reference included

A context graph makes video navigable

A context graph is the proposed durable, queryable representation connecting video moments, entities, appearances, relationships, timestamps, metadata and corpus context. Its foundation is the time-bounded moment: a scene or shot that serves as an evidence unit. Above it, an appearance records where and when an entity occurs.

A minimal application record can express that distinction directly. In this illustrative JSON, the identifiers and time range are teaching values; the appearance links an entity to a particular moment without collapsing the two into one object.

json

{
  "moments": [
    {
      "id": "moment-1",
      "video_id": "video-a",
      "start_ms": 12000,
      "end_ms": 18000
    }
  ],
  "entities": [
    {
      "id": "person-1",
      "type": "person"
    }
  ],
  "appearances": [
    {
      "id": "appearance-1",
      "entity_id": "person-1",
      "moment_id": "moment-1"
    }
  ]
}

Entities can be people, brands, places or concepts. Relationships add co-occurrences, sequences between places, causality and timelines. Corpus-level context then describes themes, patterns, gaps and coverage across the collection.

Different questions enter this structure at different points. A simple search can go directly to moments. A person-centered workflow starts with an entity and expands into its appearances. A storyline follows relationships across time. Memory is the navigable structure over the video volume, rather than merely a stored answer to the last question.

7:437:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:43 · section reference included

Design the memory for reuse

The first design principle is ingest once, reason many times. Move expensive interpretation into ingestion instead of starting from scratch with every query. The database analogy is useful: an application request should not require reparsing its entire source dataset.

The remaining principles specify what ingestion should produce and how applications should consume it:

  • Store primitives, not just answers. Moments, entities and appearances can support search, editing and analytics after the original question has changed.
  • Ground every claim. An answer should point back to a specific timestamp in the source video.
  • Let intent shape memory. Sports analysis, brand safety, compliance review and creator analytics need different primitives from the same footage. Developers need to configure what matters.
  • Keep the layer composable. API-first access to structured, grounded metadata lets applications build their own workflows above the memory layer.
9:039:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:03 · section reference included

Put the model inside a video worker

Memory alone does not define how work gets done. An isolated model call starts fresh and produces an answer; a video worker operates inside a controlled system that knows what information is available. Its workflow is sequential:

  1. Plan the task.
  2. Retrieve evidence.
  3. Inspect relevant moments.
  4. Synthesize a result.
  5. Validate it.
  6. Return the required output.

The worker’s knowledge of available memory, relevant evidence and necessary inspection depth determines both spending and the output contract it must satisfy.

Task planning distinguishes search from summarization or multi-step reasoning. Retrieval selects the evidence appropriate to that task. Expert tools let the worker inspect further: zoom in or out, compare frames, or enrich content with additional metadata. These are capabilities the surrounding system supplies, rather than assumptions that one model response will handle everything.

An operating envelope places explicit limits on time, cost, depth, scope and autonomy. An output contract specifies whether natural language is sufficient or whether the application requires structured data with references and timestamps. Evaluation then asks three separate questions: Did retrieval find the right evidence? Did synthesis preserve it? Did the worker stay within budget?

10:3610:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:36 · section reference included

Sports: distinguish near misses from goals

Le moves from architecture to three demonstrations built with Jockey, TwelveLabs’ video agent product. For the sports demo, Le reports ingesting 67 videos from the 2022 World Cup in Qatar. The first request asks for near misses: shots that almost became goals, an explanation of why each failed, and explicit exclusion of actual goals. That request needs an event’s outcome, not simply visual similarity to a shot on goal.

The narrated results include a ball hitting the woodwork, goalkeeper saves and an offside disallowance. These illustrate different reasons that an apparent scoring opportunity does not count as a goal.

The next request reverses the constraint: find dramatic actual goals, show the build-up and finish, and describe each sequence. Le highlights a returned passing chain from Álvarez to Mac Allister to Di María. This is the Di María goal in the final—Argentina’s second, as recorded in the official FIFA match report, although the narration calls it the first. The useful demonstration is the sequence of participants and actions leading to the finish.

A Richarlison example extends the request into a follow-up about a skillful build-up: identify the player who made the return pass. The interaction moves from locating a goal to inspecting the relationships within the play.

12:4813:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:48 · section reference included

Follow a player across the corpus

The sports demo then shifts from events to an entity: track Lionel Messi throughout the corpus and describe the camera framing. Le shows a highlight in which Messi dribbles past a sliding defender, then a scene of his goal against Australia in the round of 16, followed by his finish for Argentina’s third goal in the final. The results appear as playable clips with descriptive cards.

Demo interface showing three result cards with football video players and descriptions, including “Dribble past a sliding defender” and “Rebound finish in the Final.”Open full source frame
Player-tracking results displayed as football clips with descriptive cards.

This is the entity-to-appearances traversal made visible: the person remains the subject while the footage, action and framing change. The examples demonstrate the interaction, without supplying a quantified tracking evaluation.

14:4914:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:49 · section reference included

Surveillance: counts, events and busy intervals

For the surveillance demo, Le reports ingesting eight publicly available camera clips. The footage includes traffic jams, suburban scenes and urban aerial views. He asks Jockey to count and classify every vehicle at an intersection, break the result down by vehicle type, and include pedestrians. He describes outputs containing vehicle counts and peak foot traffic, but gives no numerical counts or accuracy measurements in the narration.

The demonstration then turns to safety events. Le points out a red SUV turning and nearly being struck, another vehicle turning left into an oncoming car, and a red-light entry. These requests depend on movement and sequence: the relevant evidence is how vehicles interact over an interval.

He next shows a crowded aerial scene in Bangkok and footage under rainy conditions, before identifying the busiest intersection vehicle window. The workflow progresses from classifying objects, to finding interactions, to selecting a meaningful period of activity.

15:2915:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:29 · section reference included

Advertising: find the right moment for a placement

The advertising demo uses what Le describes as a five-minute Adidas commercial. He asks Jockey to classify potential advertising placement points using reveals, hard cuts, impacts and energy peaks. Here, ingestion and retrieval serve a different intent: identifying moments whose visual and temporal character makes them useful to an advertiser.

The demonstration locates player appearances, high-impact action and the Adidas logo. Le then calls out slow-motion hero scenes, hard cuts on a beat and peak action as opportunities for contextual placement. The desired output is a set of candidate moments for brand content; the demo does not show ads being inserted.

17:0317:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:03 · section reference included

Build applications above the memory layer

The demonstrations lead to four kinds of application work: discovering moments, building reasoning experiences, organizing content across libraries, and taking action through scene assembly, compliance review or data operations. The same foundation serves different verticals:

DomainWorkflows
Media, entertainment and sportsSegmentation and highlight generation
Commercial securityEvidence review and contextual analysis
AdvertisingBrand safety and creative intelligence

The reusable layer is the grounded representation of video; the application determines what to do with it.

Le closes by describing the product capabilities beneath those applications: a knowledge store for video memory, configurable ingestion to shape extraction, a corpus digest to describe the library, entity resolution, agentic search and a Responses API. The product boundary is explicit. This is video cognition infrastructure with memory and worker harnesses, not an editing platform or a compliance application.

Diagram linking a video worker, references, knowledge store, and Responses API to three applications, alongside capability and product-boundary bullets.Open full source frame
Jockey’s video cognition infrastructure supports search, content assembly, and content organization.

At the time of the talk, the product was in private beta, with interest registration through an on-screen QR code or a conversation with Le. The current documentation calls it a research preview, with availability, limits and APIs subject to change. His invitation centers on concrete workflows: content assembly and organization, media archives, creator libraries on YouTube and TikTok, sports analysis and broader media operations. Those are the applications the memory layer is intended to make possible.

18:2218:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:22 · section reference included

Resources

From the talk

Updates since the talk

  • Entry point for creating a knowledge store and querying Jockey for cited answers or ranked video moments. Access and APIs remain subject to research-preview changes.

Transcript download
Full transcript

Loading transcript…