AI Engineer World's Fair 2026

When Agents Meet Physical Data: The Other Physics of Agent Harnesses

Loading slides…

Read the talk

When Agents Meet Physical Data: The Other Physics of Agent Harnesses

Video, sensor data and robot telemetry need more than generated code: they need schemas, recoverable execution, cheap verification and memory of work already done.

From a talk by Dmitry Petrov

Before you start: Basic familiarity with Python, database tables and object storage will help you follow the schema and execution examples.

What does an agent see in a bucket of videos?

How should a coding agent work with video recordings, sensor data and robot telemetry—especially when one project combines all three? A directory listing reveals filenames, but not the objects, events or relationships inside those binaries. Generating a script is only the beginning of understanding and processing them.

Even structured analytics needs specialized context. Petrov points to Anthropic’s self-service analytics work: its internal analytics-question evaluations reported accuracy no higher than 21% without skills. That figure concerns those internal questions, not data projects in general. OpenAI’s in-house data agent uses six layers of context. Both systems start with the advantages of structured data: tables, warehouses and existing execution engines. Physical-data projects must first make sense of messy files.

Slide contrasts a structured warehouse with schemas, indexes and a query engine against petabytes of files in S3/GCS, including video, sensor data and robot telemetry. A black hole illustration appears above.Open full source frame
Physical data contrasts structured warehouses with petabytes of files in S3/GCS.

Dmitry Petrov built Data Version Control, which he describes as Git for data, and now works on DataChain. His architecture starts with the LLM as the brain, then supplies the missing capabilities around it: sight to understand data, legs to execute processing, hands to verify results, and memory to retain important datasets and reuse earlier work. The harness must make the data understandable, executable, testable and memorable.

0:020:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:02 · section reference included

A small file inventory can conceal millions of objects

A few thousand files may look manageable until processing exposes their internal hierarchy. Videos contain clips; clips contain frames; frames contain detected objects; objects carry confidence scores, types, classes and labels. Relationships between those levels are part of the data too. Counting files therefore gives a poor estimate of the analytical workload.

Petrov compares this density to a neutron star: roughly city-sized on the surface, yet more massive than the Sun. His illustrative data equivalent is 2,000 video files yielding millions of internal objects. The storage inventory looks small while the representation needed for analysis expands dramatically.

2:322:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:32 · section reference included

Turn metadata into rows without splitting the programming model

The first response is often to write metadata as JSON beside the images or videos in S3. As the extracted object count grows, this creates millions of small metadata files, with latency, efficiency and consistency problems. Moving the metadata into a centralized database improves the storage model, but can leave researchers managing two systems, two languages and two stacks.

ApproachMetadata representationMain consequence
JSON sidecarsFiles beside the binariesMany small reads and consistency work
Separate database stackCentralized metadata recordsResearchers manage a second programming environment
Pydantic plus Python-to-SQL translationTyped Python schemas backed by rowsSchemas and processing stay in Python

Pydantic supplies the common language for schemas and code. Python expressions can then be translated into SQL, keeping the database’s query capabilities without scattering separate SQL islands through the codebase.

Slide titled “One language. One schema: Pydantic.” shows Python models for File, BBox and Detection beside boxes labeled typed rows, schema, versions and lineage.Open full source frame
Pydantic schemas organize file and detection data into typed, versioned rows.

This solves a particular boundary problem: extracting structure from physical binaries so ordinary analytical tools can operate on it. A business dataset that already lives in warehouse tables starts on the other side of that boundary.

3:564:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:56 · section reference included

Scope a dashcam analysis before spending compute

The demonstration uses open-source DataChain as a data harness for Claude Code. Setup proceeds in order:

  1. Install DataChain with pip.
  2. Install its skill for the selected coding agent.
  3. Launch the agent and describe the task.

Petrov says the integration also supports three other coding agents. He skips permission prompts to accelerate the demonstration; that is a demo choice, not a requirement of the architecture.

The task is motion analysis on an open dashcam-recording dataset. Video makes a useful example because it is common—and sometimes the dominant modality—in physical-AI projects, while still exposing the difficulty of turning a binary recording into useful observations.

Before processing, the harness asks questions that determine both cost and output shape:

  • Detection model: It proposes YOLO, and Petrov selects the smallest model size.
  • Velocity estimation: He selects the simplest available method; an alternative would take more compute time.
  • Granularity: He chooses per-frame output rather than per-track output.
  • Scope: He keeps the January subset of 91 clips and declines the offer to expand it.

These choices establish what the resulting dataset will mean before the expensive work begins.

5:526:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:52 · section reference included

Pay for extraction, then query the result

Petrov reports that analyzing roughly 90 videos took 24 minutes. With processing complete, he asks how many videos contain people. The demonstration returns people detected in 82 of 91 clips. These are results from the selected dashcam task; the recording does not establish a reproducible hardware configuration for the timing.

The answer comes from the local database, not another pass over the videos or a download-and-parse loop through JSON sidecars. The generated Python loads the dataset, filters for the person label, counts the relevant clips and compares that count with the total. The expensive perception step has already produced the information needed for the analytical question.

Petrov reports that roughly 90 videos generated about 100,000 records. This is the hidden density made visible. He projects that thousands of videos can reach millions of records, and that deeper analysis of objects can multiply the count by ten, twenty or even hundreds. Those projections describe how the representation can expand, rather than additional measured runs.

8:398:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:39 · section reference included

The data model connects detections to their source

Inspecting the generated code reveals the central artifact: a Pydantic model derived from the task requirements. A detection carries its video file, frame ID, timestamp, class ID, confidence and bounding box. The class identifies a person, car or another object; the bounding box is itself a nested structure.

The file is also structured data. It carries a path, size and available storage metadata such as checksums, version information and an ETag. Those fields should remain distinct: an S3 ETag is not universally a whole-object MD5 checksum, particularly for multipart uploads. A compact Python representation of the described fields is:

python

from pydantic import BaseModel


class FileRecord(BaseModel):
    path: str
    size: int
    checksum: str | None = None
    version_id: str | None = None
    etag: str | None = None


class BoundingBox(BaseModel):
    x1: float
    y1: float
    x2: float
    y2: float


class Detection(BaseModel):
    file: FileRecord
    frame_id: int
    timestamp: float
    class_id: int
    label: str
    confidence: float
    bbox: BoundingBox

The nested Python object becomes queryable database data rather than another JSON file in S3.

Once these fields are rows, familiar analytical operations apply: filter labels, group observations, count matching clips and relate results back to source files. The binary remains in storage; its extracted meaning becomes structured data. That separation is what makes repeated questions inexpensive relative to repeated perception.

11:3211:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:32 · section reference included

Connect typed Python to distributed execution

Schemas reveal what is inside the files and how large the problem is. Processing terabytes still requires an execution engine. It must use resources efficiently, account for potentially expensive LLM calls and preserve completed work when a job fails. A structured-data warehouse already provides an execution engine; an unstructured-data system needs to supply that capability around its files.

Teams may assemble orchestration and distributed compute with Ray or Spark. Petrov particularly likes Dask’s model of connecting computation to structured data. His corresponding design for physical data joins Python functions, Pydantic input and output schemas, metadata in a warehouse, and files in object storage. Together, these give the engine the information needed to distribute work across machines or threads according to the files being processed and the schemas being produced.

At the processing boundary, a file from S3 becomes a collection of detection objects destined for the warehouse. The user writes an ordinary typed Python function that yields those objects. Petrov’s point about avoiding extra function annotations is that no special execution decorator is needed: the specified input and output types still matter. The engine connects storage, execution and persistence around that function.

Resource allocation belongs outside the detection logic. Petrov illustrates the intended simplicity by asking to run on 40 machines. The saved configuration visible in the demonstration instead shows parallel=4 and workers=30; it is a configuration example, not evidence of a completed 40-machine run.

Function shapeOutput per input fileUse
MapperOne objectA file-level result
GeneratorMultiple objectsDetections within a file

The pipeline applies the function and saves its output as a named dataset—a database table underneath. In the visible code, detection generation feeds the persisted dashcam-jan dataset.

Code editor shows detection objects being yielded and a pipeline that reads video storage, sets parallel=4 and workers=30, loads a model, generates detections and saves dashcam-jan. The status line confirms the file was written.Open full source frame
The saved Python pipeline connects detection generation to a persisted dataset.
13:3913:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:39 · section reference included

A failed run should not erase completed work

Processing heavy files can take a long time, especially when understanding their contents requires LLM calls. If a bug or API failure interrupts a run over hundreds of thousands of files, restarting everything discards both time and money. Repeating that failure compounds the waste.

Checkpointing and incremental processing preserve different kinds of work. Checkpoints let a repaired run recover completed results. Incremental processing lets a later run handle newly arrived files without recomputing existing outputs. Petrov describes both as essential. Current DataChain examples make the operational boundary explicit: recovery retains committed batches, an uncommitted batch may repeat, and the incremental example enables delta=True. A rerun therefore needs the appropriate recovery or delta configuration; merely invoking a script again is not a guarantee of zero repeated computation.

18:0118:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:01 · section reference included

Give the agent a cheap way to check its answers

Coding agents use tests both for quality control and to reason about the environment. These are the harness’s hands: a way to touch the system and check what happened. Data work needs that feedback too, but a test that repeatedly processes raw videos can be prohibitively slow. Petrov draws on Anthropic’s distinction between software tasks with many acceptable implementations and data questions that usually have one correct answer.

The first improvement is to stop treating every question as a new Python computation over raw binaries. Build metadata datasets around the raw data so the relevant questions can be answered quickly. Dimensional and multidimensional modeling, star schemas and the one-big-table approach offer established ways to organize that layer. The choice of layout serves the same purpose: make the next analytical operation cheap enough to support iteration.

The harness changes the agent’s procedure before it answers:

  1. Check whether existing datasets contain the metadata needed for a single SQL-like query.
  2. If the metadata is missing, build the necessary layer.
  3. Make that layer broad enough to answer related questions, rather than only the current wording.
  4. Retain it for the user and teammates to reuse.

This turns a question into an opportunity to improve the data representation. Future checks can operate on the accumulated metadata instead of repeatedly paying for extraction.

19:1019:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:10 · section reference included

Remember why a dataset exists, not just its name

Once useful slices exist, they can answer questions without rerunning expensive compute. But that saving disappears if another person or agent cannot discover them. Petrov describes teams paying twice, three times or four times for the same analysis. Coding agents such as Copilot, Codex and PI already use source-code context and indexes; a data harness needs comparable memory of the data work already performed.

A dataset name alone is insufficient. Memory should preserve why the dataset was built, the session context that motivated it and a useful description, often enriched by an LLM. It should also retain the producing source code, whose importance Petrov connects to OpenAI’s data-agent work. The knowledge base must expose this information to both agents and people. In the demonstration, it takes the straightforward form of Markdown files.

The record for the newly created dataset includes:

  • Description and rationale: What the dataset contains and why the conversation led to its creation.
  • Storage dependencies: The source storage location and directory.
  • Preview and schema: Sample data and its structure.
  • Statistics and source code: A sense of the dataset’s contents and the program that produced it.

The code is especially valuable because it explains the transformation, not merely the resulting column names.

These records connect source data in the bucket, producing code in the knowledge base and results in the warehouse into data lineage. Sharing that knowledge makes the existing dataset—and the resources already spent creating it—discoverable to teammates. Petrov’s intended behavior is that a later question about the same directory uses a suitable existing result instead of launching the extraction again. The harness’s memory therefore changes the next execution decision, not just the documentation available afterward.

21:5022:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:50 · section reference included

Build the stack around the cost of physical data

The complete stack begins with the mass of unstructured files in object storage. A compute engine performs expensive extraction, including LLM calls where needed. A Dataset DB organizes the resulting metadata into datasets and useful slices. A knowledge base makes those results and their context available for reuse. Each layer makes the work beneath it more accessible without requiring that work to be repeated for every question.

Four-tier pyramid labeled Raw files · S3, Compute engine, Dataset DB and Knowledge base, with build-once and read-forever arrows and the footer “Build it once. Read it forever.”Open full source frame
The stack connects raw files, compute, a dataset database and a knowledge base.

Copilot, Codex and Claude Code can generate the processing code, but their default habits do not necessarily account for the cost and density of physical data. Petrov’s closing prescription is therefore not simply to choose a stronger model: teams are already using frontier models. The missing capability is a harness that understands these constraints and supplies the surrounding data system. DataChain implements some of those principles as an open-source project. Giving an agent that infrastructure is what Petrov means by adding mass to it: the ability to build on data work already done.

25:5225:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:52 · section reference included

Resources

From the talk

  • Maps coding-agent context and memory onto persistent datasets and a shared data knowledge base.

  • Explains object integrity checks and when an ETag does or does not represent a whole-object MD5 digest.

Updates since the talk

Transcript download
Full transcript

Loading transcript…