AI Engineer World's Fair 2026
When Agents Meet Physical Data: The Other Physics of Agent Harnesses
Loading slides…
Read the talk
When Agents Meet Physical Data: The Other Physics of Agent Harnesses
Video, sensor data and robot telemetry need more than generated code: they need schemas, recoverable execution, cheap verification and memory of work already done.
From a talk by Dmitry Petrov
Before you start: Basic familiarity with Python, database tables and object storage will help you follow the schema and execution examples.
What does an agent see in a bucket of videos?
How should a coding agent work with video recordings, sensor data and robot telemetry—especially when one project combines all three? A directory listing reveals filenames, but not the objects, events or relationships inside those binaries. Generating a script is only the beginning of understanding and processing them.
Even structured analytics needs specialized context. Petrov points to Anthropic’s self-service analytics work: its internal analytics-question evaluations reported accuracy no higher than 21% without skills. That figure concerns those internal questions, not data projects in general. OpenAI’s in-house data agent uses six layers of context. Both systems start with the advantages of structured data: tables, warehouses and existing execution engines. Physical-data projects must first make sense of messy files.
Dmitry Petrov built Data Version Control, which he describes as Git for data, and now works on DataChain. His architecture starts with the LLM as the brain, then supplies the missing capabilities around it: sight to understand data, legs to execute processing, hands to verify results, and memory to retain important datasets and reuse earlier work. The harness must make the data understandable, executable, testable and memorable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small file inventory can conceal millions of objects
A few thousand files may look manageable until processing exposes their internal hierarchy. Videos contain clips; clips contain frames; frames contain detected objects; objects carry confidence scores, types, classes and labels. Relationships between those levels are part of the data too. Counting files therefore gives a poor estimate of the analytical workload.
Petrov compares this density to a neutron star: roughly city-sized on the surface, yet more massive than the Sun. His illustrative data equivalent is 2,000 video files yielding millions of internal objects. The storage inventory looks small while the representation needed for analysis expands dramatically.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn metadata into rows without splitting the programming model
The first response is often to write metadata as JSON beside the images or videos in S3. As the extracted object count grows, this creates millions of small metadata files, with latency, efficiency and consistency problems. Moving the metadata into a centralized database improves the storage model, but can leave researchers managing two systems, two languages and two stacks.
| Approach | Metadata representation | Main consequence |
|---|---|---|
| JSON sidecars | Files beside the binaries | Many small reads and consistency work |
| Separate database stack | Centralized metadata records | Researchers manage a second programming environment |
| Pydantic plus Python-to-SQL translation | Typed Python schemas backed by rows | Schemas and processing stay in Python |
Pydantic supplies the common language for schemas and code. Python expressions can then be translated into SQL, keeping the database’s query capabilities without scattering separate SQL islands through the codebase.
This solves a particular boundary problem: extracting structure from physical binaries so ordinary analytical tools can operate on it. A business dataset that already lives in warehouse tables starts on the other side of that boundary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scope a dashcam analysis before spending compute
The demonstration uses open-source DataChain as a data harness for Claude Code. Setup proceeds in order:
- Install DataChain with
pip. - Install its skill for the selected coding agent.
- Launch the agent and describe the task.
Petrov says the integration also supports three other coding agents. He skips permission prompts to accelerate the demonstration; that is a demo choice, not a requirement of the architecture.
The task is motion analysis on an open dashcam-recording dataset. Video makes a useful example because it is common—and sometimes the dominant modality—in physical-AI projects, while still exposing the difficulty of turning a binary recording into useful observations.
Before processing, the harness asks questions that determine both cost and output shape:
- Detection model: It proposes YOLO, and Petrov selects the smallest model size.
- Velocity estimation: He selects the simplest available method; an alternative would take more compute time.
- Granularity: He chooses per-frame output rather than per-track output.
- Scope: He keeps the January subset of 91 clips and declines the offer to expand it.
These choices establish what the resulting dataset will mean before the expensive work begins.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pay for extraction, then query the result
Petrov reports that analyzing roughly 90 videos took 24 minutes. With processing complete, he asks how many videos contain people. The demonstration returns people detected in 82 of 91 clips. These are results from the selected dashcam task; the recording does not establish a reproducible hardware configuration for the timing.
The answer comes from the local database, not another pass over the videos or a download-and-parse loop through JSON sidecars. The generated Python loads the dataset, filters for the person label, counts the relevant clips and compares that count with the total. The expensive perception step has already produced the information needed for the analytical question.
Petrov reports that roughly 90 videos generated about 100,000 records. This is the hidden density made visible. He projects that thousands of videos can reach millions of records, and that deeper analysis of objects can multiply the count by ten, twenty or even hundreds. Those projections describe how the representation can expand, rather than additional measured runs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The data model connects detections to their source
Inspecting the generated code reveals the central artifact: a Pydantic model derived from the task requirements. A detection carries its video file, frame ID, timestamp, class ID, confidence and bounding box. The class identifies a person, car or another object; the bounding box is itself a nested structure.
The file is also structured data. It carries a path, size and available storage metadata such as checksums, version information and an ETag. Those fields should remain distinct: an S3 ETag is not universally a whole-object MD5 checksum, particularly for multipart uploads. A compact Python representation of the described fields is:
python
from pydantic import BaseModel
class FileRecord(BaseModel):
path: str
size: int
checksum: str | None = None
version_id: str | None = None
etag: str | None = None
class BoundingBox(BaseModel):
x1: float
y1: float
x2: float
y2: float
class Detection(BaseModel):
file: FileRecord
frame_id: int
timestamp: float
class_id: int
label: str
confidence: float
bbox: BoundingBox
The nested Python object becomes queryable database data rather than another JSON file in S3.
Once these fields are rows, familiar analytical operations apply: filter labels, group observations, count matching clips and relate results back to source files. The binary remains in storage; its extracted meaning becomes structured data. That separation is what makes repeated questions inexpensive relative to repeated perception.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect typed Python to distributed execution
Schemas reveal what is inside the files and how large the problem is. Processing terabytes still requires an execution engine. It must use resources efficiently, account for potentially expensive LLM calls and preserve completed work when a job fails. A structured-data warehouse already provides an execution engine; an unstructured-data system needs to supply that capability around its files.
Teams may assemble orchestration and distributed compute with Ray or Spark. Petrov particularly likes Dask’s model of connecting computation to structured data. His corresponding design for physical data joins Python functions, Pydantic input and output schemas, metadata in a warehouse, and files in object storage. Together, these give the engine the information needed to distribute work across machines or threads according to the files being processed and the schemas being produced.
At the processing boundary, a file from S3 becomes a collection of detection objects destined for the warehouse. The user writes an ordinary typed Python function that yields those objects. Petrov’s point about avoiding extra function annotations is that no special execution decorator is needed: the specified input and output types still matter. The engine connects storage, execution and persistence around that function.
Resource allocation belongs outside the detection logic. Petrov illustrates the intended simplicity by asking to run on 40 machines. The saved configuration visible in the demonstration instead shows parallel=4 and workers=30; it is a configuration example, not evidence of a completed 40-machine run.
| Function shape | Output per input file | Use |
|---|---|---|
| Mapper | One object | A file-level result |
| Generator | Multiple objects | Detections within a file |
The pipeline applies the function and saves its output as a named dataset—a database table underneath. In the visible code, detection generation feeds the persisted dashcam-jan dataset.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A failed run should not erase completed work
Processing heavy files can take a long time, especially when understanding their contents requires LLM calls. If a bug or API failure interrupts a run over hundreds of thousands of files, restarting everything discards both time and money. Repeating that failure compounds the waste.
Checkpointing and incremental processing preserve different kinds of work. Checkpoints let a repaired run recover completed results. Incremental processing lets a later run handle newly arrived files without recomputing existing outputs. Petrov describes both as essential. Current DataChain examples make the operational boundary explicit: recovery retains committed batches, an uncommitted batch may repeat, and the incremental example enables delta=True. A rerun therefore needs the appropriate recovery or delta configuration; merely invoking a script again is not a guarantee of zero repeated computation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the agent a cheap way to check its answers
Coding agents use tests both for quality control and to reason about the environment. These are the harness’s hands: a way to touch the system and check what happened. Data work needs that feedback too, but a test that repeatedly processes raw videos can be prohibitively slow. Petrov draws on Anthropic’s distinction between software tasks with many acceptable implementations and data questions that usually have one correct answer.
The first improvement is to stop treating every question as a new Python computation over raw binaries. Build metadata datasets around the raw data so the relevant questions can be answered quickly. Dimensional and multidimensional modeling, star schemas and the one-big-table approach offer established ways to organize that layer. The choice of layout serves the same purpose: make the next analytical operation cheap enough to support iteration.
The harness changes the agent’s procedure before it answers:
- Check whether existing datasets contain the metadata needed for a single SQL-like query.
- If the metadata is missing, build the necessary layer.
- Make that layer broad enough to answer related questions, rather than only the current wording.
- Retain it for the user and teammates to reuse.
This turns a question into an opportunity to improve the data representation. Future checks can operate on the accumulated metadata instead of repeatedly paying for extraction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remember why a dataset exists, not just its name
Once useful slices exist, they can answer questions without rerunning expensive compute. But that saving disappears if another person or agent cannot discover them. Petrov describes teams paying twice, three times or four times for the same analysis. Coding agents such as Copilot, Codex and PI already use source-code context and indexes; a data harness needs comparable memory of the data work already performed.
A dataset name alone is insufficient. Memory should preserve why the dataset was built, the session context that motivated it and a useful description, often enriched by an LLM. It should also retain the producing source code, whose importance Petrov connects to OpenAI’s data-agent work. The knowledge base must expose this information to both agents and people. In the demonstration, it takes the straightforward form of Markdown files.
The record for the newly created dataset includes:
- Description and rationale: What the dataset contains and why the conversation led to its creation.
- Storage dependencies: The source storage location and directory.
- Preview and schema: Sample data and its structure.
- Statistics and source code: A sense of the dataset’s contents and the program that produced it.
The code is especially valuable because it explains the transformation, not merely the resulting column names.
These records connect source data in the bucket, producing code in the knowledge base and results in the warehouse into data lineage. Sharing that knowledge makes the existing dataset—and the resources already spent creating it—discoverable to teammates. Petrov’s intended behavior is that a later question about the same directory uses a suitable existing result instead of launching the extraction again. The harness’s memory therefore changes the next execution decision, not just the documentation available afterward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the stack around the cost of physical data
The complete stack begins with the mass of unstructured files in object storage. A compute engine performs expensive extraction, including LLM calls where needed. A Dataset DB organizes the resulting metadata into datasets and useful slices. A knowledge base makes those results and their context available for reuse. Each layer makes the work beneath it more accessible without requiring that work to be repeated for every question.
Copilot, Codex and Claude Code can generate the processing code, but their default habits do not necessarily account for the cost and density of physical data. Petrov’s closing prescription is therefore not simply to choose a stronger model: teams are already using frontier models. The missing capability is a harness that understands these constraints and supplies the surrounding data system. DataChain implements some of those principles as an open-source project. Giving an agent that infrastructure is what Petrov means by adding mass to it: the ability to build on data work already done.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's internal analytics architecture, including curated skills, semantic definitions, evaluation and maintenance.
Six context layers combining table usage, human annotations, code enrichment, institutional knowledge, memory and runtime inspection.
Open-source Python library for typed datasets over object storage, with installation instructions and processing examples.
Further reading
- DataChain as a data harnessDocumentation
Maps coding-agent context and memory onto persistent datasets and a shared data knowledge base.
- Amazon S3 checksums and ETagsDocumentation
Explains object integrity checks and when an ETag does or does not represent a whole-object MD5 digest.
Updates since the talk
Current guides to DataChain's processing engine, dataset database and agent integrations.
Full transcript
Loading transcript…