AI Engineer World's Fair 2024
The Hierarchy of Needs for Training Dataset Development
Loading slides…
Read the talk
The Hierarchy of Needs for Training Dataset Development
Dataset development connects evaluation, selection, and human judgment to an infrastructure problem: filtering, shuffling, and streaming large samples without slowing research.
From a talk by Chang She and Noah Shpak
Before you start: Familiarity with model training, SQL, and basic storage concepts such as rows, columns, and object storage will help.
What should the model train on?
What should a model train on, and how can a team tell whether its choices helped? Answering those questions requires both data research and data infrastructure. Chang She introduces his infrastructure background through pandas, large data systems, and recommender systems, leading to his work as co-founder and CEO of LanceDB, a database for multimodal AI.
At Character.ai, the question connects directly to a consumer product. Noah Shpak describes leading its AI data platform, supporting both internally trained foundation models and the service people use. Training choices must produce results on academic benchmarks, but also in A/B tests and user engagement. His team builds tools to accelerate that research. A suitable storage format matters because researchers need to inspect, change, and reuse the data they train on.
The level of inspection changes across training stages. Pre-training tends to begin with broad composition and quantity; post-training demands a closer look at individual tasks.
| Decision | Pre-training emphasis | Post-training emphasis |
|---|---|---|
| Composition | Domains such as books and chat | Specific tasks and their contexts |
| Scale | Model size and token requirements | Coverage of relevant examples |
| Granularity | Broad domain mixture | Difficulty of a math or multiple-choice problem |
These emphases overlap. The practical difference is how much detail researchers need to understand about each example, across the whole dataset.
Several research problems cut across that distinction: achieving good results with less data for a similarly sized model, choosing useful metrics for sampling, and measuring diversity. Diversity is especially difficult: having many examples does not by itself establish that they cover meaningfully different behaviors. These questions motivate a hierarchy of tools for understanding and improving the training set.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build from clean data and evaluations
Clean data and evaluations form the base of dataset development. Shpak’s team moves quickly from cleaning to evaluation because it needs a compass for deciding whether a change helped. As post-training work expands, managing datasets becomes a research requirement in its own right. A mixture is a collection of datasets together with choices about how they enter training sets and batches. Knowing that one component is Wikipedia is not enough; researchers need to know what examples it actually contributes.
That need leads to analytics. Token counts and sequence lengths describe the basic shape of a mixture. For code, language labels such as Python or Java are only a beginning: task difficulty, the number of functions involved, and the number of classes an answer must generate can reveal more about what the model is being asked to learn. Shpak identifies reading data as a particularly valuable practice. The useful analytics automate distinctions researchers first discover by inspecting examples, model outputs, and performance together.
Above that foundation, language models help improve the data used to train language models. The operations serve different purposes:
- Dataset selection: Match the distribution of available data to desired model behavior. Embeddings, retrieval, and clustering help find examples relevant to the evaluation goals.
- Quality scoring: Use in-house classifiers to assess desired properties, or use prompted classification as a simpler route than building and evaluating a dedicated classifier.
- Synthetic data: Combine big-data tools such as Spark and Trino with GPU-backed services for prompting, embedding, and classification. This makes enrichment and augmentation part of the data platform rather than a separate manual process.
Synthetic preference pairs, for example, let researchers explore a method before they have its best possible dataset. Their immediate value is signal about which types and shapes of data might work. Shpak explicitly treats synthetic data as imperfect: once the approach looks promising, human labeling can improve it. Human work also improves the classifiers themselves and rewrites flawed synthetic examples or flawed existing data. The hierarchy therefore ends with human judgment refining the processes below it, and the whole sequence motivates tools that make these operations easy for researchers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make dataset materialization part of training
The platform interface Shpak shows is YAML containing a SQL block. That arrangement makes dataset selection explicit at the point where a training job requests its data. Conventional training artifacts—TFRecords or JSON Lines, for example—can become a source of confusion: a job consumes files, produces strange results, and leaves the researcher asking what those files actually contain.
Materialization belongs in the training workflow, with responsibilities separated from the training code. A query expresses the data choice; a materialization service produces the artifacts the job will consume. For a hypothetical table of code examples, a small YAML recipe could express a language-and-length slice like this:
yaml
sql: |
SELECT text, language, token_count
FROM training_examples
WHERE language = 'Python'
AND token_count BETWEEN 128 AND 2048
The particular table and thresholds are illustrative; the important boundary is between selecting and materializing data and running the training computation.
Multimodal data makes that boundary more consequential because both the volumes and the problems grow. The materialization service has a simple external contract: send a request, receive lists of files. Loading those files efficiently remains a separate problem, especially when the dataset becomes large.
Lance provides the random access that lets the loader shuffle references to rows instead of shuffling the rows themselves. For example, a loader can permute row references from [0, 1, 2] to [2, 0, 1] while leaving the stored samples in place, then fetch samples in that order. This does not eliminate reading the samples; it avoids moving their full payloads merely to establish a randomized order. The objective is faster research iteration while keeping GPUs supplied with work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filter, shuffle, and stream require different storage strengths
The loading problem leads into She’s infrastructure argument. AI workloads combine access patterns that traditional data warehousing and analytical systems do not necessarily optimize together. Distributed training provides a concrete example:
- Filter: Scan the raw dataset to select appropriate samples.
- Shuffle: Draw random rows from the filtered set.
- Stream: Move the selected text, images, or videos from object storage into GPUs.
The steps are sequential, but each puts a different demand on storage.
| Training step | Required capability |
|---|---|
| Filter | Fast scans |
| Shuffle | Fast random access |
| Stream | Efficient handling of large binary blobs |
A format that scans efficiently can still be expensive for scattered reads. A system that retrieves small records quickly may struggle with large video payloads. The training workflow needs all three capabilities together.
She argues that existing formats and infrastructure commonly satisfy only one or two of these requirements well. He calls the tension a “new CAP theorem” for AI data, using it as an engineering analogy rather than a formal impossibility theorem. It motivates the Lance format and, around that format, LanceDB.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Larger rows and faster data creation compound the problem
Multimodal data increases the cost of moving an individual row. She illustrates the progression with rough size estimates:
| Representation | She’s illustrative size comparison |
|---|---|
| Simple scalar tabular row | About 150 bytes per row |
| With embeddings | Roughly 20–25 times larger, depending on dimensions |
| With images | Another roughly 20-fold increase |
| With videos | Much larger again |
These are contextual estimates, not a benchmark with a specified dataset, encoding, or compression setting. Their purpose is to show why moving full rows becomes increasingly expensive as more modalities enter the dataset.
The number of rows can grow faster too. She describes generated data arriving at thousands of tokens per second, rather than being limited by the pace of human interaction. His scale comparison extends to organizations: teams of 10–20 people may manage tens of terabytes through petabytes of data. Together, larger records and faster creation make storage efficiency relevant even to relatively small teams.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine a columnar file with versioned table operations
Lance addresses the storage requirements at several layers. First, it is a columnar file format: like Parquet, it supports scans, while also targeting efficient lookups of selected rows. The removal of Parquet-style row groups that She describes belongs to the Lance v2 design, which separates column pages to accommodate wide values and inline blobs. This was a design in transition at the time of the talk: the later Lance File 2.1 retrospective dates v2 becoming the default to fall 2024.
Second, Lance provides a lightweight table format. Added data is automatically versioned, and new columns can be added without copying the original dataset. That is particularly useful when the original columns contain large images or videos: an experimental feature column should not require rewriting those payloads. She calls this zero-copy schema evolution and connects it to trying features and rolling them back later.
Time travel provides another recovery mechanism. If a change introduces bad data or an error, the table can return to a previously known good version before the problem propagates into downstream training. She describes that rollback as instantaneous; the talk does not specify conditions or a latency measurement. The practical benefit is that researchers can revise dataset state without treating every experiment as an irreversible rewrite.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An index must make the selected rows cheap to fetch
The third layer is indexing. An index can identify which rows match a request, but that only solves half the problem: storage must still retrieve those rows efficiently. She contrasts Lance with Parquet on this basis. The distinction is performance, not the absolute presence of random access: Parquet’s optional page indexes support row-index navigation and selective page reads, while implementation support and page-level read amplification affect the cost of fetching individual values.
Lance’s indexing extensions cover several access patterns:
- Embedding indexes: She describes billion-scale vector search directly from S3.
- Scalar indexes: Accelerate filtering on metadata columns.
- Full-text indexes: Support keyword or fuzzy search over an S3 dataset.
For these search workloads, She proposes that a separate Elasticsearch cluster can become unnecessary. The broader goal is a single table that supports several kinds of access, rather than a separate data copy for each task.
| Data and operation | Interface described in the talk |
|---|---|
| SQL over metadata or time-series columns | DuckDB, Trino, or Spark |
| Training on blobs and tensors | PyTorch |
| Similarity search over embeddings | Embedded vector index |
These interfaces connect exploration, retrieval, and training to the same underlying table. A researcher can analyze the dataset, search for relevant examples, and use its stored payloads for fine-tuning or training without making each activity a separate storage system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the research loop fast as requirements grow
LanceDB is the database layer built around the format. She claims distributed search across billions of vectors with low latency, high QPS, and an order of magnitude less infrastructure than other vector databases. That is a vendor performance claim without a specified comparison setup in the talk, rather than a measured result readers can apply to their own workload.
The intended scope of multimodal infrastructure is broader than image or video generation. She distinguishes three dimensions:
- Data: Features, audio waveforms, images, and vectors—including text and image embeddings.
- Workloads: OLAP SQL, vector search, full-text search, filtering, and data-frame operations.
- Scenarios: Production RAG, search and retrieval, personalization, model training, and data-lake analysis or exploration.
The same format is intended to connect those dimensions, allowing data to serve operational applications as well as research and training.
Shpak closes by making research speed the strategic priority. His team’s experience is that tools slow down under load and under new multimodal requirements. The platform’s job is to sustain the loop: understand the data, change it, train, evaluate, and try again. Storage choices matter because they determine how readily researchers can keep doing that as the workload grows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and getting-started examples for the Lance data format.
Further reading
Explains the 2024 design for independent column pages, efficient point lookups, and storage without row groups.
A 2024 Python walkthrough for streaming, tokenizing, storing, and retrieving training data by index.
- Apache Parquet page indexesDocumentation
Specification for page skipping, selective reads, and navigation by row index in Parquet.
Updates since the talk
A 2025 paper coauthored by Chang She examining columnar random access, scans, and structural encodings on NVMe storage.
Full transcript
Loading transcript…