AI Engineer World's Fair 2024

The Hierarchy of Needs for Training Dataset Development

Chang She· CEO, LanceDBNoah Shpak· Member of Technical Staff, Character.ai16:32

Loading slides…

Read the talk

The Hierarchy of Needs for Training Dataset Development

Dataset development connects evaluation, selection, and human judgment to an infrastructure problem: filtering, shuffling, and streaming large samples without slowing research.

From a talk by Chang She and Noah Shpak

Before you start: Familiarity with model training, SQL, and basic storage concepts such as rows, columns, and object storage will help.

What should the model train on?

What should a model train on, and how can a team tell whether its choices helped? Answering those questions requires both data research and data infrastructure. Chang She introduces his infrastructure background through pandas, large data systems, and recommender systems, leading to his work as co-founder and CEO of LanceDB, a database for multimodal AI.

At Character.ai, the question connects directly to a consumer product. Noah Shpak describes leading its AI data platform, supporting both internally trained foundation models and the service people use. Training choices must produce results on academic benchmarks, but also in A/B tests and user engagement. His team builds tools to accelerate that research. A suitable storage format matters because researchers need to inspect, change, and reuse the data they train on.

The level of inspection changes across training stages. Pre-training tends to begin with broad composition and quantity; post-training demands a closer look at individual tasks.

DecisionPre-training emphasisPost-training emphasis
CompositionDomains such as books and chatSpecific tasks and their contexts
ScaleModel size and token requirementsCoverage of relevant examples
GranularityBroad domain mixtureDifficulty of a math or multiple-choice problem

These emphases overlap. The practical difference is how much detail researchers need to understand about each example, across the whole dataset.

Several research problems cut across that distinction: achieving good results with less data for a similarly sized model, choosing useful metrics for sampling, and measuring diversity. Diversity is especially difficult: having many examples does not by itself establish that they cover meaningfully different behaviors. These questions motivate a hierarchy of tools for understanding and improving the training set.

Data Recipes slide groups data quantity and domain composition with pre-training, task composition, example difficulty and prompting mechanisms with post-training, and shared data preparation concerns between them.Open full source frame
Data recipes span pre-training and post-training.
0:210:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:21 · section reference included

Build from clean data and evaluations

Clean data and evaluations form the base of dataset development. Shpak’s team moves quickly from cleaning to evaluation because it needs a compass for deciding whether a change helped. As post-training work expands, managing datasets becomes a research requirement in its own right. A mixture is a collection of datasets together with choices about how they enter training sets and batches. Knowing that one component is Wikipedia is not enough; researchers need to know what examples it actually contributes.

That need leads to analytics. Token counts and sequence lengths describe the basic shape of a mixture. For code, language labels such as Python or Java are only a beginning: task difficulty, the number of functions involved, and the number of classes an answer must generate can reveal more about what the model is being asked to learn. Shpak identifies reading data as a particularly valuable practice. The useful analytics automate distinctions researchers first discover by inspecting examples, model outputs, and performance together.

Above that foundation, language models help improve the data used to train language models. The operations serve different purposes:

  • Dataset selection: Match the distribution of available data to desired model behavior. Embeddings, retrieval, and clustering help find examples relevant to the evaluation goals.
  • Quality scoring: Use in-house classifiers to assess desired properties, or use prompted classification as a simpler route than building and evaluating a dedicated classifier.
  • Synthetic data: Combine big-data tools such as Spark and Trino with GPU-backed services for prompting, embedding, and classification. This makes enrichment and augmentation part of the data platform rather than a separate manual process.

Synthetic preference pairs, for example, let researchers explore a method before they have its best possible dataset. Their immediate value is signal about which types and shapes of data might work. Shpak explicitly treats synthetic data as imperfect: once the approach looks promising, human labeling can improve it. Human work also improves the classifiers themselves and rewrites flawed synthetic examples or flawed existing data. The hierarchy therefore ends with human judgment refining the processes below it, and the whole sequence motivates tools that make these operations easy for researchers.

Hierarchy of Needs diagram stacks clean data, evaluations, systems for dataset management, analytics, dataset selection, quality scoring and sampling, synthetics, and human labeling. A Start Here arrow points toward the base.Open full source frame
The dataset development hierarchy starts with clean data and evaluations.
2:563:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:56 · section reference included

Make dataset materialization part of training

The platform interface Shpak shows is YAML containing a SQL block. That arrangement makes dataset selection explicit at the point where a training job requests its data. Conventional training artifacts—TFRecords or JSON Lines, for example—can become a source of confusion: a job consumes files, produces strange results, and leaves the researcher asking what those files actually contain.

Materialization belongs in the training workflow, with responsibilities separated from the training code. A query expresses the data choice; a materialization service produces the artifacts the job will consume. For a hypothetical table of code examples, a small YAML recipe could express a language-and-length slice like this:

yaml

sql: |
  SELECT text, language, token_count
  FROM training_examples
  WHERE language = 'Python'
    AND token_count BETWEEN 128 AND 2048

The particular table and thresholds are illustrative; the important boundary is between selecting and materializing data and running the training computation.

Multimodal data makes that boundary more consequential because both the volumes and the problems grow. The materialization service has a simple external contract: send a request, receive lists of files. Loading those files efficiently remains a separate problem, especially when the dataset becomes large.

Lance provides the random access that lets the loader shuffle references to rows instead of shuffling the rows themselves. For example, a loader can permute row references from [0, 1, 2] to [2, 0, 1] while leaving the stored samples in place, then fetch samples in that order. This does not eliminate reading the samples; it avoids moving their full payloads merely to establish a randomized order. The objective is faster research iteration while keeping GPUs supplied with work.

6:096:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:09 · section reference included

Filter, shuffle, and stream require different storage strengths

The loading problem leads into She’s infrastructure argument. AI workloads combine access patterns that traditional data warehousing and analytical systems do not necessarily optimize together. Distributed training provides a concrete example:

  1. Filter: Scan the raw dataset to select appropriate samples.
  2. Shuffle: Draw random rows from the filtered set.
  3. Stream: Move the selected text, images, or videos from object storage into GPUs.

The steps are sequential, but each puts a different demand on storage.

Training stepRequired capability
FilterFast scans
ShuffleFast random access
StreamEfficient handling of large binary blobs

A format that scans efficiently can still be expensive for scattered reads. A system that retrieves small records quickly may struggle with large video payloads. The training workflow needs all three capabilities together.

She argues that existing formats and infrastructure commonly satisfy only one or two of these requirements well. He calls the tension a “new CAP theorem” for AI data, using it as an engineering analogy rather than a formal impossibility theorem. It motivates the Lance format and, around that format, LanceDB.

Slide with three overlapping circles labeled Random Access, Large Blobs, and Fast Scans, annotated with storage formats and systems. Beneath the diagram is CAP Theorem for AI Data; a speaker inset appears at lower left.Open full source frame
AI Data Challenges: random access, large blobs, and fast scans.
7:528:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:52 · section reference included

Larger rows and faster data creation compound the problem

Multimodal data increases the cost of moving an individual row. She illustrates the progression with rough size estimates:

RepresentationShe’s illustrative size comparison
Simple scalar tabular rowAbout 150 bytes per row
With embeddingsRoughly 20–25 times larger, depending on dimensions
With imagesAnother roughly 20-fold increase
With videosMuch larger again

These are contextual estimates, not a benchmark with a specified dataset, encoding, or compression setting. Their purpose is to show why moving full rows becomes increasingly expensive as more modalities enter the dataset.

The number of rows can grow faster too. She describes generated data arriving at thousands of tokens per second, rather than being limited by the pace of human interaction. His scale comparison extends to organizations: teams of 10–20 people may manage tens of terabytes through petabytes of data. Together, larger records and faster creation make storage efficiency relevant even to relatively small teams.

9:4110:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:41 · section reference included

Combine a columnar file with versioned table operations

Lance addresses the storage requirements at several layers. First, it is a columnar file format: like Parquet, it supports scans, while also targeting efficient lookups of selected rows. The removal of Parquet-style row groups that She describes belongs to the Lance v2 design, which separates column pages to accommodate wide values and inline blobs. This was a design in transition at the time of the talk: the later Lance File 2.1 retrospective dates v2 becoming the default to fall 2024.

Second, Lance provides a lightweight table format. Added data is automatically versioned, and new columns can be added without copying the original dataset. That is particularly useful when the original columns contain large images or videos: an experimental feature column should not require rewriting those payloads. She calls this zero-copy schema evolution and connects it to trying features and rolling them back later.

Time travel provides another recovery mechanism. If a change introduces bad data or an error, the table can return to a previously known good version before the problem propagates into downstream training. She describes that rollback as instantaneous; the talk does not specify conditions or a latency measurement. The practical benefit is that researchers can revise dataset state without treating every experiment as an irreversible rewrite.

10:5711:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:57 · section reference included

An index must make the selected rows cheap to fetch

The third layer is indexing. An index can identify which rows match a request, but that only solves half the problem: storage must still retrieve those rows efficiently. She contrasts Lance with Parquet on this basis. The distinction is performance, not the absolute presence of random access: Parquet’s optional page indexes support row-index navigation and selective page reads, while implementation support and page-level read amplification affect the cost of fetching individual values.

Lance’s indexing extensions cover several access patterns:

  • Embedding indexes: She describes billion-scale vector search directly from S3.
  • Scalar indexes: Accelerate filtering on metadata columns.
  • Full-text indexes: Support keyword or fuzzy search over an S3 dataset.

For these search workloads, She proposes that a separate Elasticsearch cluster can become unnecessary. The broader goal is a single table that supports several kinds of access, rather than a separate data copy for each task.

Data and operationInterface described in the talk
SQL over metadata or time-series columnsDuckDB, Trino, or Spark
Training on blobs and tensorsPyTorch
Similarity search over embeddingsEmbedded vector index

These interfaces connect exploration, retrieval, and training to the same underlying table. A researcher can analyze the dataset, search for relevant examples, and use its stored payloads for fine-tuning or training without making each activity a separate storage system.

12:1212:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:12 · section reference included

Keep the research loop fast as requirements grow

LanceDB is the database layer built around the format. She claims distributed search across billions of vectors with low latency, high QPS, and an order of magnitude less infrastructure than other vector databases. That is a vendor performance claim without a specified comparison setup in the talk, rather than a measured result readers can apply to their own workload.

The intended scope of multimodal infrastructure is broader than image or video generation. She distinguishes three dimensions:

  • Data: Features, audio waveforms, images, and vectors—including text and image embeddings.
  • Workloads: OLAP SQL, vector search, full-text search, filtering, and data-frame operations.
  • Scenarios: Production RAG, search and retrieval, personalization, model training, and data-lake analysis or exploration.

The same format is intended to connect those dimensions, allowing data to serve operational applications as well as research and training.

Shpak closes by making research speed the strategic priority. His team’s experience is that tools slow down under load and under new multimodal requirements. The platform’s job is to sustain the loop: understand the data, change it, train, evaluate, and try again. Storage choices matter because they determine how readily researchers can keep doing that as the workload grows.

14:0514:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:05 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…