Contents
  1. Fitness is a contract
    1. Fit for which use?
      1. The same collection, different contracts
      2. Requirements, not a score
    2. Turning points in data curation
  2. Know what entered the dataset
    1. Origins, times and permitted use
      1. Times answer different questions
      2. Eligibility is separate from origin
    2. Population, frame and sample
      1. Selection serves different purposes
      2. A small coverage audit
    3. From summaries back to records
      1. Three complementary inspections
      2. A statistic is a pointer, not a verdict
  3. Preserve meaning
    1. Cleaning is semantic
      1. Unknown is not zero
      2. Filtering selects a population
    2. Duplicates depend on identity
      1. Similarity proposes; identity decides
      2. From exact copies to candidates
    3. Separate related information
      1. Match separation to the claim
  4. Labels are claims
    1. Specify the label
      1. A label specification
      2. Convenient targets can answer the wrong question
    2. Disagreement is diagnostic
      1. Why 90% can mean little
      2. Resolve the cause, not only the vote
    3. Automate without surrendering review
      1. One record, several sources
      2. Suggestions alter reviewer behavior
      3. Selection has blind spots
  5. Release and maintain evidence
    1. A release is a dependency graph
      1. Recipe and state
    2. Data checks are not system success
      1. Two evidence contracts
      2. Controlled comparison
    3. Maintain fitness through change
      1. Compare releases explicitly
      2. Detection is not propagation
      3. Close the loop carefully
  6. Check understanding
  7. Open questions
  8. Selected talks
  9. References
  10. Talk library
← All topics

Data quality & curation

Data quality is not a property a dataset possesses in isolation. It is a relationship between evidence and a use. A collection may be excellent reference material for answering current policy questions yet unsuitable for training because its permissions forbid that use. A visually diverse image corpus may suit a creative generator while failing a product-catalog task that demands accurate identifiers. Even a perfectly formatted dataset can be stale, unrepresentative, mislabeled, or contaminated by information that should have remained held out.

Data curation is the engineering work of deciding what enters a dataset, what each record means, which transformations are permissible, and how those decisions remain inspectable as sources change. The work begins with a contract: the task, operating conditions, unit represented by each record, and consequences of missing or misleading information. It continues through collection, inspection, cleaning, deduplication, splitting, annotation, release, and maintenance. Passing those data checks establishes that a release meets its stated requirements. Only independent evaluation can establish whether a system using it becomes more useful.

Part I — Fitness is a contract

Fit for which use?

A dataset is a collection of records selected for a purpose. A record might represent a document, transaction, image-caption pair, patient encounter, conversation, or completed task. The unit matters because rows are not automatically independent observations: ten messages from one conversation differ from ten conversations, and ten revisions of one document differ from ten independently written documents.

Start by writing the use contract. State the decision or behavior the data supports, the conditions in which that behavior will run, the unit of observation, and the cost of omissions or misleading values. Also state the dataset’s role. Training examples influence learned parameters; reference material is consulted at application time; reserved examples test a choice without participating in its construction. A collection suitable for one role does not inherit suitability for another.

The same collection, different contracts

UseOne record representsConsequential requirementPossible failure
Creative image trainingAn image paired with descriptive supervisionStylistic and concept coverageA conventional aesthetic filter removes intentionally low-fidelity styles
Current policy retrievalA versioned policy documentAuthority, permission, and freshnessA correct but superseded document supports a stale answer
Held-out system checkAn independently reserved task and reference judgmentSeparation from fitting and selectionA near-duplicate in training makes apparent generalization ambiguous

Requirements, not a score

This contract turns “quality” into separate requirements. Validity asks whether a value obeys its representation rules. Accuracy asks whether it correctly describes the intended thing. Completeness asks whether required values or cases are present. Consistency asks whether related representations agree under declared rules. Relevance asks whether the information bears on the task. Timeliness asks whether it is current enough at the decision point. A valid date can be inaccurate; a complete record can be irrelevant; an accurate fact can be too old.

Some requirements apply to each record, such as a recognized unit or a nonempty identifier. Others apply to the collection, such as coverage of a rare but consequential condition. Write explicit accept, review, quarantine, and exclude rules where the task supports them. Do not collapse the rules into a single quality score: an average can hide a fatal weakness in the one dimension the task depends on. Dataset size is a resource, not a substitute for correctness, coverage, or freshness.

Turning points in data curation

Data curation did not begin with foundation models. Its responsibilities accumulated through several technical traditions. In 1934, Jerzy Neyman contrasted probability sampling with purposive selection that merely looked representative, showing why the selection procedure matters to assessable uncertainty. In 1969, Ivan Fellegi and Alan Sunter formalized record linkage decisions among links, non-links, and unresolved pairs, separating similarity between records from identity of the thing represented.

As web collections grew, Andrei Broder’s 1997 work used compact sketches to make near-document comparison practical at crawl scale while warning that resemblance is not transitive. The March 2018 Datasheets for Datasets proposal made purpose, composition, collection, processing, intended uses, distribution, and maintenance explicit release concerns. DataComp-LM in 2024 then framed large-model data selection as a controlled experiment: hold the training recipe fixed, vary filters or mixtures, and compare the resulting systems. These contributions address complementary problems; later work did not make sampling, identity, or documentation obsolete.

Turning points in data curation

  1. 1934Neyman sampling frameworkMade the selection procedure—not a sample’s plausible appearance—the basis for assessable uncertainty.Sources & context

    Contributors: Jerzy Neyman

    What changed: Neyman contrasted stratified random sampling with purposive selection designed to resemble known population characteristics. His analysis showed that matching selected averages need not preserve other quantities or distributions, establishing why curation must document how units became eligible and selected.

  2. December 1969Fellegi–Sunter record linkageSeparated similarity evidence from decisions that records identify the same entity, event, or object.Sources & context

    Contributors: Ivan Fellegi and Alan Sunter, Dominion Bureau of Statistics

    What changed: The framework distinguishes links, non-links, and unresolved pairs requiring review. It explains why similar attributes can belong to different entities and differing records can still describe one entity.

  3. 1997Broder document resemblanceUsed compact sketches to make near-document candidate comparison practical at web-crawl scale.Sources & context

    Contributors: Andrei Broder, Digital Systems Research Center

    What changed: The work applied shingling and compact document sketches to an AltaVista crawl. It also emphasized that resemblance is not transitive: adjacent revisions can be similar even when distant endpoints differ materially.

  4. March 23, 2018Datasheets for DatasetsMade purpose, composition, processing, intended uses, distribution, and maintenance explicit release concerns.Sources & context

    Contributors: Datasheets for Datasets working-draft authors

    What changed: The working draft proposed dataset documentation modeled on electronic-component datasheets. It framed construction, appropriate uses, and limitations as information that creators should communicate to downstream users.

  5. 2024DataComp-LMHeld the training recipe fixed while varying filters and mixtures to test curation choices.Sources & context

    Contributors: DataComp-LM team

    What changed: DataComp-LM treated dataset selection as a controlled engineering comparison. Fixing a scale-specific training recipe made downstream differences more attributable to filters or mixtures instead of simultaneous changes in data, architecture, and compute.

Notice the progression from assessable selection to identity decisions, scalable resemblance, dataset documentation, and controlled curation comparison. Dates are chronological; spacing is not to scale and does not imply causal influence.

Part II — Know what entered the dataset

Origins, times and permitted use

Provenance records where information came from and the circumstances under which it was obtained. Useful provenance identifies the source, collection mechanism, source version, original context, and relevant times. A measurement made directly by an instrument, an event emitted by an application, a record contributed by a user, and a document acquired from another organization have different observation processes and therefore different omissions and assumptions.

Times answer different questions

FieldQuestion answeredWhat it does not prove
Observation or event timeWhen did the described event occur?That the record arrived then
Collection or processing timeWhen did the pipeline handle the record?That the event occurred then
Validity intervalWhen was the assertion considered applicable?That it remains true outside that interval
Source versionWhich particular source state supplied the record?That a changing “latest” resource still has that state

Operational logs are evidence of what instrumentation recorded, not a complete account of reality. A missing event may mean the event did not happen, the producer failed, the collector was unavailable, or a permission boundary hid it. Preserve absence reasons where the domain supplies them rather than reducing every missing field to the same blank value.

Eligibility is separate from origin

Traceability does not establish permission. A source can be accessible yet restricted from training, redistribution, or a new commercial purpose. Attach relevant permissions, prohibitions, sensitivity classifications, retention limits, and customer boundaries to the source or record, then consult them during selection. The Privacy and Data Governance chapter develops the policy, access-control, deletion, and accountability mechanisms; curation owns the narrower question of whether a record is eligible for this dataset release.

Population, frame and sample

The target population is the set of situations the intended claim concerns. The collection frame is the set the collection process can reach. The retained sample is what the pipeline selects from that frame. The frame may omit target units and include units outside the target; it may also contain duplicates or misclassified entries. Therefore even a random sample from the frame cannot justify claims about target situations that the frame never represented.

Sampling starts from the collection frame

Coverage can fail before sampling: the frame may omit target cases and admit cases outside the target population.

Target, frame, and retained sample

Partially overlapping boundaries expose undercoverage and erroneous inclusions before retained units are selected.

02.557.51002468Conceptual extent (conceptual units)Conceptual extent (conceptual units)Target populationCollection frameRetained sampleTarget populationCollection frameUnreachable target casesOut-of-target inclusionsRetained sample
  • 1. Target population
  • 2. Collection frame
  • 3. Retained sample
Read coordinates and regions as data

X: 0–10 conceptual units; Y: 0–8 conceptual units, increasing up. Equal scale on both axes.

Target population (polygon)

(0.8, 2); (1.2, 5.8); (2.5, 7); (5.8, 6.8); (7, 5); (6.6, 1.5); (4.8, 0.7); (2, 0.9)

Collection frame (polygon)

(3.2, 1.3); (3.5, 5.2); (5, 6.3); (8.7, 6); (9.4, 4.2); (8.7, 1.2); (7.1, 0.6); (4.8, 0.7)

Retained sample (points)

(4.2, 2.2); (5.1, 4.8); (5.8, 2.9); (6.5, 5.1); (7.2, 3.6); (8.1, 2.1)

Target population: (1.5, 6.2)

Collection frame: (8.9, 6.4)

Unreachable target cases: (1.4, 3.6)

Out-of-target inclusions: (8.9, 3.7)

Retained sample: (6.2, 2.4)

The retained marks are selected from the collection frame, not directly from the target population. Reweighting can adjust known selection probabilities but cannot create observations in the unreachable target-only region.

Selection serves different purposes

Probability sampling gives eligible units calculable, nonzero selection probabilities. Stratified sampling partitions the frame into mutually exclusive groups and samples within them, which can protect estimates or inspection for important subgroups. Deliberately oversampling rare failures can be useful for diagnosis or learning, but the resulting sample proportions no longer describe prevalence. Preserve the original group counts, selection rule, and inclusion probabilities when known; population estimates may require design weights.

A small coverage audit

ConditionTarget countReachable countRetained countInterpretation
Standard desktop flow700690140Broadly reachable; sampled down
Mobile with accessibility settings2008040Most target cases never entered the frame
Expired-session recovery1002040Rare reachable cases were deliberately oversampled
Internal test accounts06020Erroneous frame inclusions

The counts are invented to expose the relationships, not measured results. They show why plausible sample proportions are insufficient: the mobile gap began before sampling, while recovery cases are enriched by design. More desktop records cannot repair the missing mobile support, and reweighting cannot create observations for conditions with no collected examples. Coverage should also be checked across consequential combinations, not only one attribute at a time. Generated environment combinations can broaden coverage only after each combination is validated as a coherent task.

From summaries back to records

Profiling connects collection-level summaries to actual records. Start with structural checks—types, permitted values, key uniqueness, and referential integrity—but do not confuse conformance with truth. Add counts with explicit denominators, distributions, missingness patterns, and comparisons across relevant slices, meaning subsets that share a condition important to the intended use. Automatically inferred schemas and drift thresholds require review because they can reproduce defects or flag legitimate imbalance.

Three complementary inspections

  • Random inspection — Supports broad defect estimates only under its sampling and recognition assumptions. Even if the true defect fraction were 1%, a random sample of 100 would still have about a 37% chance of finding none.
  • Slice inspection — Examines conditions where aggregate summaries can hide concentrated omissions or errors.
  • Targeted inspection — Follows anomaly, disagreement, or change signals to suspicious records. It is useful for diagnosis but does not estimate population prevalence without accounting for selection.

A statistic is a pointer, not a verdict

SignalRecords to inspectPossible explanations
Missing-value rate risesRecords by source, time, and absence reasonCollector outage, newly optional field, privacy masking, or real nonresponse
Stable overall averageContributing slices and changed recordsOpposing subgroup changes cancel in aggregate
Rare extreme valuesOriginal records, units, and provenanceValid rare observation, unit mismatch, parser error, or duplicate event

An anomaly score should create a review queue, not an automatic deletion rule. The same extreme value can be a corrupt unit conversion or the only example of a consequential rare condition. Inspection returns to the source context because that is where the distinction can be justified.

Part III — Preserve meaning

Cleaning is semantic

Cleaning operations differ in what they assert. Parsing converts a representation into fields. Normalization chooses a common representation. Unit conversion changes a scale under a declared equivalence. Correction replaces a value judged wrong. Imputation supplies a value for missing or invalid data under assumptions. Filtering removes a record from a particular collection. Quarantine retains it outside accepted output for review or reprocessing. These are not interchangeable formatting conveniences.

Unknown is not zero

Suppose a source field says a refund amount is unknown. Converting a known “USD 12.00” string into a decimal amount plus currency preserves the observation when the parser is correct. Converting unknown into 0 asserts that no refund occurred, a claim the source never supplied. Quarantining the unresolved record preserves uncertainty while preventing a consumer that requires a numeric amount from silently treating it as accepted data.

One unknown value, three curation decisions

Example

Alternative derivations can preserve the observation, add an unsupported claim, or defer acceptance without changing the source record.

Refund amount: unknownThe stable source establishes that no amount is known; it does not establish a zero refund.
Typed unknownStore the applicable absence reason and retain the source link.
Asserted zeroA numeric zero invents the substantive claim that no refund occurred.
QuarantineRetain the record for review outside output that requires an accepted numeric amount.
Three alternatives branch from the same refund record. A typed absence reason preserves what was observed; zero imputation adds a claim; quarantine retains the unresolved record outside numeric-only accepted output.
Read the diagram as text
  • Refund amount: unknown. The stable source establishes that no amount is known; it does not establish a zero refund.
  • Typed unknown. Store the applicable absence reason and retain the source link.
  • Asserted zero. A numeric zero invents the substantive claim that no refund occurred.
  • Quarantine. Retain the record for review outside output that requires an accepted numeric amount.
  • Refund amount: unknown → Typed unknown: preserves absence reason.
  • Refund amount: unknown → Asserted zero: adds unsupported value.
  • Refund amount: unknown → Quarantine: defers acceptance.

Keep the raw input where handling rules permit, store the derived representation separately, and record the transformation, configuration, and affected fields. Compatibility normalization in Unicode illustrates why “standardization” is task-dependent: it can fold distinctions that matter to meaning. Likewise, an identity normalization such as treating West Germany and Germany as one entity changes historical counting semantics and must be an explicit policy rather than an invisible string replacement.

Filtering selects a population

Filters also change composition. Studies of C4’s blocklist found benign scientific, medical, identity-related, and dialect-associated text among rejected records. For creative image generation, a conventional aesthetic filter can remove intentionally blurred or low-resolution styles. Inspect rejected records and report retention by relevant slice rather than examining only what survived. For extraction from pages and preservation of evidence locations, see Locate supporting source material.

Duplicates depend on identity

Deduplication identifies and handles repeated content or observations under a declared identity rule. Byte-identical copies, copies made identical by normalization, overlapping passages, later revisions, records about the same entity, and repeated observations of the same process are different cases. Removing one may reduce accidental repetition; removing another may erase change, frequency, or independent evidence.

Similarity proposes; identity decides

PairDecisive distinctionTypical decision
Identical payload and event identityRetransmission of one eventRemove or make processing idempotent; retain frequency metadata if operationally useful
Same text after harmless formatting normalizationEquivalent document representation under the task’s ruleGroup and retain one representation plus provenance
Mostly identical text with a changed date, number, or negationLater revision may carry different evidenceRetain both versions and their validity relationship
Similar records from separate eventsRepeated observations are distinct evidenceRetain separately; group only for analysis if justified

From exact copies to candidates

Cryptographic content hashes efficiently generate candidates for byte-identical data; compare the content when identity matters. A normalized hash finds equality only after the chosen normalization. Neither match establishes that two records represent the same event. For near duplicates, split text into contiguous fragments called shingles and represent each document as a set. Their Jaccard resemblance is J(A,B)=∣A∩B∣/∣A∪B∣J(A,B)=|A\cap B|/|A\cup B|. MinHash provides a compact estimate of this overlap, making candidate generation practical at scale. It does not verify semantic equivalence or shared event identity.

A safe pipeline generates candidates with methods appropriate to the comparison, verifies them under the identity rule, forms groups, and chooses representatives while retaining provenance and frequency information. At image scale, Krea reported using byte-oriented MD5 and perceptual pHash before more expensive embedding-based methods. These signals propose duplicate candidates; they do not establish event identity. Pairwise resemblance is not transitive: successive revisions may each resemble the next while the endpoints differ materially, so a connected cluster is not proof that every member is interchangeable.

A resemblance chain is not an identity decision

Example

Adjacent revisions can qualify as near-duplicate candidates even though the endpoints differ in a consequential field.

Revision ACancellation deadline: June 1. Candidate resemblance to B: 0.91.
Revision BCancellation deadline: June 15. Candidate resemblance to A: 0.91; to C: 0.90.
Revision CCancellation deadline: July 1. Candidate resemblance to B: 0.90; endpoint resemblance to A: 0.74.
Identity-rule reviewInspect version, event, validity, and material changes before grouping or choosing representatives.
A–B and B–C exceed the example candidate threshold, so both pairs receive review. A and C do not qualify as a candidate pair, and the visible deadline change means the connected component cannot justify one interchangeable representative.
Read the diagram as text
  • Revision A. Cancellation deadline: June 1. Candidate resemblance to B: 0.91.
  • Revision B. Cancellation deadline: June 15. Candidate resemblance to A: 0.91; to C: 0.90.
  • Revision C. Cancellation deadline: July 1. Candidate resemblance to B: 0.90; endpoint resemblance to A: 0.74.
  • Identity-rule review. Inspect version, event, validity, and material changes before grouping or choosing representatives.
  • Revision A → Revision B: candidate: 0.91 ≥ 0.85.
  • Revision B → Revision C: candidate: 0.90 ≥ 0.85.
  • Revision A → Identity-rule review: review candidate set.
  • Revision B → Identity-rule review: review candidate set.
  • Revision C → Identity-rule review: review candidate set.

Separate related information

Training data is used to fit behavior. Development data informs choices such as features, prompts, thresholds, or configurations. A held-out test set assesses the frozen choice. Leakage occurs when information crosses a boundary the assessment intends to protect. Repeated inspection and tuning against a test set turns it into development data, regardless of its filename.

Match separation to the claim

Random row assignment is insufficient when rows share a person, source document, conversation, event, or revision family. Group related records before assignment if the intended claim concerns unseen groups. Use a temporal boundary when the intended use concerns future conditions, and include only features available at the decision time. A patient-disjoint split asks about unseen people; a later-period split asks about change over time; an external-site split asks about transport to another setting. These are different claims and can be combined.

Preprocessing can leak too. Fit statistics, imputation rules, feature selection, and learned normalization using training data alone, then apply the fitted transformation to development and test records. Even computing a normalization average over all rows exposes held-out information. Deterministic record validation against a predeclared rule is different from estimating a transformation from protected data.

Group before assigning dataset roles

Example

The protected boundary concerns shared information, so related records must be grouped before split assignment and preprocessing parameters must flow from training only.

Versioned source recordsRows include stable IDs, entity or document-family keys, source versions, and event times.
Related-record groupsExact copies, revisions, conversations, or shared entities are grouped under the intended independence rule.
Split assignmentGroups are assigned to training, development, or held-out test roles.
Training recordsUsed to fit model behavior and data-dependent preprocessing.
Development recordsUsed for selection without fitting preprocessing statistics.
Held-out recordsUsed after choices are frozen.
Fitted preprocessing stateStatistics and learned transformations estimated from training records only.
Data edges carry records; control edges carry grouping, assignment, and fitted parameters. Held-out records never supply fitted preprocessing state.
Read the diagram as text
  • Versioned source records. Rows include stable IDs, entity or document-family keys, source versions, and event times.
  • Related-record groups. Exact copies, revisions, conversations, or shared entities are grouped under the intended independence rule.
  • Split assignment. Groups are assigned to training, development, or held-out test roles.
  • Training records. Used to fit model behavior and data-dependent preprocessing.
  • Development records. Used for selection without fitting preprocessing statistics.
  • Held-out records. Used after choices are frozen.
  • Fitted preprocessing state. Statistics and learned transformations estimated from training records only.
  • Versioned source records → Related-record groups: data: records.
  • Related-record groups → Split assignment: control: keep groups intact.
  • Split assignment → Training records: data: assigned groups.
  • Split assignment → Development records: data: assigned groups.
  • Split assignment → Held-out records: data: assigned groups.
  • Training records → Fitted preprocessing state: data: fit parameters.
  • Fitted preprocessing state → Development records: control: apply only.
  • Fitted preprocessing state → Held-out records: control: apply only.

Record stable IDs, grouping keys, source versions, split assignments, and the code or configuration that produced them. Then run exact and near-duplicate checks across the resulting boundaries. Historical coding-agent tasks illustrate a broader contamination problem: checking out an old commit did not isolate the task while future Git history and public issue discussions still exposed the solution. The protected boundary must include every equivalent information path, not merely one file or tool.

Part IV — Labels are claims

Specify the label

A label is an assigned target value or judgment; annotation is the process that produces it. A usable specification defines the unit being labeled, context annotators may inspect, allowed outputs, category meanings, decision rules, ambiguity handling, abstention, and whether several answers may be valid. A familiar word such as “fraud,” “resolved,” or “unsafe” is not a specification.

A label specification

QuestionRequired decision
What is the unit?For example, one login sequence rather than each individual attempt
What context is permitted?Whether later login success, multifactor authentication, and account history are visible
What does the label assert?Observed successful access, policy-defined notification need, or an analyst’s interpretation
What if evidence is incomplete?Abstain, mark unresolved, or request additional context rather than inventing a negative
When does the outcome mature?A deadline after which absence of an observed event can support the defined negative

In a password-spray example, many failed attempts and an eventual successful login begin with similar evidence but may receive different policy labels after the outcome is known. The additional feature resolves a specific ambiguity; it does not make every unsuccessful attempt harmless under every security policy. Pilot instructions on a small, varied batch, inspect disputes, and revise rules before scaling annotation.

Convenient targets can answer the wrong question

Observed outcomes, interpretations, and proxies must remain distinct. A ticket’s closed status records a workflow transition, not necessarily customer-confirmed resolution. A healthcare allocation model can accurately predict spending while spending remains a biased proxy for health need. Likewise, absence from a curated positive database is not a negative label: the unlabeled pool can contain undiscovered positives and true negatives. For coordinate transforms and spatial targets, see Preserving annotation meaning.

Disagreement is diagnostic

Independent annotation measures how consistently people apply a specification without seeing one another’s decisions. It does not create an infallible truth standard. Annotators can share the same misunderstanding, and subjective tasks can contain stable differences in perspective. Preserve individual judgments where feasible so aggregation does not destroy the evidence needed to diagnose disagreement.

Why 90% can mean little

Raw agreement is sensitive to category prevalence. If two independent random labelers each choose one category 95% of the time, their expected agreement is 0.952+0.052=0.9050.95^2+0.05^2=0.905, or 90.5%, despite having no item-specific ability. Chance-corrected measures adjust using assumptions about category frequencies and annotator behavior; they do not supply a universal acceptance threshold or validate the category definitions.

Resolve the cause, not only the vote

PatternLikely investigationPossible response
One annotator repeatedly violates a clear ruleTraining and error reviewCorrect affected labels and retrain the annotator
Disagreement concentrates in one categoryDefinition and boundary examplesRevise guidance; reconsider earlier batches
Decisions change when context is revealedContext availabilityExpose required evidence or allow abstention
Consistent perspectives remain after clarificationTask subjectivityRetain distributions or multiple labels where the use permits

Adjudication is an explicit process for resolving disputed cases into a reference annotation when the task requires one. It should record the decision and rationale, not erase the underlying disagreement. The reference remains fallible: automated candidate detection followed by human review of ImageNet labels found both majority-supported replacements and valid unusual examples among automated false alarms. Corrections should update affected records, examples, and instructions.

Automate without surrendering review

Weak supervision uses cheaper, imperfect sources such as rules, knowledge bases, or classifiers. In Snorkel’s formulation, a labeling function can emit a label or abstain, and overlapping, conflicting outputs are combined into probabilistic labels. Dependencies matter: two rules derived from the same keyword list are not two independent confirmations. A small labeled development set and a blind held-out test set remain part of the documented workflow.

One record, several sources

SourceOutputDependency or limitation
Rule A: keyword listPositiveShares the list with Rule B
Rule B: derived regexPositiveCorrelated with Rule A; not independent support
External databaseAbstainNo matching entry; absence is not negative
Model suggestionPositiveConfidence does not establish correctness
Independent reviewed judgmentUnresolvedRequired context is missing

Suggestions alter reviewer behavior

Model-assisted annotation can save work, but the interface changes the measurement process. In a Czech dependency-annotation study, annotation from scratch took almost 1.7 times as long as correcting strong parser suggestions, with broadly similar accuracy against adjudicated references in that task. Other experiments show that added effort for rejecting a suggestion can increase acceptance of incorrect suggestions. Assistance therefore needs independently assessed quality, not only throughput or reviewer agreement.

A Duolingo study inserted fabricated cheating alerts into previously certified exam sessions. Under the original guidance, reviewers rejected only half of those false suggestions; after guidance emphasized corroborating video evidence, rejection rose to 71% in a later study with different sessions. The result shows both the risk of automation bias and the value of testing whether reviewers resist known-bad suggestions. It does not isolate the guideline change as the sole cause.

Selection has blind spots

Active learning selects examples expected to improve a learner if labeled. Uncertainty sampling was studied for text classification in 1994, but uncertainty alone can miss confidently wrong regions: a 2008 hierarchical-sampling example shows a small cluster omitted by the initial sample while subsequent queries remain near the current decision boundary. Combine targeted review with random spot checks and coverage-based inspection. Agreement, confidence, and selection priority are signals for allocating review—not substitutes for correctness.

Part V — Release and maintain evidence

A release is a dependency graph

Data lineage connects source entities, transformation activities, and derived artifacts. Provenance answers where a record came from; lineage additionally shows what used it and what it generated. A reproducible dataset release needs more than a filename and script: it needs an immutable release identity, stable record identifiers, source versions or snapshots where permitted, content fingerprints, a membership manifest, processing code and configuration, annotation-guideline versions, grouping keys, and split assignments.

Recipe and state

A manifest records membership explicitly. Apache Iceberg, for example, represents a table state as a snapshot whose manifest list identifies immutable manifests and their files; readers retain the selected snapshot until refresh. DVC separates a declared pipeline recipe from recorded state such as commands, dependencies, outputs, hashes, and parameters. These mechanisms illustrate two complementary facts: what should run and what content actually participated in a particular result.

Rerunning the same query against a changing source is not reproduction. The source may add, remove, or revise records while the query text stays unchanged. Record the resolved source state and selection result. If retention rules forbid preserving raw material, retain the permissible identifiers, fingerprints, transformation records, and handling metadata—but do not claim those records can reconstruct unavailable content.

Lineage makes correction selective. If a source revision changes one record, trace it through cleaned derivatives, duplicate groups, annotations, split membership, released datasets, and downstream artifacts. Entity merges must preserve both source sets; a fact invalidated by new evidence should retain the evidence responsible for the change. Documentation such as a datasheet records purpose, composition, processing, intended uses, limitations, distribution, and maintenance so consumers can interpret the graph rather than merely reproduce its bytes.

A correction propagates through recorded dependencies

Example

Versioned lineage identifies the artifacts that require refresh while leaving unrelated artifacts untouched.

Source record S1The original version with stable source identity and content fingerprint.
Cleaned artifact C1Derived under recorded code and configuration.
Annotation A1Produced under guideline version G1.
Release R1 manifestRecords membership, grouping, split assignment, and dependencies.
Dependent index I1Built from release R1 and requires refresh after the correction.
Unrelated artifact U1Has no dependency on source S1 and remains unchanged.
Corrected source S2A new source version; S1 remains part of historical lineage where retention permits.
Release R2 manifestContains corrected derived membership and supersedes R1 for current permitted use.
Each node is an immutable entity version or activity output. Derivation edges show why corrected source version S2 creates release R2 and invalidates only downstream artifacts that depended on S1.
Read the diagram as text
  • Source record S1. The original version with stable source identity and content fingerprint.
  • Cleaned artifact C1. Derived under recorded code and configuration.
  • Annotation A1. Produced under guideline version G1.
  • Release R1 manifest. Records membership, grouping, split assignment, and dependencies.
  • Dependent index I1. Built from release R1 and requires refresh after the correction.
  • Unrelated artifact U1. Has no dependency on source S1 and remains unchanged.
  • Corrected source S2. A new source version; S1 remains part of historical lineage where retention permits.
  • Release R2 manifest. Contains corrected derived membership and supersedes R1 for current permitted use.
  • Source record S1 → Cleaned artifact C1: derived by cleaning.
  • Cleaned artifact C1 → Annotation A1: annotated under G1.
  • Annotation A1 → Release R1 manifest: included by manifest.
  • Release R1 manifest → Dependent index I1: data dependency.
  • Corrected source S2 → Release R2 manifest: corrected derivation.
  • Release R1 manifest → Release R2 manifest: superseded by.
  • Release R2 manifest → Dependent index I1: requires refresh.

Data checks are not system success

Dataset acceptance and system usefulness are different claims. A release can pass schema, provenance, permission, coverage, separation, and label-review checks yet fail to improve a model or application. Conversely, an apparent system gain can arise from leakage, a changed model, more compute, or a different evaluation population rather than the curation change.

Two evidence contracts

ClaimEvidence requiredStill unresolved
Dataset meets its release requirementsManifest, validation results, coverage report, overlap checks, label audit, and handling reviewWhether a system using it performs better
A filter agrees with its rubricIndependent labels and a held-out confusion analysisWhether selected data improves downstream tasks
Dataset version B improves the intended systemComparable system conditions and independent reserved examples across relevant slicesTransfer to untested conditions and future drift

Controlled comparison

DataComp-LM addressed attribution by fixing a scale-specific training recipe, varying filters or mixtures, and measuring downstream models. FineWeb-Edu likewise separated filter agreement with model-generated educational labels from the later question of whether retained text improved tested capabilities; the chosen threshold also changed topic composition. A filter’s score is not an intrinsic property of text, and its preferred threshold is conditional on the task, rubric, corpus, training setup, and evaluation.

Preserve the exact dataset identity, change description, system configuration, and reserved examples used in the comparison. The Evals and Benchmarks chapter owns metrics, uncertainty, experimental design, contamination analysis, and release decisions. Curation’s responsibility is to make the data intervention identifiable and independently testable.

Maintain fitness through change

Maintenance begins by classifying the change. A source outage changes availability. A schema revision changes representation. A corrected record changes content. A withdrawal changes permitted membership. A shifting population changes composition. A revised category changes label meaning. These events require different responses. Drift simply means observed data properties changed; it is a signal to investigate, not proof of degradation.

Compare releases explicitly

  • Membership — Additions and removals, with reasons and source versions.
  • Content — Changed records and fields, including before/after lineage where permitted.
  • Coverage — Counts and proportions by important source, condition, and slice.
  • Labels — Corrections, taxonomy changes, guideline version, and adjudication rationale.
  • Dependencies — Downstream datasets, indexes, evaluations, and models requiring refresh or review.

Detection is not propagation

Change feeds can support incremental maintenance by exposing inserts, deletes, and before/after updates with commit versions. They are not permanent history: retained versions can expire, and starting from an unavailable version can fail. Archive or checkpoint permissible state according to the release contract, and verify that downstream consumers actually applied the change rather than assuming event publication equals completion.

Reproducibility does not override current permission. The Stack’s removal process recorded requests and code pointers to prevent reinclusion, and required downloaders to use the most recent allowed version. An older reproducible release can therefore be an impermissible rollback target. Removing a row also does not remove its influence from an already trained model; identify affected model artifacts and hand remediation to the appropriate lifecycle owner.

Close the loop carefully

Assign an owner for source health, corrections, release decisions, and consumer notification. Prioritize new collection, repair, relabeling, or retirement by expected task impact and effort. Treat production feedback as another selected source: interfaces shape which users respond and which decisions are logged. Rubber-stamped model approvals can become misleading labels, so validate feedback against independent outcomes where possible. Maintenance preserves fitness only when it reconnects changed data to the original use contract.

Open questions

  1. How can coverage be measured when the target population is only partially observable? Practical frames omit people, events, and languages before sampling begins. Progress would require defensible proxies for missing support, explicit uncertainty about unreachable groups, and validation that collection interventions actually reach them.

  2. How should curators preserve legitimate disagreement without making downstream interfaces unusably ambiguous? Majority labels erase perspectives, while distributions or multiple references complicate training and evaluation. Progress would connect task subjectivity, annotator context, and decision costs to an explicit representation of uncertainty.

  3. How can automated curation discover confidently shared blind spots? Active learning and model-assisted review often prioritize uncertainty or disagreement, but correlated systems may confidently repeat one omission. Progress would combine independence-aware label sources, random and coverage-based audits, and outcome evidence that reveals errors outside the current model’s boundary.

  4. What is the right correction contract for derived datasets and trained models? Lineage can identify affected records and artifacts, but source deletion does not generally remove learned influence, and retention rules can forbid rollback. Progress would couple machine-readable dependencies with verified consumer updates, permitted historical state, and scoped remediation evidence.

  5. How should dataset fitness be maintained when intended use changes faster than labels mature? Operational and safety outcomes can arrive late, while input drift and source failures are observable immediately. Progress would keep pending outcomes explicitly unresolved, separate early operational signals from outcome evidence, and define review triggers without silently treating missing labels as success.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

54 matching talks

Every catalogued talk on this subject: Fine-tuning →

TalkSpeakerEventYear
Sandipan BhaumikAI Engineer Europe 20262026
Yuval Belfer, Niv GranotAI Engineer World's Fair 20252025
Jerry LiuAI Engineer World's Fair 20252025
Varsha ShahAI Engineer World's Fair 20262026
Diane LinAI Engineer World's Fair 20262026
Anant ShankhdharAI Engineer World's Fair 20262026
Kobie CrawfordAI Engineer Europe 20262026
Nathaniel Whittemore (NLW)AI Engineer Code 20252025
Vinesh GudlaAI Engineer World's Fair 20252025
Nathan SoboAI Engineer World's Fair 20252025
Louis-François Bouchard, Paul Iusztin, Samridhi VaidAI Engineer Europe 20262026
Ibragim BadertdinovAI Engineer Europe 20262026
Vincent KocAI Engineer Europe 20262026
Muktesh MishraAI Engineer World's Fair 20252025
Calvin Qi, Chang SheAI Engineer World's Fair 20252025
Mohak SharmaAI Engineer Summit 20252025
Maxime LabonneAI Engineer World's Fair 20242024
Mahmoud MabroukAI Engineer Europe 20262026
Raj NavakotiAI Engineer Europe 20262026
Brandon WaselnukAI Engineer Europe 20262026
Rafael LeviAI Engineer Europe 20262026
Denys LinkovEvent unknownUnknown
Felippe Felisola Caso, Francesco GallettaEvent unknownUnknown
Anant SrivastavaAI Engineer Code 2025Unknown
Edo LibertyAI Engineer World's Fair 20262026
Jordi SprangerAI Engineer NYC 2026 Online Talks2026
George HeEvent unknownUnknown
Michael Manapat, Jack McDonaldEvent unknownUnknown
Gil FeigAI Engineer World's Fair 20262026
Fuad AliEvent unknownUnknown
Yohan RajuAI Engineer World's Fair 20262026
Jakub HojsanAI Engineer Code 2025Unknown
Tejas KumarAI Engineer World's Fair 20262026
Jeff Koss, Leo PlatzerAI Engineer World's Fair 20262026
Rafael LeviAI Engineer World's Fair 20262026
Palak Agarwal, Omar AlhaitAI Engineer World's Fair 20262026
Nina Žižakić, Eleonora DrykinaAI Engineer NYC 2026 Online Talks2026
Kamalakannan NandagopalEvent unknownUnknown
Peter WerryAI Engineer World's Fair 20262026
Karthic RaoAI Engineer NYC 2026 Online Talks2026
Andy Wong, Andrei Bocan, Shane WolfAI Engineer World's Fair 20262026
Sofia JakovcevicEvent unknownUnknown
Dhruv NathawaniAI Engineer World's Fair 20262026
Anders SwansonAI Engineer World's Fair 20262026
Nyah MacklinAI Engineer World's Fair 20262026
Paden GayleAI Engineer NYC 2026 Online Talks2026
Yoni Michael, Brandon CallenderAI Engineer World's Fair 20262026
Sam Green, Aaron LinskyEvent unknownUnknown
Rafal WilinskiAI Engineer World's Fair 20262026
Merve NoyanAI Engineer World's Fair 20262026
Nixon DinhAI Engineer World's Fair 20262026
Karthik Ranganathan, Heather DowningAI Engineer World's Fair 20262026
Hrishi OlickelAI Engineer NYC 2026 Online Talks2026
Micah SilvermanAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
26 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
33 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. The Government Data Quality Framework

    https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework

  2. Datasheets for Datasets

    https://arxiv.org/html/1803.09010v8

  3. Cross-validation and held-out evaluation

    https://scikit-learn.org/stable/modules/cross_validation.html

  4. Training Krea 2 - What matters in generative model training.

    https://www.youtube.com/watch?v=-tviRdpmHvs&t=360

  5. Krea 2 Technical Report

    https://www.krea.ai/blog/krea-2-technical-report

  6. PROV-DM: The PROV Data Model

    https://www.w3.org/TR/prov-dm/

  7. Deduplicating Training Data Makes Language Models Better

    https://aclanthology.org/2022.acl-long.577.pdf

  8. Beyond Accuracy: What Data Quality Means to Data Consumers

    https://courses.washington.edu/geog482/resource/14_Beyond_Accuracy.pdf

  9. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models

    https://arxiv.org/html/2406.11794v3

  10. On the Two Different Aspects of the Representative Method

    https://errorstatistics.com/wp-content/uploads/2019/01/neyman-1934.pdf

  11. A Theory for Record Linkage

    https://nhis.ipums.org/nhis/resources/Fellegi69.pdf

  12. On the Resemblance and Containment of Documents

    https://www.cs.princeton.edu/courses/archive/spring13/cos598C/broder97resemblance.pdf

  13. Datasheets for Datasets: March 2018 Working Draft

    https://ai.stanford.edu/~tgebru/papers/datasheets.pdf

  14. Apache Beam glossary

    https://beam.apache.org/documentation/glossary/

  15. HL7 Terminology: DataAbsentReason

    https://terminology.hl7.org/7.0.0/CodeSystem-data-absent-reason.html

  16. ODRL Information Model 2.2

    https://www.w3.org/TR/odrl-model/

  17. Scaling Enterprise-Grade RAG Systems: Lessons from the Legal Frontier

    https://www.youtube.com/watch?v=W1MiZChnkfA&t=186

  18. Statistics Canada Quality Guidelines: Coverage and frames

    https://www150.statcan.gc.ca/n1/pub/12-539-x/steps-etapes/4147788-eng.htm

  19. Statistics Canada: Survey Methods and Practices

    https://www150.statcan.gc.ca/n1/en/pub/12-587-x/12-587-x2003001-eng.pdf

  20. Computer Use at the Edge of the Statistical Precipice

    https://www.youtube.com/watch?v=CTLa_p6iOiY&t=264

  21. TensorFlow Data Validation: Checking and analyzing your data

    https://tensorflow.github.io/tfx/guide/tfdv/

  22. NIST: Choosing a Sampling Plan with a Given OC Curve

    https://itl.nist.gov/div898/handbook/pmc/section2/pmc222.htm

  23. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift

    https://arxiv.org/html/1810.11953

  24. Statistics Canada Quality Guidelines: Imputation

    https://www150.statcan.gc.ca/n1/pub/12-539-x/2009001/imputation-eng.htm

  25. Databricks: Expectation Recommendations and Advanced Patterns

    https://docs.databricks.com/aws/en/ldp/expectation-patterns

  26. Unicode Standard Annex #15: Unicode Normalization Forms

    https://www.unicode.org/reports/tr15/

  27. RAG Evaluation Is Broken! Here's Why (And How to Fix It)

    https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=523

  28. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

    https://arxiv.org/html/2104.08758v2

  29. Introduction to Information Retrieval: Near-duplicates and shingling

    https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html

  30. CloudEvents Specification v1.0.2

    https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md

  31. Training Krea 2 - What matters in generative model training.

    https://www.youtube.com/watch?v=-tviRdpmHvs&t=612

  32. TRIPOD+AI expanded checklist: timing, labels, and evaluation separation

    https://www.tripod-statement.org/wp-content/uploads/2024/04/TRIPODAI-Supplement.pdf

  33. scikit-learn: Common pitfalls and recommended practices

    https://scikit-learn.org/stable/common_pitfalls.html

  34. SWE-rebench: Lessons from Evaluating Coding Agents on Real Software Engineering Tasks — Ibragim Badertdinov, Nebius

    https://www.youtube.com/watch?v=wcUJWP6WpGM&t=679

  35. Analyzing Dataset Annotation Quality Management in the Wild

    https://aclanthology.org/anthology-files/pdf/cl/2024.cl-3.1.pdf

  36. Google: Supervised learning foundations

    https://developers.google.com/machine-learning/intro-to-ml/supervised

  37. Stripe card-dispute timing and lifecycle

    https://docs.stripe.com/disputes/how-disputes-work

  38. Why Your Agent Disagrees With Itself (And What To Do About It)

    https://www.youtube.com/watch?v=wEc9aG7cRQc&t=563

  39. Right censoring and unresolved event outcomes

    https://scikit-survival.readthedocs.io/en/stable/user_guide/00-introduction.html

  40. Zendesk: What is the difference between a solved ticket and a closed ticket?

    https://support.zendesk.com/hc/en-us/articles/4408887712154-What-is-the-difference-between-a-solved-ticket-and-a-closed-ticket

  41. Obermeyer et al.: Healthcare cost as a biased proxy for need

    https://www.ftc.gov/system/files/documents/public_events/1548288/privacycon-2020-ziad_obermeyer.pdf

  42. Learning Classifiers from Only Positive and Unlabeled Data

    https://cseweb.ucsd.edu/~elkan/posonly.pdf

  43. Inter-Coder Agreement for Computational Linguistics

    https://aclanthology.org/J08-4004.pdf

  44. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

    https://aclanthology.org/2022.tacl-1.6.pdf

  45. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

    https://arxiv.org/pdf/2103.14749

  46. Snorkel: Rapid Training Data Creation with Weak Supervision

    https://arxiv.org/html/1711.10160

  47. Quality and Efficiency of Manual Annotation: Pre-annotation Bias

    https://aclanthology.org/2022.lrec-1.312.pdf

  48. Bias in the Loop: How Humans Evaluate AI-Generated Suggestions

    https://hdsr.mitpress.mit.edu/pub/nrcn4h7d/release/2

  49. When Machines Mislead: Human Review of Erroneous AI Cheating Signals

    https://aclanthology.org/2025.aimecon-wip.11.pdf

  50. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    https://www.youtube.com/watch?v=CDqzWpwkSls&t=359

  51. A Sequential Algorithm for Training Text Classifiers

    https://arxiv.org/pdf/cmp-lg/9407020

  52. Hierarchical Sampling for Active Learning

    https://www.cs.columbia.edu/~djhsu/papers/hier.pdf

  53. Why Your Agent Disagrees With Itself (And What To Do About It)

    https://www.youtube.com/watch?v=wEc9aG7cRQc&t=659

  54. Apache Iceberg Table Specification

    https://iceberg.apache.org/spec/

  55. DVC: dvc.yaml Files

    https://doc.dvc.org/user-guide/project-structure/dvcyaml-files

  56. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    https://www.youtube.com/watch?v=H7puB0RwJMM&t=316

  57. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    https://arxiv.org/html/2406.17557v2

  58. scikit-learn confusion_matrix

    https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html

  59. FineWeb-Edu dataset card

    https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/blob/main/README.md

  60. Databricks: Use Change Data Feed

    https://docs.databricks.com/aws/en/tables/features/change-data-feed

  61. BigCode: The Stack Datasets and Removal Process

    https://www.bigcode-project.org/docs/about/the-stack/

  62. Machine Unlearning

    https://arxiv.org/html/1912.03817

  63. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    https://www.youtube.com/watch?v=CDqzWpwkSls&t=563

  64. Build Your First Demand-Driven Context Base: Let AI Agents Tell You What They Need

    https://www.youtube.com/watch?v=_QAVExf_1uw&t=3565

  65. Your Evals Are Meaningless (And Here’s How to Fix Them)

    https://www.youtube.com/watch?v=3jwClx0Ft2E&t=672

  66. Task Fidelity Scaling Laws — Kobie Crawford, Snorkel AI

    https://www.youtube.com/watch?v=YYH0DMQr30A&t=1146

  67. Why Your Agent Disagrees With Itself (And What To Do About It)

    https://www.youtube.com/watch?v=wEc9aG7cRQc&t=389

  68. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

    https://www.youtube.com/watch?v=X4dEHRzBLmc&t=728

  69. Build Your Own Deep Research Agent + Technical Writer

    https://www.youtube.com/watch?v=mYSRn6PC1mc&t=6789

  70. Weak supervision needs evaluation beyond supervisor agreement

    https://arxiv.org/pdf/2312.09390

  71. NIST AI RMF: Monitoring, incident response, and recovery

    https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

  72. Training Krea 2 - What matters in generative model training.

    https://www.youtube.com/watch?v=-tviRdpmHvs&t=428

  73. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    https://www.youtube.com/watch?v=H7puB0RwJMM&t=20

  74. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    https://www.youtube.com/watch?v=H7puB0RwJMM&t=390

  75. Computer Use at the Edge of the Statistical Precipice

    https://www.youtube.com/watch?v=CTLa_p6iOiY&t=699

  76. Data and Environment Curation for Post-training LLMs

    https://www.youtube.com/watch?v=ewtOo0scUh0&t=564

  77. Data and Environment Curation for Post-training LLMs

    https://www.youtube.com/watch?v=ewtOo0scUh0&t=654

  78. Data and Environment Curation for Post-training LLMs

    https://www.youtube.com/watch?v=ewtOo0scUh0&t=702