AI Engineer World's Fair 2026

Data Quality is the Compute Multiplier

Ari Morcos· CEO, DatologyAI19:05

Loading slides…

Read the talk

Data Quality Is the Compute Multiplier

Better training data can improve accuracy, reduce inference cost, and make domain adaptation more effective by increasing how much a model learns from each token.

From a talk by Ari Morcos

Before you start: Basic familiarity with model training, tokens, and evaluation benchmarks is useful; FLOPs measure computation, and post-training further adapts an already trained model.

What if more compute is not available?

How do you improve a model when you cannot count on buying more compute? Ari Morcos, CEO and co-founder of DatologyAI, opens with a capacity problem. In his account of the market at the time, H100 prices had risen about 40% above their lows at the end of the preceding year. Hardware that had been becoming cheaper was becoming more expensive again.

Reasoning adds another source of pressure: thinking tokens consume inference capacity before the model produces its final answer. Morcos reports that reasoning models use eight times as many tokens as non-reasoning models and cites a projection of another fivefold increase over roughly the following year. That second figure is a forecast, not an observed increase.

The practical concern is access, not just price. Morcos points to Google capping Meta’s Gemini usage and describes OpenAI’s future token-capacity guarantees as effectively selling token futures. His warning is that frontier APIs could face restrictions because inference capacity is insufficient, with providers prioritizing their own products. These are the talk’s contemporaneous market examples; they motivate a question that remains useful independently of any particular provider policy: how much more can a model learn from the compute already available?

Slide with three panels showing a rising GPU price chart, reasoning versus non-reasoning token growth, and API capacity examples.Open full source frame
Compute scarcity: rising GPU prices, increasing token use, and potential limits on API access.
0:380:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:38 · section reference included

Make the learning curve steeper

Better data changes the return on a training budget. On the schematic learning curve, performance rises as the model sees more data. Improving data quality moves that curve upward: at a fixed budget, the model performs better; at a fixed performance target, it needs a smaller budget. Morcos treats data, compute, time, and dollars as alternative schematic budget axes. The tenfold compute example illustrates the relationship rather than establishing a measured multiplier.

The objective is maximum learning signal per token and per batch—or, more technically, maximum marginal information gain from the next data point. That definition makes quality dependent on the intended output tasks. There is no universally optimal dataset: a legal model needs a different relevance distribution from a healthcare model.

Relevance alone is insufficient. The data also needs:

  • Diversity: Different presentations of a problem help prevent a model from succeeding only when a question is worded just so.
  • Information density: Tokens should contribute useful information rather than consume budget without teaching much.
  • A suitable mixture: Combining individually valuable sources still requires deciding how much of each source the model should see.

Quality therefore belongs to the dataset and its sampling process, not only to individual documents.

2:002:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:00 · section reference included

Clean, curate, create, compose

Datology’s refinement pipeline starts with existing public, proprietary, and licensed data. Its role is to improve those tokens rather than source new ones. Morcos organizes the process into four stages, each addressing a different limitation of the incoming corpus.

  1. Clean. Apply basic heuristic filters, such as the Gopher-style filtering Morcos mentions, to remove extremely short or malformed documents. Then decontaminate the training corpus against downstream benchmarks. He describes using a low n-gram threshold to detect overlap, without specifying the exact setting. Cleaning makes the data usable and evaluations more interpretable; it does not yet make the corpus especially informative.
  2. Curate. Use quality classifiers and topic taxonomies to select and balance material. Reduce semantic redundancy, including examples that differ in wording or pixels but convey essentially the same information. Up-sample or down-sample examples according to quality and relevance, and match the distribution of the target tasks.
  3. Create. Expand the resulting high-quality but potentially small dataset through synthetic rephrasing. Different formats increase both the amount of training material and the variety of presentations.
  4. Compose. Decide how to mix these datasets and sequence them across training. Morcos describes at least three data phases as common practice for large models, with continuous curricula extending the idea beyond fixed stage boundaries.

The output is not simply a cleaner pile of documents. It is a selected, expanded, and scheduled stream of training examples.

Public, proprietary, and licensed data flow through four columns of cleaning, curation, creation, and composition steps into multi-stage training datasets.Open full source frame
The data curation pipeline: clean, curate, create, and compose.
3:564:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:56 · section reference included

Selection can change scaling behavior

Does this refinement merely produce a small improvement, or can it change how performance scales? Morcos describes roughly half of Datology’s team as researchers, while noting that commercial incentives discourage publication of valuable curation methods. The research problem is to identify which examples are worth spending training compute on.

Beyond neural scaling laws: beating power law scaling via data pruning, which Morcos coauthored at Meta, supplied the foundational proof of principle. The NeurIPS 2022 Outstanding Paper showed that suitable data selection can change scaling behavior, including the exponent, by avoiding redundant or unnecessary examples. Its pruning results are conditional, not a promise that any filter will improve every model. The practical question behind Datology’s subsequent research releases is how far that principle carries into modern training pipelines.

6:016:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:01 · section reference included

VLM training: distinguish the controlled test from the frontier comparison

The first experiment applies curation to vision-language models. Morcos describes using approximately 25 billion tokens from the Mammoth dataset, identified as MAmmoTH-VL in the related research, to train the fusion adapter between the vision and text models. This is a specific training scope, not a claim that every component of the model was trained from scratch on that token budget.

The chart plots error against log training FLOPs, spanning roughly a thousandfold compute range. Public models including Qwen3, Qwen3.5, and InternVL define a comparison frontier: the best observed error at different compute budgets. The curated-data models extend beyond that frontier in the presented results, without post-training.

Two comparisons establish different things:

ComparisonResult reported by MorcosWhat it measures
Curated versus input dataset, everything else held constantAbout 14 absolute percentage points of improvementEffect of changing the data
Curated model versus Qwen3.5-4BWithin about one percentage point, using 145× less training computeRelative performance and training cost

The first is the controlled evidence for curation. The second compares separately developed models and should not be read as an otherwise identical training experiment. The benchmark aggregation is not specified in this passage, so these remain the talk’s reported comparisons.

6:567:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:56 · section reference included

Data representation also affects inference cost

Training efficiency is only one part of the budget. How an answer is represented in training data can also influence how concisely the model responds. Morcos first compares mean tokens per response across largely the same set of VLMs; the Datology-trained models produce particularly concise answers. He then changes the cost axis from log training FLOPs to log FLOPs per response, turning the question from training expense to inference efficiency.

Morcos reports roughly similar performance to Qwen3.5 with 35× fewer FLOPs per correct answer. That claim is distinct from both mean response length and the chart’s FLOPs-per-response axis: counting cost per correct answer also connects computation to successful outcomes. The mechanism is that data representation can reduce the amount of generation needed to answer well, so curation can affect the cost paid repeatedly after training.

8:258:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:25 · section reference included

A small multilingual allocation can matter

Text models face another data problem: the internet does not represent languages, countries, or communities uniformly. A model can perform well in English while being much less useful outside Western contexts. Curation becomes a way to address this uneven usability and its fairness implications, rather than simply accepting the distribution of available web text.

The multilingual experiment plots MMLU error against log FLOPs over roughly a hundredfold compute range. The public comparison includes Qwen models, Liquid models, and Cohere’s Tiny Aya. Morcos reports that only 8% of the training tokens were multilingual, with most languages receiving at most about six billion tokens. The point is that the non-English results did not require an enormous token allocation to each language.

On the presented multilingual MMLU comparison, Morcos reports slightly better performance than Qwen3 with roughly 8× less compute. This extends the compute-multiplier argument to a setting where the composition of the corpus matters as much as its overall size.

9:129:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:12 · section reference included

Use smaller runs to reduce the risk of larger ones

The same plot also connects experiments at very different scales. Two points represent dense Llama-style models trained on one trillion curated tokens each. A third represents Arcee AI’s Trinity Large, a hypersparse mixture-of-experts model trained on 17 trillion tokens. Morcos reports that a line through the two smaller models passes approximately through Trinity Large’s result despite its roughly 50× greater training compute.

That fit suggests a practical use for small training runs: test whether the data strategy is likely to deliver the desired performance before committing to the largest run. Morcos proposes using 50–100× less compute to reduce the risk of a larger run, provided the smaller experiments simulate token scarcity appropriately. The observed alignment is evidence from this comparison, not a universal guarantee that scaling will remain predictable across architectures and data mixtures.

10:2910:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:29 · section reference included

Curation benefits can cross language boundaries

An ablation separates the benefits of curating English from those of curating the non-English data itself. The comparison starts with no curation, then measures non-English accuracy after curating only English. Non-English performance improves even though that portion of the corpus has not been curated. Curating the non-English material as well produces further gains.

Morcos attributes the effect to cross-lingual transfer: a model that learns relationships between English and Spanish, for example, can use some improvements in English when working in Spanish. He reports that the magnitude of the benefit is strongly correlated with a language’s similarity to English. Transfer also runs in the opposite direction—curating non-English data improves English performance—but with a smaller effect. A source’s value therefore includes what it teaches beyond its own language.

11:1711:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:17 · section reference included

Turn valuable documents into multiple learning formats

Synthetic data enters the pipeline after source selection. Datology’s BeyondWeb approach uses rephrasing: transform existing documents into varied training formats, including documents held privately by a customer. The goal is to make useful source material easier for a model to learn from.

The demonstration starts with a document about a corporate takeover and converts it into a series of true-or-false questions, one of hundreds of possible templates. The source document supplies the information; the rephrasing model changes how that information is presented. Morcos argues that grounding the content this way avoids model collapse, rather than relying on the generator to invent the underlying knowledge. That is his rationale for the approach, not an unconditional guarantee for every synthetic-data pipeline.

The distinction also explains why the trained model can become more capable than the rephrasing model. The rephraser needs to transform the document accurately into the requested exercise. It does not need to independently possess all the capabilities that the eventual model will develop from the resulting corpus. Repeating the transformation across many formats exposes the same source information through different learning tasks.

Slide titled “Improve data learnability with synthetic data,” showing a source paragraph on the left, a synthetic data generation arrow, and business statements with colored answer markers on the right.Open full source frame
A corporate-takeover document becomes a true-or-false business exercise.

Format diversity helps the model learn more from valuable examples, but which documents get rephrased is consequential. Rephrasing a random selection does not produce the same result as first identifying high-quality sources. Morcos reports faster, better performance than public synthetic corpora without giving numerical results in this passage, and emphasizes that the process can operate on proprietary documents inside a customer’s own environment. The refinement pipeline’s order matters: select strong material, then expand its useful presentations.

12:1412:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:14 · section reference included

A better starting model changes what post-training can achieve

The next comparison applies Thomson Reuters’ post-training process to two starting points: the mid-trained model and the default instruction-tuned model. Morcos reports that the incremental gain from post-training almost tripled with the mid-trained starting point, without changing the post-training data. The plotted quantity is a gain—a delta—not a multiplication of final accuracy.

His explanation is that mid-training gives post-training a more accurate starting policy. The model begins with stronger domain understanding, so the same subsequent training can produce a larger improvement. Pre-training, mid-training, and post-training should be designed together: a decision about the data in one stage changes the opportunities available in the next. Treating them as isolated handoffs can miss that interaction.

15:1715:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:17 · section reference included

What the Trinity Large example says about cost

The final customer example returns to Arcee AI and Trinity Large. Morcos describes it as a fully open, US-made model trained on 17 trillion tokens curated from public datasets, with neither proprietary data nor closed-model usage in the effort. He reports competitiveness with the open frontier: matching GLM-5 and Kimi on many tasks and exceeding Claude on a couple. The passage does not specify the benchmark suite or all the model versions, so this is a broad reported comparison rather than a precise ranking.

Morcos says the Arcee team had not trained a model before beginning the collaboration around the middle of the preceding year. He reports total spending below $20 million across salaries, compute, R&D, repeated runs, and several models. That scope matters: the figure is an all-in program cost, not merely the compute bill for Trinity Large’s final run.

He separately estimates that a strong model for a narrow domain can be built for high six figures to roughly $1 million. That is a feasibility estimate, not a measured cost from the Trinity case. The economic implication is that building or customizing a capable model need not always require the spending associated with the largest general-purpose training programs; improving the data can change what is feasible within a smaller budget.

16:1716:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:17 · section reference included

Optimize signal, then make curation scale

The final training decision is often whether to add more tokens or extract more learning from the good tokens already available. Morcos recommends repeating high-quality data rather than adding low-quality data once the latter stops being useful—but only up to a threshold. He supplies no numerical repetition limit. The objective remains signal per token, not repetition for its own sake or an ever-larger corpus.

For a team constrained by compute, data quality is therefore an investment in how effectively that compute is used. Making the investment work requires both research and engineering: score and understand examples across multiple dimensions, then carry out those decisions over petabytes of data. Better selection, representation, and composition become useful at model-building scale only when the curation system can process the corpus reliably at that scale.

Three bullets call for more signal per token rather than more tokens, describe data quality as an underleveraged compute multiplier, and identify scaled curation as a frontier research and engineering challenge.Open full source frame
Key takeaways: more signal per token, data quality as a compute multiplier, and curation at scale.
17:4117:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:41 · section reference included

Resources

From the talk

  • The 2022 paper explores how informative data selection can improve scaling, with theory and image-classification experiments.

  • BeyondWebPaper12:26

    Research on scaling synthetic pre-training data and jointly optimizing its generation.

Transcript download
Full transcript

Loading transcript…