AI Engineer Code 2025

Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI

Loading slides…

Read the talk

Lessons from Generating 12 Trillion Synthetic Tokens

Selected presentation frame from Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI at 70 secondsOpen full source frame
The agenda lists the talk’s topics, including BeyondWeb, engineering lessons, outcomes, and future directions.

Bogdan Gaza explains how DatologyAI seeds synthetic data with curated documents, then keeps trillion-token generation moving through metadata batching, recoverable jobs, coordinated scheduling and inference tuning.

From a talk by Bogdan Gaza

At a glance

Ideas worth remembering

  • BeyondWeb seeds generation with curated documents, then rephrases or restructures them to expand useful training material.

  • Metadata discovery can dominate startup at trillion-token scale. Batched S3 listing reduced DatologyAI’s estimated 9–11-day approach to about two hours of API processing.

  • Right-sized partitions and checkpoints limit repeated work after GPU failures; resumable execution and idempotent writes make retries practical.

  • Cross-cluster scheduling must coordinate a job’s CPU head and GPU workers, because separate pools can each have capacity without satisfying the whole job.

  • Inference parameter sweeps produced about 40% more throughput in the reported workload, while domain-specific data recipes remain an active experimental question.

Pretraining runs grow beyond the available data

Large pretraining runs need more useful data than the web can readily supply. That is the starting problem in Bogdan Gaza’s account of generating about 12 trillion synthetic tokens across web, math and code at DatologyAI. Synthetic generation supplements a finite source of training material, but producing that much text creates another problem: keeping expensive GPUs working throughout a massive data pipeline.

DatologyAI’s role begins before generation. Gaza, its co-founder and CTO, describes the service as an oil refinery for pretraining and mid-training datasets: customers bring substantial amounts of data, curation identifies the highest-quality points, and a synthetic data recipe expands those selected inputs. The selection step matters because the documents chosen here become the material the generator will transform.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

BeyondWeb expands selected documents through rephrasing

BeyondWeb is DatologyAI’s synthetic data recipe. The opening comparisons ask whether better training data can reduce the model size or token budget needed to reach a given accuracy. Models trained at 1 billion, 3 billion and 8 billion parameters provide the first comparison: the presented 3-billion-parameter BeyondWeb result matches the 8-billion-parameter Nemotron-Synth result.

The second comparison varies training tokens rather than model size. Gaza reports matching the accuracy of Nemotron-Synth with about 2.7 times fewer tokens and Cosmopedia with about 5.3 times fewer tokens. These are results from the presented comparisons, rather than universal reductions in training cost; the recording does not specify the complete evaluation conditions. Gaza describes the results as dating from around late summer 2025 and says the recipe has continued to improve.

Selected presentation frame from Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI at 246 secondsOpen full source frame
A slide compares model accuracy across training-token counts for several datasets.

The recipe starts with an existing high-quality document. Asking a model to invent a synthetic data point without a seed tends, in Gaza’s explanation, to draw from the common modes of its original training distribution. Supplying a selected document gives generation a specific starting point. Prompts prefixed to that document request rephrasings, restructurings or reframings, and models running on GPUs produce the new material. The described inputs can be text or image-text data.

This makes curation and synthesis one connected process: identify useful source material, then generate alternate presentations of it. The GPU work is straightforward to describe at this level. Predicting how quickly the entire dataset will finish is harder, because actual throughput fluctuates and falls short of the ideal rate.

2:122:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:42 · section reference included

Bring research and product jobs into the same workflow

The original setup split synthetic generation from the product pipeline. Research ran generation, training and evaluations on an H100 cluster managed by Slurm; the core product pipeline ran on Kubernetes. Separate code paths slowed experiments, introduced errors and left GPUs underused. Generated data also required manual backfilling into the product’s asset tracker, its data catalog.

The move to Kubernetes retained separate clusters with different jobs. A control cluster holds orchestration and the asset tracker. A compute cluster runs Spark and Ray work using on-demand CPUs and, when appropriate, GPUs that do not require H100s. The H100 cluster now runs under EKS on HyperPod. An internally built scheduler coordinates work across these clusters, with Ray, KubeRay and vLLM forming the core synthetic generation stack.

Two infrastructure choices support deployment beyond this particular AWS setup. The pipeline runs on Kubernetes because customer environments may be AWS, GCP or on-premises. Data lives in S3 or storage that speaks S3-compatible APIs. The architecture therefore keeps a common orchestration approach while allowing compute to live in different environments.

What becomes possible when those jobs share one workflow? Curation can run Spark jobs, feed synthetic generation, launch training and then run evaluation. The flow below shows the experiment’s dependencies. Each stage supplies the next, allowing research and engineering to use the same infrastructure and code path rather than manually reconnecting outputs between systems.

Selected presentation frame from Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI at 527 secondsOpen full source frame
A workflow diagram connects data curation, synthetic generation, training, and evaluation.

One workflow from curated data to evaluation

CurateSpark jobs select and prepare training data.
Generate synthetic dataRun generation on available GPU capacity.
TrainLaunch a training job after generation.
EvaluateRun evaluation within the same experiment workflow.
A shared scheduler connects stages that previously crossed separate research and product systems.
Read the diagram as text
  • Curate. Spark jobs select and prepare training data.
  • Generate synthetic data. Run generation on available GPU capacity.
  • Train. Launch a training job after generation.
  • Evaluate. Run evaluation within the same experiment workflow.
  • Curate → Generate synthetic data: Selected data.
  • Generate synthetic data → Train: Generated training material.
  • Train → Evaluate: Trained model.
5:426:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:42 · section reference included

Metadata can leave the GPUs waiting for days

The first bottleneck appears before inference. Gaza estimates that the web supplies about 30 trillion usable text tokens, give or take, and names FineWeb, DCLM and Nemotron as starting datasets. At this scale, data stored in formats such as Parquet spans millions of partitions. Ray must discover and manage the corresponding storage metadata before workers can efficiently process the data.

Fetching metadata through many individual S3 requests makes startup depend on request overhead and rate limits. Gaza uses a roughly 10-millisecond request as an illustration, then describes a back-of-the-envelope estimate of 9–11 days to populate the metadata for the intended scale. That estimate is the warning: a generation fleet can have ample GPU capacity and still sit idle while the Ray head assembles the information needed to dispatch work.

The change was to batch metadata discovery through S3’s listing APIs, increasing the page size to about 1,000 entries per list. More metadata arrives per API call, reducing the repeated request work. Gaza reports about two hours of API processing time with this approach. The practical lesson is to plan how bucket state enters the execution system: accelerating inference does little for a job that cannot finish discovering its inputs.

9:4210:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:42 · section reference included

Make an eight-hour partition survive a late failure

Once generation starts, hardware failure becomes part of normal operation. A machine can go down or a GPU can fail, leaving partial outputs and consuming cluster time without completing the assigned work. In this pipeline, Spark stitches prompts together with the documents to rephrase. Ray then runs GPU generation, with jobs ordinarily organized around a one-to-one mapping between input and output partitions.

Consider Gaza’s concrete failure example: one node takes eight hours to rephrase one file, then goes down in the final five minutes. Without saved intermediate progress, almost eight hours of work must be repeated. Retrying the job restores execution, but it does not recover the compute already spent. The size of the partition determines how much unfinished work is exposed to that failure.

Two controls reduce the loss:

  • Right-sized partitions. Smaller units of work limit how much progress a failed partition can discard.
  • Periodic checkpoints. Flushing intermediate progress to S3 preserves work before the entire partition finishes.

DatologyAI uses both. A failed job can retry with a tolerable amount of lost work instead of turning into a multi-day failure.

Selected presentation frame from Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI at 878 secondsOpen full source frame
A diagram shows work progressing through partitions with checkpoint and retry stages.

What changes at the moment of failure? The comparison below follows the same eight-hour job through its retry. Checkpointing preserves a completed prefix of the work; resumption must redo only work beyond the last saved point. Gaza also calls for idempotent checkpoint writes, describing a checkpoint file that can be overwritten, so retrying a save does not require a new logical output each time. The recording gives no checkpoint interval, so the remaining loss depends on that choice.

A late failure with and without saved progress

Generate without a checkpointWithout saved progress — One partition requires eight hours.
Late failure loses progressWithout saved progress — The node fails in the final five minutes; substantial work must be repeated.
Flush progress to S3With checkpoints — Periodically save completed work during generation.
Recover and retryWith checkpoints — Saved progress remains available; lost work is limited by checkpoint timing.
The same failure near the end of an eight-hour partition causes different amounts of repeated work depending on whether progress has reached S3.
Read the diagram as text
  • Generate without a checkpoint. Without saved progress — One partition requires eight hours.
  • Late failure loses progress. Without saved progress — The node fails in the final five minutes; substantial work must be repeated.
  • Flush progress to S3. With checkpoints — Periodically save completed work during generation.
  • Recover and retry. With checkpoints — Saved progress remains available; lost work is limited by checkpoint timing.
  • Generate without a checkpoint → Late failure loses progress: Node fails near completion.
  • Flush progress to S3 → Recover and retry: Failure triggers a recoverable retry.
12:1212:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:12 · section reference included

Schedule the CPU head and GPU workers together

Synthetic generation does not necessarily need fast networking between GPU nodes, which makes geographically distributed capacity useful. Curation can remain centralized while customers’ generation jobs run across several GPU locations. The difficulty shifts to coordinating the resources each job needs across those clusters.

Separate resource availability can produce a scheduling mismatch. One job may obtain its CPU resources while its GPU workers cannot be scheduled. A later job may find GPUs available but no CPU capacity. Either resource pool can look usable by itself while the combination needed for useful execution is unavailable. Gaza describes this as a bin-packing problem.

The solution keeps dedicated pools for different demands, including Spark CPUs and Ray heads, then coordinates the CPU and GPU requirements of a Ray job in a more atomic fashion. Separating pools makes the resource roles clearer; scheduling the head and workers together addresses the mismatch between independently allocated pieces. The talk describes this scheduling principle rather than a specific cross-cluster reservation algorithm.

Selected presentation frame from Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI at 1054 secondsOpen full source frame
A diagram shows separate compute pools and a Ray head with GPU workers.

The important unit for scheduling is therefore the working job: a CPU head and the GPU workers it needs. That requirement persists when capacity spans multiple providers or clouds. Increasing either pool alone will not resolve every queueing problem if the scheduler cannot assemble both sides at the same time.

14:4215:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:42 · section reference included

Benchmark inference settings for the batch workload

The final bottleneck sits inside inference. DatologyAI runs substantial batch generation through vLLM and is also testing SGLang. This workload differs from online serving: the task is to process large volumes of generation work, so familiar serving settings need to be benchmarked against that workload.

Gaza recommends a dedicated benchmark harness and grid searches over inference parameters. Two named controls are batch size and speculative decoding configuration. The harness makes their throughput effects observable, allowing the team to choose settings from measured combinations rather than accepting defaults.

The reported gain is about 40% more throughput from flag tuning. That result belongs to the workload and configurations tested; the recording does not give the model, selected flags or baseline settings needed to reproduce it. The useful decision is to treat inference configuration as an experimental variable alongside the data recipe.

17:4118:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:41 · section reference included

From 30 billion tokens to 12 trillion—and the next question

At the end of 2024, DatologyAI could produce around 30 billion synthetic tokens in limited Slurm runs. Gaza’s closing comparison describes later customer jobs producing about 7 trillion web tokens plus another 5 trillion math and code tokens. Metadata discovery, recoverable execution, coordinated CPU/GPU scheduling and inference sweeps together supported the move to this much larger production scale.

Selected presentation frame from Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI at 1160 secondsOpen full source frame
A slide shows two charts and a domain coverage list beside the outcomes heading.

The ending leaves a substantive question open: how should the synthetic recipe change across domains? Web, multilingual, math, code and legal data may require different treatment, and Gaza identifies domain coverage as important without developing those differences here. The team continues to run ablations and search for better recipes. A pipeline that can curate, generate, train and evaluate in one workflow makes those experiments easier to carry through, while the volume generated alone does not settle which recipe works best for each domain.

18:4219:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:42 · section reference included

Transcript download
Full transcript

Loading transcript…