AI Engineer World's Fair 2025
Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek-R1/V3) — Vibhu Sapra
Loading slides…
Read the talk
Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek-R1/V3)
A foundational Paper Club curriculum leads into DeepSeek’s training recipe: verifiable rewards, longer reasoning, assistant post-training and distillation into smaller models.
From a talk by Vibhu Sapra
Before you start: Basic familiarity with language models, pretraining and fine-tuning is helpful; the article explains the reinforcement-learning and distillation mechanisms as they arise.
A weekly discussion becomes a foundational curriculum
What should an AI engineer read to understand the foundations beneath each new model release? That question grows out of an established habit: Vibhu Sapra reports that Paper Club had run every Wednesday at noon for a year and a half without missing a week, usually drawing about 100 people and reaching 300 for DeepSeek V3. Volunteers organized the sessions, and authors from NVIDIA, Meta, Allen AI, Amazon, Together and Writer joined to explain their work and answer questions.
The proposed Test of Time Paper Club adds a curriculum alongside that weekly discussion of new research. The existing format stays: a current paper, roughly half an hour of presentation, then discussion. The second club asks which concepts remain useful after the release cycle moves on—attention, sequential generation in GPT-2, optimizers, speculative decoding, FlashAttention, Stable Diffusion and Whisper.
The initial plan is a July–December run of roughly 24 weekly sessions, covering two to four papers each and about 50–100 papers overall. A speech session could connect recognition with synthesis; an image session could move from CLIP through diffusion to video generation. San Francisco gets an in-person gathering, with remote participation continuing alongside it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Organizing the papers around mechanisms
The foundational schedule would remain relatively fixed even when a new release takes over the original club. Its proposed sequence begins with deep learning, moves through language-model history and then connects training choices to model behavior.
- Deep-learning foundations: attention, optimization, ReLU, gradient descent and basic reinforcement learning.
- Language-model foundations: RNNs, LSTMs and bidirectional RNNs, followed by BERT, GPT-2 and generative models such as Llama 3 and DeepSeek.
- Pretraining and post-training: Chinchilla scaling, distillation, overtraining and the different training requirements of small and large models. The Phi family provides a way to discuss how reinforcement learning changes with model size.
A training topic might occupy one or two sessions, with several papers presented together so their assumptions and trade-offs can be compared.
The remaining tracks connect those foundations to practical techniques: CLIP, Sora, Segment Anything and diffusion for generative systems; agent papers; LoRA, QLoRA, DPO, RL and GRPO for adaptation; Whisper for voice; and speculative decoding and FlashAttention for efficiency. Evaluation gets its own track, with Eugene Yan and other contributors helping select material. These are proposed categories, open to revision by the community.
Paper recommendations, topic suggestions and volunteer presenters would come through a Google Form and the Discord Paper Club channel. Sapra plans to assemble a rough schedule, settle a San Francisco venue and retain Zoom for remote sessions. Author visits and extended discussion remain central, even if covering several papers pushes a session beyond an hour. Flo might cover music generation; Eugene might cover state-space models. The intended archive is a reusable entry point into a field: a selection of papers, a presentation and a discussion available on YouTube whenever someone needs the foundations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A date-stamped R1 release with substantial changes
The session then returns to its usual business: reading a paper. After a Mochi dog-slide interlude, Sapra reuses the DeepSeek walkthrough that had drawn a large live audience and, he reports, more than 1,000 subsequent YouTube views. There is a new reason to revisit it: DeepSeek-R1-0528, the May 28 update. Despite rumors of R2, this is a dated revision of R1. Its understated name prompts a detour through Simon Willison’s critique of model naming, his keynote surveying recent releases and his PelicanBench evaluation.
Additional post-training can substantially change a model without changing its generation name. Sapra attributes the update to better post-training of the V3 foundation and emphasizes stronger reasoning. The release card reports AIME 2025 pass@1 rising from 70.0% to 87.5%, a gain of 17.5 percentage points. Sapra calls this AIME 2024 in the talk; the release table identifies it as AIME 2025. These manufacturer-reported estimates use 16 samples per query, temperature 0.6, top-p 0.95 and a 64K generation cap.
The release also reports average AIME reasoning length increasing from about 12,000 to 23,000 tokens. Sapra rounds the new length differently during the session, but the important distinction is task-specific output length: these are generated tokens, not individually measured reasoning steps. Longer outputs accompany increased post-training compute and algorithmic improvements; the comparison does not isolate output length as the sole cause of the gain.
This opens another scaling dimension. Training can first optimize capability for a fixed training budget, then spend more upfront to improve a model of a given size, and finally teach that model to spend more inference compute on difficult questions. The release also targets coding, JSON output and function calling. Sapra reads the benchmark chart as putting the revision near OpenAI o3 and Gemini 2.5 Pro; the release describes it as approaching those models overall. He singles out a visible improvement on Humanity’s Last Exam without giving a score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Transferring the improvement into an 8B model
The second release applies the stronger teacher to a smaller student. Earlier R1 distillations used models from the Qwen and Llama families. This time, DeepSeek-R1-0528-Qwen3-8B starts from Qwen3-8B and learns from the updated teacher’s reasoning traces. The student is a dense 8B reasoning model.
The student’s reported AIME 2024 score rises from Qwen3-8B’s 76.0% to 86.0%. That gives a concrete meaning to Sapra’s otherwise unspecified ten-point improvement. His comparison with Qwen3-235B-A22B, a much larger mixture-of-experts thinking model, needs the same task boundary: the student scores 86.0 versus 85.7 on AIME 2024, but 61.1 versus 71.1 on GPQA Diamond. Strong transfer on one reasoning benchmark does not imply equal capability across domains.
The mechanism is supervised learning from teacher-generated examples. Sapra also uses the phrase logit matching, but the documented recipe is SFT on generated traces, not matching a teacher’s full token-probability distributions. The student does not receive native RL in this recipe. No corresponding refreshed 32B or 70B student is discussed. The release supplies open weights and benchmark results, while leaving much of the detailed post-training recipe unspecified.
An audience question raises the obvious evaluation risk: if benchmarks are public, can models simply be trained to score better on them? Sapra acknowledges the problem and points toward checking overfitting and general performance. He does not provide a concrete contamination test or a separate diagnostic here. That leaves the reported benchmark gains useful, but narrower than a complete evaluation of the models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why teach a model to spend more time on a question?
The original DeepSeek-R1 paper addresses a question that was difficult to answer from proprietary reasoning-model releases: how much of the behavior comes from search, supervised reasoning traces or reinforcement learning? Around OpenAI o1, plausible explanations included Monte Carlo tree search and explicitly generated chain-of-thought supervision. DeepSeek exposed two useful experiments: R1-Zero, which applies GRPO directly to a base model, and R1, which combines a supervised cold start, reasoning RL, rejection sampling with SFT and final RL.
The economic motivation starts with training compute. In the Chinchilla framing, a fixed budget informs the balance between model parameters and training data. Later model development increasingly emphasized inference economics: train a model of a chosen size for longer so it carries more capability into every subsequent inference. Sapra contrasts roughly a trillion training tokens for early Llama with 15 trillion for Llama 3, then uses a hypothetical expansion from 45 trillion to 450 trillion tokens to illustrate the cost of another tenfold increase.
Sapra treats escalating training cost and data demand as pressure to find another axis, rather than continually enlarging the same training run. R1-Zero asks whether a pretrained model can develop reasoning behavior through reinforcement learning without a preliminary supervised reasoning stage. DeepSeek starts with DeepSeek-V3-Base, applies GRPO and observes reflection and longer solution attempts. R1 adds supervised stages to make that behavior more usable.
Test-time scaling lets harder queries consume more computation before the final answer. The model still generates tokens autoregressively; RL changes the behavior it learns to produce. A short question can receive a short response, while a hard math or coding problem can elicit a longer attempt. Verifiable answers make those attempts trainable: a math answer can be checked, and a programming solution can be evaluated against known requirements. The additional computation happens during inference; it does not mean the model’s weights are updated while answering each user.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Research access and serving reliability are separate
Two research directions follow: use RL to elicit reasoning from a strong base model, and transfer a strong teacher’s behavior into a smaller student. Sapra points to Phi-4-Mini reasoning work and Qwen thinking models as examples of the wider movement. He attributes the Phi result to roughly 6,000 RL samples, although the exact experiment is not identified. The distinction among DeepSeek’s releases remains useful: R1-Zero demonstrates reasoning, R1 adds assistant-oriented training, and the Qwen/Llama students learn through distillation rather than student RL.
Open weights make these experiments available for inspection and deployment, but do not provide the entire training process. Sapra describes an MIT-licensed release while also noting the absence of training data and training code. His experience of the hosted API is less positive: he retracts his initial enthusiasm about its speed and price because of downtime and problems creating API keys.
Sapra reports that DeepSeek accounted for about 10% of OpenRouter API usage, without specifying a measurement window. He also recalls launch claims of tenfold speed and price advantages, then emphasizes that unreliable access undermined their practical value. Those are historical observations about adoption and service availability, separate from the model’s reasoning quality.
The next comparison is architectural. GPT-4o provides a familiar baseline for direct responses; o1 illustrates spending longer on reasoning before responding. Earlier approaches assembled that extra work externally through process rewards, beam search, Monte Carlo tree search or multiple trees of thought. Sapra describes a system exploring ten such trees and stitching the results together. His assessment is that these approaches did not reach the capability he saw from scaled native RL: the model itself learns when to continue, reconsider and try another path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
V3 supplies the pretrained foundation
DeepSeek-V3 is the non-reasoning foundation behind this work. Its mixture-of-experts architecture has 671 billion total parameters, with 37 billion active per token. Activating only a subset reduces computation, but does not make the full checkpoint an ordinary laptop-sized model: the inactive parameters still have to be stored and made available. Sapra characterizes its general capability as comparable to GPT-4o.
The low training-cost figure needs an accounting boundary. The V3 report estimates $5.576 million for its official training run at $2 per H800 GPU-hour, including pretraining, context extension and post-training but excluding prior research and ablations. It is not the total cost of developing DeepSeek or R1. Sapra interprets GPU restrictions as an incentive for constrained labs to improve efficiency and explore alternative scaling strategies; that is his explanation of the engineering pressure behind the work.
V3 combines several efficiency choices. It trains on 14.8 trillion tokens and adopts multi-head latent attention, previously validated in V2. Multi-token prediction supplies additional training signal, which Sapra connects to making better use of the available data. The training system uses FP8, and context extension proceeds through 32K and 128K stages. SFT and conventional reinforcement learning then shape the chat model. This foundation precedes R1; the later reasoning work changes how the pretrained capability is elicited and refined.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
R1-Zero: reward correctness and format
R1-Zero begins with the completion model, DeepSeek-V3-Base, without a preliminary SFT stage. Pretraining teaches next-token prediction; it does not by itself establish the behavior of a helpful user-facing assistant. DeepSeek applies Group Relative Policy Optimization, or GRPO, an algorithm introduced in its earlier DeepSeekMath work.
The reward has two practical components:
- Accuracy: reward a verifiably correct answer. Mathematics supplies known results; programming tasks supply checkable solutions. Compilation is one check Sapra mentions, while LeetCode-style tasks also require satisfying the problem’s tests.
- Format: reward the expected structure, including separating reasoning with
<think>tags.
This makes it possible to reward successful outputs without prescribing every intermediate step. The result can reason and answer while still lacking the readability and conversational behavior expected from an assistant.
Sapra largely skips the GRPO equations, but retains the contrast with PPO: GRPO does not require a separate critic model. It uses rewards from a group of sampled outputs to establish a relative learning signal, with constrained updates and a KL-divergence term helping control training. The key connection is between an outcome that can be scored and a policy update that makes successful behavior more likely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Longer attempts create room to reconsider
Sapra reads the original R1-Zero results as stronger than o1-mini and near, though somewhat below, o1 on the math comparisons shown. As training progresses, reasoning outputs grow from hundreds to thousands of tokens. The interesting change is behavioral: the model has room to revisit earlier steps instead of immediately committing to an answer.
A longer attempt can include a failed approach, a check of that approach and a different route. Sapra describes the emergence of reflection: revisiting assumptions, reevaluating previous steps and continuing after an initial strategy fails. His informal example is six unsuccessful steps followed by a seventh that works. The paper’s interpretation is that incentives can produce useful problem-solving strategies without explicitly supervising each strategy. This is the connection he draws to subsequent small-model RL work and Qwen’s reasoning models.
The concrete illustration is a nested-square-root problem with a parameter greater than one. In Sapra’s walkthrough, the generated solution notices an x² term, pauses with “Wait, wait, wait,” and revisits the algebra by considering squaring both sides. The significant event is the change of approach inside the generated solution. The talk’s partial reading does not supply enough of the equation to work through the complete problem here.
That behavior does not resolve the assistant problem. R1-Zero can produce awkward text and mix languages during its reasoning. A model rewarded for getting a mathematical answer right has not necessarily learned how to present it clearly or converse usefully. Those limitations motivate R1’s additional training stages.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four stages turn reasoning into assistant behavior
R1’s training alternates supervised learning with reinforcement learning. Each stage addresses a different weakness: establish readable behavior, learn to solve hard problems, collect good examples and then refine usefulness.
- Cold-start SFT. Train the base model on a few thousand long reasoning examples. Prompt–answer pairs, human review and consistent formatting establish a readable starting point before RL.
- Reasoning RL. Apply the R1-Zero approach to hard, verifiable math and coding problems, improving the model’s ability to sustain useful solution attempts.
- Rejection sampling and SFT. Generate candidate completions, evaluate them and retain correct, suitable responses. Use the resulting examples for supervised training.
- Final RL. Combine reasoning rewards with objectives for helpfulness, harmlessness and human preferences.
The third stage does not train on failures merely because they were rejected. It filters generated candidates to build a stronger supervised dataset.
The stage-three dataset contains approximately 600,000 reasoning examples and 200,000 non-reasoning examples, for about 800,000 total. Sapra connects the generate–evaluate–retain pattern to Llama 3. A small Python example makes the filtering operation concrete: for the teaching prompt What is 7 × 8?, retain a correct, readable candidate and exclude an incorrect one before constructing SFT pairs.
python
prompt = "What is 7 × 8?"
candidates = [
{"id": "a", "answer": "54", "readable": True},
{"id": "b", "answer": "56", "readable": True},
]
accepted = [
candidate
for candidate in candidates
if candidate["answer"] == "56" and candidate["readable"]
]
sft_examples = [
{"prompt": prompt, "completion": candidate["answer"]}
for candidate in accepted
]
The code builds training examples; it does not update model weights. The full training procedure applies richer correctness and quality checks to much longer responses.
Final RL preserves the ability to solve difficult problems while improving ordinary interactions. A complex question may deserve a detailed answer; a request for a basic summary should stay short. Those preferences are not captured by mathematical correctness alone. R1 therefore combines verifiable reasoning tasks with preference signals for general chat.
Returning to the May update, Sapra adds reduced hallucination to the reasoning and coding improvements. The dated May 28 release announcement also announces JSON output and function calling. These are historical release capabilities, not an API execution demonstrated in the session or a specification of today’s endpoint behavior. The stronger teacher then feeds the refreshed 8B distillation, connecting assistant post-training back to smaller-model quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why distill before applying small-model RL?
The original student recipe uses supervised fine-tuning on R1-generated reasoning examples. It transfers successful behavior from a larger model into a smaller one without applying RL to the student. Sapra proposes RL-based distillation as a further direction, saying he knows of no such work at the time. His invitation to reproduce and extend the method goes beyond what the release itself supplies: the published recipe and checkpoints should not be confused with a complete release of all training traces.
| Student family | Released sizes | Training route |
|---|---|---|
| Qwen | 1.5B, 7B, 14B, 32B | SFT on teacher-generated examples |
| Llama | 8B, 70B | SFT on teacher-generated examples |
Sapra emphasizes the gains over the starting models and broadly compares the distills with GPT-4o. The useful experimental question is what a small model gains from a strong teacher versus what it can discover through RL on its own.
In the reported Qwen-32B experiment, direct RL for 10,000 steps underperformed distillation. Sapra connects this to the R1 cold start: supplying a useful initial behavior can make subsequent learning more effective than expecting the smaller model to discover everything from outcome rewards. He also discusses QwQ-32B, distinguishing weaker base-model and stronger chat-model results in his account, but does not establish a controlled base-versus-chat ablation in the walkthrough. The concrete conclusion is the reported advantage of teacher supervision over that particular direct-RL run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What remains unresolved
The original R1’s weaknesses included function calling, multi-turn use and JSON output relative to V3. Sapra treats the updated structured-output and tool-use capabilities as progress on that list. Language mixing remains an open question for him: he has not established how the new revision behaves there. Prompt sensitivity also remains relevant. He relays an OpenAI-researcher view that users’ continuing need to scaffold reasoning models should push labs to improve the models themselves.
Sapra considers the old engineering-task comparison outdated after the refresh, but supplies no fresh task-level engineering result in this passage. He closes the research discussion by pointing toward open reproductions, including Hugging Face’s Open R1 and Bespoke Labs. He does not know the current status of the latter. Reproduction matters because the released models demonstrate what is possible while leaving parts of the training process unavailable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A discussion club, smaller reasoning models and shared ownership
The closing returns to the curriculum with a useful clarification: Test of Time is a Paper Club, not a hands-on course. The proposed topics expand to include Adam and mid-training alongside attention, model foundations, scaling, diffusion, voice, optimization and fine-tuning. Presentations and open discussion are the format; workshops, building exercises and active practice belong to a different learning experience.
The participation form invites people to present, recommend papers and help organize. Sapra refers to a June launch in this closing invitation, after initially describing a July kickoff, so the launch timing remains tentative. The curriculum would build on the Discord community’s existing weekly paper list.
The technical ending returns to the two releases: the stronger R1 teacher and the DeepSeek-R1-0528-Qwen3-8B student trained on its reasoning traces. Sapra’s final chart discussion places the student alongside Gemini 2.5 Flash Thinking and o3-mini-medium, and ahead of Phi-4, Qwen3-8B and Qwen3-32B on the comparisons he is highlighting. These are chart-based comparisons without spoken task-by-task conditions; the earlier AIME and GPQA contrast shows why they should not become a claim of universal parity.
The 8B checkpoint also changes the deployment scale. Sapra describes it as runnable on-device or on a laptop, although he gives no hardware configuration, quantization, memory budget or measured speed. The practical possibility is a smaller reasoning model learned from a much larger teacher. Both releases retain the R1 name: this is a date-stamped refresh, not R2, and the improvement comes from developing how the models reason and transferring that behavior.
The final invitation belongs to the community that made the reading possible. Sapra brings attention to regular contributors—Eugene, Era, RJ and Flo—along with remote volunteers and visiting authors, and thanks Eugene Yan, Sam and swyx. He emphasizes that Paper Club is not his project alone or swyx’s: volunteers sustain the weekly conversation. The participation code stays onscreen for anyone who wants to recommend a paper, present one or help build the next set of discussions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original account of R1-Zero, R1's four training stages and smaller distilled models.
Architecture, training infrastructure and cost accounting for the foundation used by R1.
Release benchmark results, evaluation settings and usage guidance for the May 2025 R1 refresh.
Historical announcement of improved reasoning, JSON output and function calling.
Official 8B reasoning checkpoint distilled from R1-0528, with benchmark results and deployment guidance.
Open reproduction effort with training, evaluation and reasoning-data generation recipes.
Further reading
Organizer's event listing introducing the planned Test of Time Paper Club announcement.
Full transcript
Loading transcript…