Stanford CS329A Part 6 — Train-Time Scaling and Scaling RL

Generating more model outputs is not yet self-improvement. A train-time scaling loop must evaluate those outputs, turn feedback into a learning signal, update the model, and then show that later behavior improved. Stanford CS329A Part 6 explains that loop through STaR, DeepSeekMath with GRPO, and DAPO.

This page is a teaching companion to the complete 1:12:38 lecture session from October 10, 2025. The front-matter date is this wiki entry’s publication date. The page combines the complete manual-caption transcript, 29 inspected visual cues, the official course schedule, and the three primary papers.

One source correction is load-bearing: an early lecture slide labels DeepSeekMath-RL 7B’s 51.7% result as AIME. The DeepSeekMath paper reports 51.7% on MATH, and 60.9% on MATH with self-consistency over 64 samples. This page follows the primary paper. DAPO’s separate AIME 2024 result is labeled with its own model and evaluation setup.

Download the Slides

The Core Loop

Train-time self-improvement is a closed system:

sample model outputs
        |
        v
verify or score
        |
        v
construct learning signal
        |
        v
update model parameters
        |
        +----------> sample again

Every credible claim needs four answers:

1. What behavior was sampled?
2. What verifier distinguished better from worse?
3. What signal reached the optimizer?
4. What evidence shows that the update improved the intended behavior rather than the metric alone?

If one answer is missing, “self-improvement” is underspecified.

1. Keep Four Compute Regimes Separate

Test-time search can improve an answer without changing the model. Train-time scaling changes the distribution that will generate later answers. These are different mechanisms and require different evaluation.

A full-system result also cannot be attributed automatically to the last training stage. DeepSeekMath combines model initialization, continued pretraining data, SFT, and RL.

2. Verifiability Is the Feedback Bottleneck

Mathematics is attractive for self-training because many final answers can be checked mechanically. That produces frequent outcome feedback without requiring a human to label every generation.

But the verifier’s boundary is narrow:

rationale + final answer -> answer checker -> pass / fail
       |
       +-> rationale faithfulness remains unverified
       +-> safe behavior remains unverified
       +-> tool-path validity remains unverified

A correct answer can hide an invalid derivation. A coding test can miss a vulnerability. A citation checker can reward a correctly formatted but unsupported reference. The policy learns what the verifier rewards, including its blind spots.

3. STaR Bootstraps Rationales

STaR: Bootstrapping Reasoning With Reasoning starts with a small rationale demonstration set and a larger answer-labeled dataset.

generate a rationale
        |
        v
does it yield the correct answer?
   |                         |
  yes                       no
   |                         |
retain it          show the answer and rationalize
   |                         |
   +-----------+-------------+
               v
          fine-tune
               |
               +----------> repeat

Correct-answer rationales enter the training set directly. Failed examples receive a second attempt conditioned on the known answer. If that rationalized attempt succeeds, it can also enter the training set.

This converts answer labels into rationale data, but the answer checker does not independently validate each reasoning step.

4. Rationalization Helps and Can Mislead

An answer hint can recover examples that ordinary sampling missed. It can also invite a plausible post-hoc explanation.

The durable lesson is not that rationalization is invalid. It is that correctness and faithfulness are different claims.

5. Self-Training Has a Support Boundary

The lecture raises a capability-ceiling hypothesis: self-training can make useful behavior already reachable under sampling more frequent and consistent. It does not guarantee discovery of a genuinely absent or out-of-distribution strategy.

before: useful strategy appears rarely
after:  useful strategy appears often

unknown: did the update create a new strategy,
         or only increase one already in support?

This is a diagnostic hypothesis, not a universal impossibility theorem. Test it with support-sensitive evidence:

• pass@k across multiple sampling budgets;
• performance under distribution shift;
• strategy diversity rather than answer frequency alone;
• transfer to tasks that require a new representation or tool;
• controlled comparison against additional data, search, or curriculum.

6. DeepSeekMath Is a Full Stack

DeepSeekMath is not “GRPO applied to a generic language model.” Its reported system combines:

DeepSeek-Coder initialization
        -> 120B math-related web tokens
        -> supervised fine-tuning
        -> GRPO reinforcement learning

The primary paper reports:

These results describe the full training stack. They do not isolate a causal gain from GRPO alone.

7. GRPO Uses Relative Group Feedback

PPO-style training commonly includes a policy, reference model, reward model or rule, and a learned critic. GRPO removes the separately trained critic and estimates a relative baseline from a group of outputs sampled for one prompt.

prompt
  -> sample outputs o1 ... oG
  -> assign rewards r1 ... rG
  -> center or normalize rewards within the group
  -> update the policy using relative advantages

The core comparison is within the sampled group. This reduces the critic-related memory burden described by the paper, but it does not remove dependence on a trustworthy reward function.

8. Relative Learning Needs Reward Variation

Suppose four outputs receive rewards:

all wrong:   [0, 0, 0, 0] -> no within-group distinction
mixed:       [0, 1, 0, 1] -> useful relative distinction
all correct: [1, 1, 1, 1] -> no within-group distinction

When every reward is identical, centered relative advantages collapse. Generation compute was spent, but the group offers no comparative update signal.

DAPO’s dynamic sampling keeps groups with mixed outcomes to increase signal density. This is not neutral preprocessing: it changes the effective training distribution by downweighting tasks that are currently too easy or too hard.

9. Majority@k Is Not Pass@k

Consider [A, A, A, B, C*, D], where C* is the only correct answer. Majority@6 returns A and fails. Pass@6 succeeds because C* exists in the candidate set.

A higher majority score can mean that an existing successful behavior became more consistent. It does not, by itself, establish that the model acquired a new solution mode. This is a useful diagnostic distinction, not a universal theorem about all RL runs.

10. DAPO Stabilizes Large-Scale RL

DAPO combines four controls:

The paper’s cumulative recipe moves from about 30 to 50 on AIME 2024 average@32 using a Qwen2.5-32B base model. That number is separate from DeepSeekMath’s MATH result. It is a setup-specific cumulative result, not a universal threshold and not an isolated effect of any single control.

11. Monitor the System, Not Loss Alone

Useful operational signals include:

• task reward and held-out correctness;
• response length and truncation rate;
• token entropy and semantic diversity;
• fraction of all-correct, mixed, and all-wrong groups;
• pass@k and majority@k at fixed sampling budgets;
• verifier disagreement and suspected reward hacking.

Exact entropy or length thresholds depend on model, tokenizer, task, decoding policy, and infrastructure. A number copied from one lecture slide is not a universal operating target.

12. Choose by Signal, Not Fashion

Use four gates before scaling a loop:

1. VERIFIER:  Is the intended target actually checkable?
2. SUPPORT:   Can the current policy discover useful candidates?
3. VARIATION: Do sampled groups contain comparative signal?
4. UPDATE:    Can optimization improve behavior without collapse?

A failed gate is an engineering result. Add supervision, change the curriculum, widen exploration, repair the verifier, or stop the loop.

The Lecture in 12 Teaching Moves

Teaching Questions

1. What must happen between model generation and a defensible claim of self-improvement?
2. Why is test-time search not train-time scaling?
3. What does a final-answer checker certify, and what does it miss?
4. Which STaR examples enter the training set directly?
5. How can answer-conditioned rationalization create a post-hoc story?
6. What experiment would distinguish consistency gain from strategy-support expansion?
7. Why can the DeepSeekMath result not be attributed to GRPO alone?
8. Which component does GRPO avoid relative to a critic-based PPO stack?
9. Why do all-correct and all-wrong groups carry no relative signal?
10. Give a candidate set where majority@k fails but pass@k succeeds.
11. Which DAPO control changes the effective task distribution?
12. When should a team stop rather than scale a self-improvement loop?

Homework: Design and Audit a University Self-Improvement Loop

Choose one university task: algebra tutoring, code feedback, citation checking, assignment planning, or another instructor-approved task.

Produce these deliverables:

Conclude with a one-page argument answering: What evidence would show that students learn better, rather than that the agent only scores better?

Official Reading Set

1. Eric Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning, arXiv:2203.14465.
2. Zhihong Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv:2402.03300.
3. Qiying Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, arXiv:2503.14476.

Source Versus Teaching Synthesis

Directly sourced from the lecture and primary papers:

• the train-time feedback-loop framing;
• the STaR generation, filtering, rationalization, fine-tuning, and iteration procedure;
• DeepSeekMath’s model/data/SFT/GRPO stack and MATH metrics;
• GRPO’s group-relative baseline and lack of a separately trained critic;
• DAPO’s Clip-Higher, dynamic sampling, token-level loss, soft overlong punishment, and setup-specific cumulative result;
• the lecture’s majority@k versus pass@k distinction and discussion of capability ceilings.

Teaching synthesis in this page:

• the four-regime comparison table;
• the verifier, support, variation, update diagnostic;
• the university examples and monitoring checklist;
• the support-sensitive capability tests;
• the homework and stop-rule framing.

The capability-ceiling and majority-versus-pass arguments are presented as diagnostics to test, not settled universal laws.

Related Reading

• Stanford CS329A — Self-Improving AI Agents: Public Lecture Learning Path
• Stanford CS329A Part 5 — Planning and Multi-Step Reasoning
• Stanford CS329A Part 7 — Self-Improvement and Deep Research Agents
• CS329A Agent-System Literacy — The Next Layer of AI Education
• Stanford CS329A Part 2 — Test-Time Compute Scaling