MODEL EXPLORER / LANGUAGE MILESTONES
See what changes during reasoning post-training
What changes in the learning process while the base architecture stays the same?
RECORDED NUMERICAL EXAMPLE
The milestone is a learning process.
Distinguish R1-Zero, R1’s four stages, and supervised distillation. A separate original toy GRPO implementation provides reproducible arithmetic without claiming to reproduce the published model.
The highlighted stage describes the original January 2025 paper. It is an editorial process map, not execution status. Below it, the same independent four-candidate toy GRPO batch explains reward advantages, policy ratios and the reference penalty. It is not a simulation of the selected SFT/RL stage or evidence of learned reasoning.
- DeepSeek-V3-Base
Begin with a pretrained base model; no cold-start SFT.
- Direct GRPO
Use accuracy and format rewards on sampled responses. This is the R1-Zero path, separate from R1.
Bars share one scale within this example. Values are rounded to four significant digits.
- Toy GRPO loss · before update
- 5.551e-17
- Clipping epsilon
- 0.2
- KL coefficient
- 0.04
Verified by the local recipe
- Group advantages center rewards, preserve reward shifts/scales and vanish for equal rewards
- Clipping handles positive and negative advantages with the expected saturated and unsaturated gradients
- The sampled reference penalty is nonnegative and zero for identical policies
- One toy GRPO update increases the best-rewarded completion probability while old/reference policies remain frozen
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Separate R1-Zero, R1’s four stages and supervised distillation; inspect a reproducible toy GRPO objective beside the process explanation.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1]
Input 1: [1,3] · float32 randn
Input 2: [4] · int64 ones
Input 3: [4] · float32 randn · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above ToyGRPO in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Separate R1-Zero, R1’s four stages and supervised distillation; inspect a reproducible toy GRPO objective beside the process explanation.
Training method · 2025
DeepSeek-R1 post-training
Original GRPO arithmetic for a categorical linear policy over four synthetic one-step completion IDs. Population reward standard deviation, epsilon for equal rewards, clipping 0.2 and KL coefficient 0.04. The process map distinguishes the published training paths.
Source, capture & limitations +
A toy objective and paper-described process, not DeepSeek-V3/R1 weights, architecture or a reproduction of reasoning training. The four-candidate group is fixed and enumerated; a real GRPO trainer samples responses from its old policy and handles variable token lengths. No critic, language generation, benchmark result or SFT dataset is included. The numerical batch is independent of the highlighted historical stage. Distilled models in the paper use supervised fine-tuning, without a further RL stage.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
26c2243f53b24384- Source SHA-256
5afc1a480ea9332bf34dcbe39f192d39a06267193c57807498da933d97fba721- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
Toy four-choice, one-prompt GRPO objective with frozen old/reference policies. Advantage invariants, clipping values/gradients, sampled KL penalty and one SGD update checks. Editorial process stages describe the original 2025 paper; no language generation.
Read the model manifest ↗