MODEL EXPLORER / MODERN LLMS
Scale without centering
What does normalization change in one token vector?
RECORDED NUMERICAL EXAMPLE
Keep the vector. Change the normalization.
Recorded CPU calculations with unit gains, zero LayerNorm bias, and epsilon 0.000001. Choose an input transformation.
Both formulas agree for this zero-mean vector when their gains and epsilon match.
Bars share one scale within this example. Values are rounded to four significant digits.
- LayerNorm output mean
- 0
- RMSNorm output mean
- 0
Verified by the local recipe
- LayerNorm removes a common offset; RMSNorm preserves its effect
- RMS normalization and positive rescaling agree within epsilon tolerance
- Zero input stays finite and learned gain rescales the result
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Compare LayerNorm and RMSNorm on the same eight features, then change the input's offset and scale.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [2,1,4,8]
Input 1: [1,4,8] · float32 randn · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above NormalizationComparison in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Compare LayerNorm and RMSNorm on the same eight features, then change the input's offset and scale.
Component · 2019
RMSNorm
Eight-feature implementation of the normalization equations; unit gains at initialization.
Source, capture & limitations +
Normalization type and placement are separate choices. This forward comparison does not establish training stability or speed. The recorded experiment uses float64; the captured graph input uses float32.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
57dce70f3f8f53cb- Source SHA-256
cb7f385265455049f0b476eb05031c6efb964dec3f92b8420bb1d4a32ddef028- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
One seeded CPU forward plus recorded numerical invariants. Untrained, reduced teaching implementation; no accuracy claim.
Read the model manifest ↗