MODEL EXPLORER / MODERN LLMS
Bring the modern decoder pieces together
Where do RMSNorm, RoPE and SwiGLU meet in an actual decoder block?
RECORDED NUMERICAL EXAMPLE
See the components in one decoder.
A small synthesis of LLaMA-1 architectural ingredients. It uses ordinary multi-head attention; later LLaMA-family models vary their head-sharing and training choices.
These are recorded activations for the final token of the same six-token untrained decoder. RMSNorm controls feature scale inside each branch; residual additions preserve a direct input path. The vocabulary IDs have no language semantics.
Bars share one scale within this example. Values are rounded to four significant digits.
- Feature RMS
- 0.958
- Model width
- 16
- Attention heads
- 4
Verified by the local recipe
- Later token changes leave earlier causal logits unchanged
- Zero attention/FFN output projections make the entire pre-norm block an identity
- RoPE preserves each head's query/key norm
- Prefix evaluation matches the corresponding full-sequence outputs
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Follow tokens through a reduced LLaMA-1-style decoder and inspect the activations where its familiar components connect.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,6,32]
Input 1: [1,6] · int64 ones · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above TinyLlama in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Follow tokens through a reduced LLaMA-1-style decoder and inspect the activations where its familiar components connect.
Architecture · 2023
LLaMA architecture synthesis
One width-16 layer, four MHA heads, SwiGLU hidden width 40 and a 32-token vocabulary. Uses pre-RMSNorm and rotary Q/K, with a final RMSNorm and untied output head.
Source, capture & limitations +
A reduced, randomly initialized architecture; no tokenizer, pretrained LLaMA weights or generated-language quality. It models LLaMA-1-style multi-head attention. Hidden dimensions and training settings differ from the original models.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
14549022286b4d78- Source SHA-256
66b559f494b1aea5da2a6dd8821e514150f696f08b43df73118108034614a783- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
One untrained decoder layer with integer vocabulary IDs. Causality, prefix equivalence, norm preservation and identity-path checks; no language benchmark.
Read the model manifest ↗