MODEL EXPLORER / MODERN LLMS
One token at a time
How does a decoder form a next-token distribution?
RECORDED NUMERICAL EXAMPLE
A token can look back, but not ahead.
Recorded head-0 attention and vocabulary probabilities for the fixed token IDs [1, 2, 3, 4, 5, 6], using seed-0 untrained weights.
Head 0 at position 0 can consult positions 0 through 0. Later keys receive zero weight. Token IDs identify synthetic vocabulary entries; this model has no learned language ability.
Bars share one scale within this example. Values are rounded to four significant digits.
- Visible prefix length
- 1
- Attention row sum
- 1
- All 32 vocabulary probabilities sum to
- 1
Verified by the local recipe
- Changing future token IDs leaves earlier predictions unchanged
- Every attention head gives future positions exactly zero weight
- Attention rows and vocabulary distributions each sum to one
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Start with token IDs, add position, restrict attention to the visible prefix, and return a distribution over a small vocabulary.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,6,32]
Input 1: [1,6] · int64 ones · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above TinyDecoder in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Start with token IDs, add position, restrict attention to the visible prefix, and return a distribution over a small vocabulary.
Architecture · 2017
Transformer decoder
One reduced decoder-only block: width 16, four attention heads, FFN width 64, vocabulary 32. Keeps sinusoidal positions, causal masking and post-normalization; omits the original encoder, cross-attention and dropout.
Source, capture & limitations +
This is a teaching adaptation of the Transformer, not its full original translation model. It has no tokenizer or trained weights. Predictions are distributions over synthetic token IDs.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
11db495736878eab- Source SHA-256
71dc54bd4d8dce7255cd674f020b03f48daba49120fab62e63b7aa232c1fb25b- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
One CPU forward with integer token IDs, plus causality and probability checks using varied tokens. Untrained teaching decoder; no text-generation quality claim.
Read the model manifest ↗