MODEL EXPLORER / MODERN LLMS
Put position into attention
Where does token position enter the attention score?
RECORDED NUMERICAL EXAMPLE
Move position, keep content fixed.
Recorded float64 rotations of the same synthetic Q/K vectors. The diagram shows their first feature pair (one radian per position); the score uses all four pairs.
At position zero, all rotation angles are zero. The query and key vectors remain unchanged.
- Q before (1, 0.5)
- K before (0.2, 1)
- Q at position 0 (1, 0.5)
- K at position 0 (0.2, 1)
Both axes use the same scale. The circle has radius 1. Full vectors and the score across all eight features appear below.
Bars share one scale within this example. Values are rounded to four significant digits.
- Key position − query position
- 0
- Scaled Q/K score · all 8 features
- 0.2475
- Query norm
- 1.6
Verified by the local recipe
- Every paired rotation preserves the vector norm
- Shifting both positions equally preserves the Q/K dot product for fixed content
- Changing relative position changes the score in this example
- Zero Q/K yields a uniform mixture of unrotated values
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Rotate query and key feature pairs, compare relative positions, and follow the unchanged value path to the output.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,6,8]
Input 1: [1,6,8] · float32 randn · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above RotaryAttention in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Rotate query and key feature pairs, compare relative positions, and follow the unchanged value path to the output.
Component · 2021
RoPE / RoFormer
One unmasked attention head with width 8 and four paired rotations using base 10,000. Rotates Q/K only; isolates position encoding from causal masking.
Source, capture & limitations +
This component is not a trained RoFormer model. The diagram visualizes only the first feature pair; other pairs use slower frequencies. Relative-position algebra does not guarantee context-length extrapolation in a trained model.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
e52c0735ba4f7e05- Source SHA-256
a5a791e4b252ae091485fec653a9bd53798407bb6df40d663255744f24a53b7e- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
One unmasked CPU attention forward plus norm, relative-position and unchanged-value checks. Numerical rotations use fixed synthetic Q/K vectors in float64.
Read the model manifest ↗