MODEL EXPLORER / MODERN LLMS
Learn a small update to frozen weights
How can two small matrices adapt a frozen linear layer?
RECORDED NUMERICAL EXAMPLE
Fit an update, then merge the branches.
Recorded deterministic SGD on a synthetic rank-two task. W + (alpha/r) B A gives the same output after merging, up to floating-point rounding.
A fixed eight-example toy task asks for a rank-two weight change. The base is frozen throughout; only A (8 → 2) and B (2 → 8) are fitted. At initialization B is zero. This records adapter arithmetic, not fine-tuning quality on language tasks.
Bars share one scale within this example. Values are rounded to four significant digits.
- Mean squared error · all eight examples
- 0.01563
- Maximum merge error
- 0
- Trainable adapter parameters
- 32
- Frozen base parameters
- 64
Verified by the local recipe
- Zero-B initialization exactly preserves the frozen base output
- Only B has a nonzero adapter gradient at the first step; the base receives no gradient
- Toy training reduces fitting loss while base weights remain byte-identical
- Merged weights reproduce the two-branch output and the update rank is at most two
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Follow the frozen base and rank-two adapter, inspect actual toy training checkpoints, and verify the merged output.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,4,8]
Input 1: [1,4,8] · float32 randn · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above LowRankAdapter in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Follow the frozen base and rank-two adapter, inspect actual toy training checkpoints, and verify the merged output.
Training method · 2021
LoRA
One frozen bias-free 8 × 8 base matrix, trainable A (2 × 8) and B (8 × 2), alpha = rank = 2. Standard random-A / zero-B initialization.
Source, capture & limitations +
The base is randomly initialized, not pretrained. Toy fitting demonstrates low-rank adaptation and gradients; it does not reproduce language-model fine-tuning. Benefits depend on chosen rank and adapted layers.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
a844ca6fe86121f9- Source SHA-256
2348feed277b52dd4ab33eecebfd6fbf9a677b37afd2da5376c09ae104dcdf88- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
Original width-8 rank-2 LoRA layer. Exact merge and gradient checks plus deterministic SGD on an eight-example synthetic task.
Read the model manifest ↗