MODEL EXPLORER / ALTERNATIVE ARCHITECTURES
Carry context through a selective state
How does the input decide what a recurrent state retains?
RECORDED NUMERICAL EXAMPLE
Read, write and retain a recurrent state.
Recorded selective-state evolution in a tiny Mamba-style block. Streaming state size is fixed for this architecture; these counts do not include weights or training activations.
The recurrence has processed tokens 0 through 0. This table shows channel 0’s three state coordinates; the complete state holds eight channels. Delta, B and C come from the current convolved token. Exact diagonal ZOH gives Ā = exp(ΔA) and B̄ = (exp(ΔA)−1)B/A. This is a recorded CPU loop, without the fused parallel scan or measured GPU speed.
Min -0.01054 · Max 0.002818
Recorded cell values
| Row / col | 0 | 1 | 2 |
|---|---|---|---|
| 0 | 0.002818 | -0.01054 | -0.007842 |
Min -0.01827 · Max 0.01017
Recorded cell values
| Row / col | 0 | 1 | 2 |
|---|---|---|---|
| 0 | 0.002818 | -0.01054 | -0.007842 |
| 1 | -0.001974 | 0.0079 | 0.006224 |
| 2 | -0.003395 | 0.01017 | 0.00655 |
| 3 | 0.003722 | -0.01209 | -0.008144 |
| 4 | 0.001197 | -0.003539 | -0.002266 |
| 5 | -0.001828 | 0.007008 | 0.005324 |
| 6 | 0.004022 | -0.01297 | -0.008697 |
| 7 | 0.006005 | -0.01827 | -0.01186 |
Each heatmap uses its own scale. Mint is positive, rust is negative, and the lightest color is zero. Table values are rounded to four significant digits.
Bars share one scale within this example. Values are rounded to four significant digits.
- SSM state values per sequence
- 24
- Convolution history values per sequence
- 16
Verified by the local recipe
- Future changes leave every earlier output unchanged
- Token streaming with a 16-value convolution cache and 24-value SSM state matches the full forward
- The recurrent state matches an independent sum of decayed past writes
- Negative diagonal A and positive input-dependent delta give decay factors between zero and one; selection maps receive gradients
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Follow a causal convolution into input-dependent state dynamics, then replay the scan and compare its fixed-size streaming state.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,6,4]
Input 1: [1,6,4] · float32 randn · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above TinyMamba in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Follow a causal convolution into input-dependent state dynamics, then replay the scan and compare its fixed-size streaming state.
Architecture · 2023
Mamba selective state-space models
Width four, expansion eight, three state coordinates per channel, causal depthwise convolution width three. LayerNorm wrapper, SiLU gate and residual projection. Exact diagonal zero-order-hold discretization follows the paper’s Eq. 4.
Source, capture & limitations +
Untrained single teaching block, without token embeddings or a language head. CPU sequential loop with exact ZOH B discretization; the released fast implementation uses a simplified delta × B input term. No fused parallel scan, custom backward or measured speed claim. The static graph groups the recurrence in a Python node; the panel records its actual state history. This model does not universally replace attention.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
dae2c22380683e3d- Source SHA-256
cc6ddffd335c7c6f37f908ea898b9826b5e199f49ca0fcc76d494fca0734bd85- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
Reduced Mamba-style block with causal convolution, input-dependent delta/B/C and diagonal ZOH scan. Causality, token streaming, closed-form recurrence and selection-gradient checks.
Read the model manifest ↗