MODEL EXPLORER / MODERN LLMS
A gate inside the feed-forward block
Why does this feed-forward block split into two branches?
RECORDED NUMERICAL EXAMPLE
One branch controls another.
Recorded CPU calculations using the same input and seed-0 initialization. The gate is the only changed branch.
Only the gate weights change. SiLU gates can be negative and are not probabilities. These initialized weights illustrate arithmetic, not learned language behavior.
Bars share one scale within this example. Values are rounded to four significant digits.
- Ordinary FFN parameters
- 384
- SwiGLU parameters
- 384
Verified by the local recipe
- Both comparison branches contain exactly 384 parameters
- Gradients reach the gate, value, and output projections
- A zero gate suppresses SwiGLU without changing the ordinary FFN
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Trace a SiLU gate, multiply it with a second projection, and compare a conventional FFN at the same parameter count.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [2,1,4,8]
Input 1: [1,4,8] · float32 randn · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above GatedFeedForward in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Trace a SiLU gate, multiply it with a second projection, and compare a conventional FFN at the same parameter count.
Component · 2020
SwiGLU
Bias-free width-8 FFNs: conventional hidden width 24 and SwiGLU hidden width 16, each with 384 parameters.
Source, capture & limitations +
These are untrained feed-forward components. Equal parameter counts do not imply equal quality or runtime. SiLU gate values are not probabilities and can be negative.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
2e269c77c5e058b6- Source SHA-256
610c6468be240a995ad5285135094c1c71ec0204c6ede4db5d72ccd721e09b78- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
One seeded CPU forward plus recorded numerical invariants. Untrained, reduced teaching implementation; no accuracy claim.
Read the model manifest ↗