MODEL EXPLORER / LANGUAGE MILESTONES
Choose source context for the next decoder step
How does a recurrent decoder decide which source positions to consult?
RECORDED NUMERICAL EXAMPLE
Ask the source a different question at each step.
Recorded additive alignment for a fixed source sequence under three decoder states. In translation, the evolving decoder state makes this context change at every generated token. This lesson captures one such step.
The same five synthetic source IDs produce fixed bidirectional annotations. Only the previous decoder state changes here. The learned additive score determines the weighting used by this step. These untrained alignments do not map translated words or demonstrate translation quality.
Min -0.3219 · Max 0.6347
Recorded cell values
| Row / col | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| 0 | -0.1275 | -0.01018 | 0.2421 | -0.006486 | -0.1751 | 0.006959 | 0.1005 | 0.08204 |
| 1 | -0.08045 | 0.1993 | 0.2912 | 0.3643 | 0.1429 | 0.287 | -0.2102 | 0.2492 |
| 2 | 0.01838 | 0.3667 | 0.3144 | 0.5997 | 0.2724 | 0.4881 | -0.1182 | 0.1885 |
| 3 | -0.3219 | 0.2817 | 0.4326 | 0.6004 | 0.2303 | 0.6347 | 0.1154 | 0.02379 |
| 4 | -0.1695 | 0.36 | 0.4325 | 0.5784 | 0.1242 | 0.1659 | -0.06991 | 0.07757 |
Min 0.1867 · Max 0.2096
Recorded cell values
| Row / col | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| 0 | 0.1867 | 0.199 | 0.2096 | 0.2075 | 0.1972 |
Each heatmap uses its own scale. Mint is positive, rust is negative, and the lightest color is zero. Table values are rounded to four significant digits.
Bars share one scale within this example. Values are rounded to four significant digits.
- Alignment weight sum
- 1
- Most weighted source position · zero-based
- 2
Verified by the local recipe
- Alignment weights are positive and sum to one over source positions
- The context equals an independent weighted sum of bidirectional annotations
- A changed final source token can affect the first annotation through the backward GRU
- Changing decoder state changes alignment; zero scores give the mean source annotation
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Trace bidirectional annotations into additive alignment, then change the previous decoder state and inspect its weighted source context.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,16]
Input 1: [1,5] · int64 ones
Input 2: [1,8] · float32 randn
Input 3: [1] · int64 ones · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above AdditiveDecoderStep in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Trace bidirectional annotations into additive alignment, then change the previous decoder state and inspect its weighted source context.
Component · 2014
Bahdanau additive attention
Five source tokens with width-four embeddings; bidirectional GRU annotations of width eight; additive alignment and one width-eight GRUCell decoder update. A simple linear vocabulary head replaces the original output architecture.
Source, capture & limitations +
Untrained attention subsystem and a single decoder step, not a complete translation system. It omits the paper's exact decoder/maxout output formulation, variable-length padding and training recipe. Publication began as a 2014 preprint and appeared at ICLR 2015.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
6f6805ea39bc1506- Source SHA-256
5fadad4ddf6e107b09d66c42307c4804cae1bbd5346ef8e4c927b6db0034a44c- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
One reduced additive-attention decoder step with a bidirectional GRU encoder. Alignment normalization, context-sum, bidirectional influence and decoder-state sensitivity checks.
Read the model manifest ↗