tensorViz
← All models

MODEL EXPLORER / LANGUAGE MILESTONES

Learn from both sides of a masked token

What changes when the encoder can see both left and right context?

Curated by TensorVizBERT pretraining · 4-step tour
Static architecture
What changes when the encoder can see both left and right context?
Opening the interactive graph…
Architecture snapshot · No Python requiredDownload graph ↓

RECORDED NUMERICAL EXAMPLE

Use both sides of context, predict selected targets.

Recorded MLM and NSP objectives on synthetic token IDs. The encoder can attend across both segments. Target selection defines the loss; unselected token logits are ignored by MLM. No language understanding is claimed.

Toy token IDs: CLS=1, SEP=2 and MASK=3. Segments are A,A,A,A,B,B. Only selected positions contribute to MLM loss. This controlled example always replaces selected words with MASK; original BERT selected 15% of tokens with an 80/10/10 mask/random/unchanged rule. The NSP labels here are illustrative, without a real sentence corpus.

Bidirectional attention · head 0

Min 0.1108 · Max 0.246

Recorded cell values
Row / col012345
00.17620.17730.17550.16280.15330.1548
10.18780.18530.1760.17210.13960.1393
20.18640.16890.17170.17830.14550.1491
30.15080.15340.14520.2460.16330.1413
40.13880.18780.15290.21760.16860.1342
50.18610.22980.19580.15890.11860.1108

Each heatmap uses its own scale. Mint is positive, rust is negative, and the lightest color is zero. Table values are rounded to four significant digits.

Original token IDs
1
4
5
2
6
2
Corrupted token IDs
1
4
3
2
6
2
MLM loss positions · 1 means selected
0
0
1
0
0
0
Prediction at first masked position · 16 token probabilities
0.0005717
0.7849
0.0001438
0.003133
0.0001385
0.06069
0.0003843
0.0001423
0.002378
0.1114
0.00005626
0.0007285
0.004733
0.002439
0.02759
0.0005293
Sentence pair · IsNext / NotNext probabilities
0.4712
0.5288

Bars share one scale within this example. Values are rounded to four significant digits.

Masked-token cross entropy
2.802
Sentence-pair cross entropy
0.7525
Combined pretraining objective
3.554
Verified by the local recipe
  • A later token can change earlier token logits through bidirectional attention
  • MLM cross entropy ignores all unselected token positions
  • The MLM decoder shares the token-embedding weight and receives a learning signal
  • Packed MLM/NSP logits have the declared shapes and sentence-pair probabilities sum to one

Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.

ABOUT THIS EXAMPLE

Explore a bidirectional BERT-style encoder, then inspect which token targets and sentence-pair labels contribute to its pretraining objectives.

Architecture · 2018

BERT pretraining

One post-LN encoder, width eight, two heads, fixed length six and vocabulary sixteen. Token/position/segment embeddings, tied MLM decoder, GELU/LayerNorm MLM transform and a tanh-pooled sentence-pair head.

Source, capture & limitations +

Untrained fixed-length model without padding or a tokenizer; dropout is omitted. The numerical cases always mask chosen tokens to isolate the MLM objective, rather than reproduce the complete 15% and 80/10/10 corruption sampler. NSP uses synthetic labels. The recipe packs two output heads into one 98-value tensor for verification.

Original TensorViz teaching example. PyTorch provides the underlying operators.

No separate redistribution license has been declared for these project examples.

Content revision
0c48027dcf5a807d
Source SHA-256
fa041d05ccb2a15cb1e139d53180ab5ca4930dd11ba12f3fd3aec170228f0fc5
Captured
2026-09-17 · Python 3.13.13 / Torch 2.7.1

One reduced BERT pretraining forward with MLM/NSP heads, bidirectional influence, ignored-target loss behavior and tied-weight gradient checks. Output packs 96 MLM logits and two NSP logits.

Read the model manifest ↗