tensorViz
← All models

MODEL EXPLORER / VISION MILESTONES

Turn an image into tokens

Where does an image become a sequence for a Transformer?

Curated by TensorVizVision Transformer · 4-step tour
Static architecture
Where does an image become a sequence for a Transformer?
Opening the interactive graph…
Architecture snapshot · No Python requiredDownload graph ↓

RECORDED NUMERICAL EXAMPLE

An image becomes a sequence of patches.

Recorded image-to-token conversion and class-token attention. A stride-four patch projection is equivalent to flattening each RGB patch and applying the same linear map. Learned positions preserve where each patch belongs.

Move a bright 4×4 patch in a synthetic 16×16 image. The attention map shows head 0's class-token weights to the sixteen patch tokens, excluding its self-weight. These are untrained weights and are not a saliency explanation or object detector.

Input image · red channel, exact pixels

Min 0 · Max 1

Recorded cell values
Row / col0123456789101112131415
01111000000000000
11111000000000000
21111000000000000
31111000000000000
40000000000000000
50000000000000000
60000000000000000
70000000000000000
80000000000000000
90000000000000000
100000000000000000
110000000000000000
120000000000000000
130000000000000000
140000000000000000
150000000000000000
Class-to-patch attention · head 0

Min 0.05044 · Max 0.06928

Recorded cell values
Row / col0123
00.069280.050440.055320.058
10.060140.053890.057860.06021
20.059230.055070.061030.06262
30.051010.053390.054740.0647

Each heatmap uses its own scale. Mint is positive, rust is negative, and the lightest color is zero. Table values are rounded to four significant digits.

Bright patch embedding · 16 features
-0.01606
-0.726
0.4327
1.147
0.5944
-0.1982
-0.3692
0.1407
-0.1708
0.1352
0.7761
-0.2284
1.188
0.8734
-0.5369
0.04988
Three untrained class logits
-0.09848
0.8794
0.5208

Bars share one scale within this example. Values are rounded to four significant digits.

Patch tokens
16
Tokens including class token
17
Class-token self-attention weight
0.07306
Total class attention row weight
1
Verified by the local recipe
  • Strided patch convolution equals independent flatten-and-linear projection
  • Sixteen patch embeddings plus one class token form a seventeen-token sequence
  • Without positions, patch permutation preserves the class output; fixed learned positions break this symmetry
  • Changing one patch changes only its embedding, but global attention can change the class output

Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.

ABOUT THIS EXAMPLE

Inspect patch projection, learned positions and a class token, then compare its attention across a synthetic image.

Architecture · 2020

Vision Transformer

A 16×16 RGB input becomes sixteen 4×4 patches with width 16, plus a class token. One pre-LN encoder, two attention heads, GELU MLP, learned positions and a linear three-class head.

Source, capture & limitations +

The paper title refers to one original patch-size configuration; this teaching model uses 4×4 patches to keep the example small. All weights are untrained. Attention maps are recorded probabilities, not saliency or evidence of recognition quality.

Original TensorViz teaching example. PyTorch provides the underlying operators.

No separate redistribution license has been declared for these project examples.

Content revision
0d36b50d0b83ec26
Source SHA-256
3d8cc36b80819465f0cf328570d80777db7ff486981777d425ee16eaa9ced676
Captured
2026-09-17 · Python 3.13.13 / Torch 2.7.1

Reduced ViT CPU forward with independent patch-projection equivalence, permutation/position tests and patch-local versus global influence checks.

Read the model manifest ↗