MODEL EXPLORER / MULTIMODAL MILESTONES
Turn a prompt into candidate masks
How do image features and a user prompt become a mask?
RECORDED NUMERICAL EXAMPLE
Reuse the image; change the prompt.
Recorded prompt-to-mask computation from cached synthetic image features. Explore point location, prompt type and a box while the two-way decoder updates image and token features.
The same cached 2 × 2 image embedding is reused for each prompt. Coordinates are normalized (x,y) in [0,1]. Type IDs are 0 = background, 1 = foreground, 2 = top-left box corner, 3 = bottom-right. Three mask logits express the multi-mask interface; these untrained outputs do not segment a real object and quality scores are not calibrated IoUs.
Min -0.01213 · Max 0.1301
Recorded cell values
| Row / col | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.08099 | -0.005838 | 0.07064 | -0.001285 | -0.002424 | 0.04551 | 0.01376 | 0.06102 |
| 1 | 0.1226 | 0.02539 | 0.08357 | 0.04811 | 0.0727 | 0.01617 | 0.08024 | 0.003487 |
| 2 | 0.01623 | 0.05366 | 0.01485 | 0.003523 | -0.01213 | 0.004863 | 0.1039 | -0.00614 |
| 3 | 0.05469 | 0.02535 | 0.07576 | 0.04806 | 0.05914 | 0.06118 | 0.1301 | 0.02121 |
| 4 | 0.05909 | -0.0005823 | 0.07052 | -0.005873 | 0.03205 | -0.001703 | 0.02437 | 0.06232 |
| 5 | 0.101 | 0.03357 | 0.1183 | 0.02822 | 0.09264 | 0.04094 | 0.08509 | 0.0003713 |
| 6 | 0.02238 | 0.01484 | 0.00647 | 0.008072 | 0.01646 | 0.03088 | 0.01036 | -0.0005467 |
| 7 | 0.02148 | 0.09487 | 0.0569 | 0.05986 | 0.03509 | 0.06094 | 0.07948 | 0.0499 |
Min -0.1918 · Max 0.01852
Recorded cell values
| Row / col | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| 0 | -0.1014 | -0.01623 | -0.09132 | -0.01659 | 0.000458 | -0.07277 | -0.02307 | -0.08953 |
| 1 | -0.183 | -0.03976 | -0.1188 | -0.06355 | -0.1086 | -0.0202 | -0.1157 | -0.001932 |
| 2 | -0.02913 | -0.07802 | -0.01702 | -0.02561 | 0.01852 | -0.02849 | -0.1314 | -0.01501 |
| 3 | -0.07399 | -0.02796 | -0.1172 | -0.06851 | -0.09769 | -0.0879 | -0.1918 | -0.03402 |
| 4 | -0.07423 | -0.0204 | -0.08769 | -0.01655 | -0.03836 | -0.02082 | -0.03741 | -0.09056 |
| 5 | -0.1503 | -0.04886 | -0.1774 | -0.04352 | -0.1418 | -0.05984 | -0.121 | 0.002725 |
| 6 | -0.03601 | -0.0292 | -0.007868 | -0.02868 | -0.02843 | -0.04952 | -0.01015 | -0.02267 |
| 7 | -0.02525 | -0.1195 | -0.08716 | -0.08184 | -0.04586 | -0.07538 | -0.1249 | -0.07218 |
Min -0.01925 · Max 0.1718
Recorded cell values
| Row / col | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.0074 | 0.1275 | 0.02228 | 0.09883 | 0.01379 | 0.09014 | 0.03307 | 0.07709 |
| 1 | 0.1718 | 0.04578 | 0.08481 | 0.02256 | 0.1025 | 0.001276 | 0.0903 | -0.01291 |
| 2 | 0.04948 | 0.06398 | -0.007182 | 0.1156 | -0.01925 | 0.1223 | 0.01663 | 0.1229 |
| 3 | 0.0351 | -0.01819 | 0.1287 | 0.04985 | 0.1342 | 0.06716 | 0.17 | 0.04258 |
| 4 | 0.006906 | 0.1148 | 0.003323 | 0.1294 | -0.006559 | 0.1247 | 0.03979 | 0.07404 |
| 5 | 0.1395 | 0.04028 | 0.1708 | 0.04718 | 0.1493 | 0.05048 | 0.0869 | -0.01731 |
| 6 | 0.04554 | 0.05912 | -0.0006644 | 0.1018 | 0.0441 | 0.06195 | -0.01438 | 0.127 |
| 7 | -0.006936 | 0.01281 | 0.09195 | 0.04309 | 0.01373 | 0.0007318 | 0.1456 | 0.05748 |
Each heatmap uses its own scale. Mint is positive, rust is negative, and the lightest color is zero. Table values are rounded to four significant digits.
Bars share one scale within this example. Values are rounded to four significant digits.
- Image encoder calls per changed prompt
- 0
- Candidate masks
- 3
Verified by the local recipe
- Reordering sparse prompts preserves mask and quality outputs
- Moving a point or changing foreground/background type changes outputs without modifying cached image features
- Hypernetwork masks equal explicit per-pixel feature-weight dot products
- Mask gradients reach prompt coordinates and the image-to-token attention path
Download the source and run.py to reproduce these checks. Choosing an example here replays recorded values.
THE SAME MODEL, ON YOUR MACHINE
Pick up where
the graph leaves off.
Reuse a cached image embedding, encode points or box corners, and follow two-way attention into three dynamic mask predictions.
Use in VS Code ↗1. Save these files in one folder
2. Reproduce the example
Use Python 3.13 in a dedicated environment, matching the tested recipe. These commands are for macOS / Linux shells.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run.pyExpected output: [1,195]
Input 1: [1,8,2,2] · float32 randn
Input 2: [1,2,2] · float32 ones
Input 3: [1,2] · int64 ones · CPU / eval · seed 0
3. Open the source in TensorViz
Install the preview, open this folder in local VS Code, and choose Open as Graph above PromptableMaskDecoder in model.py. Select your Python environment. Save before choosing Validate.
Installation and supported setup ↗
The recipe was run with Python 3.13.13 and Torch 2.7.1. This is an untrained example; a successful shape check does not measure accuracy.
ABOUT THIS EXAMPLE
Reuse a cached image embedding, encode points or box corners, and follow two-way attention into three dynamic mask predictions.
Component · 2023
Segment Anything · promptable decoder
Width eight, two two-way blocks with one attention head, three mask tokens and one quality token. Random Fourier coordinate features plus point/box type embeddings. Two transposed convolutions upscale a 2 × 2 image embedding to 8 × 8 mask features.
Source, capture & limitations +
Untrained prompt/decoder subsystem, not a complete SAM model. Input is a cached synthetic embedding; the pretrained ViT image encoder, dense-mask encoder, tokenizer, image resizing and mask losses are omitted. The attention residual pattern follows the paper-level description with one head; this is not a parameter-compatible port. Only three candidate mask tokens are modeled, without the released implementation’s extra single-mask token. Quality scores are uncalibrated.
Original TensorViz teaching example. PyTorch provides the underlying operators.
No separate redistribution license has been declared for these project examples.
- Content revision
06fa171ea414ac32- Source SHA-256
bba1c4d0f25557c69579fade1e1ce55a8dd70d4af374ed69438d9a3652291d3e- Captured
- 2026-09-17 · Python 3.13.13 / Torch 2.7.1
SAM-inspired prompt-encoder and two-way mask-decoder subsystem with cached synthetic image features. Prompt permutation/location/type, hypernetwork dot-product and gradient checks. Output packs three 8 × 8 mask logits and three quality scores.
Read the model manifest ↗