ViT: An Image Is Worth 16x16 Words
TL;DR: Cut an image into 16x16 patches, flatten them, project to embeddings, prepend a [CLS] token, add position embeddings, and feed the sequence to a standard transformer encoder. That’s ViT — a vision model with almost no image-specific inductive bias. The central finding: ViT underperforms CNNs when trained on small datasets (the convolutional inductive bias helps), but overtakes them when trained on large datasets (the bias eventually hurts). At scale, ViT matches or beats the best CNNs at 2-4x less compute.
These paper reviews are written more for me and less for others. LLMs have been used in formatting
This post draws from An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Dosovitskiy et al., 2021).
The Method
Take an image of size $H \times W \times C$. Reshape it into $N$ non-overlapping patches of size $P \times P$:
\[N = \frac{H \times W}{P^2}\]Each patch is flattened into a vector of dimension $P^2 \cdot C$ and linearly projected to the model’s hidden dimension $D$. These are the patch embeddings — the “words” in the sequence.
Concretely for ViT-B/16 at 224×224: $N = 196$ patches, each of dimension $16 \times 16 \times 3 = 768$, projected to $D = 768$. With a prepended [CLS] token, the sequence length is 197.
A critical scaling fact: sequence length is inversely proportional to the square of the patch size. ViT-L/16 processes 4x more tokens than ViT-L/32, with quadratically more attention cost. The “/16” vs “/32” in model names is a compute knob.
The rest is a standard transformer encoder: multi-head self-attention, MLP with GELU, pre-norm (LayerNorm before each block), residual connections. Position embeddings are learnable 1D vectors added to the patch embeddings. The [CLS] token’s final representation goes through a classification head.
The model configurations are taken directly from BERT: Base (86M), Large (307M), and a new Huge (632M).
The Inductive Bias Argument
This is the intellectual core of the paper. The authors state it precisely:
In CNNs, locality, 2D neighborhood structure, and translation equivariance are baked into every layer. In ViT, the self-attention layers are global from the start — every patch can attend to every other patch in every layer. The 2D structure of the image is used only twice: once when cutting the image into patches, and once when interpolating position embeddings for different resolutions during fine-tuning.
Other than that, the position embeddings at initialisation carry no information about the 2D positions of the patches. All spatial relations must be learned from scratch.
The empirical finding: the model does learn them. The learned position embeddings converge to encode 2D topology — nearby patches end up with similar embeddings, and row/column structure emerges naturally. This is why hand-crafted 2D-aware embeddings (which they tried) don’t help — the model already discovers the structure.
The reframe: an inductive bias isn’t universally “good.” It’s a bet that a certain structure holds — useful when you lack the data to learn it yourself, and costly when you don’t.
The Data-Scale Crossover
The paper’s central experiment: pre-train on datasets of increasing size (ImageNet 1.3M, ImageNet-21k 14M, JFT 303M), then fine-tune and compare ViT against ResNets.
On ImageNet (1.3M images): ViT-Large underperforms ViT-Base, despite regularisation tuning. Big models overfit small data.
On ImageNet-21k (14M): Base and Large perform similarly.
On JFT (303M): the full benefit of larger models shows — ViT overtakes ResNets.
A controlled version of this experiment (train on random JFT subsets of 9M/30M/90M/300M, same hyperparameters, no additional regularisation) confirms: ViT overfits more than ResNets at comparable compute on small data, but wins at 90M+.
The paper’s conclusion: “the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial.” That “even beneficial” is the strong claim — the bias eventually hurts.
The Hybrid Finding
Instead of raw image patches, you can form the input sequence from CNN feature maps — run a ResNet first, then feed its spatial features as tokens to a ViT. This is the hybrid variant.
Finding: hybrids beat pure ViT at small compute budgets, but the advantage vanishes as models get larger. The paper calls this “somewhat surprising, since one might expect convolutional local feature processing to assist ViT at any size.” Another data point for “learned beats designed at scale.”
Compute Efficiency
When plotting transfer accuracy vs total pre-training compute across 7 ResNets, 6 ViTs, and 5 hybrids (all pre-trained on JFT), three findings emerge:
- ViT dominates on performance/compute — approximately 2-4x less compute for the same accuracy.
- Hybrids slightly outperform at small compute, but the gap vanishes for larger models.
- ViT does not saturate within the range tested — motivating further scaling.
The practical compute story matters: ViT-L/16 on ImageNet-21k (publicly available) trains in ~30 days on a standard 8-core cloud TPUv3 — reproducible outside Google, which mattered enormously for adoption. The equivalent ResNet (BiT-L) requires ~15x more compute.
What the Model Learns
Three things emerge from inspecting trained ViTs:
Attention distance. Some heads attend to most of the image already in the lowest layers — structurally impossible for a CNN, which expands its receptive field gradually over depth. Other heads stay consistently local. In the second half of the network, most heads attend widely. The local heads in early layers serve the same function as early convolutional layers — in hybrid models that apply a ResNet first, this local attention is less pronounced.
Position embeddings. The model learns to encode distance within the image — closer patches get more similar embeddings. Row/column structure and sinusoidal patterns emerge naturally. This is why hand-crafted 2D-aware embeddings provide no benefit — the 14×14 grid is small enough that any reasonable scheme learns the spatial relations equally well.
Fine-tuning at higher resolution. Keep the patch size fixed, get a longer sequence. The pre-trained position embeddings no longer match the new grid, so they’re 2D-interpolated to the new positions. This and the initial patch extraction are the only two points where a 2D bias is manually injected — the paper is precise about this.
Self-Supervision (a Preview)
The paper includes a preliminary experiment: masked patch prediction, mimicking BERT’s masked language modeling. ViT-B/16 achieves 79.9% on ImageNet — a +2% improvement over training from scratch, but still 4% behind supervised pre-training.
They left contrastive pre-training to future work. That gap is exactly what MAE, DINO, and BEiT later closed — turning ViT into a strong self-supervised learner and removing the dependence on labelled data.
A Methodological Lesson
An initial attempt at replacing the [CLS] token with global average pooling (ResNet-style) performed very poorly. Investigation showed the failure was entirely explained by needing a different learning rate — at their respective optimal LRs, both work similarly well.
This is a clean reminder: an architectural ablation that isn’t learning-rate-tuned per configuration can produce a completely spurious conclusion. Worth remembering whenever you read ablation tables.
Key Takeaways
- ViT applies a standard transformer to sequences of image patches with minimal vision-specific inductive bias.
- The central finding: convolutional inductive bias helps on small datasets, hurts on large ones. At sufficient data scale, learning beats designing.
- ViT achieves 2-4x better compute efficiency than comparable ResNets on transfer tasks.
- The model spontaneously learns 2D spatial structure, local-then-global attention patterns, and patch-level basis functions — all without being told to.
- Hybrids (CNN + ViT) help at small scale but add nothing at large scale.
- The self-supervision gap (4% behind supervised at the time) was the open problem that MAE/DINO subsequently solved.
- ViT-B/L/H configurations are borrowed from BERT — the architecture is literally a text transformer applied to image patches.