CLIP: Connecting Vision to Language at Scale
TL;DR: Train an image encoder and a text encoder to map matching (image, text) pairs close together in a shared embedding space, using contrastive learning on 400 million internet-sourced pairs. The result: a visual representation that can be queried in plain English. Zero-shot CLIP (no task-specific training) matches a supervised ResNet-50 on ImageNet, closes the robustness gap by up to 75% on distribution-shifted benchmarks, and matches 4-shot learning on its own features. The method is 13 lines of pseudocode.
These paper reviews are written more for me and less for others. LLMs have been used in formatting
This post draws from Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021).
Why Natural Language Supervision
Prior work described near-identical approaches as unsupervised, self-supervised, weakly supervised, and supervised. CLIP’s argument: the label doesn’t matter — what’s common is using natural language as a training signal.
Two advantages over alternatives:
- Scales easier than crowd-labeling. You don’t need annotations in a “machine learning compatible format” — you learn passively from text that already exists alongside images.
- Connects representation to language. Unlike SimCLR or other self-supervised methods that learn good embeddings you still can’t query, CLIP produces embeddings queryable in English. This is what enables zero-shot transfer.
The Dataset: WIT
Existing options were insufficient — MS-COCO and Visual Genome have ~100K images each (too small), and YFCC100M’s metadata is mostly autogenerated filenames and camera settings (after filtering to images with actual English descriptions, it shrank 6x to 15M).
So they built WIT: 400 million (image, text) pairs from public internet sources. The construction is deliberate:
- Search for pairs whose text includes one of 500,000 queries — all words occurring ≥100 times in English Wikipedia, augmented with high-PMI bigrams and all Wikipedia article names
- Cap at 20,000 pairs per query for approximate class balance
- Total word count roughly matches WebText (GPT-2’s training data)
The query-list construction is a conscious attempt at broad visual-concept coverage rather than scraping whatever’s popular, and the 20K cap is a cheap balancing mechanism.
The Method
The entire method fits in pseudocode:
1
2
3
4
5
6
7
8
9
10
11
12
I_f = image_encoder(I) # [n, d_i]
T_f = text_encoder(T) # [n, d_t]
I_e = l2_normalize(np.dot(I_f, W_i), axis=1) # [n, d_e]
T_e = l2_normalize(np.dot(T_f, W_t), axis=1) # [n, d_e]
logits = np.dot(I_e, T_e.T) * np.exp(t) # [n, n]
labels = np.arange(n)
loss_i = cross_entropy_loss(logits, labels, axis=0)
loss_t = cross_entropy_loss(logits, labels, axis=1)
loss = (loss_i + loss_t) / 2
Two encoders, two linear projections, L2 normalization, a scaled dot product, and symmetric cross-entropy with the diagonal as labels. In a batch of $N$ pairs, you have $N$ positives and $N^2 - N$ negatives.
The image encoder is either a modified ResNet or a ViT. The text encoder is a GPT-2-style transformer (12 layers, 512-wide, 63M params). The [EOS] token’s final-layer activation is the text representation.
Why contrastive beats generative here: predicting the exact words of accompanying text is very hard — a huge variety of descriptions co-occur with images. Predicting only which text as a whole pairs with which image is a much easier proxy that still forces understanding of image content. Prior findings they cite: generative image models need over an order of magnitude more compute than contrastive models for the same representation quality.
Simplifications that work at scale (all justified by dataset size making overfitting a non-concern): no pre-trained initialisation (trained entirely from scratch), only a linear projection instead of the nonlinear head SimCLR uses, and image augmentation reduced to a single random square crop.
Training uses a batch size of 32,768 — enormous, because batch size directly determines the number of negatives in the contrastive loss. The flagship model (ViT-L/14@336px) trained for 12 days on 256 V100s.
Prompt Engineering: +5% for Free
An unexpectedly large practical contribution.
Problem 1 — polysemy. A bare class name has no context. ImageNet contains construction cranes and cranes that fly. Oxford Pets has a “boxer” that’s a dog breed, not an athlete.
Problem 2 — distribution gap. In WIT, text is usually a full sentence, not a single word. A bare class name is off-distribution for the text encoder.
Fixes:
- Default template “A photo of a {label}.” → +1.3% on ImageNet by itself
- Task-specific customisation: “a type of pet” for Oxford Pets, “a satellite photo of” for satellite imagery, quotes around text for OCR datasets
- Ensembling over multiple prompts (“A photo of a big {label}”, “…small {label}”, etc.), averaged in embedding space — so the ensemble costs the same as a single classifier at inference since you cache one averaged text embedding. They ensemble 80 prompts on ImageNet for +3.5%
- Combined: nearly +5% — comparable to the gain from 4x more compute, but essentially free
Zero-Shot Transfer
CLIP’s definition of zero-shot is broader than usual: not just unseen categories, but unseen datasets — a proxy for unseen tasks.
vs. supervised ResNet-50 linear probe (27 datasets): CLIP wins on 16 of 27. The pattern is informative:
- CLIP dominates on tasks where language provides rich supervision — action recognition, geo-localisation, fine-grained categories people actually write about (cars, food)
- CLIP loses on specialised, abstract, or synthetic tasks — satellite imagery, medical imaging, counting, traffic signs, distance estimation. Nobody captions internet photos with “this is a lymph node metastasis”
vs. few-shot learning: zero-shot CLIP matches 4-shot logistic regression on its own features. The explanation: the zero-shot classifier is generated via language, which lets visual concepts be directly specified. Few-shot learning must infer concepts from examples, and with few examples many hypotheses are consistent with the data.
Data efficiency: a median of 5.4 labelled examples per class are needed to match zero-shot CLIP, but the range is wide — from <1 (Flowers102) to 184 (FER2013).
Robustness: The Most Important Section
Setup: 7 natural distribution shifts from ImageNet (ImageNetV2, Sketch, ObjectNet, ImageNet-A, ImageNet-R, etc.). A standard ResNet-101 makes 5x as many mistakes on these shifts as on ImageNet validation.
Headline comparison — ResNet-101 vs zero-shot CLIP, both at exactly 76.2% on ImageNet:
| Dataset | ResNet-101 | Zero-Shot CLIP |
|---|---|---|
| ImageNet | 76.2 | 76.2 |
| ImageNetV2 | 64.3 | 70.1 |
| ImageNet Sketch | 25.2 | 60.2 |
| ObjectNet | 32.6 | 72.3 |
| ImageNet-A | 2.7 | 77.1 |
| ImageNet-R | 37.7 | 88.9 |
Same in-distribution accuracy, dramatically different out-of-distribution. CLIP closes the robustness gap by up to 75%. ImageNet-A is the extreme: 2.7% vs 77.1%.
The genuinely surprising follow-up: adapt CLIP to ImageNet by fitting logistic regression on CLIP features using the ImageNet training set. ImageNet accuracy jumps +9.2% to 85.4% — but average accuracy under distribution shift slightly decreases. The paper’s honest response: “How is it possible to improve accuracy by 9.2% on ImageNet with little to no increase in accuracy under distribution shift? … We do not have confident answers to these questions at this time.”
The continuum: as you go from 0-shot → 1-shot → … → fully supervised, effective robustness fades monotonically. Their summary: high effective robustness seems to result from minimizing the amount of distribution-specific training data — at the cost of dataset-specific performance.
The MNIST Confession
CLIP scores 88% on MNIST — worse than logistic regression on raw pixels. Nearest-neighbour retrieval confirms almost nothing resembling handwritten digits exists in the training data.
The paper’s statement is unusually clear-eyed: “CLIP does little to address the underlying problem of brittle generalization of deep learning models. Instead CLIP tries to circumvent the problem and hopes that by training on such a large and varied dataset that all data will be effectively in-distribution. This is a naive assumption that, as MNIST demonstrates, is easy to violate.”
Bias: Documented, Not Hand-Waved
Evaluated on FairFace with denigration probes (adding classes like ‘criminal’, ‘animal’, ‘gorilla’ alongside real demographic classes):
- 4.9% of images misclassified into a non-human class. Black images had the highest rate at ~14%.
- 16.5% of male images misclassified into crime-related classes vs 9.8% of female.
- People aged 0-20 had the highest misclassification rates across both categories (~14-18%).
- Adding a ‘child’ class dramatically reduced under-20 misclassification (30.3% → 2.3%) — demonstrating that class design is a key determinant of both performance and harm, and with CLIP any developer can define classes freely.
The paper notes that FairFace’s categories are themselves reductive, that benchmark fairness metrics are a weak proxy for real-world fairness, and that higher accuracy on underrepresented groups can paradoxically justify deployments that then harm those groups.
Limitations the Paper States
- Zero-shot CLIP is competitive with a linear probe on ResNet-50 — well below SOTA. Reaching SOTA zero-shot would need ~1000x more compute.
- Data efficiency: 12.8 billion images seen over 32 epochs. At one image per second, that’s 405 years.
- Limited to choosing among concepts you put in the classifier — unlike captioning, which can generate novel outputs.
- The few-shot drop from zero-shot remains unexplained.
- The 27-dataset evaluation suite is “somewhat haphazardly assembled” and “undeniably co-adapted with CLIP’s capabilities” — they call for a proper zero-shot benchmark.
Key Takeaways
- Contrastive learning on 400M (image, text) pairs produces a visual representation queryable in plain English — enabling zero-shot transfer to new tasks via prompt engineering.
- The method is remarkably simple: two encoders, a dot product, and symmetric cross-entropy. The scale of the data does the heavy lifting.
- Prompt engineering (+5% from templates and ensembling) is a practical contribution as important as the architecture.
- The robustness finding is the paper’s strongest result: same ImageNet accuracy as a ResNet, but dramatically better under distribution shift — suggesting that language supervision produces more general representations than class-label supervision.
- Adapting to in-distribution data (even linearly) trades away robustness — a finding the authors flag as important but unexplained.
- CLIP is honest about its failures: 88% on MNIST, poor on specialised domains, and documented bias in denigration probes. The MNIST confession — “this is a naive assumption” — is worth internalising.