Self-supervised vision: five label substitutes, and what each one costs

Self-supervised visual representation learning has produced five distinct answers to one question: when nobody labels anything, what stands in for the human label? Each answer is defined as much by what it cannot see as by what it learns, and the field's own sequence reads as a relay in which one substitute's blind spot becomes the next substitute's job. The five substitutes are not competitors and not a leaderboard; each is the previous one's shortfall answered, and each carries a price that can be stated in one sentence.

The label has to be manufactured, and the network learns whatever you manufactured

A hand-made label always teaches exactly the thing that was hand-made, so the fake task's own accuracy is the wrong score to optimise. The lecture opens with a picture rather than a definition: a child handling the physical world learns how it behaves without anyone annotating it.

Caption: The deck's own opening analogy for a self-supervised system: a small human learns a world model by interacting with the world and observing causes, with no external annotation. The picture carries no architecture, data or formula, and nothing about any specific method follows from it.

The analogy is where the difficulty starts rather than where it ends, because a machine has no hands. Something has to be turned into a label by a person, and the cleanest specimen of that act is a jigsaw puzzle: crop a 225×225 window from a photograph, cut it into a 3×3 grid, take one random 64×64 tile from each cell, reorder the nine tiles by a permutation drawn from a predefined set, and ask the network to name the index of that permutation.

Caption: The manufactured label drawn end to end. A car photo is cut by a 3×3 dashed grid and its cells numbered 1 to 9; a "Permutation Set" table pairs an index with a tile order (the visible row reads 64 → 9,4,6,8,3,2,5,1,7); nine parallel branches with shared weights take the reordered tiles, their features are concatenated at fc6, and a softmax produces a distribution over 64 permutation indices. The label is that index, and a person wrote down every part of how it is produced.Jigsaw

The evidence that this label is the wrong thing to optimise comes from the paper's own permutation-set ablation, which reports the puzzle score and the downstream score side by side for the same network.

Along the permutation-count axis the trend runs one way: six permutations are solved almost perfectly and transfer to only 49.7 mAP, while a thousand permutations drop to 71% and reach 53.2. That axis is not the whole story, though: the paper says the best performing permutation set is a trade-off between how many permutations are used and how dissimilar they are from each other, and squeezing the average Hamming distance down to 6.35 makes the task harder (62%) while transfer also gets worse (51.3). The pattern repeats in the shortcut ablation, where dropping the gap between tiles raises puzzle accuracy to 98% and drops detection mAP to 47.7, while three anti-shortcut measures together give 88% and 52.6. The authors compress the lesson into one sentence: "A good self-supervised task is neither simple nor ambiguous."Jigsaw

Once that is visible, the cost of the whole family is easy to state. Rotation prediction, relative patch position, jigsaw, inpainting and colourisation are not five variations on a theme that get progressively better; each one is a different hand-designed θ, a specific transform whose knobs (how many permutations, how large the gap between tiles, how much colour jitter) were all chosen by a person and tuned against the downstream task they cared about. The representation inherits that choice. The jigsaw authors say so themselves: a self-supervised system "might lead to representations that are suitable to solve the pre-text task, but not the target tasks", and they call those solutions shortcuts. A transfer result makes the inheritance concrete, though it also shows the ceiling: pretraining on the puzzle for 2.5 days on a single Titan X reaches 67.6% classification and 53.2% detection mAP on PASCAL VOC 2007, against 78.2% classification for full ImageNet supervision. The Doersch and Pathak rows in that same table are reproduced from Pathak et al., not measured by the jigsaw authors.Jigsaw

For a reader who wants names for kinds of substitute rather than mechanisms, one 2026 proposal offers a three-way split by the nature of the objective: alignment-based methods, which match representations of observed views; reconstruction-based methods, which recover masked input signals; and predictive representation learning, which predicts unobserved components in latent space.

Caption: One proposed taxonomy, drawn as a category tree. "Self Supervised Learning" fans out to "Alignment Based SSL" (which splits into "Contrastive" and "Non Contrastive"), "Reconstruction Based SSL", and a green-bordered "Predictive Representation Learning" whose only child is "JEPA style architectures". The figure carries category names only, so no method, date or accuracy can be read off it.PRL Survey

This is one paper's proposal, submitted in April 2026, and it stops there: nothing after April 2026 is covered, and neither DINO nor iBOT appears in its body, so any placement of the self-distillation family on that axis is an editorial inference rather than the paper's claim. The useful part is the axis itself, because it makes the next move legible. If the label is a hand-made θ, then the way out is to stop designing θ and start deciding something instead.

Same-or-different decisions replace the hand-made task, and put a price on negatives

Dropping the designed task costs a supply of negatives, and where those negatives come from becomes the next design decision in the relay. The replacement decision is minimal: are these two views of the same object or not, and push the same together while pushing the different apart. The positive pair is manufactured by augmentation rather than by a puzzle.

Caption: Where the label comes from once the task is gone. One image x splits by two augmentations sampled from the same family into two correlated views; a base encoder f maps each to h, a projection head g maps each to z, and the objective maximises agreement between the two z vectors. The head is discarded after training and h is what downstream tasks receive. The figure shows no loss, no temperature and no negatives; the fact that the rest of the batch supplies the negatives lives in the paper's prose.SimCLR

Written out once, the objective is the reader's own N-way cross-entropy with the positive as the target class:

$$\mathcal{L} = -\mathbb{E}\left[\log \frac{\exp s!\left(f(x), f(x^{+})\right)}{\exp s!\left(f(x), f(x^{+})\right) + \sum_{j} \exp s!\left(f(x), f(x_j^{-})\right)}\right]$$

The gloss is the paper's own: the loss is "the categorical cross-entropy of classifying the positive sample correctly", which is another way of saying the model is learning to pick the true sample out of N candidates. The information-theoretic reading follows in one sentence and is not derived here: minimising this loss maximises a lower bound on the mutual information between the context and the true future, $I(x_{t+k}, c_t) \ge \log N - \mathcal{L}_N$, and the bound tightens as N grows, which is the formal reason a large candidate set helps.CPC

Then the bill arrives in the same breath as the recipe. The negatives are not sampled: they are "the other 2(N−1) augmented examples within a minibatch", so the number of negatives is a function of the batch size and of nothing else. A batch of 8192 gives 16,382 negatives per positive pair, the paper sweeps batch sizes from 256 to 8192, and it trains on 32 to 128 TPU v3 cores depending on the batch size, with about 1.5 hours for a ResNet-50 at batch 4096. Memory is the binding constraint, and the paper reports no controlled footprint measurement, so the cost is an inference from those two facts rather than a measured number.SimCLR

The answer is to decouple the dictionary from the minibatch, which needs a queue and a second encoder that is only allowed to lag.

Caption: The picture in which the negative count stops being a function of the batch. The left branch is the single trainable encoder producing a query q from one input; the right branch is a momentum encoder producing keys k from many past inputs, kept in a queue with the current minibatch enqueued and the oldest dequeued. The two branches meet only at the similarity node that feeds the contrastive loss. The queue brace is not a count of negatives, and nothing in the figure implies that the momentum encoder is trained by back-propagation.MoCo

The mechanism is two lines. Keys are pushed into a FIFO queue and the oldest batch falls out; the key encoder is never updated by gradients but by

$$\theta_k \leftarrow m,\theta_k + (1-m),\theta_q$$

with m = 0.999 by default, and the key branch runs under a detach so gradients reach only the query encoder. The ablation shows why the lag matters: with m = 0 training fails to converge, and the reported sweep runs 55.2 at m = 0.9 up to 59.0 at m = 0.999 for a ResNet-50 under the frozen-feature linear protocol. The paper's own framing is a dictionary "much larger than a typical mini-batch size", set independently as a hyper-parameter, and the price of the alternative is stated in the same paper: on 8 Volta 32GB GPUs the largest end-to-end minibatch is 1024, and accuracy drops about 2 points without linear learning-rate scaling. That momentum encoder is the first appearance of a mechanism the rest of the arc keeps reusing: a copy of the network that trails the trained one, never updated by gradients, and useful precisely because it lags.MoCo

The same objective also has a second face, on the time axis. Contrastive Predictive Coding keeps the InfoNCE form but replaces the positive with the real future: an encoder maps each step to a latent, an autoregressive model compresses everything up to time t into a context vector, and a time-dependent score $s_k(z_{t+k}, c_t) = z_{t+k}^{\top} W_k c_t$ with a different $W_k$ per prediction step decides which candidate is the true successor. On a 100-hour LibriSpeech subset, a linear probe on the frozen context vector reaches 64.6% on 41-class phoneme classification against 27.6% for random initialisation and 74.6% supervised, and 97.4% on 251-speaker classification. Its image variant, which treats rows of overlapping patches as a sequence, reaches 48.7% ImageNet top-1 with a ResNet-v2-101, which the paper reports as a 9-point absolute top-1 gain over the state of the art of its time. That reading uses a different backbone and protocol from the later instance-contrast results, so the two are not directly comparable. Instance contrast and sequence contrast are still one objective sampled two ways rather than two rival schools, and this tutorial's reading — not a measured head-to-head — is that instance contrast wins on images because a second augmentation is a cheap, dense, free positive while a future frame is a scarce and sometimes ambiguous one.CPC

Reconstructed pixels make the label free, and blind to meaning

Masking makes the label free, but what you ask the model to produce at the masked patch decides whether it learns meaning or only texture. The pipeline is a denoising autoencoder rebuilt on a ViT: cut the image into patches, remove most of them, encode only the ones that remain, and predict the removed pixels from a small decoder.

Caption: Read the left edge as the label, the middle as the reason it is cheap, and the right edge as the answer key. A flamingo photo is cut into a 4×4 grid with most cells covered by grey placeholders; only the surviving tiles enter the encoder, which is why the encoder never sees the full sequence; the placeholders reappear after the encoder and a small decoder reconstructs the whole image, labelled "target". The only words printed are input, encoder, decoder and target, so the masking ratio, the loss and the asymmetry argument have to come from the text.MAE

The number that makes the asymmetry worth having is the masking ratio. The paper recommends 75%, against BERT's 15% and 20 to 50% in earlier vision work, because a high ratio removes redundancy and leaves a task that cannot be solved by extrapolating from neighbours. The asymmetry is what makes such a ratio affordable: if the encoder is fed mask tokens as well, linear-probing accuracy drops 14% and training FLOPs rise 3.3×, because the encoder spends pretraining looking at a token pattern that will never appear on a clean image. Wall-clock on 128 TPU-v3 cores for ViT-L at 800 epochs goes from 42.4 hours to 15.4 with the visible-only encoder and an 8-block decoder.

Then comes the turn, in the authors' own words: "our MAE reconstructs pixels, which are not semantic entities". Pixels are recorded light, with no decomposition into the visual analogue of words, so any semantics the features pick up is a by-product of a rich hidden representation rather than the thing being predicted. The repair keeps the mask and changes the answer key. In DINOv2's patch-level objective, the student sees a masked image and the teacher sees the unmasked one, and the target at a masked position is the teacher's own distribution over learned prototypes. The papers say prototype scores or teacher patch tokens; "semantic token" is a convenient shorthand rather than their vocabulary.

The two targets are the same act of filling a hole at two very different levels of abstraction (predicting pixels is diffing raw bytes, predicting the teacher's tokens is diffing the parsed syntax tree).MAEDINOv2

There is an evaluation consequence worth naming here, because it recurs in everything that follows. MAE's headline numbers are end-to-end fine-tuning numbers: 83.6 / 85.9 / 86.9 / 87.8 top-1 for ViT-B / L / H / H at 448. Under a linear probe on frozen features the same ViT-L reaches 75.8, below MoCo v3's 77.6, and the paper argues that linear separability is not the sole metric because fine-tuning a single block lifts accuracy from 73.5 to 81.0. A reader who compares the fine-tuned number to a linear-probe number from another paper is comparing two different measurements.MAE

A teacher that lags behind the student supplies the label, and scale erodes dense detail

A model can label itself from its own recent past, which removes the negative bill entirely and hands the field a different one: the features that get better at global recognition can get worse at dense prediction. DINO is the one figure in the whole arc where both of the article's recurring mechanisms are the same object, because the target of the similarity is produced by a slow copy of the network being trained.

Caption: Student and teacher share an architecture and differ only in parameters. One image branches into two views; the student path runs softmax to p1, the teacher path runs centering then softmax then a stop-gradient to p2, and an "ema" arrow runs from the student box to the teacher box. The loss between the two distributions is written at the top. There is no queue and no negative anywhere in the picture, and the figure carries no accuracy numbers.DINO

The update is the MoCo rule with the queue removed, $\theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s$ with λ following a cosine schedule from 0.996 to 1. "Slow" is a measurable design variable rather than a metaphor, and the paper measures it with teacher variants under k-NN top-1 on ViT-S at 300 epochs: using a straight copy of the student gives 0.1, using the previous iteration gives 0.1, using the previous epoch's weights gives 66.6, and the momentum teacher gives 72.8. A copy that updates instantly collapses; a copy that lags by an epoch almost works; a copy that lags continuously works best. On an engineering reading, this is a read replica that trails the primary and never writes back, and the lag is the feature: it answers with a snapshot close enough to agree with the primary but far enough behind to have smoothed out the noise the primary is still fitting.

Two teacher-side tricks hold the thing together and they fail in opposite directions. Centering stops one dimension from dominating but pushes the output toward uniform, sharpening does the reverse, and the paper reports that both are needed, and only work in the presence of a momentum teacher. What the objective buys was not asked for: the attention of the [CLS] token concentrates on objects, and thresholding those attention maps against ground-truth masks on PASCAL VOC12 gives a Jaccard of 45.9 for DINO against 27.3 for a supervised ViT and 22.0 for random weights.

Caption: [CLS] self-attention from a self-supervised ViT with 8×8 patches, drawn as eight photo and heat-map pairs. The heat concentrates on the object in all eight cases. These are attention maps, not segmentation masks and not class predictions, and the paper notes they are "smooth and not optimized to produce a mask", so the claim is that segmentation information is present in the features, not that the model understands objects. The measurement is the Jaccard figure; this strip is the illustration.DINO

The family's next node folds the missing pieces together. DINOv2 keeps the DINO image-level term, adds the iBOT masked patch-level term as the answer key for masked positions, and adds KoLeo, a nearest-neighbour regulariser that keeps features spread out. The ablation separates the jobs cleanly: removing the masked-patch term costs 3 points of ADE-20k mIoU (47.1 to 44.2) while leaving retrieval intact, and removing KoLeo costs 8 points of Oxford-M mAP (63.9 to 55.6) while leaving dense tasks intact. It trains on LVD-142M, 142,109,386 curated images, and the curation matters more than the count: at equal budget, the curated set scores 85.8 / 73.9 / 47.7 / 64.6 against 83.3 / 59.4 / 48.5 / 54.3 for a random uncurated sample of the same size. This is also where the family fixes its read-out: frozen backbone, linear probe on top, k-NN reported alongside, and the paper states that k-NN is the quantity it actually optimises.DINOv2

The third node is where adding scale makes something worse. DINOv3 takes the data from 142M to 1,689M curated images and the teacher from 1.1B to 6.7B parameters, ratios of roughly 12× and 6× that are derived from the two papers rather than printed in either. Global linear-probe accuracy on ImageNet-1k rises monotonically throughout training while VOC linear-segmentation mIoU peaks around 200k iterations and then declines, with the 7B model eventually falling below its own early level. The paper calls this dense feature degradation and diagnoses it as a real trade-off between learning discriminative features and keeping local consistency.

Caption: What a per-patch Gram matrix looks like, and what varying the Gram teacher buys. Panel (a) shows two photographs each followed by three square patch-similarity matrices labelled 256, "Downsamp." and 512, with the 512 and downsampled versions visibly smoother and more concentrated around the marked reference patch. Panel (b) is the ablation table: baseline 88.2 IN1k linear / 50.3 ADE mIoU / 0.307 NYU RMSE, and GRAM at a 200k-iteration teacher with two resolutions 88.0 / 55.7 / 0.281. The IN1k column is a linear-probe number, not a fine-tuned one, and a 1M-iteration Gram teacher is deliberately worse on dense metrics than a 200k one.DINOv3

The repair works on the geometry rather than on the features. Take an early checkpoint of the teacher as a frozen Gram teacher, compute the matrix of all pairwise dot products of L2-normalised patch features for both student and teacher, and penalise the gap:

$$L_{\text{Gram}} = \lVert X_S X_S^{\top} - X_G X_G^{\top} \rVert_F^2$$

Local features are then free to move as long as their similarity structure stays put, which is why ADE mIoU rises from 50.3 to 55.7 while the ImageNet linear number stays flat at 88.0. Two details matter for anyone quoting this: the penalty is a squared Frobenius norm, and the teacher has to be early, because a 1M-iteration Gram teacher has itself lost the patch-level consistency being anchored.

Borrowed labels from text survive distribution shift, and a fitted head hands that robustness back

Language moves the label source outside the image, and that changes what a reported number means: the classifier nobody trained is the one that survives a distribution shift, and the classifier you fit buys ImageNet back at a measurable price. Nothing is manufactured here, because the words were already written down next to the image.

Caption: Three panels, three stages of one argument. The left panel is the status quo being left behind, an image backbone feeding a linear classifier over a fixed label set. The middle panel is the training signal, a batch of N image-text pairs whose image features and text features land in one shared space, with the N×N cosine matrix and its diagonal marked as the target. The right panel is the consequence for evaluation: the text encoder receives prompts such as "a photo of a dog" and the argmax over cosine similarities returns the label, so the read-out is a sentence rather than a trained layer.CLIP

The scale is what makes the borrowed label usable: 400 million (image, text) pairs, built from about 500,000 queries capped at 20,000 pairs each. For contrast, the paper puts MS-COCO and Visual Genome at roughly 100,000 training photos each, and notes that YFCC100M's 100 million photos shrink by a factor of six to 15 million once filtered for usable natural-language descriptions, which is about ImageNet's size. The objective is the same N-way cross-entropy as before, made symmetric: both encoders project into one L2-normalised space, an N×N matrix of scaled cosine similarities is computed, cross-entropy is applied along both axes, and the two are averaged, $\mathcal{L} = (\mathcal{L}_i + \mathcal{L}_t)/2$, with a temperature that is learned rather than set. Two design choices are instructive by contrast with the second movement: the projection is linear rather than the nonlinear head SimCLR needed, because at 400M pairs overfitting is not the concern, and the batch itself supplies the negatives, at 32,768.

The read-out is where the claim actually lands, and it requires correcting a number that circulates widely. CLIP's 76.2% ImageNet top-1 is a zero-shot figure, produced by a classifier that was never trained. Fit a linear probe on the same frozen features and the number rises to 85.4. The two are different measurements of the same model, and reporting one as the model's accuracy without saying which is the mistake this whole arc keeps inviting.

Fitting the head buys 9.2 points on ImageNet and gives back robustness in every shifted column that matters: 1.9 points on IN-A, 2.8 on IN-Sketch, 6.1 on ObjectNet (72.3 against 66.2 in the table above) and 4.7 on IN-R. The paper's own summary is that zero-shot CLIP improves the state of the art on five of seven natural distribution shifts, and that the gap between ImageNet accuracy and accuracy under natural distribution shift narrows by up to 75%. The reason is structural rather than incidental. A classifier synthesised from class names was written by the world, so it travels with the world; a classifier fitted on one dataset's training distribution learns that distribution's peculiarities along with its classes.CLIP

A comparison column that circulates in teaching material lists 64.3 / 37.7 / 32.6 / 25.2 / 2.7 and a 77.1 for IN-A. None of those values appear in the paper's robustness table, and the paper's IN-A figure is 77.2. The CLIP-side numbers above are the paper's; the comparison column is not verifiable from it.

One measuring stick judges all five, and only two mechanisms ever recur

The numbers that judge these methods are chosen rather than given, and once that is clear the five movements collapse into two mechanisms, which is enough to place any new method in one cell. The ruler every method above has been judged by is simple to state: pretrain without labels, then freeze the backbone and fit a linear probe or fine-tune it, then read classification, detection and segmentation. Nobody grades the fake task, because the fake task's own accuracy runs in the opposite direction to the score anyone cares about, as the permutation ablation showed. That is why one ruler applies to all five substitutes.

The ruler is itself a choice, and one 2026 study measures the alternative. Pretrain-then-finetune (PFT) runs two sequential stages; joint training (JT) optimises the self-supervised and supervised losses together in one loop, $L_{\text{total}} = L_{\text{SSL}} + L_{\text{sup}}$. Across eight SSL methods, eleven datasets and four task types at 10 / 20 / 50 / 100% labels, the verdict is conditional, and the split tracks the method family rather than the dataset.

Reconstruction-oriented and auxiliary-task methods, meaning MAE, Rotation and Colourisation, generally gain from joint training; contrastive and redundancy-reduction methods, meaning MoCo, SimCLR, DINO and Barlow Twins, generally transfer better under PFT. The paper is careful to say its relative verdict depends on task, label availability and domain complexity, and its own PFT arm is not uniform: classification, image quality assessment and two of the segmentation sets use a frozen encoder, COCO detection fully fine-tunes YOLOv12, and JSRT initialises from SSL weights. There is no single clean gap to quote, which is exactly the point. The ruler is a methodological convention, not a law.PFT-vs-JT

With that settled, the five movements fit in four cells. The axes are the two things that keep recurring: a similarity score raised on positives and lowered on negatives, and a teacher that is a lagging copy of the student.

DINO sits in the top-right cell, and it is the only method where both mechanisms are literally one object: the similarity is pushed up between two views of the same image and never pushed down on anything, because the training signal is the teacher's own output. MAE sits in the bottom-right, with no negatives and no teacher, which is why its ceiling was set by what pixels can express. Language supervision is the one entry that does not fit the relay at all, since its label never came from the image, and it lands in the bottom-left cell structurally, as a symmetric contrastive objective whose negatives are the other pairs in the batch.

The lagging copy has also survived to foundation scale, and that is the strongest 2026 evidence available for the second mechanism.

Caption: One picture carries the chain from a multi-view bound to an actual loss. The top band states the inequality and its decomposition into per-patch information minus total correlation; the middle band is the teacher-student pair over shared patch grids, with a dashed EMA arrow running from the student encoder to the teacher encoder and outputs p and q; the right band is three numbered loss panels for alignment, a global information term and a structural term. The dashed EMA arrow is the same lagging copy that DINO's teacher uses, now drawn as one arrow among three loss paths; the figure contains no benchmark number and no scale.JEM

JEM trains a patch-level student-teacher objective stably at 303M, 1.13B and 6.7B parameters, with a teacher that is an EMA of the student at momentum 0.994 and no negatives anywhere, and it evaluates everything with the same frozen-backbone probe discipline this article has used throughout. Its own ViT-L ablation keeps the claim honest: removing the EMA teacher changes ImageNet accuracy by less than a point, 83.0 against 82.8, so what the evidence supports is that a scalable lagging-copy recipe exists, not that the lagging copy is individually indispensable. That distinction is the difference between describing a mechanism and over-claiming one.

Nine sub-topics, read in an order that respects what each inherits, look like this.

Two regroupings of that order are safe rather than arbitrary. The temporal work belongs together, so video colourisation and sequence contrast read better as one sitting than as two separated stops, because both are about correspondence across time. And the puzzle era's verdict belongs next to the switch it triggers, because the permutation ablation only becomes a lesson at the moment contrastive learning answers it. Both are reading advice, not findings from any paper.

What survives the whole arc is smaller than the list of methods: five substitutes, each priced in the specific thing it cannot see, one ruler that is itself a choice, and two mechanisms wearing different clothes. That is enough apparatus to place a new method on sight. Ask what it uses instead of a label, ask what that substitute structurally cannot see, and ask which cell of the map it lands in.

References

1. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles — arXiv:1603.09246
2. Representation Learning with Contrastive Predictive Coding — arXiv:1807.03748
3. A Simple Framework for Contrastive Learning of Visual Representations — arXiv:2002.05709
4. Momentum Contrast for Unsupervised Visual Representation Learning — arXiv:1911.05722
5. Masked Autoencoders Are Scalable Vision Learners — arXiv:2111.06377
6. Emerging Properties in Self-Supervised Vision Transformers — arXiv:2104.14294
7. DINOv2: Learning Robust Visual Features without Supervision — arXiv:2304.07193
8. DINOv3 — arXiv:2508.10104
9. Learning Transferable Visual Models From Natural Language Supervision — arXiv:2103.00020
10. Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training? — arXiv:2607.13192
11. Scalable Patch-Level Self-Supervised Learning — arXiv:2610.10013
12. From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning — arXiv:2604.13518