Deeper yet trains worse: five fixes from init to transfer

Stack more layers onto a plain convolutional network and the training error can go up. That sentence is strange enough to deserve a second look. More parameters, more capacity, yet the network fits its own training set worse than a shallower twin. Deep CNN training is full of failures shaped like this: not "the model is too small" or "the model is overfitting", but "the numerical problem the optimizer was handed is badly conditioned, so it cannot make progress".

This tutorial is about the sequence of fixes that turned deep CNNs from an aspiration into a routine, and about the single pattern behind all of them. Each fix takes a badly conditioned or high-variance learning problem and swaps it for a better-behaved one. The five fixes run in a fixed order, from initialization through architecture, optimizer, regularization, and finally transfer, and each one exposes the next.

Why a CNN trains at all, and why the answer is five separate fixes

Most people meet CNNs through convolution arithmetic: a filter slides over an image, produces an activation map, and a stack of those layers builds up features. The mechanics are well understood by now. The harder question is why any of it trains at all. A network with millions of parameters sitting on a non-convex loss is not obviously going to cooperate, and in the years before the fixes below it often did not.

Start with the simplest failure, from a one-hidden-layer network that uses sigmoid activations. Sigmoid outputs live between 0 and 1, and its derivative peaks at 0.25 near the origin and collapses toward zero at both tails. A unit whose pre-activation is large in magnitude is saturated: it outputs almost 0 or almost 1, and the gradient flowing back through it is nearly zero. A top layer that pins itself to 0 at the start of training does not merely produce a bad value. It blocks the gradient that the layers below need in order to learn. This is a numerical failure with a clean shape: the signal that should flow backward is dead. Glorot and Bengio documented exactly this pattern in sigmoid networks, where the top hidden layer drifts into the lower saturation regime almost immediately and the depth-5 model never escapes it during trainingXavier.

The same recap exposes a second, subtler failure. Sigmoid outputs are always positive, so every activation feeding a weight is non-negative. During backpropagation the gradient with respect to each weight is proportional to the incoming activation. If all incoming activations share a sign, then every weight in that layer gets pushed in the same direction on a given step, and the only way to correct an overshoot is to overshoot on the next step. The optimization zig-zags instead of settling.

Two distinct problems, and neither is about capacity. Both are about the conditioning of the problem the optimizer is asked to solve. That distinction is the spine of everything that follows. Deep CNN training is not one clever optimizer trick. It is a stack of five conditioning decisions, and each one repairs a different numerical failure.

Here is the whole map before any single stop is unpacked:

The first decision is signal conditioning: keep activations and gradients alive so the network can learn at all. The second is architecture: change how depth is parameterized so that a deep model stays optimizable. The third is the optimizer: shape noisy and unevenly-scaled gradients into stable steps. The fourth is generalization: close the gap between training error and error on data the network has never seen. The fifth is transfer: when the target dataset cannot support training from scratch, reuse a representation someone else already trained.

Each stage follows the same three-beat rhythm. Name a failure, apply a conditioning move, and hand a sharper question to the next stage. Fixing the initialization is what makes depth affordable, and that immediately raises the question of how to make depth train well. If a better optimizer keeps lowering the training loss, it guarantees that the model's error on unseen data also decreases. And when none of the conditioning helps because there is simply not enough target data, the remaining move is to reuse a backbone. Trainability, in other words, is a stack of five conditioning decisions, and the same fix-a-numerical-failure pattern recurs at every stage.

Keeping activations alive: from initialization to batch normalization

The first stop is making sure the signal survives the forward and backward passes. Initialization sets this up statically, at step zero. Batch normalization maintains it dynamically, for the whole training run. Together they form the signal-conditioning stage.

Initialization is not a cosmetic detail, because the scale of the weights decides whether the signal grows or shrinks as it travels through the stack. Consider a single dense layer with input $x$, weight vector $w$, and no activation, so that $y = \sum_{k} x_k w_k$. Under the assumption that the inputs and weights are independent and identically distributed with zero mean, the variance of the output is

$$\mathrm{Var}(y) = D_{in},\mathrm{Var}(x),\mathrm{Var}(w).$$

Read that line as a compounding statement. Every layer multiplies the variance of its input by $D_{in},\mathrm{Var}(w)$, where $D_{in}$ is the number of inputs to the layer. Stack $L$ layers and the activation variance is the input variance multiplied by that factor $L$ times. If the factor is below 1 the signal decays toward zero by the time it reaches the output. If it is above 1 the signal explodes. Either way, the deep end of the network stops carrying usable information, and the backward gradient is squeezed by the same factor in reverse.

The fix is to force that per-layer factor to be 1, which requires $\mathrm{Var}(w) = 1/D_{in}$, or equivalently $\mathrm{std}(w) = 1/\sqrt{D_{in}}$. That is the simple equal-width form of the rule the deck uses. The full proposal in the paper is a compromise between the forward constraint $n_i,\mathrm{Var}(W^i) = 1$ and the backward constraint $n_{i+1},\mathrm{Var}(W^i) = 1$, which cannot both hold when the layers have different widths. The compromise is $\mathrm{Var}(W^i) = 2/(n_i + n_{i+1})$, giving the uniform initialization $W \sim U(-\sqrt{6/(n_i + n_{i+1})}, \sqrt{6/(n_i + n_{i+1})})$Xavier. When all layers share a width the two constraints coincide and the simpler $1/\sqrt{D_{in}}$ form is exact. For ReLU the same derivation needs a factor of two, because half the units are inactive at initialization; that is the He initialization extension, and it is the reason modern ReLU networks use a scale twice as large as the tanh-era rule.

It is worth seeing why the compounding is the whole story. If the per-layer factor wanders off 1, the signal is repeatedly multiplied by a gain as it travels through the stack, the way a running value fed back into a loop grows without bound when the loop multiplies by 1.1 and decays to nothing when it multiplies by 0.9. The starting value barely matters; the per-layer gain decides everything. That is precisely why the target is a per-layer factor of exactly 1 rather than "small weights" or "large weights".

Why naive initialization breaks depth is visible directly. The figure below overlays the distribution of activations in each of five layers of a tanh network, once under standard initialization and once under the normalized scheme.

Caption: With standard initialization (top panel) the five layers' activation histograms march apart as depth increases: the first layer stays broad, while deeper layers collapse into taller and narrower spikes pinned near zero. With normalized initialization (bottom panel) all five layers sit on top of one another as a single bell centered near zero. This is the visual form of the failure the variance argument predicts: under the wrong scale the signal concentrates rather than propagates, and the deeper the network the worse it gets.

The bottom panel is what "the signal survives" looks like. Every layer sees an input with roughly the same spread, so no layer is starved and no layer is saturated.

Preserving variance at initialization fixes the distribution at step zero, but the distribution does not stay put. As the parameters of earlier layers change during training, the input distribution seen by each later layer drifts too. Ioffe and Szegedy named this drift internal covariate shift and proposed a direct, learnable remedy: normalize each layer's pre-activations to zero mean and unit variance, then let the network learn how much to undoBatchNorm. The transform is four lines. Compute the mini-batch mean $\mu_B$; compute the mini-batch variance $\sigma^2_B$; normalize $\hat x_i = (x_i - \mu_B)/\sqrt{\sigma^2_B + \epsilon}$; then apply a learned scale and shift $y_i = \gamma \hat x_i + \beta$.

The learned $\gamma$ and $\beta$ matter more than they look. Because the network can set $\gamma = \sqrt{\mathrm{Var}(x)}$ and $\beta = \mathbb{E}[x]$, batch normalization can represent the identity transform, so it does not remove any representational capacity. It reparameterizes the layer, it does not restrict it. The layer is placed after the linear or convolutional operation and before the nonlinearity, because $W u + b$ has a more symmetric, more Gaussian distribution than the output of a previous nonlinearity, which is exactly the quantity you want to standardize. Since the mean is subtracted anyway, the bias $b$ is redundant and can be dropped.

Batch normalization pays off because it holds the distribution flat instead of letting it drift. The figure below shows what it is actually fixing.

Caption: Each panel tracks the 15th, 50th, and 85th percentiles of one sigmoid input over the course of training, on a small MNIST network with a sigmoid activation. Without batch normalization (left) the percentile bands wander as training proceeds. With it (right) the same bands stay roughly flat. The bands are a running picture of the input distribution each layer receives, and flat bands are the direct picture of the "make the distribution fixed" principleBatchNorm.

The practical consequences are large. In the paper's ImageNet experiments, batch normalization let the authors raise the learning rate by 5 to 30 times and reach the Inception baseline accuracy in 14 times fewer steps, because each step no longer had to fight the drift of the input distribution. The paper's best ensemble reports 4.82% top-5 error on the ILSVRC test set, a number the paper describes as exceeding the estimated accuracy of human raters on that benchmark. mixup

There is one more asymmetry to keep, because it is the first appearance of a pattern that returns twice more. During training, the normalization statistics come from the current mini-batch, so each example's normalization depends on the other examples in its batch. At inference there are no batches to speak of and no randomness wanted, so the transform is frozen: the layer uses population statistics $\mathbb{E}[x]$ and $\mathrm{Var}[x]$ accumulated as running averages during training, and the whole operation collapses to a fixed linear map that can be folded into the preceding layer. Training uses per-batch statistics, inference uses fixed running statistics. The first fix, then, is to make the distribution fixed: at initialization through the weight scale, and during training through normalization.

Depth: when more layers make training worse, and the reparameterization that fixes it

The second stop is architecture. Even with the signal conditioned, plain depth has its own failure mode, and it is the counterintuitive one from the opening: a deeper network can have higher training error than a shallower one.

This is the degradation problem, and the shape of the evidence matters. Take a plain stack of convolutional layers, train it at 20 layers and again at 56 layers on CIFAR-10, and watch the curves. The 56-layer network has higher training error throughout, not just higher test error. That single fact rules out overfitting as the explanation. A model that overfits fits the training set better, not worse. What is happening is an optimization failure: adding layers made the optimization problem harder to solve even though the deeper network contains the shallower one as a special caseResNet. If the deeper network could simply learn the identity for the extra layers, it should never do worse. The problem is that it cannot easily find that solution.

ResNet's move is to change what the layers are asked to fit. Instead of asking a block to learn the desired mapping $H(x)$ directly, ask it to learn the residual $F(x) = H(x) - x$, and recover the original mapping through an identity shortcut:

$$y = F(x, {W_i}) + x.$$

When the best thing for a block to do is nothing, driving the residual branch to zero is easy: push the weights toward zero and the block becomes the identity. Fitting the identity through two stacked convolutional layers with their own nonlinearities, by contrast, is not something gradient descent does quickly. The reparameterization does not change the set of functions the network can represent. It changes how hard each function is to reach. This is a conditioning move on the parameterization, and it is why depth becomes affordable.

Why depth comes at essentially no cost is visible in the two figures below. The first shows a single residual block.

Caption: The block takes an input $x$, runs it through a stack of weight layers to produce $F(x)$, and adds the original $x$ back through a shortcut branch before the next nonlinearity. In the basic form the shortcut carries no parameters at all; it is a plain identity path, and the only learned computation is the residual branch.

The shortcut is what makes the whole argument work. Because the identity path carries no parameters in its basic form, the residual variant adds depth without adding capacity, and a network that has learned nothing useful at a block has done nothing rather than broken the signal. When the block needs to change dimensions, the shortcut becomes a small projection, but in the common same-shape case it stays free.

The second figure shows what that buys at the level of whole networks, comparing three ImageNet architectures side by side.

Caption: Three network architectures for ImageNet shown as layer stacks. VGG-19 on the left is the deep reference at 19.6 billion FLOPs. The middle column is a plain 34-layer network at 3.6 billion FLOPs. The right column is the same 34-layer network redrawn with curved identity shortcuts wrapping each pair of convolutions, making it ResNet-34 at the same 3.6 billion FLOPs. The residual variant adds the shortcuts and essentially no computationResNet.

The comparison columns carry the message directly: the residual column has the same FLOP count as its plain counterpart, so nothing was paid for in computation. Once the parameterization is conditioned this way, depth stops being a liability. Residual networks were trained at 152 layers on ImageNet and at 1202 layers on CIFAR-10, depths that a plain stack cannot reach. The 152-layer ensemble reached 3.57% top-5 error on ImageNet and won the ILSVRC 2015 classification track.

Depth conditioned this way did not stay a classification trick. The same backbones swept the 2015 detection and segmentation competitions, and the deck reports the margin by track.

Caption: The MSRA result slide from ILSVRC and COCO 2015: first place in all five main tracks, with ImageNet classification won by "ultra-deep" 152-layer networks, and the detection, localization, and segmentation tracks won by margins of 16%, 27%, 11%, and 12% better than the runner-up respectively. The four per-track margins are as reported on this slide.

Residual learning makes depth affordable because it conditions the parameterization on proximity to the identity, so the added layers start from doing nothing rather than having to learn everything. That is the architecture fix. It leaves a new question open: having made a very deep network trainable in principle, how do you actually update all of its parameters efficiently? That falls to the optimizer.

Shaping the updates: why plain SGD struggles and what momentum, AdaGrad, RMSProp, and Adam change

The third stop is the optimizer. The optimizer's job is to turn a stream of noisy, unevenly-scaled gradient measurements into a stable sequence of steps. Plain stochastic gradient descent struggles on three kinds of terrain, and it is worth naming them as a teaching frame: the conditioning problems the adaptive methods were built to address, rather than a taxonomy that any one paper states outright.

The first is ill-conditioned curvature. When the loss surface is much steeper in some directions than others, a single global step size cannot serve both. A step large enough to move quickly along the flat directions will oscillate wildly along the steep ones, and a step small enough to be stable on the steep directions will crawl on the flat ones. The second is stationary points where the gradient vanishes. In high-dimensional non-convex losses, saddle points are far more common than local minima, and near a saddle the gradient gives almost no usable direction. The third is minibatch noise. The gradient computed on a mini-batch is a noisy estimate of the true gradient, so the update direction jitters from step to step even when the underlying landscape is smooth.

Momentum is the first conditioning move, and it attacks the noise and the oscillation together. Instead of stepping along the current gradient, it maintains a running mean of recent gradients and steps along that:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \qquad \theta_t = \theta_{t-1} - \alpha, m_t.$$

The running mean smooths a noisy mini-batch estimate rather than following it, so a single unlucky batch no longer dominates the step. This is structurally the same move as acting on an exponential moving average of a spiky sensor reading instead of the raw reading: one noisy sample barely moves the average, while a consistent trend accumulates. In the classical velocity picture, $\beta_1$ plays the role of persistence and $1 - \beta_1$ the role of friction.

Momentum fixes the noise, but it still applies the same step size to every parameter. The adaptive methods fix that by rescaling each coordinate separately. AdaGrad divides each parameter's update by the square root of the historical sum of its squared gradients, so coordinates that have consistently received large gradients get damped and coordinates that have received small ones get accelerated. The problem is that the denominator only ever grows, so the effective step size for a parameter that keeps receiving gradient signal decays toward zero as training proceeds, and learning stalls. Adam

RMSProp keeps the per-parameter rescaling but replaces the cumulative sum with an exponential moving average of squared gradients. Now a coordinate's denominator reflects its recent gradient magnitude rather than its entire history, so the step size adapts without decaying to zero permanently.

Adam combines both moves and adds one more piece. It maintains the momentum-like first moment $m_t$ and the RMSProp-like second moment $v_t$ in parallel, then rescales each parameter's update using the ratio of the two. The subtlety is the one that trips people up: both accumulators start at zero, so in the first few steps they are both biased toward zero. Dividing by an underestimated second moment produces a step that is far too large, which is exactly what destabilizes early training or, with $\beta_2$ close to 1, causes divergence. Adam corrects for this explicitly with a time-dependent rescaling:

$$\hat m_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat v_t = \frac{v_t}{1 - \beta_2^t}, \qquad \theta_t = \theta_{t-1} - \alpha,\frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$

The correction is not optional decoration. It exists only because both moments start at zero, and it is what makes the same defaults work out of the box across many problems. The full algorithm, with its default settings, is compact enough to read line by line.

Caption: The Adam update as published, with its default settings listed in the caption: step size $\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, and $\epsilon = 10^{-8}$. The loop's seven lines split into three roles: two parallel exponential moving averages (the first and second moments), two bias-correction lines that divide those moments by $1 - \beta^t$, and the parameter update, which rescales the first moment by the square root of the secondAdam.

Reading the loop vertically makes the lineage visible. Lines three and four are two exponential moving averages running side by side, one for the gradient and one for the squared gradient. Lines five and six are the bias correction, which only matters because both averages were initialized to zero. Line seven is the per-parameter rescaling that uses both moments to set the step for that coordinate. The whole progression lines up as one table:

The through-line is not four competing tricks but one goal: condition noisy, unevenly-scaled gradients into a stable step. Adam is simply the point where both moments and the zero-initialization bias are handled explicitly, which is why it became a common default.

The detailed geometry of the loss surface and the full dynamics of momentum, which the table compresses into one row, are their own story in From SGD's conditioning problem to momentum and adaptive optimizers.

The generalization turn: from training loss to error on unseen data

The optimizer gets the training loss down. That success quietly changes the question. Once a network can fit its training set, the number that matters becomes the error on data it has never seen, and lowering training loss is not the same as lowering that error. This is the fourth stop, and it calls for a different set of tools.

The tools form a small toolbox, and they all attack the same gap in different ways. Early stopping halts training before the model starts fitting noise, using a held-out set to decide when. Checkpoint selection keeps the parameters from the point of best validation error rather than the point of best training error. Ensembling averages several trained models, which reduces variance without changing what a single model can represent. Explicit penalties on the weights, whether the sum of squared weights (L2, often implemented as weight decay) or the sum of absolute weights (L1), shrink the model toward simpler solutions. These are not competing philosophies; they are different entry points into the same gap.

Dropout is the worked example that makes the mechanism clear. During training, each unit is kept with a fixed probability $p$ and dropped otherwise, independently, freshly resampled every mini-batch. The dropped units contribute nothing to the forward pass and receive no gradient. The network that survives one mini-batch is a thinned sub-network of the full architecture. Dropout

Caption: A standard two-hidden-layer network on the left, and one example of the thinned network produced by applying dropout on the right. Units marked with a cross have been dropped for this pass, and the surviving units and their remaining connections form the active sub-network. Because the mask is resampled every mini-batch, a different sub-network is active each timeDropout.

Two things follow from that picture. First, no unit can rely on any particular partner being present, because the partner set is resampled on every step. Each unit has to be useful on its own or in small combinations, which discourages the fragile, tightly coupled solutions a plain network tends to find. Second, the randomness introduced during training has to be dealt with at inference. At test time the masks are gone and every unit is always present, so the full network's outputs would be systematically too large compared with what each unit produced during training. The standard fix is to scale the outgoing weights by the keep probability $p$ at test time, so that each unit's test-time output matches its expected output under the training-time dropout distribution. An equivalent formulation, now the more common one in frameworks, scales the retained activations by $1/p$ during training and leaves the test-time network untouched. The two are the same rule written in different places, and it matters only that you pick one and stay with it.

Both of dropout's effects point at the same underlying move: inject randomness during training, then average it away at inference. Training-time randomness is exactly what the dropout mask is, and the test-time scaling is exactly the averaging that removes it. Lowering training loss is not the same as closing the gap to new data, and the first tool for that gap works by randomizing during training and averaging the randomness away at inference.

The deeper readings of dropout, its role in breaking feature co-adaptation, its interpretation as training a very large ensemble of parameter-sharing sub-networks, and the full train-versus-test scaling algebra, are developed in Beyond Training Loss: Dropout as Noise, Redundancy, and Model Averaging.

One pattern, many tools: randomize during training, average at test time

Once the train-randomize, test-average shape is visible in dropout, it turns out to appear all over the regularization toolbox. The unifying sentence is short: training injects randomness, and inference averages it out. Four techniques that look unrelated are four places the same pattern is applied to a different object.

Batch normalization is the first instance. During training, each layer's normalization statistics come from the current mini-batch, so a given example's normalization is randomized by whichever other examples happen to share its batch. At inference, the transform swaps those random mini-batch statistics for fixed running statistics accumulated over training, which is the average made deterministic. Same randomness-during-training, fixed-at-test shape, just applied to the normalization statistics rather than to units.

Data augmentation is the second. During training, each image is randomly flipped, cropped, rescaled, or color-jittered, so the network never sees the identical input twice and is forced to be invariant to those transformations. At inference the randomness is gone and the model is evaluated on unmodified images, or in a deterministic multi-crop scheme that averages predictions over several fixed crops.

Mixup is the third, and it pushes the pattern from the units to the data itself. Instead of training on single examples, mixup trains on convex combinations of pairs:

$$\tilde x = \lambda x_i + (1 - \lambda) x_j, \qquad \tilde y = \lambda y_i + (1 - \lambda) y_j,$$

with $\lambda$ drawn from a Beta distribution, $\lambda \sim \mathrm{Beta}(\alpha, \alpha)$, and the labels one-hot. When $\alpha$ approaches zero the mixing concentrates on the endpoints and mixup reduces to ordinary training. Larger $\alpha$ pushes the training pairs deeper into the region between examples, which biases the learned function to behave linearly in between them. On a two-class toy problem, that bias shows up as a smooth, near-linear decision boundary instead of the abrupt, oscillating boundary ordinary training produces.

Caption: A two-class toy problem where green points belong to class 0 and orange points to class 1, with blue shading indicating the predicted probability of class 1. Training on convex combinations of example pairs (here with the mixing strength set to 1) produces a smooth decision boundary that transitions gently through the region between the two class clouds, rather than jumping sharply between them. Interpolating examples biases the learned function toward in-between behaviormixup.

The empirical gains are real but the point is the shape. Mixup improved ImageNet top-1 error from 22.0% to 20.8% for a ResNet-101 and CIFAR-10 error from 5.6% to 4.2% for a PreAct ResNet-18, and it improved four of six UCI tabular datasets while never hurting the other two, which is the evidence that the principle is data-agnostic rather than image-specific.

Once the pattern is the payoff, the rest of the family is a single screen of breadth, each line perturbing a different object. Fractional pooling randomly mixes pooling regions. Stochastic depth randomly drops entire layers during training and uses the full network at test. Cutout randomly zeros out image regions. Convolutional augmentation adds learned transformation policies on top of the hand-designed ones. Batch normalization randomizes statistics. Mixup randomizes examples and labels. These are not separate tricks to memorize. They are the same train-randomize, test-average move aimed at the pooling regions, the layers, the image regions, the statistics, or the examples.

The broader family of augmentation and stochastic regularization methods is its own treatment in Randomize, Then Marginalize: A Unified View of CNN Regularization.

When your own data cannot support training: reuse a pretrained backbone

Everything so far assumes there is enough target data to train on. The fifth stop is the case where there is not. On a small target dataset, a network trained from scratch from random initialization will overfit long before it learns anything transferable. The remaining move is to skip the from-scratch problem entirely and reuse a representation that someone else already trained on a much larger dataset. The pretrained CNN is not just a starting point for more training. It is a usable feature in its own right.

DeCAF made this concrete with the simplest possible recipe and no fine-tuning at all. Take a CNN trained on ImageNet, freeze all of its weights, run images through it, and use the activations of a hidden layer as a feature vector to feed a linear classifier. The second-to-last fully-connected layer, a 4096-dimensional vector known as DeCAF_6, turned out to be consistently the best feature, beating the earlier convolutional layers and the final layer. With this frozen feature and a linear support vector machine, Caltech-101 reached 86.9% mean per-class accuracy, roughly 2.6 points above the prior state of the art built on hand-engineered featuresDeCAF. Even one-shot learning, with a single training image per class, reached 33.0% with a linear classifier, against chance of under one percent. The same frozen features closed a large part of the domain gap on the Office adaptation benchmark and set a new state of the art on fine-grained bird recognition when pooled per detected part.

The transfer story is exactly what the pipeline figure shows in miniature: a frozen feature extractor plugged into a downstream task pipeline, extracting features from image regions and pooling them, with no retraining of the CNN at all.

Caption: A frozen CNN used as a feature extractor inside a downstream pipeline: deformable part detections localize regions of a bird image, the detected parts are cropped, and the frozen features extracted from those crops are pooled into a pose-normalized representation for the final classifier. The CNN weights never change; only the small task-specific layer on top is trainedDeCAF.

Reusing a pretrained backbone works out to the same conditioning logic applied one level up. It is structurally the choice of depending on a well-tested library rather than reimplementing every part of a system from scratch: you keep the proven internals and swap out only the small task-specific layer on top, and you only reach for the internals when the small layer is not enough. In the transfer setting, the frozen trunk is the library, the reinitialized head is the task-specific layer, and how much of the trunk to unfreeze is a choice about how much the target task departs from what the library already handles.

That choice is the framework the deck lays out, combining how much target data you have with how similar the target domain is to the pretraining domain:

The diagonal pattern is the whole idea: the less target data you have, and the closer the target domain is to the source, the less of the pretrained network you should disturb. When there is little data and the domains are far apart, no pretrained recipe works well, and that corner deserves to be labeled honestly as the hard case. The rule of fine-tuning at a smaller learning rate, commonly one tenth, is later practice rather than a DeCAF result; the paper proves only the frozen-feature floor and explicitly notes that a full back-propagation pass may do better still.

The same reuse pattern is what scaled up afterward. Rather than one ImageNet backbone for one modality, later systems pretrain a shared representation across vision and language and then adapt it to a task, which is the modern limit of the same idea: reuse a large pretrained representation, and only specialize the small part that the target task actually needs.

When signal, architecture, optimizer, and generalization have all done their work on a large source dataset, the resulting backbone is itself a reusable representation. Transfer is the same conditioning logic applied as an initialization choice on a small target dataset, and it closes the arc that began with the very first question of whether a deep network could learn at all.

参考文献

本专题引用的核心资料如下，建议深入学习时阅读原文：

1. Understanding the difficulty of training deep feedforward neural networks (Glorot & Bengio, AISTATS 2010) — https://proceedings.mlr.press/v9/glorot10a.html
2. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (Ioffe & Szegedy, ICML 2015) — arXiv:1502.03167
3. Deep Residual Learning for Image Recognition (He et al., CVPR 2016) — arXiv:1512.03385
4. Adam: A Method for Stochastic Optimization (Kingma & Ba, ICLR 2015) — arXiv:1412.6980
5. Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., JMLR 2014) — https://www.jmlr.org/papers/v15/srivastava14a.html
6. mixup: Beyond Empirical Risk Minimization (Zhang et al., ICLR 2018) — arXiv:1710.09412
7. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition (Donahue et al., ICML 2014) — arXiv:1310.1531