Neural Nets by Hand: From a Straight Line to an Image Classifier, One Gradient at a Time

A linear classifier draws one straight line. A convolutional network reads a handwritten digit. What turns the first into the second? This tutorial walks the road by hand, and every chapter closes the gap the previous one exposed, using numbers you can redo on paper. The middle chapters are the heart of it, because backpropagation and the choice of activation function are where the machinery really lives.

Training a model is just walking downhill: one real gradient step, by hand

Every machine-learning course starts from the same abstraction, two functions:

def train(images, labels):
    # Machine learning!
    return model

def predict(model, test_images):
    # Use model to predict labels
    return test_labels

That is the framework the CS231n notes build their linear-classifier material on, and it is deliberately a black box.CS231n NN1

Caption: Two code blocks defining the two-step picture of learning: train returns a model, predict returns labels. Everything below unpacks what train does when it says # Machine learning!.

Take the smallest learnable baseline, a linear classifier. A 32x32 color image has $32 \times 32 \times 3 = 3072$ values, so the input $x$ is a 3072-dimensional vector and the weights form a $10 \times 3072$ matrix producing ten class scores. Scores are not probabilities, so softmax exponentiates each one and divides by the sum. The deck's example for one image:

Comparing those probabilities against the one-hot target $1, 0, 0$ gives the cross-entropy loss. Now make gradient descent a verb. The update is $w \leftarrow w - \eta,\partial J/\partial w$, and the loss surface is a bowl.

Caption: A symmetric bowl-shaped surface over two weight axes, lowest near the center and rising toward the edges. Gradient descent is the rule for stepping down this surface toward the low region.

Take one slice of that bowl. Suppose $J(w) = (w-3)^2$, so the minimum sits at $w=3$. Start at $w=5$: the slope is $2(w-3)=4$. With learning rate $\eta=0.3$, the update gives $w \leftarrow 5 - 0.3 \times 4 = 3.8$, and the loss drops from $4$ to $0.64$. Repeat and you walk into the minimum. A real digit classifier has 13,002 such weights and biases, which is why solving for zero slope by hand is hopeless and iterative steps are the practical answer.3Blue1Brown

So training is walking downhill. The rest of the tutorial asks what happens to that walk once several layers sit between input and loss.

Why a straight line cannot draw XOR, and why another linear layer changes nothing

Take the four XOR points.

The two positive points sit on opposite corners of a square and the two negative points on the others, so no single straight line separates them. Goodfellow's textbook works this out exactly: minimizing the squared error of $f(x) = x^T w + b$ on these four points returns $w=0$ and $b=1/2$, so the model outputs a constant $0.5$ everywhere.Goodfellow

The fix is to build a new feature from the inputs, their product, and classify on that. A hidden layer does it automatically. Goodfellow gives one exact solution with ReLU units:

$$W = \begin{bmatrix} 1 & 1 \ 1 & 1 \end{bmatrix}, \quad c = \begin{bmatrix} 0 \ -1 \end{bmatrix}, \quad w = \begin{bmatrix} 1 \ -2 \end{bmatrix}, \quad b = 0$$

Feed the four points through $h = \max{0, W^T x + c}$ then $y = w^T h + b$. The batch product $XW$ gives rows $[0,0],[1,1],[1,1],[2,2]$; adding $c$ gives $[0,-1],[1,0],[1,0],[2,1]$; the ReLU clips the first to $[0,0]$; multiplying by $w$ gives $0,1,1,0$, exactly the XOR targets. The hidden layer collapsed both positive points to the same place, so a straight line now works.

Caption: Two small plots. In the original input space the corners carry the pattern 0, 1, 1, 0 and no straight line separates them. In the learned hidden space the two points that must output 1 have been pulled onto the same level, so a horizontal line now divides the classes. The figure shows point positions only, not the network or its weights.Goodfellow

In layer form that network is $h = \sigma(W_1 x + b_1)$ and $o = w^T h + b_2$, which Goodfellow draws two ways, once with each unit as its own node and once with each layer collapsed to one node.

Caption: The same one-hidden-layer network drawn unit-by-unit and layer-by-layer. The layer view labels the edges with the parameters, a matrix $W$ from input to hidden and a vector $w$ from hidden to output, which is the layout the deck's equations assume.Goodfellow

Now the trap. Drop the nonlinearity and set $h = W_1 x + b_1$. Then $o = w^T(W_1 x + b_1) + b_2 = (w^T W_1)x + (w^T b_1 + b_2)$, a single linear transformation. Two linear layers collapse into one, so depth alone buys nothing. As the CS231n notes say, the non-linearity is where we get the wiggle.CS231n NN1

So the nonlinearity is what adds power, and that raises the central question: how do we train a network with several of these layers? The answer is backpropagation, easiest to see on a network small enough for one page.

How does the error travel backward? Backprop on a 2-2-1 network, done by hand

The deck's example uses two inputs, two hidden units, and one output. The inputs are $2$ and $3$ and the true output is $1$.

Caption: Two input values, 2 and 3, separated from a single target output of 1. There is no formula here; it only fixes the numbers the update will move toward.

Before the algebra, hold one picture. 3Blue1Brown frames the chain rule as three ratios for a single weight: how much the pre-activation moves when the weight moves, the activation when the pre-activation moves, and the loss when the activation moves.3Blue1Brown

$$\frac{\partial C}{\partial w} = \frac{\partial z}{\partial w} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial C}{\partial a}$$

Each layer you pass contributes another factor of the same shape, so backprop is just this rule applied from the output backward. Formally, define the error of a neuron as $\delta_j^l = \partial C/\partial z_j^l$. Nielsen's four equations are the whole algorithm:Nielsen

• BP1, output layer: $\delta^L = \nabla_a C \odot \sigma'(z^L)$
• BP2, hidden layers: $\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l)$
• BP3, bias gradient: $\partial C/\partial b_j^l = \delta_j^l$
• BP4, weight gradient: $\partial C/\partial w_{jk}^l = a_k^{l-1} \cdot \delta_j^l$

BP4 is the one to remember, because it splits a weight's gradient into two quantities you already have: the activation coming from the previous layer, and the error leaving the next.

Caption: A single weight drawn as two nodes joined by one line, with the equation above: the cost's gradient with respect to this weight equals the input activation times the output error. It is a one-line reminder, not a picture of a whole network.Nielsen

Here is the network those equations walk over.

Caption: An input layer of two nodes, a hidden layer of two nodes, and one output node, with six labeled weights and the output marked as the prediction. Running inputs of 2 and 3 forward through this structure once produces 0.191.

Feed $2$ and $3$ forward with the initial weights and you get $0.191$. The target is $1$, so the network is far off. Now run the chain rule backward. The output error $\delta^{\text{out}}$ is the gap between prediction and target scaled by the output activation's slope. BP2 hands that error to each hidden unit, transposed through the outgoing weights and shrunk by that unit's own slope. BP3 turns each hidden error into a bias gradient, and BP4 into a weight gradient by multiplying by the activation that fed it. One backward pass over six weights, and every gradient is on the page.

Then take one step, $w \leftarrow w - \eta,\partial C/\partial w$ for all six weights and both biases. The prediction moves from $0.191$ toward $1$; after several iterations the deck reports $0.26$, closer but not there yet.

Caption: A second drawing of the same three-layer network with the six weights marked on the connections and the output arrow labeled as the prediction, showing the structure both the forward and the backward pass walk over.

One piece of magic is worth naming. When the output is a softmax paired with cross-entropy, the $\sigma'(z)$ in BP1 cancels against the loss derivative, leaving $\partial C/\partial w_j = \frac{1}{n}\sum_x x_j(\sigma(z) - y)$. The gradient now grows with how wrong the network is instead of shrinking as the output saturates.Nielsen

Look again at those $\sigma'(z)$ factors in BP1 and BP2. Each is a multiplier on the way home. If one is near zero, the signal dies there.

Which activation survives the trip back? Comparing gradients at x = -10, 0, +10

Every $\sigma'(z)$ in the backward pass is the slope of the activation you chose, so a zero slope multiplies the gradient to nothing. The two classic choices are sigmoid, which squashes inputs into $(0,1)$, and tanh, which squashes into $(-1,1)$ and is a rescaled sigmoid: $\tanh(x) = 2\sigma(2x) - 1$.

Caption: Two S-shaped curves. One is compressed into 0 to 1, so its output is never negative; the other is symmetric about the origin and spans -1 to 1. Both flatten into near-horizontal tails at the far left and right, where their slopes collapse.CS231n NN1

Evaluate the slopes at three points and the story appears.

For sigmoid and tanh both tails give essentially zero, so a neuron pushed into either tail passes nothing back. This is "saturation kills the gradient" as a number you can check yourself, and it matches what Nielsen's equations predict. ReLU, defined as $f(x)=\max(0,x)$, refuses to saturate on the positive side: for every positive input its slope is exactly 1, so an incoming gradient passes through unchanged.

Caption: A plot of the rectified linear function: flat at zero for negative inputs, rising in a straight line of slope one for positive inputs. Goodfellow calls it the default activation recommended for most feedforward networks. It shows the function, not its gradient.Goodfellow

The variants each patch a weakness. Leaky ReLU keeps a small negative slope so the negative side is not dead. ELU uses a smooth exponential there. At the far end sits SELU, a scaled ELU with fixed constants $\alpha \approx 1.6733$ and $\lambda \approx 1.0507$, whose selling point is that a deep stack keeps activations near zero mean and unit variance without batch normalization.Klambauer

Caption: Four activations on one axis. All four rise linearly for positive inputs, with SELU slightly steeper because of its scale factor. For negative inputs they diverge: ReLU is flat at zero, Leaky ReLU keeps a slight downward slope, and ELU and SELU curve back toward a floor, with SELU's floor lower. The plot was generated from the paper's own constants.

So the backward pass only works if the activation still has a slope along the route home. ReLU keeps one for positive inputs; sigmoid and tanh eventually do not. The next chapter asks what that costs in practice, and what ReLU gives up.

Why ReLU replaced sigmoid in practice, and when a ReLU stops firing

Two concrete problems pushed sigmoid out of the hidden layers. The first is the zig-zag. Sigmoid outputs are always positive and never centered on zero, so when a neuron's input is consistently positive, every weight gradient for that neuron carries the same sign, because $\sigma'(z)$ is itself positive. When all gradients point the same way, the update cannot head straight for the optimum; it steps along one axis, then the other, tracing a zig-zag. Averaging over a minibatch is what smooths it out.CS231n NN1

The second is the empirical result, and the deck's own Playground experiments settle it. Train the same six-hidden-layer network on the same ring-shaped dataset and change only the activation.

Caption: A TensorFlow Playground screenshot with six hidden layers and Sigmoid selected. The decision surface is still a flat wash of one color, so the network has not separated the inner cluster from the outer ring. The reported test loss is 0.503 at epoch 234, near what a coin flip scores.

Swap Sigmoid for ReLU and change nothing else.

Caption: The same six-layer network with ReLU selected. A clear region has formed around the inner cluster and the reported test loss has dropped to 0.000 by epoch 394. Same architecture and data, one change to the activation.

Krizhevsky's 2012 ImageNet result is the historical version of this. The CS231n notes report that ReLU accelerated convergence of stochastic gradient descent by a factor of about 6 compared with sigmoid and tanh, credited to its linear, non-saturating form.CS231n NN1

Caption: On the left, ReLU itself. On the right, a training-error-rate plot against epochs with two curves falling at clearly different rates. The image carries no "six times" label; that figure comes from the source's caption and the paper.CS231n NN1

ReLU is not free, though. A ReLU pushed far enough into negative territory can stop activating on any input, and then its gradient is forever zero, so it never recovers. The property that rescues the gradient on the positive side turns the negative side into a permanent dead zone. The CS231n notes describe neurons dead across the entire training set, and recommend a small positive bias such as 0.01 to nudge units toward firing at initialization.CS231n NN1

ReLU took over not because it is more correct, but because in a deep stack it is the option that lets the gradient survive the whole trip back. The price is that it may switch off, and the fix is a careful learning rate or a small positive bias.

What is a convolution layer actually computing? Filters, stride, padding, parameters

Now scale the same reasoning up to images, where the problem is not gradients but size. A fully connected hidden layer over a 3072-dimensional vector costs thousands of weights per unit, and stacking layers only worsens it. A convolution layer changes what each neuron may look at: instead of every input pixel, a small local patch, but one that reaches through the full depth of the input.

Caption: A wide input volume with a small block lifted out of it, standing for the local region one convolution neuron reads. Lines fan out to a row of output neurons. The patch is small across width and height but reaches through all input channels, and the row represents different filters watching the same patch.CS231n CNN

Concretely, take an input of $32 \times 32 \times 3$ and a filter of $5 \times 5 \times 3$. Line the filter up with a patch and take the dot product: $5 \times 5 \times 3 = 75$ multiplications plus one bias, giving a single number. Slide it to every valid position and you get a $28 \times 28$ activation map. Six filters give six maps stacked into $28 \times 28 \times 6$. Each filter costs $5 \times 5 \times 3 + 1 = 76$ parameters, so the layer has $6 \times 76 = 456$, a tiny fraction of the fully connected alternative.

Output size depends on two more choices. On a $7 \times 7$ input with a $3 \times 3$ filter, stride 1 gives $5 \times 5$, stride 2 gives $3 \times 3$, and stride 3 does not fit evenly since $(7-3)/3 + 1 = 2.33$. The general rule is

$$H_{out} = \left\lfloor \frac{H_{in} - K_h + 2P}{S} \right\rfloor + 1$$

where $K$ is the filter size, $S$ the stride, and $P$ the zero padding around the border.

Caption: Two formulas giving the output height and width, each the floor of the input size minus the kernel size plus twice the padding, divided by the stride, plus one. The floor brackets state that the division must come out to a whole number for the configuration to be valid.

Padding keeps the volume from shrinking every layer: convolving $32 \times 32$ with $5 \times 5$ filters repeatedly gives $32 \to 28 \to 24$, throwing away edge information fast. Pad the $7 \times 7$ input with one border pixel and the same $3 \times 3$ filter with stride 1 returns $7 \times 7$. The size-preserving convention is $P = (F-1)/2$, so a $3 \times 3$ filter gets 1 pixel, $5 \times 5$ gets 2, and $7 \times 7$ gets 3.

Redo the deck's example with padding. Input $32 \times 32 \times 3$, ten $5 \times 5$ filters, stride 1, padding 2: the spatial output is $(32 + 2\times2 - 5)/1 + 1 = 32$, the depth is 10, so the output is $32 \times 32 \times 10$ and the parameter count is $10 \times (5\times5\times3 + 1) = 760$.

LeNet-5 is the classic worked example of this arithmetic.

Caption: A block diagram of LeNet-5 from a 32 by 32 by 1 input: a 5 by 5 convolution with 6 filters, 2 by 2 average pooling, a 5 by 5 convolution with 16 filters, another 2 by 2 average pooling, then fully connected stages of 120, 84, and 10 units. The diagram poses the question of how many parameters the network must learn.

The deck simplifies the original. LeCun's 1998 paper has seven trainable layers, and the per-layer counts are 156, 12, 1,516, 32, 48,120, and 10,164, giving a headline of about 60,000 trainable parameters across roughly 340,908 connections, with weight sharing doing the compression.LeNet-5

Caption: The full LeNet-5 stack: a 32 by 32 input, six 28 by 28 planes, six 14 by 14 planes, sixteen 10 by 10 planes drawn on a diagonal to hint at sparse connectivity, sixteen 5 by 5 planes, then vector stages of 120, 84, and 10. Two caveats: this is a community redraw, not the paper's original figure, and it labels the subsampling stages as max pooling, whereas the paper's subsampling is a learnable weighted sum.dl-visuals

A one-dimensional picture shows how stride changes output size while the filter stays fixed.

Caption: A filter sliding across a fixed one-dimensional input. With stride 1 it produces five outputs; with stride 2 it produces three. The filter itself, shown on the right, never changes. The picture isolates stride and does not show the padding cells.CS231n CNN

So a convolution layer trades a global view for a much smaller parameter count, at the cost of each neuron seeing only a small patch. The last chapter asks how small that patch is.

How far can one neuron see? Receptive fields and why we shrink the feature map

Every neuron in that $28 \times 28$ activation map looks at a $5 \times 5$ region of the input, its receptive field. Notice what else holds: all $28 \times 28$ neurons in one map share the same 75 weights. This is parameter sharing, the reason a convolution layer can cover an entire image with a handful of weights. But a $5 \times 5$ view of a 32 by 32 image cannot recognize a digit on its own, so we stack layers, because each layer widens what the next layer's neurons effectively see. Araujo's Distill article gives the recurrence:

$$r_{l-1} = s_l \cdot r_l + (k_l - s_l)$$

Here $r_l$ is the receptive-field size at layer $l$, $k_l$ its kernel size, and $s_l$ its stride. Padding does not appear, because it does not change how many input pixels a neuron depends on. Solving the recurrence gives the closed form $r_0 = \sum (k_l - 1)\prod s_i + 1$, and the numbers for real networks are striking.

Stack enough layers and one output neuron ends up depending on a surprisingly large patch of the original image.

At the input a neuron's pattern is tiny; at the output of a deep network its response depends on hundreds of pixels, shaped by every kernel and stride between.Araujo

While the receptive field grows, most networks also shrink the spatial size, and pooling does that. Max pooling with a $2 \times 2$ window and stride 2 keeps the largest value in each block and discards the rest.

Caption: A single depth slice, a 4 by 4 grid of numbers split into four 2 by 2 blocks. Each block reduces to its largest value: 6 top-left, 8 top-right, 3 bottom-left, 4 bottom-right, giving a 2 by 2 output. Depth is unchanged; only the spatial size is halved.CS231n CNN

On the deck's example, the slice

$$\begin{bmatrix} 1 & 2 & 2 & 4 \ 5 & 6 & 7 & 8 \ 3 & 2 & 1 & 0 \ 1 & 2 & 3 & 4 \end{bmatrix}$$

pools to

$$\begin{bmatrix} 6 & 8 \ 3 & 4 \end{bmatrix}$$

because 6, 8, 3, and 4 are the maxima of the four blocks. Pooling works independently on each depth slice, so $28 \times 28 \times 6$ becomes $14 \times 14 \times 6$: depth preserved, spatial footprint halved, and the strongest responses kept.

What the whole road taught

Put the chapters back to back and one pattern repeats. A straight-line model fails on XOR, so we add a hidden layer. Two linear layers still collapse, so we make the hidden layer nonlinear. A nonlinear network cannot be trained layer by layer, so we push the error backward with the chain rule. That backward push multiplies the gradient by the local slope at every step, so the activation must keep a slope alive; ReLU does, and it took over as a result, at the cost of possibly dying. Then the same push for scale: a fully connected layer over every pixel explodes in parameters, so convolution shares one small filter across a whole image, and we grow the receptive field by stacking layers while pooling shrinks the feature map. It is one long answer to the opening question, built from gradients, slopes, and filters you can compute by hand.

参考文献

The core references cited in this tutorial are listed below; reading the originals is recommended for going deeper.

1. Neural Networks and Deep Learning (Michael Nielsen, Determination Press, 2015) — http://neuralnetworksanddeeplearning.com/chap2.html
2. 3Blue1Brown Deep Learning series (Grant Sanderson, 2017) — https://www.3blue1brown.com/lessons/backpropagation-calculus
3. Deep Learning, Chapter 6: Deep Feedforward Networks (Goodfellow, Bengio, Courville, MIT Press, 2016) — https://www.deeplearningbook.org/contents/mlp.html
4. CS231n: Neural Networks Part 1 (Karpathy, Li, Johnson, Stanford, 2016) — https://cs231n.github.io/neural-networks-1/
5. CS231n: Convolutional Neural Networks (Karpathy, Li, Johnson, Stanford, 2016) — https://cs231n.github.io/convolutional-networks/
6. Gradient-Based Learning Applied to Document Recognition (LeCun, Bottou, Bengio, Haffner, Proc. IEEE 86(11), 1998) — http://yann.lecun.com/exdb/lenet/
7. Self-Normalizing Neural Networks (Klambauer, Unterthiner, Mayr, Hochreiter, NIPS 2017) — https://arxiv.org/abs/1706.02515
8. Computing Receptive Fields of Convolutional Neural Networks (Araujo, Norris, Sim, Distill, 2019) — https://distill.pub/2019/computing-receptive-fields/