Image Classification Can't Be Hand-Coded: k-NN, Linear Scores, and Neural Nets Form One Pipeline

1. Why Can't We Hand-Code Image Classification?

Suppose you want a program that looks at a photo and decides whether it shows a cat or a car. The naive instinct is to write classify_image: a function full of rules. If the pixels are pointy at the top and have whiskery edges, call it a cat. The claim of this article is that this instinct is structurally doomed, and the fix is not better rules but a different kind of program. An image classifier is not a single formula. It is a continuous pipeline: represent the image as numbers, score each candidate class, measure the result with a loss, optimize the parameters, differentiate efficiently to make optimization cheap, and finally compose nonlinearly when straight lines stop being expressive enough. Everything in this article is one stage of that pipeline, each stage patching the blind spot of the one before it.

The naive idea: a function called classify_image

The trouble begins with what an image actually is. A photo arrives as a grid of integer pixel values, a tensor of numbers. A CIFAR-10 image, for instance, is 32×32 pixels tall and wide with three color channels, and it is conventionally flattened into a single column vector of 3072 numbers before a classifier touches it. The same object can be viewed two ways at once: semantically it is "a cat," computationally it is 3072 integers. Every hand-written rule has to bridge that gap, and rules written directly in pixel space collapse under variation.

The canonical inventory of failure modes from the CS231n lecture notes names seven ways reality breaks a fixed pixel rule: viewpoint variation, scale variation, deformation, occlusion, illumination conditions, background clutter, and intra-class variation. Move the camera and every pixel changes. Zoom in and every pixel changes. A cat in shadow, a cat behind a fence, a white cat and a black cat: no fixed combination of pixel thresholds survives all of them. A category is not a pixel template. It is a distribution over all these variations, and a hand-coded rule can only ever capture one point of that distribution. The lesson is that the difficulty of image classification is not programming. It is that the category lives far away from the pixel grid, and the only bridge that scales is learned from data rather than written down.

The data-driven answer is to stop specifying what a category looks like and instead provide many labeled examples and let the program figure out the decision rule itself. This is supervised learning. In the standard textbook formulation, a program learns from experience E with respect to task T and performance measure P if its performance at T, as measured by P, improves with experience E. That is the definition Mitchell gave in 1997, and it is the definition Goodfellow, Bengio and Courville open their machine-learning chapter with. The training set is the experience, the classifier's accuracy on unseen images is the performance measure, and the whole pipeline below exists to convert experience into performance. CS231n Deep Learning

Linear algebra as the computational language

Before any of that can happen, images must become objects that support arithmetic. That is where linear algebra enters: a flattened image is a vector in 3072-dimensional space, and the vocabulary of vectors gives us the operations a learner needs. The most primitive of them is the dot product, which turns two vectors into a single number by summing the products of their matching components:

Caption: The dot product $a^T b = \sum_i a_i b_i$ turns two vectors into one number. It is the primitive with which "resemblance" becomes arithmetic: matching components reinforce, opposite components cancel.

From the dot product comes orthogonality: when the dot product is zero, the two vectors share no component of each other, they are directionally unrelated under this measure:

Caption: The orthogonality condition $a^T b = 0$: two vectors whose dot product vanishes carry none of each other's direction. This is the vocabulary point at which "these two things have nothing in common" becomes a computable statement.

This is the standard notation used throughout Strang's Introduction to Linear Algebra, the same convention the rest of this pipeline inherits. Strang

Then come the two matrix products. A matrix-vector product $c = Ab$ computes each output coordinate as a dot product of one row of $A$ with the vector $b$; a matrix-matrix product $C = AB$ chains two such mappings, with $C_{ik} = \sum_j A_{ij} B_{jk}$. Rounding out the vocabulary with matrix norms and the eigenvalue equation $Ax = \lambda x$ (the engine behind PCA and compression), linear algebra becomes the language in which the entire pipeline speaks: every later stage, from the nearest neighbor to the neural network, is a sentence in this language.

So the answer to "why can't we hand-code it" is not that the rules are hard to write but that pixel space and category space are different spaces. Rulers break; learned mappings survive. The cheapest possible learner that actually runs is the one that keeps the data and votes on it. We turn to it next.

2. The Zero-Parameter Baseline: k-NN Memorizes the Training Set and Votes by Distance

Nearest neighbor: memorize, then ask the closest point

The simplest classifier that works at all is the nearest-neighbor rule: take the entire training set and store it. To classify a test image, find the stored image closest to it under some distance, and output that neighbor's label. Training is literally storage. Prediction is a loop that compares the query against every stored example, using L1 distance (the sum of absolute pixel differences, also called Manhattan distance) or L2 distance (Euclidean distance). This rule is old enough to be a theorem: Cover and Hart formalized it in 1967, assigning to each unclassified point the classification of the nearest previously classified point, and proved that in the large-sample limit its error is bounded above by twice the Bayes error, the best any classifier could do. Cover & Hart

The obvious refinement is to make k neighbors vote. Instead of trusting the single closest image, take the k closest stored images and let them vote on the label by majority. The general rule: find the top k closest images and have them vote on the label. CS231n

Larger k smooths the decision boundary

The choice of k is not cosmetic. It visibly changes what the classifier draws. With k = 1, every training point claims a small island of territory around itself, and a single outlier creates a tiny patch of likely-misclassification in the middle of another class. With k = 5, the vote needs five neighbors to agree, so the decision regions smooth into broader zones and stray points no longer carve out islands. The CS231n notes state it directly: higher values of k have a smoothing effect that makes the classifier more resistant to outliers. CS231n

Caption: Three side-by-side panels on the same two-dimensional data. The second panel shows the nearest-neighbor classifier: small, jagged decision regions hugging every training point. The third panel shows the 5-nearest-neighbor classifier: larger, smoother regions, with gray zones marking tied votes. The contrast is the geometric meaning of "larger k smooths the boundary." CS231n

On CIFAR-10 (50,000 training images, 10 classes), this zero-parameter baseline reaches 38.6% accuracy with L1 distance and 35.4% with L2. That already beats the 10% a pure guess would score, which proves the pipeline concept works: store, compare, vote. But it is nowhere near useful, and the reasons are structural, not a matter of tuning. CS231n

The cost of carrying the whole training set

Two structural problems make k-NN a dead end as a real image classifier, and both will motivate the next stage. The first is cost. Every prediction scans the entire training set and computes a distance against every stored example: each query is O(N·D) work where N is the number of training images and D their dimensionality. There is no way around it, because the classifier's entire knowledge lives inside the data. The classifier cannot even be shipped somewhere: it has to carry the dataset wherever it goes. The second problem is semantics. Pixel distance does not track meaning. Two pictures of the same object shifted by a few pixels can have a huge pixel distance, while two pictures that are numerically similar can show completely different things. The classifier memorizes the training set without understanding anything in it. Choosing k and the distance metric carefully helps only marginally, and doing it honestly is its own problem: k is a hyperparameter, and picking it well requires a protocol that never lets the test set influence the choice.

That is the zero-parameter baseline in full: it fits nothing, understands nothing, and still works, purely by memorizing and voting. Its blind spot is that the entire dataset has to travel with it. The next stage trades that memory for a much smaller object: a matrix of parameters.

3. Trading Memory for Parameters: The Linear Score Function s = Wx + b

From stored examples to a weight matrix

The parametric move replaces storage with arithmetic. Instead of keeping 50,000 images around, keep one matrix $W$ and one bias vector $b$, and define a linear score function that maps an image vector to a vector of scores, one per class:

$$s = Wx + b$$

Prediction is then simply the class with the largest score. For CIFAR-10, $x$ is the flattened image of shape [3072×1], $W$ is [10×3072], and the scores come out as [10×1]. The single matrix multiplication $Wx$ is effectively evaluating 10 separate classifiers in parallel, one for each class, with each classifier being one row of $W$. CS231n

The score machinery is exactly the matrix-vector product defined earlier, written with a new name:

Caption: The matrix-vector product $c = Ab$, where each entry $c_i$ is the dot product of row $i$ of $A$ with the vector $b$. The linear classifier $s = Wx$ is this same shape: each row of $W$ is a class template, and each score is one dot-product match between the template and the flattened image.

This is the standard matrix-vector product notation, exactly as the linear-algebra textbooks write it for a row of dot products. Strang

The row-of-$W$-as-template reading is the intuition that sticks. The "cat" row of $W$ is a prototype cat; when it is dotted against an actual cat image, matching pixels pile up and the score rises. The linear classifier is template matching by inner product. The field-standard textbook opens its deep-learning chapters with exactly this pipeline, representation through a parameterized score function first, losses and optimization after. Deep Learning

Note the shift in responsibility this introduces: the input data $(x_i, y_i)$ is given and fixed, and the only thing we control is the setting of the parameters $W, b$. The data is no longer carried around; it is distilled.

The bias trick: affine becomes linear

Strictly, $s = Wx + b$ is an affine map: a matrix multiply plus an offset. There is a classic trick to absorb the offset into the matrix. Augment the input vector $x$ with one extra coordinate held constant at 1, and extend $W$ by one matching column. Then the model collapses to a single pure matrix multiply, $f(x, W) = Wx$, with no separate $b$ anywhere. The bookkeeping gains one dimension and loses an entire moving part; the rest of the linear-algebra machinery then applies to the whole model unmodified. The same parameterization opens the door to every loss that follows, since both the margin-based and the probabilistic losses take score vectors as their input. CS231n

Prediction without touching the data

Here is the structural break in behavioral terms. A k-NN prediction is a full walk over the training set. A linear classifier prediction is one matrix multiply, after which the entire training set can be discarded and only the learned parameters kept. CS231n

The programming analogy writes itself. k-NN answers every query by scanning every row of the whole database: correct but linear in the number of rows per query. The linear classifier is the model that precomputes a small parameter file once, then serves queries from it in constant time, at the price that everything not captured inside the parameters is simply forgotten. That is the trade: the model shrinks from gigabytes of images to a few tens of thousands of numbers, and prediction becomes fast enough to deploy anywhere. A linear classifier, unlike a k-NN, can be embedded in a phone.

But the moment the model becomes a matrix, a new question appears that k-NN never had to face. Plenty of matrices will separate the training data. Which one is better? Nothing in $s = Wx + b$ says so. The next stage supplies the ruler: a loss function.

4. What Makes One W Better Than Another? The Hinge Loss and the Δ-Margin

A ruler over score vectors

A loss function is a number attached to $W$ that says how badly it failed. The multiclass hinge loss does this in one clean form: for each training example, sum over every incorrect class $j$, and add a penalty whenever class $j$'s score intrudes on the correct class's lead:

$$L_i = \sum_{j \neq y_i} \max(0,; s_j - s_{y_i} + \Delta)$$

The requirement encoded inside is: the correct class's score must beat every other class's score by at least $\Delta$. If it does, the term inside $\max$ is negative and contributes zero. The deeper an incorrect class penetrates into the margin, the more loss accumulates. The geometric picture behind this formula is what carries the intuition. The design idea has a famous ancestor: the support-vector network of Cortes and Vapnik (1995), which selected the linear decision surface with the widest margin between classes and defined support vectors as the training points sitting exactly on that margin. Cortes & Vapnik

The loss is quick to feel with numbers. Take scores [13, −7, 11] with the correct class at index 0 and $\Delta = 10$. The first incorrect class contributes $\max(0, -7 - 13 + 10) = 0$: it is safely below the margin. The second contributes $\max(0, 11 - 13 + 10) = 8$: it sits inside the margin, 8 units deep. Total loss for this example: 8. One number now summarizes, for this image, how badly the classifier failed to be confident enough. CS231n

The margin geometry is a one-dimensional picture of the same idea:

Caption: A schematic score axis. The green vertical lines mark the scores of the other classes; the blue line marks the score of the correct class; the red segment is the margin $\Delta$ that must stay empty. Any other-class score falling inside the red region accumulates hinge loss. CS231n

Keep that red gap in mind, because it leads straight to a genuinely strange consequence about what zero loss actually pins down.

Scale ambiguity: many Ws tie at zero loss

Now the subtlety that forces the next design decision, and it is genuinely counterintuitive the first time you see it. Suppose some $W$ achieves zero loss on every training example: the correct class always wins by at least $\Delta$. Then $2W$, or any $\lambda W$ with $\lambda > 1$, also achieves zero loss, because scaling all scores uniformly preserves every ordering and stretches every margin by the same factor. Zero loss therefore pins down nothing. There is an entire ray of equally good parameters pointing through $W$. CS231n

Read that twice, because it is the hinge on which the whole idea of regularization turns: when the data no longer distinguishes between $W$ and $2W$, the data alone cannot finish the job. Something else has to, namely a preference over parameters themselves, a prior that says smaller or simpler $W$ is better. That preference, expressed as a regularization term, rounds out the objective into "data loss plus parameter preference." The loss answers "did the correct class win by enough," and when many $W$ tie at zero loss, the problem of choosing among them moves beyond the data.

So now we can measure $W$ with a ruler, and we know the ruler alone does not uniquely decide. But we have not said a word about how to find a good $W$ in the first place. That is the optimization stage.

5. How Do We Find a Good W? Random Search, Gradient Descent, and Mini-Batch SGD

Random search as the deliberate floor

Optimization answers "how do we search the space of parameters." The honest baseline is the dumbest possible search: generate random $W$s, evaluate the loss on each, keep the winner. On CIFAR-10 this algorithm reaches about 15.5% accuracy. That is slightly above the 10% random guess, and it is the floor that makes directed search necessary. If guessing in parameter space barely beats guessing in class space, then "try and hope" is not an option when the space has tens of thousands of dimensions. CS231n

Gradient descent: direction and step size

The directed alternative uses the gradient, the vector of slopes of the loss with respect to each parameter. The gradient points uphill; stepping against it reduces the loss. The gradient-descent loop is one line repeated:

$$w \leftarrow w - \text{lr} \cdot \nabla L$$

Two different quantities ride in that one line, and they answer two different questions. The gradient $\nabla L$ supplies the direction: which way in parameter space does the loss drop most steeply. The learning rate lr supplies the distance: how far to go along that direction. Get the direction right and the step wrong and the loop still fails. CS231n's printed table makes the failure mode concrete: already at a step size of 1e-5, the loss after a single update rises from the original 2.20 to 2.84, and at 1e-3 it explodes to about 254 — the loss grows instead of shrinking with each update, because every step jumps clean over the valley and lands higher on the far wall. That is the overstep failure, the distinguishable signature of a learning rate that is too large rather than an optimizer that is broken. CS231n

Caption: The gradient-descent loop. Evaluate loss and gradient at the current parameters, step in the negative-gradient direction, repeat. The loop only settles if each step lands closer to the bottom; a too-large learning rate overshoots the valley and the loss rises instead.

That loop is the whole optimizer in its purest form. Everything that follows is about making its two ingredients, the gradient and the step size, cheaper and safer to compute.

Numerical vs analytic gradient, and the gradient check

Where does $\nabla L$ come from? There are two ways, and they are a study in opposites. The numerical way estimates each slope by finite differences: perturb one parameter, re-evaluate the loss, measure the change. For $d$ parameters that is $d$ loss evaluations per gradient, and with millions of examples per loss, this strategy is not scalable. The analytic way computes the exact gradient with calculus in a single symbolic pass: instantaneous and exact, but easy to get subtly wrong when implemented. The discipline that reconciles them is the gradient check: compute the analytic gradient and compare it against the numerical one on a handful of random parameter settings. If they match, trust the fast path. This is the standard safety net, and it is the reason every deep-learning framework tells you to verify your gradient code once before relying on it forever. CS231n

Mini-batch SGD when N is large

The full-data gradient requires evaluating the loss over the entire training set for a single update, which is wasteful when the dataset runs to millions of examples. Mini-batch gradient descent estimates the gradient from a small random batch instead, typically 32 to 256 examples per step, and stochastic gradient descent (SGD) is the extreme case of a batch of one. The estimate is noisy, but it is noisy in the right direction, and modern vectorized hardware makes a batch dramatically cheaper per example than the same number of one-off updates. In practice people say SGD even when they mean mini-batch gradient descent; the boundary is fuzzy and nobody important minds. CS231n

So the pipeline now has a way to measure $W$ (the loss) and a way to move $W$ (gradient descent with a sane learning rate). But gradient descent presumes a gradient, and for the linear score function that gradient is a short calculation. The moment the model gets more interesting, computing gradients by hand through every term becomes a nightmare. That is the next blind spot, and it is plugged by computation graphs and reverse-mode backpropagation.

6. Where Do the Gradients Come From? Computation Graphs and Reverse-Mode Backprop

A computation graph decomposes the function

The trick is to stop treating the loss as one giant algebraic expression and instead decompose it into primitive operations: additions, multiplications, max operations. Each operation becomes a gate in a computation graph. The forward pass runs the graph from inputs to output, computing every gate's result and caching the intermediate values. The backward pass then walks from the output back to the inputs, applying the chain rule in reverse order and consuming the cached values as it goes. The forward pass is where the values are made; the backward pass is where the gradients are collected.

Consider the tiny function $f = (x + y) \cdot z$, small enough to see every step:

Caption: The computation graph for f = (x+y)·z. Forward pass: x and y feed the add gate, producing q, which is cached; q and z feed the multiply gate, producing f. Backward pass: the gradient flows from f back to q and z, then from q onto x and y.

Forward: $q = x + y$, then $f = q \cdot z$. Backward: the output gradient is 1 at $f$; the multiply gate sends partial gradients to $q$ and $z$; the add gate then splits $q$'s gradient onto $x$ and $y$.

The three gate rules

Backpropagation reduces to three local rules, one per gate type, and every gate knows only its own rule. An add gate always takes the gradient on its output and distributes it equally to all its inputs, because $\partial(x+y)/\partial x = \partial(x+y)/\partial y = 1$. A max gate routes the gradient to exactly one input: the one that had the highest value during the forward pass, since the smaller inputs had zero slope. A multiply gate takes the two input activations, swaps them, and multiplies by its output gradient, because $\partial(ab)/\partial a = b$ and $\partial(ab)/\partial b = a$. Every gate can compute its own local gradient immediately and independently, without knowing anything about the rest of the circuit. CS231n

Our example composes those rules: the multiply gate says $\partial f/\partial q = z$ and $\partial f/\partial z = q$ (swap the inputs). The add gate says $\partial q/\partial x = \partial q/\partial y = 1$, so by the chain rule $\partial f/\partial x = \partial f/\partial q$ and $\partial f/\partial y = \partial f/\partial q$. Three local rules, applied in reverse order, reproduce the chain rule over the whole graph without ever writing out the full expression.

The debugging analogy is almost too fitting to resist. Backprop is like tracing how an output value was produced by walking backward along its dependency chain. The forward pass retaining intermediate values is exactly a debugger holding every intermediate variable in memory; the backward pass reads them back in reverse order, multiplying local gradients as it goes. Each gate is responsible only for its own local rule, just as each function in a call chain owns only its own state. The two passes are fully decoupled, and the cache is the contract between them.

Reverse-mode accumulation: backprop in one sentence

The sentence to keep is this: backpropagation is not "the chain rule." The chain rule is the mathematics; backpropagation is the organization of the chain rule as reverse-mode accumulation on a computation graph, one forward pass that caches and one backward pass that accumulates, with cost proportional to the number of parameters rather than to the size of a symbolically expanded expression. The field-standard textbook frames it the same way: back-propagation and its modern generalizations compute exactly these gradients over a computation graph. Deep Learning

This is the procedure Rumelhart, Hinton and Williams put on the map in their 1986 Nature letter: a learning procedure that repeatedly adjusts the weights of a network to minimize the difference between the actual and the desired output, thereby letting internal hidden units come to represent useful features of the task. Rumelhart et al.

Every modern automatic-differentiation framework, PyTorch, JAX and TensorFlow included, implements exactly this heart: a graph, a forward pass that caches, a backward pass that accumulates. Nothing about it has changed since 1986 except the scale and the software.

With gradients now cheap to obtain, nothing stops us from asking the structural question we have been avoiding: how expressive is a machine built only from linear stages? The answer is embarrassing, and its repair forces the last stage of the pipeline.

7. When a Straight Line Is Not Enough: XOR, the One-Hidden-Layer MLP, and Why Nonlinearity Is Structural

XOR: the smallest proof that linear boundaries fail

Take the exclusive-or function: four points, (0,0), (0,1), (1,0), (1,1), where the label is 1 exactly when the two coordinates differ. Try to separate the 1-points from the 0-points with a single straight line. There is no such line. Any line you draw leaves one pair of opposite corners mixed together. That is the smallest possible demonstration that a single linear decision boundary has a hard expressive ceiling, and it applies verbatim to the linear classifier of the previous chapters: there exist perfectly simple two-class functions it cannot represent at all, regardless of how well you optimize. CS231n

The textbook version of the argument lands the same blow: if you drop the activation from a one-hidden-layer network, the whole forward pass collapses into a single linear function of the input. Deep Learning

One hidden layer: nonlinearity is structural

The repair is one step of algebra away. Put a nonlinearity $\sigma$ between two linear maps. The one-hidden-layer network writes the forward pass with explicit dimensions:

$$h = \sigma(W_1 x + b_1), \qquad o = w_2^T h + b_2$$

where $h$ is a vector of hidden activations, $W_1$ maps the input into hidden space, and $o$ is a linear combination of hidden units, with the hidden width a free hyperparameter. Why must $\sigma$ be there? Because without it, the composition of the two affine maps is itself affine:

$$o = w_2^T (W_1 x + b_1) + b_2 = W'x + b'$$

The two matrices merge into one product, $w_2^T W_1$. Add a third linear layer and it merges the same way. Any stack of linear stages is a single linear stage, and the matrix-multiplication rule that makes this visible is the $C = AB$ pattern:

Caption: Matrix multiplication $C = AB$, where each entry $C_{ik}$ is the dot product of row $i$ of $A$ with column $k$ of $B$. Composition of two linear maps is itself a linear map: applying $W_1$ then $W_2$ is the single product $W_2 W_1$, which is exactly why removing the activation collapses the network into one affine function.

So nonlinearity is structural, not cosmetic. It is the only thing standing between a "deep" network and the single linear score function of chapter 3. It is literally where the model gets its wiggle: the expressive freedom of a neural network lives entirely in the nonlinearity. The architecture that realizes this is a stack of fully connected layers:

Caption: A one-hidden-layer network: three inputs, a hidden layer of four units, and an output layer of two units, fully connected between adjacent layers. The diagram shows only connectivity; the weights, biases, and the activation that make the middle layer nonlinear live in the forward equations, not in the picture. CS231n

With this architecture in hand, the natural question is how far a single hidden layer can actually go, and the honest answer has an important caveat attached.

Universal approximation, with its honest caveat

The same algebra that condemns XOR to failure for linear classifiers is the reason hidden layers matter generally. Cybenko's 1989 theorem makes the claim precise: one-hidden-layer networks with a continuous sigmoidal activation can approximate any continuous function on a compact domain to arbitrary accuracy. Weighted sums of sigmoidal units, $\sum_j \alpha_j \sigma(y_j^T x + \theta_j)$, are dense in the space of continuous functions on the unit cube. Cybenko

But the paper's own warning must travel with the result. Cybenko states plainly that his theorem is existence only, and that the important questions concern feasibility, namely how many terms are required, suspecting quite strongly that the overwhelming majority of approximation problems will require astronomical numbers of terms. Universal approximation guarantees that a solution exists somewhere; it does not guarantee that the solution is affordable. Expressive capacity and trainable feasibility are different claims, and this is why depth and width are engineering decisions rather than free gifts.

The whole pipeline, in one breath

Retrace the chain and notice that it is one continuous object, not a menu of methods. Images must be represented as numbers, which makes linear algebra the language of everything that follows: represent. A score function assigns one number per class, replacing k-NN's stored dataset with a parameter matrix: score. A loss measures whether the correct class wins by at least $\Delta$, and its scale ambiguity forces a preference over parameters themselves: loss. Gradient descent searches in the direction the loss gradient points, with mini-batches when the data is large: optimize. Computation graphs make those gradients cheap through reverse-mode accumulation: differentiate. And XOR proves a straight line is not enough, so a nonlinearity is inserted between linear maps, at which point the pipeline has become a neural network: compose nonlinearly.

Each stage existed to patch the previous stage's blind spot. k-NN could not predict fast, so parameters replaced memory. The linear scorer could not say which $W$ was good, so the loss appeared. The loss's tied solutions demanded a preference over parameters. The search demanded gradients; the gradients demanded cheap differentiation; and cheap differentiation is exactly what made a nonlinear stack trainable. Every named algorithm in this article, nearest neighbor, hinge loss, backpropagation, multilayer perceptron, is a special case of one of these six stages. For a first pass, the map is the prize: represent, score, loss, optimize, differentiate, and only then compose nonlinearly.

If you want to go deeper, each stage has a fuller treatment ready: the validation protocol that picks hyperparameters honestly is worked out with cross-validation in Hyperparameter Tuning, Representation Limits, and Linear Score Functions; the hinge loss's full derivation with worked numerical examples lives in Multiclass SVM Hinge Loss, Margins, and Scale Ambiguity; and the softmax cross-entropy loss, the mechanics of regularization, and multilayer backpropagation are developed in Regularization, Softmax Loss, Backpropagation, and Multilayer Networks.

References

The core sources cited in this tutorial, for deeper study:

1. CS231n: Convolutional Neural Networks for Visual Recognition — Lecture Notes — https://cs231n.github.io/
2. Goodfellow, Bengio, Courville — Deep Learning (MIT Press, 2016) — https://www.deeplearningbook.org/
3. Strang — Introduction to Linear Algebra, 5th Edition (2016) — https://math.mit.edu/~gs/linearalgebra/ila5/indexila5.html
4. Cover & Hart — Nearest Neighbor Pattern Classification (IEEE TIT, 1967) — https://ieeexplore.ieee.org/document/1053964
5. Cortes & Vapnik — Support-Vector Networks (Machine Learning, 1995) — https://link.springer.com/article/10.1007/BF00994018
6. Rumelhart, Hinton & Williams — Learning representations by back-propagating errors (Nature, 1986) — https://www.nature.com/articles/323533a0
7. Cybenko — Approximation by superpositions of a sigmoidal function (MCSS, 1989) — https://link.springer.com/article/10.1007/BF02551274