Modern AI, plainly: a stack of learned weights that predicts the next word

You hear "AI" everywhere. It writes essays, drives cars, holds conversations, suggests a diagnosis. It is tempting to picture it as a super-fast calculator or a machine running clever pre-set instructions. Both pictures are wrong, and this whole tutorial exists to replace them with a truer one. By the time you finish, you should be able to explain, in one sentence and without hand-waving, what the thing everyone calls AI actually is: a large stack of numbers, called weights, that was not written by a human programmer but slowly discovered from data, and that does one surprisingly humble thing, predicting the next word over and over. Everything else, we will see, is a consequence of scaling that humble habit.

We build to that conclusion step by step: what AI is and is not, why everyone chose to learn from data rather than write rules, what the smallest learning unit computes, why it has to be bent out of "linear" shape, why going deep was hard, and how a set of tricks unlocked first generation and then the language models that read and write with you.

1. What is AI, and what it is not

Fast and automatic are not the same as intelligent

A hand calculator adds a billion numbers a second. A washing machine runs its full cycle on its own. Nobody calls either of them "intelligent", and the reason is worth making precise, because it is the trap most beginners fall into. A fixed program machine does exactly what its designer anticipated. Give it an input it was not designed for, and it produces garbage or just stops. It does not change its behaviour in response to fresh experience, and it cannot be reused for a task its builder never imagined.

A second generation of counter-examples blurs the line, which is exactly why they are worth examining. Chess engines like Deep Blue and quiz machines like Watson (we will meet them properly in the next chapter) are automatic and very fast. Are they AI? A useful interim answer comes from how they were made. Deep Blue and Watson were largely programmed: experts encoded knowledge and search rules by hand. The thing that today wears the label "AI" was mostly learned: nobody wrote down the rules for recognizing a cat; a system was shown millions of labelled cat pictures and adjusted its numbers until it could do the job. That "learned numbers instead of written rules" trait is the central thread of this tutorial.

We can now state a working definition. AI is about getting machines to do tasks that normally demand human intelligence, and the tasks split roughly along five capability dimensions: perceiving (seeing images, hearing speech), learning (improving from experience), reasoning (connecting facts logically), language (understanding and producing text and speech), and acting (moving through and affecting the world). A single system rarely needs all five; most applications pick the subset the task calls for.

Three ingredients turn "AI as an idea" into AI that works: data (examples to learn from), compute (the machine muscle to crunch them), and algorithms (the learning procedures). In the modern era the balance among the three shifted hard toward data and compute. The weights we described above are extracted from data, so more, better data and more compute generally buy a more capable system.

Turing's operational trick: judge behaviour, not inner sparks

If you ask "can a machine think?", philosophy departments could argue forever, and Alan Turing (1950), in the paper Computing Machinery and Intelligence, said roughly that. He judged the question itself too vague and meaningless to settle by debate. His move was to replace the abstract philosophical question with an observable behavioural one, a test later called the imitation game. In his original setup three people sit in separate rooms and can communicate only through a typewriter. An interrogator asks questions and tries to tell which of the two respondents is the man and which the woman; the man tries to deceive, the woman to help. Turing then asked: what happens when a machine takes the man's place? If, after five minutes of typed conversation, the interrogator cannot reliably tell the machine apart from a human, the machine has passed a meaningful, checkable bar for "intelligent". The invented philosophical worry "does it really understand?" is set aside; what matters is that its behaviour is indistinguishable from a human's Turing.

Notice how radical this reframing is, and keep it in mind, because we will come back to it at the very end. Turing did not claim that passing the game proves the machine has an inner conscious experience. He proposed that, for practical purposes, how the machine behaves is the only testable thing. "Behaves like a person" and "understands like a person" are two different claims, and Turing explicitly parked the second one. Almost everything that follows in this tutorial is an exercise in applying this behaviourist test to deeper and deeper machinery.

So at the end of chapter one, hold two ideas: modern AI is a system that learns numbers from data rather than runs hand-coded rules, and the criterion by which we judge it is behaviour, not inner understanding. The next question naturally rises: why did the field ever decide that learning from data was the right road in the first place?

2. Two roads to being smart: write rules, or learn from data

The road of written rules

In the early days there really was a mainstream view that intelligence could be encoded. A computer is very good at following instructions, so why not write down, in painstaking detail, the knowledge and reasoning an expert human uses? This camp is called symbolic AI. Build a big enough database of facts and a set of if-then rules, the thinking went, and intelligence would emerge from formality. The road produced genuine milestones, best seen as a timeline:

All of these were, in essence, elegant ways to run human knowledge faster. And they hit a wall that is worth feeling in your gut: there are too many things a human expert just cannot articulate as crisp rules. What are the rules for recognizing a friend's face? For telling a poem from a news article? Most of our expertise is not conscious and rule-like; we cannot type it out even though we can reliably perform it. If intelligence can only be encoded by writing down rules, and most intelligence cannot be written down, the whole project stalls. Human experts could technically update rule-bases, but the effort grows without bound for tasks that "just feel obvious". Turing

The other road: learn from examples

There was a second, quieter track, and this is the one that won. Instead of being told the answer, a machine is shown many examples and adjusts itself until it produces the right behaviour. In machine learning terms, the computer learns by example: it is handed inputs together with desired outputs and nudges its internal numbers so that, increasingly, it produces the desired output for a new input it has never seen. Nobel

The biological inspiration was the brain. In the 1940s, researchers began asking whether the brain's network of neurons could be described mathematically. Around the same time, the psychologist Donald Hebb proposed a simple rule for how learning happens: when neurons fire together, the connection between them is strengthened. That idea, now called Hebbian learning, survives to this day as one of the basic ingredients in how artificial networks update themselves. In the artificial version, your nodes stand in for neurons and can hold numerical values, and the connections between them stand in for synapses and can be strengthened or weakened. Whether you look at a real brain or an artificial one, the story is the same: activation together strengthens the link.

Caption: The Royal Swedish Academy's official illustration sets a biological neuron (left) beside an artificial network of numerical nodes and connections (right). In both, ties are labelled stronger and weaker: when units are active together, the connection between them is reinforced, which is the Hebbian rule in picture form. Nobel

This "learn by adjusting connections" line is called connectionism. It nearly died twice. Around the end of the 1960s, discouraging theoretical results led many to doubt neural networks would ever be practically useful. Interest revived in the 1980s, driven in part by two physicists you will meet repeatedly. John Hopfield (1982) built a network whose stored patterns could be retrieved from partial or noisy input, by thinking of the whole network as rolling a ball down a landscape of valleys. Geoffrey Hinton and Terrence Sejnowski then extended Hopfield's ideas with statistical physics into what they called a Boltzmann machine (1985). Then came a second trough in the 1990s, when many researchers again lost faith, before a 2006 technique called layerwise pre-training — training restricted Boltzmann machines one stacked layer at a time — revived the field once more.

To see what those three names actually differ on, look at the official diagram. A Hopfield network connects every node to every other node. A Boltzmann machine separates its nodes into visible ones that receive input and hidden ones that influence the network's overall state. A Restricted Boltzmann machine removes the links between nodes within the same layer, and precisely because same-layer nodes cannot talk to each other, you can chain several of them: train the first, then feed its hidden nodes' values into the next, and so on. That chaining is exactly how the 2006 revival worked.

Caption: Three network shapes drawn by the Royal Swedish Academy: a Hopfield network with all-to-all connections, a Boltzmann machine with visible (solid) plus hidden (shaded) nodes, and a Restricted Boltzmann machine with no links within a layer, which is why several can be stacked and trained in sequence. Nobel

The effort was not pointless. Around 2010, three things converged: vastly more data, vastly more compute (graphics processors, CUDA), and correct formulations. Together they flipped the field. To feel how far it came, consider one number trip. Hopfield's original 1982 network used 30 nodes; counting node values and connections, it had under 500 adjustable parameters. Today's large language models are estimated to hold more than a trillion parameters. Nobel

That last achievement is why the 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton, "for foundational discoveries and inventions that enable machine learning with artificial neural networks". The connectionist road, twice written off as a dead end, had become the mainstream answer to Turing's question — learning from data beat writing down rules.

So the question that will occupy the next several chapters: that "thing that learns numbers from examples", what does it actually compute on a single run? We zoom in on its smallest unit now.

3. The atom that learns: one neuron and its forward pass

The biological neuron of chapter two is, for our purposes, only an inspiration, and this is worth saying plainly: an artificial "neuron" is not a brain cell. It is a small arithmetic operation, beloved because it can be stacked a million times. The historical bridge between the two is the McCulloch–Pitts unit from the 1940s, a threshold device that either fires or does not. Our modern artificial neuron is its smoothed-out descendant. Recall the official picture from the last chapter: an artificial network is nodes holding numbers, connected by weighted links. Now we give that picture its one-line mathematics.

The weighted sum and the soft switch

Suppose a neuron receives several inputs. Call them $x_1, x_2, x_3$. Each arrives along a link with its own strength, its weight $w_i$: think of the weight as "how much this input should count". The neuron first gathers everything into a single tentative score:

$$ z = w_1 x_1 + w_2 x_2 + w_3 x_3 + b $$

The plus $b$ is the bias: it sets the threshold of the decision, like sliding where "on the fence" sits. If the pattern "has four legs, pointy ears, a wet nose" produces a high $z$, and "round red fruit" produces a low one, then the neuron is roughly deciding dog versus no dog. But $z$ can be any real number from $-\infty$ to $+\infty$, and usually we want a clean answer between 0 and 1 that can act like a confidence. So we pass $z$ through a squashing function. The classical one is

$$ \sigma(z) = \frac{1}{1 + e^{-z}} $$

called the sigmoid, from its S-shape. Its whole job is to compress $(-\infty, +\infty)$ into $(0,1)$. Watch what it does at the extremes and in the middle: for very large $z$ it hugs 1 ("yes, dog"), for very negative $z$ it hugs 0 ("no"), and near $z=0$ it varies smoothly through the middle. A physical switch has no transition zone: on or off, nothing between. The sigmoid is a soft switch whose decision is continuous and differentiable, which is exactly the property that lets us later nudge it with mathematics. Every symbol here, the weighted sum, the bias, and the squashing function, is defined so you could hand-compute a single neuron if you had to. D2L

Many neurons, stacked into layers

A single neuron can utter only a yes/no about one narrow thing. A real system needs thousands of these units cooperating. So we arrange many neurons into layers: an input layer receives the data, one or more hidden layers each contain many neurons of exactly the kind above, and an output layer produces the final answer. Crucially, every neuron in one layer connects to every neuron in the next, which is why this is called a fully connected network; the architecture is a feed-forward one because the signal moves strictly forward, input to hidden to output, on a single pass. The word "deep" in deep learning refers to having many such hidden layers stacked. Each neuron in each layer is doing the same two-step dance we just derived: weight its inputs, add a bias, apply a squashing activation. Nobel

By now you can mechanically recite what one run does: take inputs, multiply by weights, add biases, squash each through an activation, repeat layer by layer, and read off an answer in the output. The mystery of "what is it doing?" is, at the single-step level, no mystery at all — it is arithmetic. But a worrying thought may have stuck with you from the last chapter's triumphant depth. If each neuron is just a weighted sum, and a deep network is just many weighted sums in a row, is stacking them actually giving us anything a single fat layer could not? Why shouldn't a deep network just collapse into a shallow one? That question is the whole subject of the next chapter, and its answer is the first genuinely surprising thing in this tutorial.

4. Why stacking layers of math is not enough: a nonlinear gate is required

Here is the desire: "more layers equals more clever", because a deeper network can build ever more nested concepts. Here is the problem: if each layer is only a weighted sum, then multiple layers mathematically collapse into one, and depth buys you nothing.

Proving that plain linear layers collapse

Take two layers. The first takes input $\mathbf{x}$ and produces a hidden value; the second takes that and produces output. Written out:

$$ \mathbf{o} = ( \mathbf{x} W^{(1)} + b^{(1)} ) W^{(2)} + b^{(2)} $$

where $W^{(1)}, W^{(2)}$ are the weight matrices and $b^{(1)}, b^{(2)}$ the biases of the two layers. Now expand, because matrix algebra obeys distributivity:

$$ \mathbf{o} = \mathbf{x} W^{(1)} W^{(2)} + b^{(1)} W^{(2)} + b^{(2)} $$

The product $W^{(1)} W^{(2)}$ is just another matrix, call it $W$, and the messy bias terms $b^{(1)} W^{(2)} + b^{(2)}$ are just another vector, call it $b$. So

$$ \mathbf{o} = \mathbf{x} W + b $$

which is the equation of a single linear layer. Two layers collapsed into one, with no loss. Repeat the argument for a hundred layers and you keep collapsing until you are back at one. The unmistakable conclusion: a deep network with no nonlinearity between layers is nothing more than a single linear model, a kind of linear regression no matter how many layers you pile on. Diagonal line in, diagonal line out, forever. D2L

Here is the point where reality is genuinely counter-intuitive, so say it plainly: more layers is not automatically more powerful. Without a nonlinear gate between layers, a hundred layers are no stronger than one. The desire for depth only becomes meaningful once each layer bends the signal out of "straight line" shape before passing it on. The gate that does this bending is called an activation function, and you now see why every hidden layer must apply one after its weighted sum.

The three classic activation gates

Three gates dominate the textbooks. We already met the sigmoid, $\sigma(z) = 1/(1+e^{-z})$, which squeezes into $(0,1)$. Closely related is tanh, which rescales the same S-curve to sit in $(-1,1)$ and is symmetric about zero. And the modern workhorse is ReLU (rectified linear unit):

$$ \text{ReLU}(z) = \max(0, z) $$

ReLU is brutally simple: it passes positive values through untouched and turns everything negative into zero. Its appeal is twofold. Computing it costs only a comparison, next to nothing. And its derivative behaves beautifully: on the positive side it is exactly 1, on the negative side exactly 0. A ReLU gate either lets a correction signal pass through cleanly or blocks it, never throttling it into a vanishingly small value. Compare that to the sigmoid, whose steepest slope, at $z = 0$, is only $0.25$, and which flattens out to essentially zero slope as $z$ grows large in magnitude. Chains of sigmoids therefore choke gradients, a problem we will dissect in the next chapter; that saturation is precisely why ReLU displaced the sigmoid inside hidden layers. (The sigmoid survives where you need a clean probability between 0 and 1, such as the final output layer of a binary classifier — or inside the gating of recurrent networks.) D2L

The three gates in one glance:

The conceptual payoff of this chapter is compact and powerful: linear layers collapse, nonlinear gates make depth real. Only once the signal is bent into curves at every layer can a deep stack represent more intricate functions than any single layer could. But this hard-won depth brings with it a fresh catastrophe that chapter five is about to set you up to meet.

5. Wanting to go deep is not the problem — vanishing gradients were. Here is the fix.

You now have the desire (depth buys expressive power) and the tool (nonlinear gates keep layers from collapsing). So: just make the network very deep and enjoy. Reality refuses. Around 2015, researchers pushing to ever deeper plain networks hit a wall: make a network with more layers and it did not get better, it got worse. And the worsening was not overfitting, the usual suspect. On CIFAR-10, a 56-layer plain network achieved a higher training error than a 20-layer one — it could not even fit the training data it had seen, let alone generalize. The problem was that these deep nets were simply hard to optimize. ResNet

Where the training signal goes: the gradient is multiplied layer by layer

To understand why deep networks resist training, you need to know how a network "learns" in the first place. A network produces an output, an algorithm compares that output with the correct answer, computes an error, and then sends a correction signal backward through the layers telling each weight how much to change to reduce the error. That backward correction signal is the gradient. The crucial fact you need is not calculus but intuition about multiplication: as the signal travels from the output back toward the early layers, at every boundary it is multiplied by a local sensitivity factor, roughly the local slope of that layer's activation. Follow a chain of such factors and you are multiplying, not adding.

Now press on the two directions. Suppose the factors are mostly small, around $0.5$ in magnitude. Raise that to the power of twenty layers:

$$ 0.5^{20} \approx 0.000001 $$

The correction signal has shrunk to nearly nothing by the time it reaches the early layers. Those layers receive no meaningful feedback, so they never learn, and the "deep" part of the network is effectively dead weight. This is the vanishing gradient. In the other direction, suppose factors mostly exceed $1$, around $2$:

$$ 2^{20} \approx 1{,}000{,}000 $$

Now the signal explodes into absurdly large updates that send the weights careening, and training never settles — the exploding gradient. Either way a key fact: multiply a number twenty times and you are no longer in normal arithmetic territory. Gradient-based training only works when the per-layer factors multiply to something manageably near one; deep stacks routinely violate this. D2L

The fix: learn the residual

The breakthrough that tamed depth was the residual network (ResNet). The core re-framing is quiet but deep. Conventionally a stack of layers is asked to learn the true mapping $H(x)$ directly. ResNet instead asks the stack to learn the residual $F(x) = H(x) - x$, the difference between the desired output and the input itself; the true output is then reconstructed as

$$ H(x) = F(x) + x $$

Why is this easier? The critical case to reason about is when the ideal mapping for those layers turns out to be the identity — meaning the "right" thing for the layer to do is pass its input through unchanged. Learning that directly means pushing all the weights of a deep stack to reproduce the identity, which is fiddly. Learning the residual instead means pushing $F(x)$ to zero, which you accomplish by simply shrinking the layer weights toward zero. Driving something to zero is dramatically easier than sculpting a general identity transform. The empirical evidence lands on the same side: the residual formulation let a 152-layer network train successfully on ImageNet, in fact eight times deeper than the 19-layer VGG net yet with markedly lower FLOPs. The headline result on ImageNet, however, belonged to an ensemble of such residual nets, which reached a 3.57% top-5 error and won the ILSVRC 2015 classification competition. ResNet

Where exactly does the magic happen? Look at the residual block's structure. The input $x$ travels down two roads. One road passes through a couple of weighted layers and a ReLU, emerging as $F(x)$. The other road is an identity shortcut: the input jumps straight ahead, untouched, and is simply added to $F(x)$ before one final activation. That shortcut is a gradient expressway. Because it carries the signal forward without any multiplication by weights, the error signal flowing backward along it also encounters no shrinking or exploding factors. Gradients now have a fast lane back to the earliest layers, and the deep network finally becomes trainable.

Caption: The paper's building block. An input x passes through a weight layer, a ReLU, and another weight layer to produce F(x); meanwhile an identity shortcut carries x unchanged to be added to F(x), and the sum then goes through a final ReLU. Learning the residual F(x) = H(x) - x, with the shortcut preserving the gradient path, is what makes deep stacks trainable. ResNet

Caption: Three ImageNet architectures compared in the paper: VGG-19, a plain 34-layer net, and a residual 34-layer net at the same complexity (3.6 billion FLOPs each). The residual version's extra dotted shortcuts are what let it optimize where the plain net stalls. ResNet

The residual idea became the load-bearing wall of all modern deep networks — including, as we will see, the Transformer. By restoring the gradient's path to shallow layers, ResNet made it safe to be deep, and once you can be deep, you can aspire to model vastly more complex things. Which raises the next ambition: what if the network did not merely classify the world, but produced new content from nothing? We are ready to leave judging and enter the world of generating.

6. From judging to inventing: two 2014 recipes for generation

Everything up to here has been discriminative in spirit: give the machine an input, and it tells you something about it — this is a dog, this sentence is positive, this pixel is a face. But a much stranger ambition waited in the wings: could a machine make something it was never shown, a new face, a new sentence nobody had written? That goal is generation, and as with most of this field, its modern form was crystallized in 2014, by two simultaneously published papers with sharply different philosophies.

Recipe one: the Variational Autoencoder (VAE)

The first recipe is reconstruction with a bottleneck. Think of an autoencoder: an encoder squeezes an input into a compact internal code, and a decoder tries to reconstruct the input from that code. The compression forces the code to capture what is essential. The variational autoencoder makes one small but transformative change: instead of compressing each input into a single fixed code, the encoder outputs a distribution of likely codes — a cluster of possible hidden values — and the generative act draws one sample from that cluster.

Concretely, let $\mathbf{x}$ be a data point, say an image. The encoder maps it to a hidden (latent) variable $z$ described by a distribution, in the simplest case a gaussian with a mean and a spread, so there is a whole family of codes that could plausibly explain $x$. Generation then has a clean grammar: sample one code $z$ from that distribution, and let the decoder map $z$ back to a plausible $\mathbf{x}$. Because $z$ can be drawn freely — you do not have to start from a real image — the model can invent genuinely new outputs that read like the data it was trained on. VAE

The one technical subtlety a beginner should feel, rather than slog through, is the reparameterization trick. Sampling costs a gradient: if $z$ is literally drawn at random, we cannot differentiate through the draw to update the encoder. The trick writes the sample as

$$ z = \mu + \sigma \varepsilon $$

where $\mu$ and $\sigma$ come from the encoder and $\varepsilon$ is fixed random noise drawn from a standard normal. Now the randomness has been moved into the noise term, and the path from $\mu, \sigma$ to $z$ is a clean, differentiable arithmetic ride, so ordinary gradient training can flow through it. VAE

Generation flips this to a shortcut: drop the encoder and sample a fresh z straight from the latent distribution, then run only the decoder to produce a new output.

The VAE sits squarely in a distinguished lineage: as an early generative model it learns the distribution behind the data and samples from it. But its samples, while structurally sound, tend to look slightly blurry, because forcing a gaussian over every code smooths away crisp details. What if, instead of carefully modelling a distribution, you just let two networks fight until one is forced to be an expert forger?

Recipe two: the Generative Adversarial Network (GAN)

The second recipe is a game with two players. A generator takes random noise and turns it into a fake data point. A discriminator looks at a data point and tries to say whether it is real (from the true data) or fake (made by the generator). They train against each other in a loop: the generator tries to fool the discriminator; the discriminator tries to catch it. Pushed to its equilibrium, the generator's fakes become indistinguishable from real data, and the discriminator, now unable to tell the difference, is stuck answering "fifty-fifty" — a perfect coin flip is the signature that the generator has won. GAN

Caption: The paper's four-panel intuition figure: as training proceeds the discriminator curve (dashed) homes in on telling real from generated data, the generator's sampled distribution p_g (solid green) is pushed toward the true data distribution p_data (dotted black), until the two match and the discriminator outputs 1/2 everywhere, meaning it is defeated. GAN

Framed that way the contrast with the VAE is crisp. The VAE explicitly models a latent distribution and pays a reconstruction price; the GAN refuses to write down any explicit density and instead learns "for our purposes, indistinguishable" through adversarial pressure. VAE outputs trend blander but controllable; GAN outputs trend sharper but notoriously unstable to train. Both are 2014 answers to the same underlying impulse, learn the distribution of the data and sample from it, and together they are the conceptual preamble to everything that claims to "generate" today. GAN

Hold that equation in mind: generation = learning a distribution and sampling from it. Yet neither the VAE nor the GAN is what actually powers today's chatty, essay-writing, image-crafting AI. Their 2014 ambitions set the stage, but the engine that let "creating new content" scale to behave like reading and thinking arrived in 2017 with a different architecture entirely, one that trades every earlier device for a single mechanism: paying attention.

7. Everything collapses to one habit — predict the next word. Here is what that buys, and hides, in the world.

From sequences to attention: the Transformer

Natural language, images, audio: at bottom, they all arrive as long sequences, and until 2017 the leading architectures processed sequences one elementary step at a time, feeding the output of each step into the next. That sequential, step-by-step design was slow to train and struggled with very long contexts. In June 2017, a paper titled Attention Is All You Need swept the recurring structure away and replaced it with a single principle called attention, presenting the Transformer. Transformer

What is attention? Give it to you as intuition first. When you read "the bank by the river", you do not treat every earlier word as equally relevant to "bank"; you flick your eyes back to "river" and up-weight it, because it disambiguates "bank". Attention is that up-weighting, made into a machine. Each position in the sequence computes how relevant every other position is to it, and then pulls, from the whole context, a weighted blend weighted by that relevance. The paper implements this with three roles for every word: a query (what am I looking for), keys (what each context word offers), and values (the content to actually pull). The compatibility between a query and each key sets a weight, and the output is a weighted mix of the values. The formula — scaled dot-product attention — is:

$$ \text{Attention}(Q, K, V) = \text{softmax}!\left( \frac{QK^T}{\sqrt{d_k}} \right) V $$

where dividing by $\sqrt{d_k}$ keeps the inner products from ballooning into huge numbers that choke the softmax. Doing this in several parallel "heads", each attending with different learned projections, lets the model look at different relationships at once; that parallel multi-head trick is what protects the model from averaging all its information into mush. By the paper's own framing, multi-head attention "allows the model to jointly attend to information from different representation subspaces". Crucially, all positions can attend to all others in a fixed, parallel number of operations — no sequential waiting. Every token looks at the whole context, at once. Transformer

One device inside the Transformer deserves a sentence of its own, because it quietly seeds the entire "predict the next word" story. Because attention contains no recurrence, nothing tells the model the order of words; positional information must be injected explicitly as numbers added to the input. And because the Transformer was designed as a translation engine, its decoder is deliberately auto-regressive: while generating, it consumes the symbols it has already produced as additional input to predict the next one, and its self-attention is masked so a position can only look at earlier positions, never at the future it is supposed to guess. That masked, "given the preceding tokens, predict the next token" object is the exact ancestor of the objective every large language model is trained on. Transformer

Caption: The paper's overview of the pure-attention Transformer: an encoder stack and a decoder stack, each built from multi-head attention and feed-forward sub-layers wrapped in residual connections, with input and output embeddings plus positional encoding at the bottom and a linear-plus-softmax head turning the decoder output into next-token probabilities. Its bottom-to-top flow hides nothing about the operative process: tokens in, next tokens out. Transformer

If you suspect the Transformer is doing something familiar to chapter five, you are right: its residual connections and layer normalization are the ResNet idea, applied block by block inside an attention machine. Equally important, notice which half of the architecture the later state of the field kept. GPT-style language models discard the encoder entirely and keep only an auto-regressive decoder: an input prefix, an attention layer, a probability over what comes next.

The habit at scale: GPT-3

Here is where the whole tutorial snaps shut. A language model's training target is a single, almost embarrassingly plain habit: given the tokens before, predict the next token. Watch it write. It looks at "The cat sat on the", guesses the next word is likely "mat", appends it, and repeats with the now-longer prefix, forever. Whatever this model does — answer a question, write the start of a story, translate a sentence — it is the same habitual next-token prediction, run again and again. Translation is "predict the next token" conditioned on the foreign sentence. A factual answer is "predict the next token" conditioned on your question.

What rescues this from being a party trick is scale. In 2020, OpenAI's GPT-3 took the Transformer and scaled the identical next-word objective to 175 billion parameters — ten times larger than any earlier non-sparse language model — trained on hundreds of gigabytes of text. The astonishing observation was in-context learning: at this size, you feed the model a task plus a few worked examples typed straight into the prompt, and without any weight update, without turning off gradient updating (there is none at inference), the model generalizes to new instances of the task simply by "predicting what comes next". Larger models were markedly more fluent at this. GPT-3

The sobering framing you need: a large language model is not a database you poke for stored facts, and "ChatGPT chats" and "predict the next word" are the same event, not two different capabilities. The habit is the engine. Scale brings out the fluent, few-shot behaviour that makes it feel like an agent that understands your intent, but underneath is that single statistical habit running uncounted times. The LLMSurvey's summary of model history makes the convergence visually obvious: after a lineage of statistical, neural, and pre-trained language models, the "large" generation takes that same language-model objective and applies it at a scale where behaviour changes qualitatively — from pure language modeling to solving tasks. LLMSurvey

Caption: The LLMSurvey's evolution diagram from a task-solving perspective: statistical LMs and neural LMs gave way to pre-trained LMs and finally to large language models, where scaling past pre-trained sizes is associated with the shift "from language modeling to task solving". Stage boundaries are approximations, not hard dates. LLMSurvey

By late 2022 ChatGPT carried that habit to the general public, and the resulting wave is what most of us now call "AI". Along the timeline, instruction tuning, RLHF-based alignment, and reasoning steps were layered on top, but none of them replaced the auto-regressive core; they steered and wrapped it. LLMSurvey

What this buys, and hides, in the world

Finally we step out of the mechanism and into your daily news feed, because the same habit, deployed into different domains, is the economic and social event of the current decade. And here the pattern is strikingly uniform once you know to look for it: a single general base model, plus domain-specific data and workflows, plus human oversight, yields a copilot for that domain. The medical chat copilot advising on a clinical case, the financial analyst copilot citing filings, the tutoring copilot explaining a proof, and the tourism guide — different surfaces, same pipeline underneath: one general-purpose learned base, adapted to a field, kept under human supervision.

Consider the medicine and finance cases the course runs on: a domain-specialized clinical copilot built around a general base, guiding diagnosis; finance copilots that pull research and filings; education platforms and language apps that tutor; mental-health bots like Wysa and Woebot that hold supportive conversations; and cultural-tourism audio guides. All of them are the general foundation model in a domain outfit, not bespoke inventions from scratch. The uniformity is not an accident of marketing; it is the economics of a single reusable base. Independent data supports the wave's scale. Stanford's AI Index 2026 reports that by 2025, 88% of surveyed organizations had adopted AI, with generative AI used in at least one business function at 70% of them; in the United States, four in five high-school and college students report using AI for schoolwork; ambient "AI that writes the clinical note" systems saw physicians spend up to 83% less time on note-writing in one hospital system. AI Index

Even an ordinary consumer meets the pattern. Google's Talking Tours – India Edition (launched February 2026) runs Gemini inside a Street View cultural-tourism experience: you navigate a 360° view, press a snapshot button, and Gemini reads the visual scene plus your location, generates a context-aware script about the landmark, turns it into natural audio in English and Hindi, and poses follow-up questions. At launch it covered more than 200 iconic locations such as the Red Fort and the Taj Mahal, per its own announcement. What it really demonstrates is the reusable base again: Gemini the general multimodal model, plugged into a brand-new setting (a historical site) with new context data (imagery and GPS), producing a domain-specific audio guide. Talking Tours

But the honest epilogue, and the bookend to Turing, is the raw material these systems are made of. A machine whose entire skill is predicting the next token has no inner compass that tells it whether the confident-sounding next token is true. It can hallucinate: emit fluent, self-assured sentences that are factually invented, because next-token probability and truth are different currencies. It can encode unfairness and bias inherited from the text it learned from, in ways that land differently on different groups. Its answers about private matters are generated without any guarantee of privacy discipline. And society is scrambling to govern it: the AI Index 2026 documents that documented AI incidents rose from 233 in 2024 to 362 in 2025, and that experts and the public diverge sharply — 73% of experts expect AI to affect jobs positively versus 23% of the public, a fifty-point gap. Meanwhile China and the UAE have mandated AI education starting in the 2025–26 school year, and in the United States barely half of schools have any AI policy at all, with only 6% of teachers calling it clear. AI Index

None of which should be overstated into mystical fear. These systems offer genuine, measurable value: a conversational tool is not a database to query, and we should hold in our heads simultaneously that it can be stunningly useful and that it can calmly assert a falsehood. Governance, fairness, privacy, and accuracy under pressure are exactly the frontier that research on large models lists among its open problems: why next-token-prediction produces general problem solvers at all, the fairness and safety of alignment, the hallucination that is a symptom of probabilistic modeling. LLMSurvey

Which brings us back, full circle, to the 1950 Turing test. A modern chatbot passes large portions of Turing's imitation game embarrassingly often, and that is real progress on the behavioural criterion. But Turing himself separated "behaves like a human" from "understands like a human," and parked the second. What we have built is behaviour: a trillion numbers that predict the next word shockingly well. Whether fluent behaviour is understanding is the exact question Turing declined to answer, and it is as open today as it was in 1950. The machine has learned, at enormous scale, to talk like us. What it has not yet agreed to answer is whether anything like a mind lives behind the words.

Summary

If this entire tutorial could be compressed into three sentences, they would be these. Fast and automatic execution is not intelligence; modern AI is a stack of numbers, its weights, learned from vast data rather than written by a programmer. Each smallest unit inside that stack computes a weighted sum pushed through a nonlinear "soft switch", and those switches only earn their keep once nonlinearity keeps deep layers from collapsing — while skip connections keep deep stacks trainable when gradients would otherwise vanish. And the many systems you meet in the news are, at bottom, one engine scaled up: a network trained, over and over, to predict the next word, then pointed at medicine, finance, education, therapy, and tourism, where it behaves impressively and honestly like a person — while the question of whether it understands, the question Turing set aside in 1950, remains unanswered to this day.

References

1. Computing Machinery and Intelligence (A. M. Turing, Mind, 1950) — courses.csail.mit.edu
2. The Nobel Prize in Physics 2024 — Popular science background ("They used physics to find patterns in information") — nobelprize.org
3. Dive into Deep Learning (D2L, online PyTorch ed. v1.0.3) — d2l.ai
4. Deep Residual Learning for Image Recognition (He et al., 2015) — arXiv:1512.03385
5. Auto-Encoding Variational Bayes (Kingma & Welling, 2013) — arXiv:1312.6114
6. Generative Adversarial Nets (Goodfellow et al., 2014) — arXiv:1406.2661
7. Attention Is All You Need (Vaswani et al., 2017) — arXiv:1706.03762
8. Language Models are Few-Shot Learners (Brown et al., 2020) — arXiv:2005.14165
9. A Survey of Large Language Models (Zhao et al., LLMSurvey) — arXiv:2303.18223
10. Artificial Intelligence Index Report 2026 (Stanford HAI) — hai.stanford.edu/ai-index/2026-ai-index-report
11. Google Blog: Talking Tours – India Edition (2026) — blog.google