One word at a time, then all at once: the four bottlenecks sequence models remove

Sequence modeling did not advance by accumulating capability. Each architecture from the first recurrent network to the Transformer and the Vision Transformer removed one specific structural limit that the previous design had left in place, and each removal is what forces the next one: a hidden state makes variable-length input possible, but it creates a single channel that all information must pass through; backpropagating through that channel over many steps makes long-range gradients unstable; the need to compress a whole input into one vector then caps what the model can carry; and satisfying that cap by re-reading the input everywhere locks the computation into a serial order. The four bottlenecks are variable-length structure, unstable long-range gradients, fixed-context compression, and sequential computation. The payoff of holding them in one frame is a single question you can ask of any new sequence-model paper: which of the four is it removing?

1. Variable-length input forces a hidden state: one vector, rewritten at every step

A single hidden-state vector, reused at every time step, is the smallest mechanism that lets one network consume an input of any length. The same reuse is also the first ceiling: whatever a later step needs must survive the squeeze into the final state.

A fixed network that maps one input to one output cannot express the task shapes that sequence work actually needs. Captioning turns one image into a sentence, and classifying an action turns a sequence of frames into one label. Below, the same three-layer stack is drawn for a one-to-one mapping, a one-to-many mapping, and a many-to-one mapping. Only the middle and right panels carry the horizontal connections that let state flow from one time step to the next, which is exactly the difference between a fixed-size network and a sequence model.

Caption: The one-to-many and many-to-one panels connect adjacent hidden units along the time axis, while the one-to-one panel has no such links. That horizontal connection is the whole mechanism: input and output lengths become free the moment a value can be carried from one step to the next.

The transferable object here is the hidden state, a vector $h_t$ that carries everything the model has seen so far, updated by a function applied identically at every step, $h_t = f_W(x_t, h_{t-1})$. Because the same parameters $W$ are used at every position, parameter count stops depending on sequence length. The hidden state is a running summary rather than a per-position parameter, and the cleanest way to hold it in mind is an accumulator that gets passed forward and overwritten as the sequence is consumed, not a fresh variable allocated per position.

Unrolling the cell into an explicit computation graph is what makes shared weights and time-indexed states visible as one graph rather than many copies of a network. That matters here because it isolates the quantity the next bottleneck is about: the same weight matrix, applied at every arrow between consecutive hidden states. The fully derived vanilla RNN equations and the unrolled graph mechanics get their own treatment in RNN Fundamentals: State, Unrolling, Sequence Tasks, and the LSTM Remedy; for the bottleneck argument, one property is enough, and it is that the arrows between steps all use the same $W$.

Glue a many-to-one graph to a one-to-many graph and you get the classical recurrent sequence-to-sequence model: an encoder reads the source sequence, a decoder expands an output from what the encoder produced. The two halves are joined by one fixed-dimensional vector $v$, taken from the encoder's last hidden state, and the decoder's entire job is defined relative to it:

$$p(y_1, \ldots, y_{T'} \mid x_1, \ldots, x_T) = \prod_{t=1}^{T'} p(y_t \mid v, y_1, \ldots, y_{t-1})$$

Caption: The encoder block on the left reads the source tokens, and the decoder block on the right emits the target tokens. Every arrow between them is accounted for by the single vector $v$, which is the bottleneck that the next two moves in this article exist to remove. This is a pre-attention architecture: there is no alignment here, and the diagram shows the reversed source ordering that the paper's training trick relies on.Seq2Seq

The single vector is concretely sized in that work: a 4-layer LSTM with 1000 cells per layer represents a whole sentence as 8000 real numbers. Reversing the source tokens before feeding them to the encoder was worth a large jump on its own, moving test perplexity from 5.8 to 4.7 and BLEU from 25.9 to 30.6, and the full ensemble reached 34.81 BLEU on WMT'14 English-to-French with 36.5 after rescoring. The reason to keep those numbers in view is not the scores themselves. It is that a system this large was, at the time, the first pure neural translation system to beat a phrase-based statistical machine translation baseline by a sizeable margin, while still routing every piece of source information through one arrow.Seq2Seq

2. The same weight multiplied over and over: long-range gradients vanish or explode, and LSTM buys an additive path

Backpropagation through time applies the recurrent weight once per step, so a gradient travelling $N$ steps is scaled by a product of $N$ such terms. When the typical factor is below one the signal decays to nothing, when it is above one it blows up, and both behaviours are the same multiplication seen from two sides. LSTM's response is a separate cell state whose update is additive and a set of multiplicative gates, which gives error a path that is not repeatedly rescaled. That makes long-range learning easier. It does not eliminate the gradient problem, and every honest description of LSTM has to say so.

The mechanics of backward flow are worth stating once because they are the entire diagnosis. Forward through the whole sequence to compute the loss, then backward through the whole sequence to compute the gradient: that is full backpropagation through time. In the simplified linear view the chain of Jacobians collapses to powers of the recurrent weight matrix, so singular values below one produce vanishing gradients and values above one produce exploding ones. Gradient clipping caps the explosive end, and it does nothing for the other end, because a long-range signal that has already been multiplied down towards zero is not recoverable by rescaling the step size. A useful picture is a signal passing through a chain of amplifiers: a factor slightly below one at each stage gives a signal that fades out over distance, a factor slightly above one gives one that saturates, and limiting the output gain protects against the second failure only.

The 1997 LSTM cell answers this with structure rather than with a better optimizer. A memory cell sits on a linear unit with a fixed self-connection of weight 1.0, so that error arriving at the cell can circulate without being scaled on the way through; the paper names this the constant error carousel. Two multiplicative gates guard it: an input gate that decides how much of a new candidate value is written in, and an output gate that decides how much of the cell's contents is exposed to the rest of the network. The update itself is additive:

$$s_{c_j}(t) = s_{c_j}(t-1) + y^{\text{in}}j(t) \cdot g(\text{net}{c_j}(t))$$

Caption: The cell body contains the candidate squashing node $g$, the input-gate multiplication, the self-recurrent loop labelled 1.0 that forms the constant error carousel, and the output-gate multiplication. The two sigmoid units below are the input gate and the output gate. The self-recurrent weight of exactly 1.0 is the point of the design: a gradient passing through that loop is not rescaled.LSTM

The additive update is what makes the carousel useful rather than just stable: because the new state is added to the old one instead of replacing it, the derivative of the cell state with respect to its previous value is close to identity, and the gradient is not forced through a squashing nonlinearity at every step. The 1997 formulation has an input gate and an output gate but no forget gate; the forget gate arrives with Gers, Schmidhuber and Cummins in 2000, so any description that lists "input, forget, output" is describing the later cell. The practical cost is also worth naming: the update is $O(1)$ per weight and per time step, and it is local in space and time, which is why this is an engineering fix rather than a theoretical construction.LSTM

The boundary of the claim is part of the claim. The guarantee in the paper is conditional, holding for error flow inside the carousel provided truncated backpropagation cuts off error trying to leak out of the memory cells; outside the carousels error flow still tends to vanish quickly, and for the same reason full backpropagation through time does not outperform the truncated version. The course material states the same limit in one line, and it is the wording to reuse rather than paraphrase upwards: "LSTM doesn't guarantee that there is no vanishing/exploding gradient, but it does provide an easier way for the model to learn long-distance dependencies."CS231n The empirical headline, bridging minimal lags in excess of 1000 steps, comes from engineered long-time-lag tasks with distractors, not from natural-language data, and should not be generalised past that.

Once the problem is stated as "the same weight multiplied at every step", LSTM stops looking like a stronger RNN and starts looking like a targeted intervention at exactly one point in the graph. That is the pattern the remaining three removals follow.

3. One fixed vector cannot hold a sentence: attention re-reads the input at every output step

The first attention mechanism was not a new network. It replaced the single compressed vector $v$ with a fresh weighted read of the encoder states at every decoder step, so that step 40 of the output is no longer required to retrieve its information from the same 8000 numbers as step 1.

The mechanism is small enough to state in one pass. Keep the encoder's per-position states instead of discarding all but the last one. At each decoding step, score every encoder state against the decoder's current state, turn the scores into a distribution with a softmax, and take the weighted sum of the encoder states. That weighted sum is the context vector for this step, and it is recomputed from scratch for the next one. In the figure below, the alignment block produces one set of weights per decoder step and each step receives its own context vector, so $c_1$ and $c_4$ are different vectors even though both are functions of the same encoder states.

Caption: The encoder states on the left are scored, normalised by a softmax, and summed into a separate context vector for each decoding step. The four context vectors $c_1 \ldots c_4$ feed the decoder states $s_0 \ldots s_4$ that emit the Italian translation one word at a time. Compare this with the single-arrow encoder-decoder diagram above: the fixed vector has been replaced by a per-step weighted read.CS231n

The same operator appears twice in the 2014-2015 literature, once per modality, which is what shows that the principle is not an artifact of translation. On the machine-translation side, the alignment score comes from a small feed-forward network that takes the previous decoder state and a source annotation and returns a scalar; the resulting weight matrix over source and target positions can be plotted, and it comes out largely diagonal with instructive exceptions, one panel showing the English phrase "European Economic Area" being reordered into "zone économique européen". The critical methodological point is that nobody supplies those alignments as labels. The alignment function is part of the model, so the gradient of the translation loss flows through it and trains it jointly with everything else.

The empirical case that the fixed vector really was the problem, and not just an inelegant detail, comes from comparing two otherwise similar systems. On WMT'14 English-to-French, the plain recurrent encoder-decoder variant scores 17.82 BLEU on the full test set, while the attention variant scores 26.75, and the gap persists on the subset without unknown tokens (26.71 against 34.16). The accompanying BLEU-versus-length curve is the part that matters: the no-attention model degrades sharply as sentences get longer, while the attention models stay flat, which is what a bottleneck diagnosis predicts rather than merely a nicer diagram.Bahdanau

On the vision side the same recipe runs over a set of spatial region vectors produced by a convolutional feature map instead of over encoder states, and the decoder emits one word per step from a weighted sum of regions. The alignment score there is a function of a region vector and the previous decoder state, which is what makes the weights move as the sentence unfolds: where the model looks next depends on the words it has already produced, not on a fixed plan. The gain over the translation setting is visibility, since the weights can be drawn on top of the image and inspected while the caption is generated.ShowAttend

Caption: A convolutional feature map supplies a set of annotation vectors, one per spatial region. Attention weights select and combine a subset into a context vector that a recurrent decoder consumes one word at a time, and the small heat map on the image shows the region being read as each word is emitted. The attention here is conditioned on the previous decoder state, which is why the gaze moves with the sentence.ShowAttend

A caution belongs with these gaze maps: a caption can be right for the wrong reason, meaning the model produced a plausible sentence while looking at the wrong region. Reading those gaze maps as evidence about what the model actually learned, and the interpretability arguments built on top of them, is a topic of its own in Attention and the Transformer: from translation alignment to vision; the part that matters for the bottleneck story is only that the weighted read is differentiable and needs no alignment labels.

The bottleneck has changed character rather than disappeared. At the start of the arc the question was how much fit into one vector. Here the question becomes how many times the decoder is allowed to look, and the answer is once per output step.

4. Query, key, value: attention becomes a layer, and every token can be its own query

Writing the same alignment computation as query, key, value is what promoted attention from a fix attached to an RNN decoder into a standalone layer with its own inputs and outputs. Once every position in a sequence is allowed to emit its own query, that layer no longer needs a recurrent decoder at all, and the model gains global access in a single parallel step. The price is that order disappears and has to be added back explicitly.

The abstraction is short. Take a set of input vectors, project them into queries, keys and values, compare each query against all keys, normalise the similarities, and use them to weight the values:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The division by $\sqrt{d_k}$ is not cosmetic. With zero-mean unit-variance components the dot product of a query and a key grows with the dimension, so large similarities push the softmax into a saturated regime where its gradients are tiny; scaling keeps the scores in a range where the distribution can still move. Multi-head attention then runs $h$ independent copies of this computation in parallel, each with its own projections, concatenates the results and applies one final linear layer. With $h = 8$ and $d_k = d_v = 64$, the total cost matches a single attention over the full dimension, and the paper's own ablation puts $h$ between 4 and 16 within roughly 0.4 BLEU of each other. Multi-head is therefore a structural choice about representation subspaces, not a hyperparameter that has been tuned to a sweet spot.

Caption: The upper panel is one attention computation: queries, keys and values are projected, matched by a matrix multiply, scaled, optionally masked, softmaxed, and used to weight the values. The lower panel is multi-head attention, which is this same pipeline instantiated $h$ times in parallel with separate projections before concatenation. Nothing is added to the operator between the two panels; only the number of copies changes.Transformer

The same pipeline in its smallest form is five lines of code, and it is instructive that the mask is one of them:

Caption: Reading the boxes in order maps the equation to operations: a matrix multiply between queries and transposed keys, a scale by $1/\sqrt{d_k}$, an optional mask applied to the scores, a softmax, and a final matrix multiply against the values. Nothing else happens between the input vectors and the output.Transformer

The implementation underneath is a single function, and the mask is one of its five lines: compute the score matrix as a matrix product divided by $\sqrt{d_k}$, overwrite the forbidden entries with a large negative number, softmax, and multiply by the values. Because masking is an operation on the scores rather than a separate component, the same function serves self-attention, cross-attention and causal self-attention, with only the mask argument changing.Annotated

Self-attention is the special case where queries, keys and values all come from the same input vectors, $Q = XW_Q$, $K = XW_K$, $V = XW_V$. That single change has two consequences that define the rest of the architecture. The first is the gain: a recurrent model needs $N$ sequential steps for position $N$ to receive information from position 1, while self-attention connects every pair of positions in one step, so the number of sequential operations per layer drops from $O(n)$ to $O(1)$. The cost moves to memory, where the attention weight matrix is $N \times N$; the source lecture works the number out for a long context, $N = 100{,}000$ with $H = 64$ heads taking 1.192 TB. The second consequence is a loss. Because the operation only compares vectors and never consults an index, self-attention is permutation equivariant: shuffle the input tokens and the output shuffles with them. It literally sees a bag of vectors rather than an ordered sequence, and a bag has no first or last element, so order has to be attached to the vectors themselves.

Attaching order is what positional encoding does. The original choice is deterministic and sinusoidal, with one geometric family of wavelengths spread across the embedding dimensions:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

Each dimension is a sinusoid of a different period, so together the dimensions encode position in a way that no single dimension could, and the pattern can be read directly off the heat map below: the low-dimensional rows alternate quickly across time steps, while the high-dimensional rows change slowly enough that they are nearly constant over the plotted range.

Caption: Time steps run along the horizontal axis and embedding dimensions along the vertical one, with colour encoding the value. Low dimensions produce dense fast stripes, higher dimensions produce progressively wider bands, and the highest dimensions barely change across the window. One dimension alone is ambiguous about position; the set of them together is not.

The original paper also tried a learned positional embedding table and found essentially no difference, reporting development BLEU of 25.7 against 25.8 for the fixed sinusoidal version, which is a useful reminder that the mechanism is what matters and the particular encoding is a design choice.Transformer

Causal masking handles the other half of the order problem, the half that says a position must not look forward. In an autoregressive model, allowing position $i$ to attend to position $i+1$ would leak the answer into the input. The fix is a single masking operation applied to the score matrix before the softmax, setting the forbidden entries to negative infinity so that they receive exactly zero weight. This is why the mask appears inside the attention computation rather than as a separate component, and it is why "masked self-attention" is not a different operator: it is the same function with a triangular mask argument.

At this point the three terms that get conflated most often can be separated precisely. An attention layer is the operator itself: take queries, keys and values, return a weighted sum. Self-attention is a wiring decision, namely that the three inputs are projections of the same sequence, as opposed to cross-attention where queries come from one sequence and keys and values from another. The Transformer is neither of those: it is an architecture that stacks blocks built around attention and does not use recurrence. Attention existed as a component for three years before the Transformer, attached to RNN decoders the whole time.

5. Stack the block and the same operator eats image patches: encoder/decoder blocks, ViT, and the closing checklist

Once self-attention is wrapped in residual connections, layer normalization and a position-wise MLP and stacked, the block stops being a language component. The Vision Transformer is the evidence: feed it image patches as tokens and a standard encoder stack classifies images, which means the endpoint of the four removals is not an architecture but a reusable operator over variable-length sequences.

The encoder block contains two sub-layers, multi-head self-attention and the position-wise feed-forward network, each wrapped in a residual connection followed by layer normalization. The decoder block adds a third sub-layer on top of the encoder's two, and that middle sub-layer is where the distinction between self-attention and cross-attention becomes concrete.

Caption: The left stack is the encoder, where every sub-layer is wrapped by an add-and-normalise residual. The right stack is the decoder, which adds a masked multi-head attention below the middle sub-layer and takes arrows from the encoder output into that middle sub-layer. Those arrows are the cross-attention: its queries come from the decoder, while its keys and values come from the encoder, so it is not self-attention.Transformer

The same module is reused three times inside one model, and only the inputs change: as the encoder's self-attention it receives the encoder state for all three arguments, as the decoder's self-attention it receives the decoder state with a causal mask, and as cross-attention it receives the decoder state as queries and the encoder's output as keys and values. A worked implementation makes this explicit by instantiating the identical attention class three times and varying nothing but which tensors are handed to it. One implementation detail is worth flagging rather than copying: the widely used annotated implementation applies layer normalization before the sub-layer and adds the residual afterwards, for code simplicity, which is the reverse of the order drawn in the original diagram. The two are equivalent up to where the normalization sits, but a reader matching code to the figure will otherwise trip on it.Annotated

The translation results are the last of the recurrent-era numbers worth keeping: 28.4 BLEU on WMT'14 English-to-German, more than 2 BLEU above the previous models including ensembles, and 41.0 on English-to-French in the paper's prose, with its own Table 2 reporting 41.8 for the same model. Both figures come from the paper, which is internally inconsistent on that one number, and the abiding fact is the compute side: the result came at a small fraction of the training cost of the ensembles it beat.Transformer

The move to vision keeps every part of the recipe and changes only what a token is. Cut an image into fixed-size patches, flatten and project each patch linearly, prepend a learnable classification token, add position embeddings, run the resulting sequence through the standard encoder stack, and read the class token out through a small MLP head. The pipeline below is the whole architecture in one frame, and the encoder drawn inside it is the block from the previous figure.

Caption: An image is split into patches, each patch is linearly projected into a token, a learnable class token is prepended and position embeddings are added, and the resulting sequence goes through an encoder stack repeated $L$ times. The enlarged block on the right shows the internal order: normalization, multi-head attention, residual add, normalization, MLP, residual add. The encoder is encoder-only: no decoder, no causal mask.

Two properties of that pipeline are worth separating. The first is what the position embeddings are: plain learnable 1D vectors added to the patch tokens, not the sinusoidal scheme from the original Transformer. The second is what happens to them during training. The similarity panel below compares one patch's position embedding against those of all other patches, and the pattern that emerges is that patches which are close in two dimensions end up with similar embeddings. The network recovered approximate 2D neighbourhood structure from data, which means that piece of inductive bias is learned rather than written into the encoding.

Caption: Each tile shows the cosine similarity between one reference patch's position embedding and the position embeddings of all other patches in the same image. Bright cells mean high similarity, and they cluster around the reference patch's own coordinates, so spatially adjacent patches end up with similar embeddings even though the embeddings are learnable 1D vectors with no spatial prior built in.ViT

The transfer numbers explain both why this was a real result and why it took years of scaling to arrive. With JFT pre-training, ViT-H/14 reaches 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100 and 77.63% on VTAB, topping a strong ResNet-based baseline on every one of those benchmarks while using about a quarter of the pre-training compute (2.5k against 9.9k TPUv3-core-days). Pre-training on ImageNet-1k alone, the same family of models underperforms ResNets, because self-attention carries less image-specific prior than convolution does: locality and translation equivariance are baked into every convolutional layer, while in ViT only the MLP layers have them and the attention layers are global. The data dependence is the price of the generality.ViT

That closes the arc, and the synthesis is what makes it reusable. The comparison below is the course material's canonical side-by-side of the three ways of processing sequences, and its trade-off rows line up one for one with the four moves described above.

Caption: The three columns place the recurrent network, convolution and self-attention side by side with the operation applied at each position. The comparison underneath is the trade-off summary: recurrence handles 1D ordered sequences with linear compute but is not parallelizable, convolution is parallelizable but needs many stacked layers to reach long-range dependencies, and self-attention reaches any input in one step at the cost of quadratic compute and memory.CS231n

The closing claim from the course material is the one to carry out of here: Transformers are the backbone of all large AI models today, used for language, vision, speech and more. That is a consequence of the four removals rather than a separate development. By the end of the arc there is nothing left in the operator that is specific to words, images or audio; a sequence is a set of vectors plus whatever positional information you attach to them, and the same block processes all of them.

References

The core sources behind this article, listed for deeper reading:

1. Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) — https://web.stanford.edu/class/psych209/Readings/HochreiterSchmidthuber97LSTM.pdf
2. Sequence to Sequence Learning with Neural Networks (Sutskever, Vinyals, Le, 2014) — arXiv:1409.3215
3. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Xu et al., 2015) — arXiv:1502.03044
4. Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho, Bengio, 2014) — arXiv:1409.0473
5. Attention Is All You Need (Vaswani et al., 2017) — arXiv:1706.03762
6. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Dosovitskiy et al., 2020) — arXiv:2010.11929
7. Stanford CS231n 10th Anniversary (2025), Lecture 7 (RNN) — https://cs231n.stanford.edu/slides/2025/lecture_7.pdf
8. Stanford CS231n 10th Anniversary (2025), Lecture 8 (Attention and Transformers) — https://cs231n.stanford.edu/slides/2025/lecture_8.pdf
9. The Annotated Transformer (Huang, Subramanian, Sum, Almubarak, Biderman, v2022; original Sasha Rush, 2018) — https://nlp.seas.harvard.edu/annotated-transformer/