A frozen image encoder, a frozen LLM, one trained layer: how a model answers about photos

Modern multimodal assistants are not one model. They are a stack of capability interfaces, each bought separately with its own training objective, and every purchase creates the bottleneck the next interface has to fix. The five interfaces are next-token prediction, human preference alignment, language-aligned visual representation, visual-token translation, and tool-mediated action. This is a map of that stack, read in the order the interfaces were built, with the same four-way trade-off recurring at every step: generality, information preservation, training efficiency, and context cost.

Everything later plugs into one recipe: predict the next token, then restate each task as text

The base that every later component plugs into is a Transformer decoder whose only job is to predict the next token, and what lets that single decoder accept every task is a two-stage training recipe plus one interface convention that rewrites each task as a contiguous piece of text. Alignment, a language-aligned vision tower, a projector, and an action head are all additions to this decoder rather than replacements for it.

The canonical first instance is GPT-1. Radford et al. describe a multi-layer Transformer decoder with masked self-attention, with the encoder and the encoder-decoder cross-attention removed from the stack: 12 layers, 768-dimensional states, 12 attention heads, 3,072-dimensional inner states in the position-wise feed-forward network, a byte-pair vocabulary of 40,000 merges, learned position embeddings rather than the sinusoidal scheme, GELU activations, and dropout at rate 0.1.GPT-1

Training comes in two stages. Stage A is ordinary left-to-right language modelling on unannotated text, here BooksCorpus with more than 7,000 books, and it lands at a held-out token perplexity of 18.4. Stage B reuses that same stack for a labelled dataset: the final block's activation goes through a single linear layer, and the language-modelling objective is kept as an auxiliary term with weight 0.5 so the fine-tuned model does not drift away from the distribution it was pre-trained on. The paper's own ablation makes the value of Stage A explicit. Removing pre-training drops the average score from 74.7 to 59.9, and swapping the Transformer for a single-layer LSTM inside the same framework drops it to 69.1.GPT-1

The two-stage structure is visible in the architecture itself, because both stages feed off the same weight stack.

Caption: A decoder-only stack, read bottom to top: text and position embeddings enter a block marked 12x, which contains masked multi-head self-attention, a residual add, layer norm, a feed-forward network, a second residual add and a second layer norm. The output forks into two heads, Text Prediction and Task Classifier. The two heads are the visible evidence that pre-training and downstream classification share one stack, and the 12x badge is the repeated block.

Stage B is a loop over labelled examples rather than a single gradient step.

Caption: The panel walks a translation task example by example, running a gradient update after each one, and only reaches a fresh unseen prompt at the very bottom. Sequence-level annotation is applied example by example, which is what "supervised fine-tuning" buys and what makes it expensive relative to an interface that only needs text in the context.

The more consequential half of GPT-1 for this article is not the training schedule but the input convention. Every structured task was reduced to one contiguous token sequence so that a single decoder could consume it. Classification and entailment concatenate their inputs with a delimiter token between segments and start and end markers around the whole thing. Similarity has no inherent order, so the paper feeds both orderings and adds the two resulting representations element-wise. Multiple choice runs the model once per candidate answer and takes a softmax over the candidates. The only parameters that change per task are the output projection and the embeddings for the delimiter tokens.

This is the same discipline as exposing one wire format on a service and serialising every business process into it. The decoder is the service, the contiguous token sequence with its start, end and delimiter markers is the wire format, and the per-task object is only the output projection. The point of the mapping is the consequence, not the resemblance: because the interface is fixed, adding a task never requires changing the model, only the way the input is written.

The family that grew out of this recipe is the frame the rest of the article climbs into.

Caption: Five nodes running left to right from GPT-1 in 2018 to GPT-4 in 2023, each with a card listing what changed. The cards are the deck's own annotations, and the parameter counts printed on them are not verified here, so read the cards as capability labels: unsupervised pre-training, larger data, few-shot capability, RLHF, and text plus image input. Only the progression of what each generation added is a takeaway.

GPT-3 then scaled the same decoder recipe to 175 billion parameters with a context window of 2,048 tokens, and pushed task adaptation entirely into the prompt text, applied without any gradient updates or fine-tuning.GPT-3 That scaling step has a literature of its own. For the map here, the relevant fact is that the decoder survived the scale-up unchanged in kind, which is why every later component in this article is something inserted into this stack rather than a different architecture.

Answering the request is a separate training stage: demonstrations, then preference rankings, then PPO

Making a model answer the request instead of continuing the prompt is not a by-product of scale. It is a separate stage with three parts: supervised fine-tuning on human demonstrations, a reward model trained on pairwise preference rankings, and policy optimisation against that learned reward.

The behaviour being fixed is a completion habit. Given the prompt "Explain the moon landing to a 6 year old in a few sentences," an unaligned model produces four more requests of the same shape, while the aligned model answers the question.

Caption: One prompt, two completions. The unaligned model returns a list of new prompts that copy the pattern of the input, and the aligned model returns the explanation that was asked for. This is a lecture illustration rather than a figure from the InstructGPT paper, so it shows what the gap looks like in user-facing text and says nothing about relative scores.

Scale alone does not close that gap, and the distinction the scaling literature draws explains why. Cross-entropy loss improves smoothly across more than seven orders of magnitude, while downstream metrics such as exact match, BLEU and accuracy stay near random over long stretches and then jump sharply once a critical scale is passed.Emergent Abilities A capability that appears at scale is therefore not the same thing as behaviour a user wants, which is exactly the gap that has to be purchased separately.

The three parts of that purchase are explicit in InstructGPT. Step one fine-tunes a pre-trained GPT-3 model with supervised learning on roughly 13,000 prompts drawn from the API and written by labelers, at 1.3B, 6B and 175B sizes. Step two collects comparisons rather than scores: labelers receive groups of 4 to 9 candidate outputs for the same input and rank them, which yields C(K,2) pairwise comparisons per prompt across roughly 33,000 prompts, and a reward model is trained so that the difference in its rewards represents the log odds that one response is preferred. A single 6B reward model was used for every PPO model, because larger reward models were less stable despite their lower validation loss. Step three optimises the policy against that reward with PPO, adding a per-token KL penalty against the SFT model with coefficient 0.02 and initialising the value function from the reward model, while the PPO-ptx variant adds a pre-training gradient term so the general capabilities of the base model are not forgotten.InstructGPT

The single most important design decision in that pipeline is that human preference enters as a ranking and not as a rating.

Caption: Three stages in a row, with blue arrows marking which data trains which component: labeler demonstrations train the supervised model, ranked comparisons train the reward model, and the reward model becomes the reward that drives the PPO policy. In the middle stage four sampled outputs are shown being compared by a labeler, which is the form the preference signal takes.InstructGPT

The measured result is a preference winrate, not a benchmark score, and that distinction matters when reading the number. On the paper's own API prompt distribution, the 175B aligned model's outputs are preferred to those of the 175B base model 85 percent of the time, and preferred 71 percent of the time to a well-crafted few-shot-prompted 175B base model. The same evaluation shows a 1.3B aligned model preferred over the 175B unaligned one despite having roughly 100 times fewer parameters, and reports hallucination on closed-domain API tasks at about half the base model's rate, 21 percent against 41 percent.InstructGPT

Caption: Model configurations on the horizontal axis and preference winrate against the 175B supervised baseline on the vertical axis, with error bars. The ordering runs from the raw base model at the bottom through prompting and supervised fine-tuning to the PPO variants, and the small 1.3B aligned model sits above the 175B base model. The vertical axis is labeler preference on this paper's own prompt distribution, not a public leaderboard score.InstructGPT

This is the stage that later gets reused verbatim on the vision side, with one substitution. When instruction tuning is ported to multimodal models, the demonstrations are generated by a strong model instead of by paid labelers, and the same arc from demonstrations to preference signal to alignment reappears.

A model can only output the classes it was trained on; CLIP lets text name them instead

A supervised vision classifier is capped by the label list that was fixed when it was trained, and the way out of that ceiling is to replace the label list with language as the supervision signal. That swap buys a classifier that is assembled at inference time from class names, and the price is that what comes back is a representation rather than an answer.

The ceiling is easy to state. In the ImageNet paradigm humans assign a fixed set of category labels, the trained model can only predict inside that closed vocabulary, and a new category means new labels and another training run. The class names themselves are isolated tokens that carry almost no semantic structure, so nothing about the label set tells the model that a previously unseen name is related to the ones it knows.

CLIP's move is to train an image encoder and a text encoder jointly so that inside a batch of N image-text pairs the N correct pairings get high cosine similarity and the N²−N incorrect ones do not, with a symmetric cross-entropy over that N by N similarity matrix as the entire objective. The supervision is whatever text co-occurs with the image on the public web, drawn from a newly collected set of 400 million pairs, not a curated label. The paper's own comparison also explains why matching and not caption generation: predicting only which text goes with which image is substantially cheaper than generating the exact caption, which is a training-efficiency argument before it is an accuracy argument.CLIP

At inference the class list arrives as text. Class names are turned into caption-like prompts, embedded by the frozen text encoder, compared with the image embedding, and the nearest prompt supplies the label. This is the same shift as moving a set of allowed values from a compile-time enumeration to a runtime configuration: the set of legal outputs used to be baked into the trained head, and now it is passed in when the model is called, so adding a category is an argument, not a rebuild. The consequence of the mapping is what matters here, namely that the set of classes is decided by whoever is calling the model rather than by whoever trained it.

The empirical anchor is a single number that has to be quoted with its full specification. The best CLIP configuration, a ViT-L/14 at 336 pixel input with prompt ensembling, reaches 76.2 percent top-1 zero-shot on ImageNet without using any of the 1.28 million training labels, which matches a supervised ResNet-50 baseline that used all of them, and zero-shot CLIP beats a supervised linear probe on 16 of 27 evaluation datasets.CLIP

The three-step structure of the whole idea fits into one figure.

Caption: Three panels read left to right. The first shows the standard supervised image model, where a backbone feeds a linear classifier over a fixed label set. The second shows CLIP training, where image features and text features from a batch of pairs meet in a shared space and the diagonal of the similarity matrix is the training target. The third shows inference, where the text encoder receives prompts such as a photo of a dog and the image encoder receives the image, and the largest similarity decides the class.CLIP

What that buys is best seen in the output format rather than in the architecture, because it makes the limitation concrete.

Caption: A grid of examples from 36 zero-shot classifiers. Each cell shows an image with the top five class-name predictions, the prompt template used to represent each class, and a probability, with the correct label highlighted in green and a wrong top prediction in orange. The only thing produced per image is a probability vector over the supplied class names, which is why this model chooses and does not speak.CLIP

So the output of this interface is a representation. It emits a vector per image and a vector per class name, and the decision is a closed-form comparison between them. It cannot converse, and because its objective aligns whole images with whole sentences, it is also weak at fine spatial relations. That is precisely why the next interface exists: something has to translate this representation into a form a language model can reason over.

A single learned projection is enough to give a language model sight

The bar for building a vision-language model is lower than intuition suggests. Freeze the vision encoder, freeze the language model, and train only the matrix that sits between them, and the model can answer questions about images. The cost of that economy is that the language model keeps every failure mode it already had, plus one new one: when the visual evidence is thin, its language prior fills the gap.

LLaVA-1.0 is the smallest recipe that works. A frozen CLIP ViT-L/14 encodes the image, a single trainable linear projection maps those features into the word-embedding dimension of a frozen Vicuna, and the projected features are concatenated with the text instruction and fed through the language model, which autoregressively produces the answer. Both large backbones stay frozen in the first training stage, so the only thing being learned is the translation between two representation spaces. The reason a single linear map is enough is that the vision tower's features are already aligned to language, which is the whole point of the previous chapter.LLaVA

Training uses two stages and a large amount of synthetic data. The first stage aligns the projector on 595,000 image-text pairs, each reformatted as a short describe-the-image instruction. The second stage tunes the projector together with the language model on 158,000 visual instruction samples that were generated by GPT-4 from image captions and boxes, split into 58,000 conversations, 23,000 detailed descriptions and 77,000 complex reasoning examples. The only human annotation in that data construction is the handful of seed examples used to prompt GPT-4.LLaVA

Caption: The minimal bridge drawn as a pipeline: an image enters a frozen CLIP visual encoder, its grid features are mapped by a single trainable linear projection matrix into tokens with the same dimensionality as the language model's word embeddings, and those tokens are concatenated with the instruction and passed into the frozen language model.LLaVA

The follow-up version isolates which of the cheap upgrades are actually worth paying for, and the answer is that three of them are. The linear projection becomes a two-layer MLP, the encoder moves from 224 to 336 pixel input, and academic VQA data is added to the instruction mix. On the cumulative ablation table these changes move the score on the MME perception benchmark from 809.6 to 1323.8 to 1531.3, but the shorthand hides two things worth knowing. The three numbers are rows 0, 2 and 9 of a table that adds one ingredient at a time, so they are not an additive sequence; and the last row also swaps the language model from 7B to 13B, so the final jump is not purely a data or resolution effect. The single largest early step is adding VQA-v2, which alone takes the score from 809.6 to 1197.0; the format-prompt row then lifts it to 1323.8.LLaVA

Caption: The upgraded bridge, showing what changed rather than what it is: the image is split into a grid of sub-images encoded by a higher-resolution CLIP encoder, the patch features pass through a two-layer MLP connector in place of the single linear projection, and the result feeds a language model that this recipe also scales up. Read this together with the previous figure to see that both changes are small and local.LLaVA

The projector is not a single trick, and the alternative routes are worth knowing by name even if their details belong elsewhere. Preserving the visual tokens and teaching the language model to read them is the LLaVA route. Compressing them at the bridge with a small set of learnable query embeddings, as BLIP-2's Q-Former does with 32 queries, trades visual fidelity for a fixed and much shorter visual prefix. Scaling the vision encoder itself, as InternVL does with a 6-billion-parameter vision tower, removes the bottleneck at the encoder and pays for it in pre-training cost. That design space gets a full treatment in the companion tutorial "VLM architecture, visual connectors, and resolution/data scaling".BLIP-2

What this interface bought, then, is a language model that can see, at the cost of a trained matrix and some token budget. What it inherited is everything the language model already was, including the habit of answering from its own prior when the evidence is thin. That habit is the hallucination tax on this whole branch of the stack.

Resolution sets what a vision model can see; the training data sets what it can do

Two things are paid for separately once a projector exists: resolution and position encoding decide what is legible in the image and how much context it costs, and the data curriculum decides which capabilities exist at all. Neither one buys the other.

The resolution ceiling comes from the standard vision transformer, which is trained at a fixed input size and carries a fixed set of learned position embeddings. Fine detail is lost at low resolution, and because position embeddings were learned at one grid size, changing the grid requires either interpolation or cropping. One common workaround is grid splitting, which tiles a large image into sub-images and adds a downsampled global view for context. A cleaner answer is to drop the fixed grid entirely.

Qwen2-VL does that with what it calls Naive Dynamic Resolution. The vision transformer removes its absolute position embeddings and uses 2D rotary position encoding to capture the two-dimensional structure of an image, so the encoder can accept a native resolution. A simple MLP after the encoder then compresses each group of four adjacent tokens into one, and the compressed sequence enters the language model as ordinary tokens. The arithmetic is instructive: a 224 by 224 image with 14-pixel patches produces 256 patches, which compress to 64 visual tokens, plus two special boundary tokens, so 66 tokens for that image. At inference two knobs bound the cost, a minimum and a maximum pixel budget, and small images are upscaled to clear the minimum, which means the same picture can cost a different number of tokens depending on those settings.Qwen2-VL

Caption: The same image shown at progressively higher minimum-pixel thresholds, which is the mechanic behind the token bill: small images are upscaled before they enter the model, so a higher threshold means more pixels and more visual tokens for the same picture. The small bar cluster on the right is an illustration only. The authoritative account of dynamic against fixed resolution is the paper's ablation table, where dynamic resolution averages about 1,924 tokens and reaches 75.89 on InfoVQA and 866 on OCRBench, matching the top-tier scores of a 3,136-token fixed setting while using roughly 40 percent fewer tokens on average.Qwen2-VL

Position encoding is the other half of the argument, because a token count that tracks image size is useless if the model has no way to represent where a token sits. M-RoPE, the multimodal form of rotary position encoding, decomposes the rotary embedding into three components, temporal, height and width. For text all three components use identical position IDs, which makes the scheme reduce to ordinary one-dimensional rotary encoding. For an image the temporal ID stays constant while the height and width IDs vary with position. For video the temporal ID increments per frame while the spatial pattern matches the image case. The measured payoff is uneven in an informative way: across image benchmarks the three-axis scheme is roughly a tie with the one-dimensional baseline, while across video benchmarks it wins clearly, which is what you would expect when the temporal axis is the one that has to carry new information.Qwen2-VL

Caption: Three position-ID streams, temporal, height and width, running along a sequence that mixes text, image patches and video frames. Text collapses onto the same ID across the three axes, image tokens hold the temporal axis constant while the spatial axes follow the patch position, and video frames step the temporal axis forward. This is the figure to hold onto when a paper says it uses M-RoPE.Qwen2-VL

In this architecture the vision side is deliberately small: about 675 million parameters shared across all model sizes, a 2 by 2 compression MLP, and the language decoder. The point of putting the intelligence on the resolution and position side is that the encoder's load stays constant no matter how large the language model gets.

The other half of this chapter is not architecture at all. Capability tracks the data that was used, and the cases are blunt about it. Reading text in an image needs OCR data, driving a user interface needs trajectories of actual interaction, and producing structured document output needs supervision that maps document elements to markup. The document case makes the token accounting vivid: DocOwl2 crops a high-resolution page adaptively, reduces it through a convolutional reducer, and then uses a two-layer cross-attention compressor with the global page feature as the query and the cropped sub-image features as keys and values, which locks every page to exactly 324 visual tokens regardless of the raw resolution. Ten pages is therefore 10 times 324, or about 3,240 tokens, which is a derived arithmetic step rather than a number stated in the paper, and the same system reaches single-page performance at less than 20 percent of the visual tokens used by comparable models.DocOwl2

Both halves of this chapter are interfaces with a bill attached. Resolution decides what can be seen and charges the context window for it, and the data curriculum decides what is possible and charges training time for it. A model missing either one will not show the capability, no matter how good the other half is. The temporal version of the same trade-off, where frame sampling competes with context length, is the subject of the companion tutorial "Video VLMs: Paying for Time with Tokens and Position".

A model that describes your screen cannot operate it; acting needs an action-and-feedback loop

The same vision-language substrate, plus grounding in absolute image coordinates and a shared function-call action space, is enough to cross from answering into acting. What the agent numbers then show is a split verdict: locating a UI element is close to solved, and acting continuously inside a real environment is not, because the missing piece is not perception but a feedback loop.

The four capability stages separate cleanly. A language model can generate and reason but is blind. A contrastive vision model can represent visual semantics but cannot converse. A vision-language model can see, speak and reason. An agent can execute actions and sustain multi-step interaction, which is a different output contract from anything before it, since the outputs of a standard vision-language model are descriptions or coordinates rather than effects in an environment.

Qwen2.5-VL is the reference point for the substrate that carries into that stage. Its vision encoder is re-engineered for cost at native resolution, using window attention in all but four of its layers, RMSNorm, SwiGLU feed-forward blocks, 14-pixel patches and 2D rotary position encoding; an MLP merger concatenates four spatially adjacent patch features before projecting them into the language model's embedding dimension; and the language side is a Qwen2.5 decoder. The one inheritance from the position-encoding story is that the temporal IDs of the multimodal rotary embedding are now aligned to absolute time rather than to frame index, so the gap between IDs carries the pace of events.

Caption: The pipeline from inputs to decoder: images and video frames at native resolution and different sampling rates are mapped to token sequences of varying length by the vision encoder, which uses window attention, RMSNorm and SwiGLU, with temporal position IDs aligned to absolute time, and the result is fed into the language decoder. The figure covers the encoder, the merger and the decoder only. It does not depict the action or function-call surface, which is where the agent half of this chapter lives.Qwen2.5-VL

The action surface itself is textual and has no figure in the paper. Grounding output is expressed in the actual image dimensions rather than in normalised coordinates, which is what keeps a click target stable when resolution changes. Agent training data comes from screenshots of mobile, web and desktop platforms, where UI elements are annotated for grounding and multi-step trajectories are reformatted into one shared function-call format across all three platform types, with reasoning content attached to each step. The important structural claim is the word shared: one action vocabulary spanning platforms, rather than a per-platform controller.Qwen2.5-VL

The benchmark split is what makes the verdict concrete rather than promotional. On UI element grounding this model reaches 87.1 on ScreenSpot and 43.6 on ScreenSpot Pro, which is a working capability. On continuous action in a live environment it reaches a 35 percent success rate on AndroidWorld and 8.83 on OSWorld. Grounding a target is largely solved; finishing a task in an environment with consequences is not, and the gap between those two numbers is the action-and-feedback loop that no amount of additional perception closes by itself.Qwen2.5-VL

One precursor is worth naming, because it is the first clear instance of a model calling a tool before answering rather than answering in a single pass. V* pairs a question-answering model with a visual search model over a working memory, letting the model recursively zoom into an image to gather evidence before committing to an answer, and on its high-resolution benchmark it reaches 75.39 overall against 48.68 for LLaVA-1.5 and 54.97 for GPT-4V.V* Fusion across language-model layers and tool-mediated visual search are the subject of the companion tutorial "Beyond Passive Vision: Deep Fusion, Visual Tools, and the Agent Boundary".

Read as a whole, the stack shows a consistent pattern: every interface is bought separately, and each purchase leaves a debt that the next interface pays. A decoder that only continues text creates the need for a preference stage. A language model that answers creates the need for a visual representation it can read, a language-aligned representation creates the need for a translation layer, a translation layer puts images on a token budget, and outputs that stop at descriptions and coordinates create the need for an action loop. The same four currencies are spent at every step, which is generality, information preservation, training efficiency, and context cost, and no configuration so far has traded one away for free.

References

The core sources behind this map, in order:

1. Improving Language Understanding by Generative Pre-Training — https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf
2. Language Models are Few-Shot Learners — arXiv:2005.14165
3. Emergent Abilities of Large Language Models — arXiv:2206.07682
4. Training Language Models to Follow Instructions with Human Feedback — arXiv:2203.02155
5. Learning Transferable Visual Models from Natural Language Supervision — arXiv:2103.00020
6. Visual Instruction Tuning — arXiv:2304.08485 and Improved Baselines with Visual Instruction Tuning — arXiv:2310.03744
7. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models — arXiv:2301.12597 and InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks — arXiv:2312.14238
8. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution — arXiv:2409.12191
9. mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding — arXiv:2409.03420
10. DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs — arXiv:2406.04334 and V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs — arXiv:2312.14135
11. Qwen2.5-VL Technical Report — arXiv:2502.13923