Detection shed its hand-designed stages; the rules moved into training

Object detection and segmentation are two shapes of the answer to one picture: a fixed map that labels every pixel, or a list whose length depends on what happens to be in the image. The decade from 2015 to 2025 was a single campaign to shorten the path to the second shape, and it proceeded by deleting one hand-designed stage at a time. The per-region forward pass went first, then the external proposal engine, then the anchors and the non-maximum suppression. Nothing was thrown away. Every deleted stage came back inside the network as something learned, and the bill arrived later as training time, a heavy image encoder, and latency that grows with how general the prompt is. What survives in the middle is a fixed set of slots, and that slot is now the part a user holds.

One image, three answers, and the two different machines they demand

A dense per-pixel map and a sparse per-object list are two answers to the same question, and each was expensive in its own way from the start. The clearest way to see the difference is to look at one street scene under all three annotation schemes before meeting a single model name.

Caption: One intersection in four states: the raw photograph, semantic segmentation with one flat colour per class, instance segmentation with a separate colour and outline per object, and panoptic segmentation with every region filled. The upper band of the instance panel is black because it is the annotation's own void area, not a crop or a rendering fault. All four panels are ground truth; none is a model output.PanopticSeg

The semantic panel paints every pixel with a class colour and merges all pedestrians into one region. The instance panel separates each person and each vehicle, gives each its own mask, and paints nothing where the annotator had no label. The panoptic panel does both at once: each countable object keeps its own region, and each uncountable class such as road or sky becomes a single region. The rule behind the fourth panel is mechanical. Every pixel carries a pair, a class label and an instance id, and for a class that has no instances the id is simply ignored. That is why panoptic segments cannot overlap: overlap is excluded by the format rather than cleaned up afterwardsPanopticSeg.

The distinction doing the work here is things versus stuff. Countable objects such as people, animals and vehicles are things and get one region each. Amorphous regions such as grass, sky and road are stuff and get one region per class. The 2018 paper that fixed this vocabulary is a task and metric paper, not a model: its own baseline combines two independent systems by post-processing and lists deeply integrated models that handle things and stuff together as future work. So it specifies the output contract. The campaign that follows is what delivers it.

The same paper also names why the field was split into two machines. Stuff classifiers are usually fully convolutional networks, while object detectors are usually region-based and driven by proposals. Those two architectures exist because the two answer shapes impose different demands. A dense map asks for one score per class per pixel, so a network has to return a tensor the same size as the input, and every pixel is a training example. A sparse list asks for a variable number of objects per image, and there the arithmetic breaks immediately: every extra object needs its own group of numbers, a busier image needs more, and a fixed-size regression head can only ever emit a fixed count. Proposals are the workaround, and with proposals comes a second problem the dense map never had, which is deciding which prediction owns which object. The rest of this piece is the story of that second problem being solved by deletion.

Every deleted stage moved the work inside the network, until only the assignment rules were left outside

Two deletions two months apart removed the per-region forward pass and the external proposal engine, and both were relocations rather than removals. Each one shortened the pipeline and dumped the displaced work onto a different component, which then became the next bottleneck.

The first target was the crop. R-CNN took roughly two thousand region proposals per image, warped each to a fixed size, and ran a full convolutional network on every one of them, so the same pixels were convolved thousands of times. On VGG-16 that cost 47 seconds per image on a GPU. The fix was to run the network over the whole image once and crop features instead of pixels. RPN

Caption: The whole image goes through one convolutional network, each region of interest is projected onto the resulting feature map, and the pooling layer turns that projected region into a fixed-size vector. Two heads then read that vector: a softmax over classes and a per-class box regressor. What the picture does not show is the proposal generator, the loss, or any timing.Fast R-CNN

The measured effect was an order of magnitude in both directions: training on VGG-16 fell from 84 hours to 9.5, and test time from 47.0 seconds per image to 0.32. The deletion is real, and it is incomplete in two places that matter for what comes next. The timings explicitly exclude proposal generation, and per-class non-maximum suppression is still applied exactly as R-CNN applied it. The pipeline got shorter and the answers still had to be pruned afterwards.

The second target was therefore the proposal engine, which at that point still ran outside the network on a CPU. Selective Search cost about 2 seconds per image and EdgeBoxes about 0.2, against a detector that now took a fraction of a second. The fix was to compute proposals with a small head on the same convolutional features the detector already had, which is what makes them nearly free rather than merely faster.

Caption: One shared feature map under two branches. The Region Proposal Network on the left emits proposals, and those proposals feed the region pooling on the right, which finishes in the classifier. The two branches are one network on one feature map, not two models, and the figure carries no anchors, no suppression step and no numbers.Faster R-CNN

The paper prices the change itself, and the table is worth reading as a cost model rather than as a scoreboard.

Those are K40 GPU numbers with the external proposal algorithm measured on a CPU, and the paper itself calls that comparison inequitable. Two lessons survive the caveat. The first two rows are paid once per image and the third is paid once per proposal, so only one term grows with how many candidate regions survive. That single distinction explains most of the trade-offs in the rest of this piece, from proposal budgets to query counts to inference latency. The paper's own wording is two modules plus a timing table; reading the first two rows as per-image is this article's framing of what that table implies.

The second lesson is that both deletions left the assignment machinery untouched. Anchors were still defined by hand, positives and negatives were still sampled by hand, and non-maximum suppression still decided which of several overlapping boxes survived. Before those rules could go, the two answer shapes had to be shown living in one network.

Caption: The same region feature feeding three branches: class, box, and a mask branch that ends in a small per-class mask tensor, labelled 28 by 28 by 80 in the right-hand panel. The mask is dense inside one region and does not cover the image, so this is not a per-pixel map over the scene. The two panels are the same head for two different backbones, not two models.Mask R-CNN

Mask R-CNN added a third per-region branch and emitted one small binary mask per class per region, and because each region gets its own mask, two objects of the same class do not merge into one. It also forced out a defect the detector had tolerated: region coordinates were being rounded to the feature grid, and that rounding misaligns features just enough to ruin pixel-accurate masks. Replacing pooling with a sampling scheme that keeps fractional coordinates moved mask AP from 26.9 to 30.2 with the same ResNet-50 backbone. The cost was modest, about twenty percent over the detector it extended, and the count of answers was still decided by the proposals. Mask R-CNN

What survived two deletions and one addition, entirely intact, was the machinery that decides which prediction owns which object. That is what the next step deletes.

Drop the assignment rules and the answer becomes a set, emitted slot by slot

Anchors and non-maximum suppression were not tuned, they were deleted, and what replaced them is an interface rather than a rule. DETR predicts a set of objects directly: a fixed number of output slots, each emitting one class-and-box pair or a single no-object label. The paper's own claim is that this removes hand-designed components that encode prior knowledge about the taskDETR.

Caption: Image features from the backbone and encoder enter the decoder from below, a small fixed set of learned object queries enters from above, and each slot leaves through a shared head as either a class plus a box or a no-object label. This is the one picture in the piece that shows the whole interface at once. The four coloured squares are schematic and do not represent the query count, and the matching step is described in the text rather than drawn here.DETR

Making that legal required one change on the training side. Because a set has no inherent order, nothing in the network says which of a hundred slots should be responsible for the third pedestrian. The answer is a one-to-one assignment computed once per image, found with the Hungarian algorithm, where each slot is matched to at most one object and the cost of a match combines the class probability with a box distance. Every unmatched slot is trained to say no object. Because the loss is invariant to permutations of the predictions, the slots can be emitted in parallel, and duplicate suppression moves from a post-processing threshold into the self-attention that lets slots see each other across decoder layers. The paper reports that DETR needs no suppression by design.

The visible consequence is that slots specialise, and the specialisation is geometric.

Caption: Twenty panels out of the hundred slots, one panel per slot, each dot a predicted box centre pooled over the COCO validation set. Green marks small boxes, red marks large horizontal boxes, blue marks large vertical boxes. Some slots concentrate in one corner of the image and some spread along a horizontal band, which is a region-and-size preference rather than a category preference: the paper finds no strong class specialisation. The panels carry no axes, so no counts or coordinates should be read from them.DETR

The price is entirely on the training side, and it is the reason the next two years of work look the way they do. The assignment depends on the model's own guesses, so early in training those guesses are poor, and the paper notes the model needs an unusually long training schedule to settle. It also inherits a hard ceiling: with a hundred slots, a hundred objects is the maximum, and when an image really does contain a hundred the model finds about thirty. Accuracy put it level with a well-tuned two-stage detector rather than ahead of it: 42.0 AP with a ResNet-50 backbone on COCO validation, against 42.0 for the equally tuned Faster R-CNN-FPN+ at a comparable parameter count (both in the paper's Table 1), with large objects much better and small objects worse.

At this point the pipeline is as short as it can get. Every remaining stage is either a learned component or the interface itself. What changes from here is not the shape of the pipeline but what gets loaded into the slot.

The same slot takes a better guess, and then it takes a word

Nothing about the interface changed after DETR. What changed is what fills the slot: first a spatial guess borrowed from the encoder's own candidate boxes, then a phrase typed by a user. The lecture this material comes from draws that progression as four panels on one axis, which is the briefest way to see that the only moving part is the content of the query.

Caption: Four states of the same interface, left to right. DETR feeds the decoder randomly initialised queries and boxes three pedestrians plus a dog. DINO feeds box-initialised queries, drawn as dashed squares, and finds the same objects. Grounding DINO conditions the queries on the word dog and boxes only the dog. SAM feeds mask tokens conditioned on a point, a box or a mask and returns the dog's pixels. The panel captions are the lecture's own wording, and each paper's own figure is authoritative for its mechanism.

The second panel is the more precise idea. DINO splits the query into two halves, a positional part and a content part, and then takes only the positional half from the encoder.

Caption: Three ways to fill the shortcut. On the left both halves are static learned parameters. In the middle the encoder's selected features replace both halves. On the right, DINO's choice, only the positional half travels down the long arrow from query selection while the content half stays a learned parameter in its own row. The figure does not show how many features are selected or what the accuracy difference is.DINO

The reason for stopping halfway is that the encoder's preliminary features are not yet clean object descriptors. A selected feature may straddle two objects or cover only part of one, so feeding it into the content half injects noise into the very thing the decoder is supposed to refine. Position, by contrast, is a useful prior even when it is rough. The paper's controlled comparison isolates the choice: keeping the content static while initialising anchors from the encoder's top candidates gives 46.5 to 47.0 AP, and small-object AP 29.6 to 31.1DINO. Its abstract headline of six AP points is a different claim, and the paper itself attributes part of that gap to engineering rather than to the three named changes. The denoising branch DINO adds exists for the instability DETR admitted: it feeds noised ground-truth boxes into the decoder as extra queries, half of them required to reconstruct the box and half required to be rejected, so the model learns to discard poor slots before the matching has to.

The third panel replaces the spatial guess with language, and that single substitution is what makes the label set a runtime input instead of a training-time constant. The paper's framing is that the solution to open-set detection is introducing language into a closed-set detectorGrounding DINO.

Caption: Three blocks: the overall model, one feature-enhancer layer, and one decoder layer. Text and image features are fused in the enhancer before any query exists, the fused features drive query selection, and each decoder layer adds a text cross-attention alongside the image one. Those are the three places language enters. No accuracy figure, loss weight or training mixture appears in the diagram.Grounding DINO

The mechanism that makes the query text-dependent is in the middle block. Image tokens are ranked by similarity to the prompt tokens, and the top nine hundred become the query positions. Which encoder tokens become queries is therefore a function of the sentence rather than a function of the image alone, and each decoder layer then carries one extra cross-attention against the text.

Caption: One scene, three prompts. Under building, river and ferris wheel the selected candidate boxes visibly differ, which is the empirical face of the selection rule: the query set is a function of the prompt. These are selected queries, not final detections, so the boxes are dense and mostly low-scoring, and the small numbers printed above them are not defined in the paper and are not confidence scores.Grounding DINO

The consequence is that a category never seen in training can be requested at inference time, because the detector scores each slot against the words it was given rather than against a fixed list. The reported zero-shot figures travel with their variants and their protocol: 52.5 box AP on COCO 2017 validation for the Swin-L model with no COCO training data, and 26.1 mean AP on ODinW-35, the paper's thirty-five-dataset zero-shot benchmark, for a Swin-L row with a different pre-training mix. The paper's zero-shot definition is its own and is narrow, and some rows in its tables add the evaluation dataset during pre-training. The ablation is the cleanest evidence that the language path is load-bearing: replacing language-guided selection with a static query set costs 2.5 points of zero-shot AP on LVIS, from 16.1 to 13.6.

The query leaves the network and becomes a handle

Three years later the query stopped being something the model supplies and became something a user or another program hands over. That shift matters because it changes who decides what to ask. Until this point the model chose its own candidates and returned whatever it found; now anything holding a point, a box or a mask can request an output, and the cost of asking is small enough to be repeated.

Caption: Three components in a row. A heavy image encoder produces an embedding; a prompt encoder turns points, boxes and text into sparse prompt embeddings while a convolutional path turns a mask into a dense one; a light mask decoder returns several valid masks, here the whole object, then a blade, then the handle, each with a score. The figure does not name the backbone, the embedding size or any latency, and the score is not a calibrated probability.SAM

The design is an asymmetry between two halves. The image encoder is expensive and runs once per image, so its output can be cached. The mask decoder is deliberately trivial by comparison, under one percent of the encoder's compute, and runs again for every promptSAM. Given a precomputed embedding, the prompt path runs in a web browser on a CPU in about fifty milliseconds. That asymmetry is the whole reason a box produced by some other component is a legal prompt: the marginal cost of one more question is tens of milliseconds, not another forward pass over the image. Prompts split into sparse ones, which are points, boxes and text, and dense ones, which are masks. A box is encoded as a pair of corner embeddings, so it sits on equal footing with a click.

The mask itself is produced by alternating attention between the prompt tokens and the image embedding, then upsampling the embedding and taking a dot product per mask. The training corpus behind it is 11 million images and 1.1 billion masks, of which 99.1 percent were generated automatically. The authors are explicit that the system as a whole is not real time once the heavy encoder is included, and that it underperforms dedicated interactive methods when a user supplies many points. What it makes cheap is the second question about the same image.

Caption: A text prompt on the left, detector boxes in the middle with a category tag on each, and filled masks on the right. The tags are the important detail: the category decision is visibly made by the detector and carried into the mask stage. This is a qualitative demonstration, not a benchmark result, and the masks come from two chained models rather than one joint model.Grounded SAM

Chaining the two gives a system that both names things and cuts them out, with a clean division of labour. The detector owns the category, because that is the half that can be asked for a word. The segmenter owns the boundary, and it never needs to know what the object is, only which region to cut. SAM's own text licenses exactly this hand-off: given a box detector for cats, cat instance segmentation follows from feeding the detector's box in as a prompt. The Grounded SAM paper adds two reasons the split is sensible rather than merely convenient. Detection labels are cheaper to collect than pixel-accurate masks, so the mask supervision is taken from the segmenter's own large corpus rather than annotated again. And a mask conditioned on a box is easier to predict than a mask conditioned on a sentence, because the location is already given.

The contrast with Mask R-CNN is instructive. Both produce boxes and masks, but one does it inside a single jointly trained network and the other serialises two independently trained ones with no shared weights. The reported number for the composed system is 48.7 mean AP on a twenty-five-dataset zero-shot segmentation benchmark, with the caveat that two of the three rows in that comparison were measured by another team. What the composition demonstrates is not a new mechanism but a new interface: the segmenter's prompt slot is stable enough that another model's output can be plugged into it.

A noun phrase now returns every matching instance, and generality is what costs the compute

The prompt has moved from a click to a noun phrase, and the same slot now returns every instance of the named concept along with a persistent identity for each. That is the current endpoint of the promptable branch, and it also shows where the bill for all this generality is paid.

Caption: One concept prompt, here a noun phrase plus an exemplar box on one of four penguins, reaching both an image-level detector and a memory-based tracker over a shared image encoder. The detector finds masks in the current frame, the tracker propagates masks from the previous one, and the two are merged into a per-instance output. The colour legend marks which component each part came from, not class identity, and this overview does not draw the presence head.SAM 3

The task is a genuine generalisation. A prompt is a short noun phrase such as red apple, or an image exemplar, or both, and one prompt returns all matching instances rather than one. Architecturally that forces a separation the earlier models did not need. Recognition is global, because deciding whether the phrase is present at all wants the whole image, while localisation is local, because a box is a local thing. SAM 3 decouples them with a single learned presence token that predicts whether the phrase occurs in the frame, multiplied into every slot's own match score. A phrase that is simply absent can therefore be suppressed everywhere at once, which is the hardest failure mode when the vocabulary is open and confident false positives are easy. That head is not drawn in the overview above; the ablation reports it as worth about 1.5 points on the paper's new metric, measured on shorter training runs than the main results. SAM 3

That metric is itself a sign of where the field is. Because average precision degrades badly when the vocabulary is large and most categories are absent, the paper proposes a composite score that multiplies a detection term by an image-level correlation term, and it releases a benchmark of 207 thousand phrases over 121 thousand images and videos. On the SA-Co/Gold split, instance segmentation scores 54.1 for SAM 3 against 24.6 for the strongest baseline and 72.8 for human annotators, with the caveat that the baseline row carries a marker because it was partly trained on LVIS.

The measurement question is not settled, and the disagreement is worth stating plainly. One branch argues that average precision is the wrong instrument for open vocabularies and invents a replacement. Another defines zero-shot narrowly as not using the evaluation split in training, which admits rows where the test dataset's own images were part of pre-training. A third, a preprint from October 2026, measures robustness to distribution shift on COCO-O and notes that this is not the same thing as recognising unseen categories, because its own training corpus contains COCO images.

That preprint is also the useful check on the other branch, the one that keeps detecting rather than segmenting. Its deployed model is a compact DETR-style detector with a three-layer decoder and three hundred queries at inference, and it reaches a real-time budget: 40.0 Fixed AP on LVIS minival at 32 frames per second for its base variant, with the largest model slower than the smallest, 32 frames against 63. What is interesting is where its contribution sits. It does not change the interface at all: the category-oriented query and the language-guided selection are exactly the arrangement Grounding DINO established. The gain comes from richer supervision at training time, in the form of object descriptions and image descriptions produced by a large language model, and every teacher-side module is deleted before deployment, so the deployed detector is unchangedRT-DETR-World. The paper describes its own numbers as competitive rather than state of the art, and it is one month old at the time of writing, with no code released and two ablation values that disagree between its main text and its appendix.

So the campaign is still running, and it is still arguing in the terms it started with: a prompt in, boxes or masks out, with the query as the carrier. What is genuinely unfinished comes in three parts.

The cost of generality is the second unfinished part, and it is measurable rather than speculative. In video, the promptable system is not real time on a single GPU: it reaches 30 FPS only by parallelising across more GPUs, since it pays per tracked object. And in the detection branch, the larger model is roughly half the speed of the smaller one for a few points of accuracy. Generality is not free at inference, only deferred to a heavier encoder or a cluster.

The third is evaluation, and there is no consensus to report, only the three incompatible positions above. A benchmark that cannot agree on what counts as unseen cannot rank methods that differ in what they were trained on.

Three of the deeper mechanics are worth reading on their own terms now that the shape of the argument is in place. For the segmentation half, from sliding windows through upsampling to the loss functions that survive class imbalance, read 《Semantic segmentation foundations: fully convolutional networks, upsampling, and…》. For the two deletions in detail, including the region-pooling arithmetic and the measured speed comparison, read 《Fast R-CNN, RoI Pool, and Faster R-CNN: sharing convolution and moving proposals…》. For the mask branch, the alignment fix it required, and the suppression step inside the two-stage branch, read 《Instance segmentation and modern detection: Mask R-CNN, YOLO/NMS, DETR-family qu…》. NMS

参考文献

1. Panoptic Segmentation — arXiv:1801.00868
2. Fast R-CNN — arXiv:1504.08083
3. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks — arXiv:1506.01497
4. Mask R-CNN — arXiv:1703.06870
5. End-to-End Object Detection with Transformers — arXiv:2005.12872
6. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection — arXiv:2203.03605
7. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection — arXiv:2303.05499
8. Segment Anything — arXiv:2304.02643
9. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks — arXiv:2401.14159
10. SAM 3: Segment Anything with Concepts — arXiv:2511.16719
11. RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection — arXiv:2610.09502