From pixel masks to prompts: four detector generations, each deleting a hand-designed stage

Every generation of detector in this line answers one question, where candidates should come from, and every answer deletes one piece of hand-designed machinery. Region proposals answer it with an explicit stage. The dense grid answers it by regressing boxes out of every cell. DETR answers it with a fixed set of learned slots. The query-conditioning line answers it with the image, with text, or with a prompt. Read as a ledger rather than a leaderboard, each deletion moves work out of inference-time engineering and into something the model must learn, and the bill always arrives somewhere you can point at.

1. A box cannot say which pixels are whose: the mask has to be a per-region output of the detector

A box localises an object and stops there, so instance segmentation is a different output rather than a sharper box. The segmentation family differs only in the rule used to group pixels: one label per category, one identity per countable thing, or both kinds of region in the same map.Mask2Former

Caption: One photograph under three grouping rules. The semantic panel colours every rider red and every horse the same tint; the instance panel gives each rider-and-horse pair its own colour and leaves background blank; the panoptic panel keeps instance colours for things and category colours for grass, trees and sky. The mask is therefore a requirement of the task, not a decorative head.

Mask R-CNN supplies it by adding a third branch alongside the two outputs Faster R-CNN already produces for every candidate region, a class label and a box offset, so that each region also emits a dense mask.Mask R-CNN

Caption: The whole recipe in one frame. The same extracted region feature splits into a class and box path and a two-convolution mask path, which is what makes the mask a parallel peer of the box rather than a post-hoc refinement.

The mask branch is a K-channel output at m x m per region, trained with a per-pixel sigmoid and binary cross-entropy defined only on the ground-truth class channel. Other channels do not compete, which decouples mask from classification. Replacing the sigmoid with a per-pixel softmax costs 5.5 mask AP (24.8 against 30.3), because softmax forces the channels to race.Mask R-CNN

Caption: The tensor anchor, read off the figure's labels rather than from a sentence in the paper. On the FPN panel the region feature is labelled 14x14x256, four convolutions hold it there, a stride-2 deconvolution takes it to 28x28x256, and a 1x1 output convolution emits 28x28x80, so m = 28 and K = 80.

Instance separation comes from the per-region structure, not from the class axis: each candidate object is its own region with its own mask, and class-agnostic masks still reach 29.7 mask AP against 30.3 for class-specific ones. Pixel accuracy depends on alignment. RoI pooling rounds continuous coordinates onto the feature grid, computing x/16 and quantising it to an integer cell, and that rounding is tolerable for classification but damaging for masks. RoI Align keeps the fractional coordinate, samples four locations per bin by bilinear interpolation, and aggregates with max or average. The paper states that the sample count and the sample locations do not matter as long as no quantisation happens, and the ablation prices the fix at 26.9 to 30.2 or 30.3 mask AP. A keypoint branch on the same head reaches 63.1 keypoint AP at 5 fps, which is the cheapest evidence that one per-region head can serve several dense targets.

2. Remove the proposal stage and the overlap bill arrives: YOLO regresses from every cell, NMS pays for it

Deleting the proposal stage does not delete the need for it. It converts a stage that selected a few well-placed candidates into an inference-time filter that has to clean up a dense field of overlapping ones.

What the deleted stage was buying is easy to state: a small set of class-agnostic regions, roughly 2000 per image from a method like selective search, worth classifying. YOLO replaces it with an unconditional answer. The image is cut into an S x S grid, each cell predicts B boxes of five numbers (centre offsets, width, height, confidence), and each cell also predicts C class probabilities shared across its boxes, giving an S x S x (5B + C) tensor.YOLOv1

For Pascal VOC the paper's configuration is S = 7, B = 2, C = 20, so each cell carries 30 numbers and the whole output is 7x7x30. B = 2 is not a detail: the arithmetic only closes at two boxes per cell, and later YOLO configurations with a larger B produce a different tensor. Responsibility is assigned by geometry, since the cell containing an object's centre owns it, and that rule is what makes one-shot prediction well defined without matching machinery. The cost is structural: 98 boxes per image for a handful of objects, and objects near cell borders or larger than a cell get localised by several cells at once.

Caption: The dense hypothesis field. Boxes with confidence and the class probability map are two parallel outputs of the same grid, and the final panel is what survives suppression. The drawing labels S x S symbolically and prints no value for S or B.

Caption: The suppression loop as four frames: many boxes, a few scored boxes, overlapping lower-scoring boxes crossed out, one box per object left. Keep-highest, remove everything overlapping it above a threshold, repeat is the standard algorithm drawn here; the YOLO paper names the step and sizes it at 2-3 mAP but never writes the loop or a threshold, and the printed scores are the panel's own example.

The counter-intuitive half is that this design also escapes the alignment problem entirely, because YOLO never resamples a region and therefore has nothing to misalign. What it takes on instead is localisation error plus a large background-precision advantage: the paper reports more localisation errors than its baselines but far fewer false positives on empty background, with 13.6% of Fast R-CNN's top detections being background, almost three times YOLO's rate.YOLOv1

3. Delete the filter instead of tuning it: DETR emits a fixed set of slots and makes matching enforce uniqueness

DETR moves the uniqueness constraint out of post-processing and into the training objective, so what used to be an inference-time filter becomes something the model is taught.

The output becomes a fixed set of N learned object queries, N = 100 in the experiments, decoded in parallel. Ground truth is padded with a no-object class to the same size, and training finds the one-to-one assignment of lowest total cost with the Hungarian algorithm. The cost is the only formula in this article that earns its space, because it is the mechanism:

$$\hat{\sigma} = \arg\min_{\sigma} \sum_{i} \left[ -,p_{\sigma(i)}(c_i) + \lambda_{\mathrm{iou}},\mathrm{GIoU}(b_i, \hat{b}{\sigma(i)}) + \lambda{\mathrm{L1}},\lVert b_i - \hat{b}_{\sigma(i)} \rVert_1 \right]$$

The class term uses a probability rather than a log-probability so that it is commensurable with the box terms, and the log-probability of the no-object class is down-weighted tenfold to counter class imbalance. Only the matched slot takes the class and box loss; every other slot takes the no-object loss, which is what makes the set prediction orderless and parallel.DETR

Caption: The set claim rather than the architecture. The whole prediction set is emitted at once, matched one-to-one to ground truth, and unmatched slots fall to no object. The figure is schematic: six squares are not the query count, and it shows no loss formula or matching implementation.

Caption: The fixed-slot regime at a readable scale, with ten slots of which six are no object. Two same-class objects can occupy two slots, so a slot is not a category and an object is not guaranteed a slot.

The deletion is a mechanism claim, not a slogan, and the paper tests it: running NMS on the per-layer outputs lowers AP in the later decoder layers because it removes true positives, and helps only at the first layer, where self-attention has not yet had a chance to inhibit duplicates. Two consequences define the next generation. Slots specialise by geometry rather than by class, clustering on image regions, box sizes and aspect ratios, and decoder attention concentrates on object extremities such as heads and legs. And the design has a hard ceiling, since DETR cannot predict more objects than it has slots, which is 100.

4. A blind query was the bottleneck: initialise it from the image and NMS-free becomes deployable

On DINO's account, the obstacle to end-to-end detection lay in the starting state of its queries. Once that state carries a spatial prior from the image, the pipeline converges fast enough to train; making it cheap enough to run without a filter took a second fix, on the encoder's cost.

DINO's diagnosis is that a static learned query carries no information about the particular image, and early in training the assignment between ground-truth objects and queries keeps changing. The fix splits the query into a positional part, expressed as a 4D anchor box, and a content part that stays learnable, and initialises only the positional half from the encoder's top-K selected features. Mixed query selection deliberately discards the selected feature itself, because a partially refined feature may cover several objects or only part of one, so it is safer as a position than as content.DINO

Caption: What the blind slot costs. On the left the links between two ground-truth objects and three queries cross and change between steps; on the right each object keeps one query. The diagnosis is DINO's, not DETR's: DETR documents slow convergence but never calls its assignment unstable.

Caption: The three initialisation schemes side by side. In the mixed panel only the hollow squares, the positional anchors, are replaced by the selection path, while the filled content squares stay static. That is the whole difference from the pure variant beside it.

The evidence that initialisation rather than architecture was the bottleneck is a training-budget comparison: DETR reaches 15.5 AP in the 12-epoch setting where DINO reaches 49.4, and cutting the decoder from six layers to two costs DINO 3.0 AP against 13.8 for a comparative variant. Contrastive denoising adds a second signal, feeding noised ground truth as positive and negative queries so that a query near but not on an object learns to answer no object.

Deployment needed a second fix, because DETR's encoder cost blocked real-time use and NMS was the thing standing between YOLO and an honest latency number. RT-DETR confines transformer attention to the deepest feature stage, fuses scales with a convolutional block, and selects initial queries by a criterion that penalises the disagreement between a feature's classification and localisation scores.RT-DETR

Caption: Where the speed comes from: attention only on the deepest stage, a CNN cross-scale fusion block, and an explicit selection step between encoder and decoder. The image carries no latency numbers and does not itself prove the NMS removal.

The NMS cost is measured rather than asserted, at 2.24 to 2.46 ms for the suppression kernel as the IoU threshold moves, and the reported 53.1 AP at 108 FPS is a T4 measurement with TensorRT FP16 at batch 1 and 640x640 input. Read the table with care: the YOLO column in the same table includes NMS while the DETR column does not, which is the paper's point and also the reason the two columns are not like-for-like.

5. Move the query outside the model: language supplies the labels, geometry supplies the region, and the box carries the category

Once the query can be conditioned on text or on a point, the interesting question stops being which network wins and becomes which stage owns the category. When a detector and a segmenter are chained, the answer is the detector, because the segmenter never sees the label.

Grounding DINO shows that language can enter a closed-set detector at three places, the neck, query initialisation and the head, and it fuses at all three. Its decoder is DINO's with one extra layer type per layer, a text cross-attention after the image cross-attention, and its classification branch is a text-region similarity: each query is dot-producted with every text token and trained with focal loss, beside a localisation loss on the boxes. Because the label set comes from the prompt, the detector is open-set, but open-set here means only that the label set is supplied by the text rather than fixed by the dataset.Grounding DINO

Caption: Language as an injection point rather than a new architecture family. The same closed-set skeleton can be opened at the neck (fusion A), at query initialisation (fusion B) or at the head (fusion C), and the paper's contribution is covering all three.

The two evaluations are not the same capability, and the gap is the useful part. Category prompts are scored by AP on detection data, while referring expressions are scored by top-1 accuracy, and the Swin-T variant moves from 50.41 to 73.98 top-1 on RefCOCO val once referring data enters pre-training; the larger fine-tuned Grounding DINO L reaches 90.56. Zero-shot COCO detection reaches 52.5 AP and ODinW averages 26.1, with the long tail weaker than the frequent classes.

SAM takes the query off the label axis entirely and puts it on geometry. A heavy image encoder runs once per image and produces an embedding worth caching, a prompt encoder turns points, boxes, text or masks into tokens and into a feature-map offset, and a deliberately light mask decoder turns those into masks. The economics are the design: the decoder costs under one percent of the encoder's compute, which is why one embedding can serve many prompts at interactive speed.SAM

Caption: Points, boxes, text and masks are alternative inputs to one decoder, and every output mask carries a quality score. The split exists so the expensive part never runs twice for two prompts on the same image.

Chaining the two is where the category question becomes concrete. Grounding DINO consumes text and emits boxes with corresponding phrases, those boxes become SAM's prompts, and SAM returns masks. The box is the only channel across the boundary, so the category travels on the detector's boxes and never reaches the segmenter, which also means a wrong label cannot be repaired downstream.Grounded SAM

Caption: Separately trained models chained at inference, with the label attached to the detector's boxes and the segmenter receiving only geometry. Each lower band swaps in another expert model on the same interface, which is the composition claim.

The measured bound belongs here too. SAM's own zero-shot instance-segmentation experiment feeds it a supervised detector's boxes and still trails that detector on mask AP, 46.5 against 51.0 on COCO, so promptable is not the same as label-free.

6. The 2026 check: concept prompts widen the query, duplicates survive inside the decoder, and the frontier model does not win in the wild

Conditioning the query on text, geometry or a concept changes what a detector can be asked for. It does not replace supervision, and it does not by itself remove the duplicates that one-to-one matching was supposed to delete.

SAM 3 moves the prompt from an instance to a set. A concept prompt is a short noun phrase, an image exemplar, or both, and the task returns masks plus unique identities for every matching instance in an image or a short video. Structurally the interesting piece is a single global presence token that answers whether the concept is in the frame at all, so that each query only has to decide whether it matches a present concept:

$$p(\text{query}_i \text{ matches } NP) = p(\text{query}_i \text{ matches } NP \mid NP \text{ present}) \times p(NP \text{ present})$$

The presence score is shared by all 200 queries, which decouples recognition from localisation inside a localisation-first decoder. On the SA-Co/Gold split the model reaches 54.1 cgF against 24.6 for the strongest baseline and 72.8 for human annotators, and its own reported LVIS zero-shot mask AP is 48.8 in the introduction and 48.5 in the paper's first table. The stated limits are narrow: weak generalisation to fine-grained out-of-domain concepts, and no long referring expressions without an external multimodal model.SAM 3

Caption: The semantic shift in one picture. On the left a point prompt selects one bird; on the right a noun phrase or a pair of exemplar boxes selects the whole set of matching instances, which is a set-valued answer rather than a single mask.

Whether matching really removes duplicates is testable, and the 2026 answer is no. Take a standard 6-decoder DETR-style model and read its predictions after only the first decoder stage, the point where duplicates are worst, evaluating once as is and once with NMS at evaluation time: AP rises by 2.4 to 3.0, so duplicates are still present after training with one-to-one matching. The proposed repair keeps the ordering that NMS provided and drops the threshold: queries are ranked by confidence, the top one becomes a pivot, queries similar to it by a mix of box IoU and class-score cosine are grouped behind it, and the process repeats until every query is ordered. A bidirectional recurrent block then takes over the decoder's self-attention slot while cross-attention and the feed-forward block stay untouched, and nothing is discarded. With that in place the gain from adding NMS collapses to 0.3, 0.2 and 0.9 AP across three baselines, and on a crowded-person benchmark the method reaches 47.6 AP with no threshold tuning where the baseline needs a hand-picked NMS threshold to move between 46.2 and 46.7.DETRNN

Caption: Competition moved inside the decoder. The unordered query set becomes an ordered sequence, and the recurrent scan occupies the slot self-attention used to fill, with cross-attention, the feed-forward block and the class and box heads unchanged.

Real data adds a second correction. A 2026 in-the-wild benchmark puts exactly these families through one dataset of 6,016 annotated 4K frames of a single bird species, with the test split held out at the level of a whole recording, 1,006 images, so that frames from the same session cannot leak across the split.Bluebird

Zero-shot open-vocabulary detection collapses on this domain, with recall at 0.0080 for YOLO-World despite precision near 0.9. Fine-tuning rescues it unevenly and model-dependently: YOLO-World reaches 0.4467 box mAP and 0.9370 F1, while OWL-ViT gains mAP but loses recall, and Grounding DINO has no adapted row at all. And a supervised Mask R-CNN still leads both SAM 3 zero-shot and the Grounded SAM chain on mask mAP, at 0.6398 against 0.2229 and 0.0000. This is one species at one site with a one-video test split, so it is a stress test of a claim, not a family ranking.

Across these rungs the axis reads as a set of paid-for trades rather than a ladder.

References

1. Mask R-CNN — arXiv:1703.06870
2. Masked-attention Mask Transformer for Universal Image Segmentation — arXiv:2112.01527
3. You Only Look Once: Unified, Real-Time Object Detection — arXiv:1506.02640
4. End-to-End Object Detection with Transformers — arXiv:2005.12872
5. DINO: DETR with Improved DeNoising Anchor Boxes — arXiv:2203.03605
6. DETRs Beat YOLOs on Real-time Object Detection — arXiv:2304.08069
7. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection — arXiv:2303.05499
8. Segment Anything — arXiv:2304.02643
9. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks — arXiv:2401.14159
10. SAM 3: Segment Anything with Concepts — arXiv:2511.16719
11. Order Matters: Competition-Guided Query Ordering for RNN-Based Object Detection — arXiv:2610.05191
12. Towards Benchmarking Western Bluebird Detection in the Wild — arXiv:2610.07802