Why can AI that chats about a video still lose "the zebra behind the fence"?

Ask a modern multimodal model to watch a short clip and tell you what is happening, and it will usually do fine. Ask the same model to return the exact pixels of "the zebra active behind the fence", frame by frame, for the whole clip, and it can quietly point at the wrong zebra. Describing a video is mostly a language problem. Localizing one object inside it, pixel by pixel, through time, is a different and much harder class of problem: the model has to decide which evidence counts, hold one identity alive while the world moves around it, and pay for precision on every single frame.

This tutorial is a map of that harder corner. We start from the scale gap that makes naive solutions impossible, then formalize the task and see why the obvious recipe fails, then tour the four solution families that each answer a different part of that failure, and finish with a lens you can use to place any new paper in five minutes. Along the way you will meet a ladder of commitments that keeps reappearing: select relevant evidence, identify the right entity, keep the mask alive in time, and stretch that identity to minute-long videos. Every method in this field is strengthening one rung of that ladder while paying for it somewhere else.

1. Why can't we just load the whole video into the context window?

One movie, fifty-four million tokens

The naive fix for video understanding sounds tempting: tokenize the video, stuff it into the context window, let the transformer do its thing. The numbers make that impossible. The token cost of video is not a few times that of text. It is four orders of magnitude larger.

Caption: The token cost of each modality on a log scale. A short text sits at about 13K tokens, a 4K image at 32K, two hours of audio at 720K, and a 90-minute movie at roughly 54 million tokens, about 4,000× a text. The purple dashed line marks Gemini 2.5's 1M context window, and the video bar towers far beyond it.

No shipping context window fits a movie. The deck that motivates this field makes the point directly: a 90-minute video is on the order of 54 million tokens, while the cited one-million-token context window already struggles with long contexts. Video is a high-dimensional, highly redundant stream; language is a compact, discrete description. Putting all of one inside the other is not an option.

That single fact forces a change of frame. Alignment between video and language cannot mean "load everything and align everything". It has to be selective correspondence: the model must pick which parts of the video answer which parts of the query, and it must decide that on the fly. Everything that follows in this field is a design decision about how to make that selection.

The task landscape, and the corner where pixels meet time

Once alignment is selective, the next question is what the output should look like. The video-language task space is usually organized by information flow. Video to language gives captioning, dialogue, reasoning and video question answering. Language to video gives text-based video generation, motion generation and editing. And the joint direction, language plus video together, contains spatiotemporal grounding and referring object segmentation, tasks where a sentence must pull a specific thing out of a specific clip. Within that joint family, outputs also differ in granularity: a whole-clip sentence or answer, a temporal event, an object trajectory, a bounding box, or a pixel mask. The further right you go on that axis, the more the task demands from the language:

Caption: Shen and Zhang's survey maps the whole family into two paradigms: RVOS methods, categorized by temporal processing and architecture, and ReasonVOS methods, categorized by how they integrate an MLLM, what reasoning paradigm they use, and how they keep temporal consistency. This is a taxonomy map, not a diagram of any single method.Shen & Zhang 2025

Referring video object segmentation, RVOS, is the entry point to the hardest corner of this map. Given a video and a text description, the model must identify the referred object and output its spatial mask through time, not just a box and not just once. The language has to select the correct entity among visually similar candidates, and the output has to preserve that entity's pixels across every frame. That is why RVOS sits at the intersection of the two hardest commitments in this whole space: semantic selection and temporal persistence, both at pixel precision.Shen & Zhang 2025

Before we look at how RVOS methods are built, it is worth anchoring what a "mask through time" even is, because one foundation model made that representation cheap enough to build on: SAM 2.

The mask engine: SAM 2 and its streaming memory

SAM 2 (Segment Anything in Images and Videos) from Meta FAIR is a promptable segmentation model for both images and videos. The key idea is streaming memory. Frames arrive one at a time and go through an image encoder. At each frame, the model cross-attends to a memory bank that holds what it has seen before: a FIFO queue of recent frames plus a small set of prompted frames, along with lightweight object pointers that summarize each tracked object. A mask decoder predicts the mask for the current frame, and a memory encoder folds that prediction back into the bank for the next frame. One sentence in the paper does a lot of work: when applied to a single image, the memory is empty and the model behaves like SAM, its image-only predecessor. An image is just a one-frame video.

Caption: SAM 2 processes video frame by frame. Each frame goes through the image encoder, cross-attends to the memory bank of previously observed frames and prompted anchors, and produces a mask; the memory encoder then writes this prediction back for future frames. Note there is no language input here: SAM 2 consumes clicks, boxes or masks, which is exactly why RVOS systems have to supply the text side themselves.SAM 2

Think of this memory bank the way you would think of a window cache in a streaming data pipeline. The cache does not hold the whole stream, only a sliding window of recent chunks plus a few items you explicitly pinned as anchors, and every new chunk is processed against that window. In SAM 2, the FIFO window holds the recent frames, the pinned anchors are the prompted frames, and the cross-attention is the lookup into the window. Nothing about this design needs to know the total length of the video in advance, which is precisely why it scales to tracks of arbitrary length.

SAM 2 sets the terms for the whole field: it gives you the mask representation, and it requires 3× fewer interactions than prior video segmentation approaches on top of being about 6× faster than SAM on images. But it cannot do RVOS alone. It does not read language. Some other component has to decide which object the sentence means and seed the mask. That division of labor between a semantic selector and a mask engine is the plot of this whole tutorial.SAM 2

The map is drawn and the mask engine is on the table. The natural next question: what exactly does this task demand, and why does the obvious recipe fail?

2. What does RVOS ask, and why does the obvious recipe fail?

The formal contract, and three tasks people confuse

RVOS has a clean formal shape. Given a video $V = {I_1, \dots, I_T}$ and a text query $Q$, output a sequence of binary masks $M = {M_1, \dots, M_T}$, one per frame, marking the pixels of the referred object. The standard score is J&F, the average of two components computed per frame: J, the region Jaccard index, which is intersection over union between predicted and ground-truth mask; and F, the boundary F-measure, which scores precision and recall of the contour pixels. A high J&F means the mask overlaps the true object and traces its edge. Averaged over all frames, it is the workhorse metric of the field.

Three tasks in this area look similar and are not, and keeping them apart early saves a lot of confusion:

• VOS, video object segmentation, works from visual signals only. A first-frame mask is given or discovered, and the model propagates it. There is no language in the loop, so VOS cannot choose between "the person carrying the red backpack" and "the person wearing the red shirt": it defaults to whatever is salient.
• RVOS adds natural language as an input. The query can be anything: a category, an attribute, a spatial relation, a person's role. The model must localize the correct target throughout the video.
• ReasonVOS keeps the same input-output shape but pushes the query from explicit descriptors to implicit ones that need multi-step inference and world knowledge, like "the rail vehicle that is pulling all the other wagons round the track". Shen and Zhang's survey names five reasoning dimensions along which such queries stretch: functional, causal, spatial, temporal and world-knowledge integration.

You can think of these three as escalating demands on the language side: no language, explicit language, implicit language. RVOS sits in the middle, and ReasonVOS is what happens when the "description" stops being a description and becomes a riddle.Shen & Zhang 2025

Describing is not selecting

Here is a qualitative result from the seminar deck that drives the point home. Two queries, three frames each, two methods. The first query is "The lamb with horns in white and brown." The second is "The zebra active behind the fence." In both cases, the highlighted phrase is the deciding modifier: horns for the lamb, active behind the fence for the zebra. A strong generic baseline (shown with blue boxes) picks a plausible object but the wrong individual: a lamb without horns, or a zebra in the open instead of the one moving behind the fence. The stronger system (red boxes) tracks the one the sentence actually names.

Caption: A two-case comparison grid. Top: "The lamb with horns in white and brown" across three frames, where the blue-box baseline selects the wrong lamb and the red-box system holds the horned one. Bottom: "The zebra active behind the fence", where the baseline drifts to the wrong zebra while the stronger system stays with the one moving behind the fence. The red-highlighted modifiers are what make the difference: "horns" and "active behind the fence".

This is the first concrete evidence for the central asymmetry of the tutorial: a model can describe a video fluently and still fail to select the one object a sentence points at. Describing is a generation problem. Selecting is a correspondence problem, and the finer the attribute or the relation, the easier it is to fail. The deck names the structural reasons: expensive fine-grained annotation, few training samples, complex visual and textual information, hard cross-modal alignment, and the absence of strong fine-grained video-text pretrained models. Its proposed strategy is to adapt image-text foundation models to video masks, which is exactly what the rest of this tutorial traces.

The obvious recipe, and its three failure modes

Given this contract, there is an obvious recipe: take a strong image-text grounding model that finds objects from a sentence, run it on every frame independently, and you have your per-frame masks. The recipe fails in three distinct ways, and those three failure modes are the reading lenses for the entire field:

1. Semantic selection. The query is a fine attribute or relation, and there are lookalikes in the frame. An independent per-frame detector has to solve the hardest version of the correspondence problem from scratch on every frame, so "the lamb with horns" and "the zebra behind the fence" are exactly where it breaks.
2. Temporal persistence. Per-frame decisions are independent, so nothing in the design knows that the object in frame 42 is the same object as in frame 41. The mask can flicker, jump to a neighbor, or silently drop the target when it is briefly occluded, then fail to pick it up again.
3. Spatial precision. A grounding detector naturally outputs boxes, and a box is not a mask. Turning box-level detections into pixel-accurate masks, and doing it consistently across frames, is its own design problem that the naive recipe does not solve.

The second failure mode is worth making visceral, because it is the one software people feel immediately. Independent per-frame detection is like a stateless API call: each request knows nothing about the previous one. You can call it a hundred times and it will never notice that the thing in frame 42 is the same object it saw in frame 41. Persisting an identity across calls is exactly what a stateful session adds, with its session state and its session identifier. A mask sequence needs that session-level memory: the model must remember "this entity is still the same entity" as it moves, occludes and reappears.

The foundation lineage: from grounding boxes to video masks

The first family of answers attacks all three failure modes at once by adapting a foundation model, and the cleanest way to see it is through its ancestry. Start with Grounding DINO, an image-text grounding model built on the DETR-style detector DINO with language fused into every stage: image and text backbones, a feature enhancer, language-guided query selection, and a cross-modality decoder. Give it an image and a phrase and it returns bounding boxes with phrase grounding. It cannot produce masks and it does not touch video, but it owns the semantic selection skill: open-set detection that can name anything a sentence can name.Grounding DINO

Caption: The Grounding DINO framework in three blocks: the overall pipeline on top (image and caption through backbones, feature enhancer, language-guided query selection, cross-modality decoder, boxes with phrase grounding), a zoom-in of the feature enhancer in the middle, and a zoom-in of one cross-modality decoder layer below. The output is boxes, not masks, and it is a single-image model.Grounding DINO

ReferDINO is what happens when you extend that ancestor into video masks. The architecture diagram color-codes the story: everything blue is inherited from Grounding DINO and kept frozen, and everything red is new:

Caption: ReferDINO's full pipeline. Blue modules are borrowed from Grounding DINO and frozen: per-frame image-text grounding that produces object queries. Red modules are new: confidence-aware query pruning that keeps only the strongest queries per layer, an object-consistent temporal enhancer that reasons across frames with time-varying text embeddings, and a grounding-guided deformable mask decoder that turns box predictions into per-frame masks. The whole chain is end-to-end differentiable.ReferDINO

The three red modules map one-to-one onto the three failure modes. Query pruning is about speed: the decoder starts with a large set of object queries, and per layer it keeps only the top fraction, scoring each query by how much other queries attend to it and how strongly the text attends to it. This is what takes inference from about 4.9 to about 51.0 FPS at 360P resolution, roughly a 10x speedup, while preserving J&F; random pruning at the same rate, by contrast, drops J&F by about 29%. The temporal enhancer is about persistence: it aligns objects between consecutive frames with a memory-augmented tracker and injects time-varying text features, so dynamic attributes like "the cat swinging its tail" can be resolved across frames instead of within one. The deformable mask decoder is about precision: it uses each object's predicted box as a location prior and does deformable cross-attention to carve out the pixel mask. One number summarizes the family payoff: ReferDINO gains about 3.9 J&F on Ref-YouTube-VOS over the strongest prior method and stays real-time.ReferDINO

Yet even this strong recipe has a ceiling, and the ceiling is visible from the long-video side of the field. Give SAM 2 a perfect first-frame ground-truth prompt, the ideal seed, and ask it to track on its own: it reaches 54.3 to 56.6 J&F on Long-RVOS, the minute-scale benchmark we will meet in Chapter 4, versus 77.3 to 80.6 on the short-clip benchmark MeViS. A roughly 25-point gap with a perfect prompt. The obvious recipe, detector plus propagation, has nowhere to hide: on long videos the target simply stops cooperating.Long-RVOS

The ladder of commitments

Before touring the solution families, fix the ladder they climb. Every method in this field is a bet on which commitments to strengthen, and each rung below corresponds to one of the recurring requirements:

Rung 1 is about evidence: which frames or external sources the model actually reads. Rung 2 is semantic selection: turning that evidence into the right entity. Rung 3 is temporal persistence and spatial precision together: keeping a pixel-accurate mask attached to that entity through time. Rung 4 is the long-video commitment: keeping identity alive across occlusions, disappearances and shot changes for a full minute or more. The four families we are about to visit each answer one part of the naive recipe's failure, and each leaves a different rung of this ladder to someone else. Our next question sets up the first pair: when the video itself cannot answer who is being referred to, where can the answer come from?

3. When the video can't name the object: search outside, or reflect inside

The failure that pixels cannot fix

Some queries cannot be answered by looking at pixels at all. Consider "Please find Labubu", where Labubu is a line of collectible plush toys, or "find the artist who took the top prize at the American Music Awards several months before the public release of Nano Banana". The referent exists in the world, not in the frame: you have to know what Labubu looks like, or know who won what and when, before you can point at anyone. The model's frozen internal knowledge is not enough, and the video alone will never tell you. This is the open-world corner of RVOS, and two agentic families attack it from opposite directions: one searches the outside world, the other reflects on its own reasoning.

Searching outside: Seg-ReSearch

Seg-ReSearch is the outward-looking answer. Its policy is an MLLM that interleaves chain-of-thought reasoning with explicit calls to a search engine: at each turn the model can emit a search of text or images, the retrieved evidence is wrapped and appended to the conversation, and the loop continues until the target is localized or a maximum of five search turns is reached. Only after that does the model emit a keyframe index plus a box and a point, which a frozen mask generator turns into the final mask. The name of the game is dynamic multimodal chain-of-thought: think, search, read, think again.

Caption: Seg-ReSearch reads evidence far beyond the pixels. The input video and instruction feed an MLLM, which runs a multi-turn dynamic chain-of-thought that alternates thinking with text and image searches against a search engine, guided by a reward manager, and only the final answer goes to a frozen segmentation model.

The worked example makes the loop concrete:

Caption: Two think-search-reason traces. On the left, "Please find Labubu": the model first runs a text search to learn that Labubu is a line of collectible plush toys with sharp teeth and large ears, then an image search to confirm the appearance, then grounds it in frame 5 of the video. On the right, "the artist who took the top prize at the AMAs before Nano Banana's release": text searches pin the release date and the winner, and the final mask lands on the right person.

The empirically interesting result is that having a search engine is necessary but not sufficient. Add the same Google Search API to a plain Qwen3-VL-8B and its J&F on the OK-VOS benchmark rises by only 1.8 points. Train the search policy with a hierarchical reward, and the same model reaches 50.0. The hard part is not retrieval, it is the decision of when to search and what to search for. The paper's reward design is the training-side answer: an initial guidance reward seeds the first query, a tapering process reward keeps the model emitting well-formed actions without looping forever, and an outcome reward scores whether the final mask is correct, with the three combined into one objective. The ablation tells the story: a sparse outcome-only reward reaches 41.3 J&F, a rigid process-imitation reward reaches 43.6, and the hierarchical design reaches 46.0 on the 4B model. OK-VOS itself is a benchmark built so that outside knowledge is required: queries answerable from the frame alone were filtered out. Its 1,000 test samples over 150 videos and 500 objects split into one-hop queries where direct retrieval suffices, multi-hop queries that connect several pieces of retrieved information, and relational queries that define the target relative to another knowledge anchor.Seg-ReSearch

Reflecting inside: Refer-Agent

Refer-Agent is the inward mirror. It keeps the agentic structure but replaces the search engine with self-checking: rather than asking the world, it asks itself whether its own intermediate answers hold up. The pipeline decomposes RVOS into four sequential stages, each handled by a dedicated agent: text-guided frame selection picks the frames that matter, intent analysis writes a concise discriminative description for each candidate target, object grounding localizes each description as a bounding box, and mask generation hands the keyframe plus boxes to SAM 2, which propagates the masklets through the whole video.

Caption: Refer-Agent's four-stage main pipeline (frame selection, intent analysis, object grounding, mask generation) runs in blue from left to right, and two orange reflection loops verify it: existence reflection checks that the target is visible and complete in the keyframe and feeds back to frame selection, while consistency reflection checks that the grounded candidate matches the query's attributes and feeds back to intent analysis.

Two design details carry the method. The first is the Dynamic Focus Layout. Stage-reasoning agents that look at a single keyframe lose temporal context, while agents that stack all sampled frames at equal size dilute the MLLM's attention. The layout instead gives the strongest keyframe a large high-resolution block and compresses the surrounding context frames into small slots:

Caption: Six examples of the Dynamic Focus Layout. Each layout composes several sampled frames into one image: the keyframe takes the large, high-resolution block, and surrounding context frames are compressed into smaller blocks at varying positions. The number inside each block is the original frame index.Refer-Agent

This is the counterintuitive part: the keyframe gets more pixels, not because it is the only frame that matters, but because concentrating resolution there gives the intent-analysis agent the best possible evidence for writing discriminative descriptions. The ablation supports the surprise: on ReasonVOS, a single keyframe reaches 67.0 J&F, uniform integration of all frames 67.7, and the Dynamic Focus Layout 69.8. Giving one frame more pixels beats giving every frame equal pixels.

The second detail is the Chain-of-Reflection, a questioner-responder pair that verifies the pipeline in two stages. Existence reflection asks whether the target is visible, complete and optimally framed in the chosen keyframe, and if not, sends instructive feedback back to frame selection, for example "the white sedan is occluded in the keyframe, find a clearer frame". Consistency reflection decomposes the query into attributes and asks multi-choice questions about category, state, appearance, shape and spatial location, and if the prediction drifts from the query, sends feedback back to intent analysis. The reflection is not decoration: removing both stages drops Refer-Agent from 69.8 to 64.5 J&F on ReasonVOS.Refer-Agent

The whole recipe is training-free and MLLM-agnostic. The default backbone is Ovis2.5-9B with SAM 2 as the segmenter, and other MLLMs drop in and out in the paper's ablations. That is the strategic contrast with supervised fine-tuning: an SFT approach trains bridge tokens into a specific MLLM and has to be redone whenever the MLLM generation changes, while Refer-Agent's staged prompts survive model swaps. The reported payoff is J&F of 71.3 on Ref-YouTube-VOS, 54.7 on MeViS and 69.8 on ReasonVOS, with a double-digit gain over the best SFT baseline on the reasoning-heavy benchmark.

Two mirrors of one failure

Stand back and the chapter's rhythm is clear. One family searches the open web for evidence, the other decomposes the problem and audits its own reasoning. Both cost the same things: inference latency that grows with every search turn or reflection round, and hallucination risk that is never fully eliminated. Both are bets on rungs 1 and 2 of the ladder, getting the evidence and the entity right, while delegating the mask itself to a frozen engine. That delegation is exactly what the next chapter questions: if you do not want reasoning latency at all, if you want precision written directly into the network, and if the video is a minute long, what then?

4. When time is the enemy: real-time precision and minute-long identity

The agentic families buy intelligence with latency. Two other families make the opposite bet: no reasoning loops, precision engineered into the network itself. They split along the axis of time. ReferDINO, which we met as the reader's first architecture, is the real-time, short-clip exponent. Long-RVOS, the minute-scale benchmark, brings its own baseline, ReferMo, which stakes out the long-video end.

The foundation-adaptation route: precision written into the network

ReferDINO is the representative of the foundation-model adaptation family. Nothing about it is agentic: no MLLM, no search, no reflection rounds. It inherits Grounding DINO's cross-modal grounding, adds the three red modules from Chapter 2, and the whole chain runs end to end at about 51 FPS at 360P resolution, with roughly 3.9 J&F gained over the strongest prior method on Ref-YouTube-VOS. Its cost is equally clear: you need a pretrained image-text grounding backbone in the loop, and the design is short-clip oriented, sampling a handful of frames per video rather than reasoning about minutes. A companion report from the same group pushes accuracy further by fusing ReferDINO's masks with SAM 2 as a refinement pass, explicitly trading real-time speed for stronger segmentation.

The pipelined variant is where SAM 2 enters the foundation family as a frozen module, and this is the clean way to think about it: frozen SAM 2 is a mature dependency library. You call it without retraining it. Your own network is responsible for the semantic decisions, the keyframe selection, the mask seeding; then you hand the mask to the library and let its tested propagation fill in the rest. You trade away control over the pixels in between for the library's robustness and zero training cost.

Long-RVOS: the benchmark where objects vanish

The long-video side of the field exists because short clips hide a failure mode. In a few seconds, the target is usually salient, visible and roughly static in identity. Over a minute, everything breaks: the object is occluded, disappears entirely and reappears, the shot changes and the appearance changes with it, or the object is identifiable only through a long-range motion pattern that no short window can capture. Long-RVOS is the benchmark built to expose this. Its numbers are the scale of the problem: 2,193 videos averaging 60.3 seconds and 361.7 frames each, 2.1 million masks over 6,703 objects in 163 object categories, with 24,689 text descriptions. The category spread matters because it is not just people: humans dominate at about 71.9%, but instruments at about 15.4% and animals at about 12.7% force the models to hold identity across very different motion and appearance regimes.Long-RVOS

Caption: A montage of objects and scenes in Long-RVOS: people, animals and instruments across indoor and outdoor settings. The diversity supports the benchmark's 163 object categories and its demand that identity survive across wildly different appearance and motion regimes.Long-RVOS

The benchmark also redefines the descriptions you evaluate against. It separates queries into Static, which rely on appearance and position such as color or "the left cat"; Dynamic, which rely on motion, state changes and interactions such as "the cat chasing a mouse"; and Hybrid, which combine both. The split is nearly balanced at 35.03%, 32.45% and 32.52%, which means a model cannot cheat by solving one type. And it introduces two metrics beyond frame-level J&F: tIoU measures temporal overlap between predicted and ground-truth non-empty mask frame sets, penalizing a model that emits a mask on frames where the target is absent; vIoU averages per-frame Jaccard over the intersection of those frame sets, exposing whether good per-frame scores actually come from consistent tracking. The finding that matters: tIoU tends to be stable across methods while vIoU is consistently low, which is the quantitative proof that high J&F does not imply temporal consistency. Frame-level scores can look great while the model is quietly tracking the wrong thing half the time.

The oracle experiment makes the point sharper than any method comparison. SAM 2 given perfect first-frame ground-truth prompts, the strongest possible seed, reaches only 54.3 to 56.6 J&F on Long-RVOS, against 77.3 to 80.6 on MeViS. A roughly 25-point gap with a perfect prompt. The missing ingredient is not a better tracker, it is the language-guided selection of what to track across the long arc of the video.

ReferMo: sparse high-resolution keyframes, frozen propagation

ReferMo is the baseline the benchmark ships with, and its design is a direct answer to the cost problem: processing a minute of video at full resolution with full temporal reasoning is simply too expensive. ReferMo instead decomposes the video into clips, each clip being one high-resolution keyframe plus a strip of low-resolution motion frames. The motion frames come from MPEG-4 motion vectors extracted during decoding, not from dense optical flow, which keeps them nearly free. Within each clip, local perception fuses text, keyframe and motion features to capture appearance and short-term dynamics. Across clips, global interaction aligns object features and applies temporal self-attention, so the identity of the referred object is decided with the whole minute in view, then carried clip to clip.

Caption: ReferMo's pipeline. The video is cut into clips of one high-resolution keyframe plus low-resolution motion frames; local perception fuses text, keyframe and motion features within each clip; global interaction aligns object features across clips with temporal self-attention; and the segmentation head predicts only the keyframe masks, which a frozen SAM 2 propagates through the rest of the frames at inference time. SAM 2 is not trained jointly with ReferMo.Long-RVOS

The supervision trick is the heart of the trade-off: the model is trained only on keyframe masks. Non-keyframe masks are produced at inference by the frozen SAM 2 propagator, which keeps training cost flat even on minute-scale videos. On the Long-RVOS test set, ReferMo reports the best numbers in the paper's comparison across every primary cell: 55.8 J&F on Static, 49.3 on Dynamic, 53.3 on Hybrid and 52.9 Overall, with 73.6 tIoU and 45.2 vIoU, at 52.5 FPS on 360P resolution on an A6000. The cost of the long temporal receptive field is equally explicit: dense per-frame processing is abandoned, and the pixel-level detail of non-keyframe frames is delegated to propagation.Long-RVOS

Put the three recurring requirements next to the long-video challenges and the recalibration is visible:

The two routes of this chapter are the two honest prices: ReferDINO-style foundation adaptation buys real-time pixel accuracy and needs a mature grounding backbone, while ReferMo-style long video buys minute-scale identity and supervises only keyframes. Each route answers the question the other one leaves open. The map looks complete, so the next question is whether the field itself has stopped moving.

5. Where is RVOS still moving in 2026?

The map we have drawn has four families, but the boundary is still growing, and two 2026 papers mark the two axes of that growth: motion on the language side, and perception before reasoning on the agentic side.

The language-side signal is EMC, Expression-driven Motion Calibration. Its observation is that existing pipelines apply motion uniformly: a static expression like "the red mug on the table" gets the same motion treatment as "the man moving to the left". EMC instead models how much motion a given expression actually depends on, and calibrates accordingly. An LLM produces an explicit motion signal, a flag and a strength value, from the expression; a calibration module gates motion's influence on keyframe selection by multiplying a base reliability score with a motion-saliency term weighted by that strength, with the motion branch shut off entirely when the flag is zero. Around that gate, spectral clustering partitions the video into expression-aware temporal stages so candidates are drawn from a compact set, and bidirectional SAM 2 propagation fills the mask sequence.EMC

Caption: EMC's three modules. STSC partitions the video into expression-aware temporal stages; MSP runs the text expression through an LLM and emits an explicit motion signal; MIC gates how strongly motion influences keyframe selection, after which bidirectional SAM 2 propagation produces the full mask sequence. The novelty sits on the language side: motion influence is calibrated per expression, not applied uniformly.EMC

Why does this belong here? It is the mirror image of ReferMo's motion story. ReferMo extends the temporal receptive field per clip on long videos. EMC calibrates motion per expression on the same keyframe-then-propagate skeleton. Two answers to the same unresolved question of how much motion really matters, one on the video axis and one on the language axis. EMC reports the best J&F in its comparisons across the six standard benchmarks, for instance 74.7 on Ref-YouTube-VOS and 77.7 on Ref-DAVIS17, which is the field's way of saying the dynamic-attribute axis is still being actively pushed in 2026.

The agentic-side signal is AgentRVOS, and it inverts the order of operations that every agentic pipeline so far has used. The standard pattern is MLLM first: the LLM picks keyframes, grounds the object, then a segmenter propagates. AgentRVOS flips it: perception first. SAM 3 generates a dense pool of candidate mask tracks from concepts extracted from the query, and only then does an MLLM reason over the pool, classifying each candidate as accepted, rejected or uncertain, while progressively narrowing the temporal scope until convergence.

Caption: AgentRVOS runs in two phases. Left: candidate mask track generation, where the MLLM extracts concepts from the query and SAM 3 produces temporally consistent candidate mask tracks, iterating for coverage. Right: iterative spatio-temporal pruning, where the MLLM classifies each candidate as Accepted, Rejected or Uncertain and the temporal scope narrows round by round until convergence.AgentRVOS

The intuition is that the MLLM was being asked to make temporal decisions before it had any object-level evidence to anchor them. Generate the tracks first, and reasoning operates on real visual facts. The measured effect: the empty-mask ratio, the share of queries where the pipeline finds nothing, drops from 12.0% in a prior training-free system to 3.8% after three iterations. With a GPT-5 backbone the pipeline reports 73.1 J&F on MeViS, 66.3 on ReVOS and 75.5 on ReasonVOS. The family contrast with Chapter 3 is one sentence: Seg-ReSearch reads the web for evidence, Refer-Agent reads its own reasoning, and AgentRVOS reads object-level mask tracks, evidence of a third kind, generated by perception itself.AgentRVOS

Both signals are short by design. They are not new families, they are the leading edges of the families we already have: EMC extends the long-video family's motion axis from the language side, and AgentRVOS extends the agentic family from the reasoning side. With the boundary drawn, one question remains: faced with a dozen methods, is there a lens that places any one of them in a minute?

6. One lens for any RVOS paper, and three routes deeper

Go back to the taxonomy map from Chapter 1 and stop reading it as a classification exercise. The map's real use is as a coordinate system: when a new paper arrives, you ask three questions about it, and the answers place it on the landscape and on the ladder.

The first question is what evidence the method reads. Pixels only, like ReferDINO. Pixels plus searched text and images, like Seg-ReSearch. Pixels plus the model's own reasoning and reflection, like Refer-Agent. Sparse high-resolution keyframes plus motion frames, like ReferMo. The second question is what representation the method preserves. A per-frame box or mask, or an actual mask track that survives through time. The third question is what accuracy-efficiency trade-off it pays. Real-time inference with a heavy pretrained backbone, agentic latency with reasoning loops, or sparse processing with delegated propagation.

Then ask the fourth question, the one that ties the lens to the ladder: which rung does this method strengthen, and which rung does it leave to someone else? Seg-ReSearch and Refer-Agent fortify rungs 1 and 2, evidence and entity, and delegate the mask. ReferDINO fortifies rungs 2 and 3, entity and pixel precision, at the cost of short-clip scope. ReferMo fortifies rung 4, minute-long identity, at the cost of per-frame detail. A paper that claims a gain on MeViS is mostly a rung 2 and 3 paper. A paper that reports tIoU and vIoU on Long-RVOS is a rung 4 paper. The lens works both as a placement tool and as a sanity check: if a paper claims to strengthen everything, look again at what it pays.

If you want to go deeper along one of the three routes, the natural next reads are the group's deep dives. The foundation-model route continues in Inside ReferDINO: From Grounding Boxes to Fast Temporal Masks. The agentic-reasoning route continues in Refer-Agent: Turning RVOS into a Team of Verifiable Decisions. The long-video route continues in ReferMo: Stretching Temporal Receptive Field Without Processing Every Frame Equally.

So here is the whole tutorial in one breath. Describing a video is easy because language tolerates vagueness. Localizing one object at pixel precision, through time, is hard because it demands selection, persistence and precision all at once, and no method gets all three for free. The model that chats fluently about the clip can still lose the zebra behind the fence. Now you know why, and you know the four families that each try to win one part of that race, and you have the three questions that will place any new paper before you read its ablations.

References

The sources cited in this tutorial, in order of appearance:

1. A survey of language-guided video object segmentation: from referring to reasoning — Springer (Vicinagearth, 2025)
2. Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search — arXiv:2602.04454
3. Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation — arXiv:2602.03595
4. ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations — arXiv:2501.14607
5. Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation (with ReferMo baseline) — arXiv:2505.12702
6. SAM 2: Segment Anything in Images and Videos — arXiv:2408.00714
7. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection — arXiv:2303.05499
8. Unlocking Motion in Expressions: Temporal Calibration for Referring Video Object Segmentation (EMC) — arXiv:2608.16332
9. AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation — arXiv:2603.23489