A robot picks up a mug nobody labelled: the map of spatial AI's unclosed loop

Spatial-intelligence work is usually introduced as a list of subfields. The reading that actually carries information is a single closed loop of four steps, where the interesting structure is not the taxonomy but what each step hands to the next. A 67-slide seminar deck by Kai Han (The University of Hong Kong / Visual AI Lab, presented at HKUST Guangzhou in October 2026) arranges roughly twenty methods under three pillars and then ties them into one loop: understand → model → interact → correct. Read as a dependency chain rather than as a list, the deck's five open research problems all land exactly where one of those hand-offs leaks.

1. Three pillars are three arcs of one loop, and each step hands the next a different artefact

The deck's frame puts three pillars side by side: spatial intelligence (feed-forward 3D reconstruction, 3D/4D generation and editing, physical simulation, 3D-aware MLLMs), foundation models (vision–language alignment, post-training for grounded reasoning, benchmarking capability ceilings, video world models), and open-world learning (knowing what the model does not know, novel and generalised category discovery, concept extraction with continual discovery). Those are not three parallel research programmes. They are three arcs of one cycle, and the cycle has four steps rather than three for one reason: describing a scene does not by itself let anything act in it.

That last claim is argued from outside the deck, which is what makes it more than a slogan. "Perception alone does not confer the ability to act: a model that can describe a scene in detail may still be unable to navigate through it or manipulate objects within it"SpatialAgents. The distinction that keeps the two apart is between symbolic grounding, which is what CLIP-style contrastive models do when they associate an image with text, and spatial grounding, which is metric understanding of geometry and physics. Perception supplies the what and the where. Agency additionally requires planning, memory and action, so a system can hold a correct scene description and still be a passive observer.

Caption: the loop is one cycle with four steps, not a list of four topics. Each arrow is a contract: the Understand step emits a point map, the Model step hands over explicit geometry, the Interact step produces text that carries position, and the Correct step returns a new concept that the Understand step can use.

The agentic machinery that has to be wired into those steps is not implicit. The figure below lays out how memory, planning and tool use land on concrete spatial task domains through named interfaces.

Caption: this is a capability-to-domain mapping, not a coordinate chart. Memory Systems reaches Navigation through cognitive maps and Scene Understanding through semantic memory; Planning Systems reaches Manipulation through task and motion planning and Geospatial Analysis through route planning; Tool Use and Action reaches the same two domains through vision-language-action interfaces and geo-data APIs. It carries no axes, no spatial-scale levels and no method positions.SpatialAgents

Read this way, the four steps behave like stages that agree on a data contract: change what one stage emits and you change what the next stage can be built against. Everything that follows is an account of those four contracts, taken one at a time, and of what each step still fails to hand over.

2. "Spatial intelligence" is a three-rung ladder from perception to extrapolation, and its benchmarks inflate how well models climb it

The term is the reader's most likely trap, because it is easy to hear "spatial intelligence" as "3D detection with extra steps". A recent survey of spatial intelligence in vision-language models makes the term operational instead of merely defining it, and the operational form is a three-level hierarchy borrowed from human spatial cognition. L-1 Perception covers individual objects plus their intrinsic spatial attributes (size, geometric structure, orientation), which is already more than category recognition. L-2 Understanding asks how objects are arranged relative to one another. L-3 Extrapolation reasons beyond the immediately visible input: inferring hidden states, predicting future configurations, or adopting viewpoints that are not shown. The task families cluster accordingly, with 3D detection, segmentation, orientation and depth on the first rung; spatial-relations VQA and spatial grounding on the second; spatial manipulation, occlusion reasoning, mental rotation, perspective-taking and spatial planning on the third. The move the survey describes is from static perception to situated understanding, not from pixels to boxesSpatialVLM.

That same survey supplies the second useful piece of vocabulary: four named root causes of spatial failure, each with a mechanism rather than a label. A suboptimal vision encoder (CLIP is the de facto default, and coarse semantic objectives do not preserve fine-grained 3D shape or layout), inefficient positional embeddings (fixed sinusoidal designs versus learnable or rotary ones, plus the suppression effect where high-magnitude vision tokens drown out subtler positional cues in attention), lack of effective modality alignment (contrastive objectives scale to web data but skip fine-grained spatial supervision, producing bag-of-concepts behaviour), and scarcity of datasets (spatial corpora lack absolute-spatial and geometric annotations, and the top 17% of relation categories account for over 90% of all spatial examples, which cannot support extrapolation).

Caption: the figure is a structure map, not a scoreboard. Problems on the left, the five places you can intervene on one VLM pipeline in the middle, evaluation and open directions on the right. It contains no numbers and no rankings, and the same survey's own measurement section finds no universally winning intervention category.SpatialVLM

The survey's third contribution is a five-category taxonomy of methods sorted by where you intervene, not by what technique you use: training-free prompting, model-centric enhancement, explicit 2D information injection, 3D spatial information enhancement, and data-centric enhancement. Sorting by location is what makes the taxonomy usable, because it exposes an asymmetry that a technique-based list hides: prompting improves perception and understanding through explicit attention guidance but degrades extrapolation, where over-specified cues become distractors rather than aids.

Caption: the five categories are not five techniques but five attachment points on one pipeline. Prompts attach at the input, 2D and 3D priors attach as extra inputs, an encoder or fusion change attaches inside the model, and data support runs underneath everything. The drawn encoder stack is the survey's abstraction of a generic VLM, and the illustrated answer is not a benchmark result.SpatialVLM

The measurement part of the survey is the part to carry forward, because a later chapter of this map depends on it. Across 9 benchmarks and 37 models the ordering is Understanding (best), then Perception, then Extrapolation (worst): models are weakest exactly where reasoning has to go beyond what is visible, and surprisingly weak at basic spatial awareness as well. The central negative result is about the instrument rather than the models: "Current spatial reasoning benchmarks disproportionately emphasize vision-based relational tasks while underrepresenting metric and object-centric reasoning. This design bias inflates apparent model performance and obscures genuine spatial reasoning limitations"SpatialVLM. The authors add their own caution that the numbers are protocol results rather than contamination-free estimates of intrinsic spatial ability, and the survey's literature window closes in October 2025 with publication in August 2026, so it is a well-organised recent survey rather than the latest word.

One axis warning is worth stating plainly, because it is the confusion this map is most likely to produce. This taxonomy sorts methods by where you intervene. The taxonomy that frames the representation question, which arrives with the geometry strand below, sorts them by what the world is represented as. They are orthogonal, and a method belongs on one axis without implying anything about its place on the other.

3. Reconstruction comes first for an interface reason; the representation you choose then decides who can render, simulate or edit

Geometry is modelled before anything is generated, and the reason is the shape of the output rather than its accuracy. DUSt3R replaced the serial pipeline with a single regression onto a new object, the pointmap: a dense field of 3D points in one-to-one correspondence with the pixels of an image. One tensor carries three things at once: the scene geometry, the relation between pixels and scene points, and the relation between the two viewpointsDUSt3R. Two images enter a shared-weight ViT encoder, two transformer decoders keep exchanging information by cross-attention at every block, and two regression heads emit two pointmaps plus confidence maps, both expressed in the coordinate frame of the first image.

Caption: this is the shape of the artefact the rest of the chain reads, not a claim about how accurate it is. One forward pass, two views, one coordinate frame, one dense 3D field per view plus a confidence map. The figure does not draw the downstream readouts, and it does not show the multi-view global-alignment stage.DUSt3R

Because the method casts pairwise reconstruction as pointmap regression, it relaxes the hard constraints of the usual projective camera model, and that relaxation is what unifies the monocular and the binocular case: the monocular experiment literally runs the network on the same image twice. The limits are as informative as the shape. Outputs are valid only up to an unknown global scale, pointmaps need not correspond to any physically plausible camera model, and one forward pass is strictly a pairwise property: a full multi-view scene needs a separate global-alignment step, which minimises 3D projection errors rather than the 2D reprojection errors that classical bundle adjustment minimises. What matters for the map is that the result is task-agnostic and scene-level. Every classical quantity (depth, focal length, relative and absolute pose, correspondences, point cloud) is read back out of the same tensor, so anything that needs a scene description can be wired to one interface. The dependency order is fixed by that interface, not by taste.

Once geometry is a cheap by-product, the interesting question moves to what the world should be made of, and that is where "world model" becomes ambiguous. The survey that classifies native 3D and 4D world modelling draws four explicit boundaries and one decisive line: 3D/4D generation synthesises spatial or spatiotemporal signals and is judged on fidelity and diversity without necessarily modelling dynamics; 3D/4D world modelling additionally predicts how a scene evolves from past observations and actions, and that prediction "need not be carried out in observation space"; scene simulation is the closed-loop interactive rollout; and a world model in the broad sense is an internal, action-conditioned predictive model of environment dynamics. Hence the sharpest statement in the survey: a generator that merely synthesises pixels or voxels is not by itself a world model, because observation-space reconstruction is sufficient but not necessaryWorldModel3D. A 3D foundation model supplies geometry; a world model supplies dynamics. Collapsing the two would also collapse the Model and Interact steps into each other.

Caption: one frame holding the whole representation question. What the world is made of (the three tracks), what the model does (four functional roles), what it is told (geometric, action and semantic conditioning), and how it is judged (the evaluation staircase). It is a structure figure with no metrics, no paper counts and no rankings.WorldModel3D

The survey's representation layer names three generation tracks — rasterised video (VideoGen), occupancy grids (OccGen) and LiDAR point clouds (LiDARGen) — and lists neural fields (NeRF / Gaussian splatting) as a fourth representation family alongside them. Each buys something specific. Occupancy encodes visibility and motion in coordinates where physics acts, at a cubic memory and compute cost that forces compression tricks. LiDAR point clouds are metrically exact but sparse and anisotropically sampled, which is why ray-drop modelling is a recurring theme. Video is densest in appearance and weakest in metric structure. The functional axis cuts across all three: a model is a Data Engine, an Action Interpreter, a Neural Simulator or a Scene Reconstructor. The reading hazard is that the tracks rename those roles, so VideoGen lists data engines, action interpreters and neural simulators, OccGen lists scene representors, occupancy forecasters and autoregressive simulators, and LiDARGen lists data engines, action forecasters and autoregressive simulators. Asking "is this a simulator or a data engine?" without naming the track is often a malformed question.

Caption: the same functional axis, renamed per track. The corner chart is the survey's own curated reading list cut at 2025 up to August, and only the 2024 and 2025 bars carry printed values, so it should not be read as a census of the field.WorldModel3D

This is where the deck's own question belongs. Points, Gaussians, meshes or latents, one or several? In the deck's mapping, RAF works on Gaussians, Sculpt4D generates 4D meshes, and SCoPE generates in latents. The point of the question is that representation is a route decision. What the world is made of decides who can be rendered, who can be simulated and who can be edited, which is why physics and editing attach after the representation is chosen rather than before. Choosing a representation is like choosing a game's save-file format: physics, rendering and editing are all readers of that file, and the format decides which readers can exist at all. Whatever form the loop's Interact step takes has to be built against one of those readers.

4. Text about a scene has to bind every phrase to pixels, and a complete caption still cannot carry order

The Understand step is not classification with a longer output. Its current specification is panoptic grounded captioning: describe both the discrete objects and the background regions of an image, and bind every referring phrase in the caption to a set of pixel masks. The set-valued part matters, because a phrase may refer to zero, one or many regions, which makes grounding a selection problem rather than a single-mask prediction. Getting there by stacking dense captioning on top of pixel grounding fails in two directions at once: "Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks"PANORAMA.

The two errors do not cancel, and the measured split is clean. Generalist VLMs caption well and mask badly (Gemini 2.5 Pro reaches a caption score of 64.7 with a grounding quality of 22.7; Qwen3-VL-235B scores 64.0 and 26.1). Specialised grounding models mask adequately and describe little (GLaMM-7B captions at 40.7 with a grounding recall of 17.0). Broad entity coverage does not translate into precise region delineation, and no single existing system is competitive on both halves.

Caption: grounding is posed as selection rather than generation, and that is what stops the two errors from being the same error. The language model decides which entity a phrase means; the segmenter supplies where its boundary is; the scorer links the two, and a phrase may end up with zero, one or several masks. The coloured overlays are the model's predicted masks, not ground truth.PANORAMA

The mechanism is worth one sentence of detail because it explains the decoupling: one [SEG] token per phrase is projected by a learned concept bridge into a concept vector, the vector conditions a pretrained promptable segmenter into a phrase-specific pool of candidate masks, and a lightweight scorer keeps every proposal above a fixed threshold independently. Replacing that selection with direct mask generation costs 7.5 points of grounding quality at matched training budget while leaving captioning almost intact, which is the ablation that shows the two capabilities were never the same capability. The scope of the claim is narrow in a way that matters for this map: this is 2D, image-level grounding. It is one rung of a ladder that continues to structured text for 3D scenes, and the survey discussed earlier finds the same gap from the measurement side, in its result that current benchmarks underrepresent metric and object-centric reasoning.

The deck's alignment question then becomes concrete, and it is not about coverage. Its phrasing is that we see the lightning before we hear the thunder, and that PanCap, HoloCap and JoVA all pair a modality with another while none of them carries timing or absence. Turned into a measurable proposition, that says the pairing paradigm has an information ceiling: a caption can be complete, correctly paired and still unable to express sequence.

Caption: the same frames in a different order, and a mean-pooled CLIP model cannot tell the two apart. The heat map values are similarity scores for this one illustrative pair, not benchmark results, and the figure shows the diagnostic rather than a training pipeline. The point is narrow: order-insensitivity under average pooling, not a claim that CLIP cannot handle video.ViTAL-X

The evidence floor for this is a diagnostic benchmark built from 11,000 held-out videos and 25,000 evaluation pairs across five temporal competencies, with a human ceiling of 85.1%. Against it, CLIP scores 44.7 and a 7B video LLM hovers near 56.0, both close to the 45.0 random baseline. Two independent causes are named: permutation-invariant frame pooling in the architecture, and static spatial shortcuts in the training data, which is why chronology is rarely the variable a model has to get right. The finding that makes this an open problem rather than an engineering detail is that scaling parameters does not close the gap, while a data-layer fix does: rewriting the video and the caption together so that time is the only thing that changed, which leaves the spatial shortcut useless, gets a 0.4B-parameter model past 6B-parameter ones on the temporal benchmarks and 67.9 on the diagnosticViTAL-X. The scope stays video-text; the conclusion should not be extended to every modality pair.

5. Knowing what you don't know is a ladder of dropped assumptions, and its deployment form has no labels at all

Open-world learning is not generic out-of-distribution detection. It is a ladder whose label-space assumption is relaxed one rung at a time, and the rung a method sits on determines what it is expected to produce. Closed-set classification assumes test classes equal training classes. Semi-supervised learning allows unlabelled data but assumes it comes from the same classes. Open-set recognition allows unknowns to appear but asks only that they be detected, with no classification among the detected items. Novel category discovery requires all unlabelled data to be new classes, with the count known in advance. Generalized category discovery drops two assumptions at once. Continual and concept-level settings drop the remaining ones: new classes keep arriving, and concepts can live inside a single image.

The ladder is defined by what each rung removes, which is why it is a better guide to the literature than a list of techniques. The rung that carries most of the weight is generalized category discovery, and its formalisation is worth stating in the protocol's own terms: a dataset splits into a labelled part and an unlabelled part, the unlabelled images "may come from labelled classes or from novel ones", and the model never sees their labels during trainingGCD. Two assumptions are gone at once: the unlabelled pool need not be homogeneous, and the number of unknown classes need not be known in advance.

Caption: the rung made visible. The unlabelled pool is neither all new nor all known, it is a mixture, and the class count is not given. The figure shows the setting and the three method steps only: it contains no accuracy, and it does not show the class-count estimation step, so the number of clusters drawn on the right is not the estimated count.GCD

This is also the rung where the deck's own contribution sits, and one attribution should be kept honest. The judgement that this protocol is still the one the field uses comes from the deck's summary of the area, not from the 2022 paper that defined it. The protocol's mechanism, meanwhile, is deliberately non-parametric: contrastively trained ViT features clustered by a constrained semi-supervised k-means, with labelled instances pinned to their ground-truth cluster and the cluster count estimated online by scoring clustering accuracy on the labelled subset only. The paper adds a caveat worth keeping: the granularity of a real-world dataset is not an intrinsic property of the images but a framework imposed by the labels, so the estimate depends on how the labelled set was built.

At deployment the same task loses its labels entirely, and that is the part that makes Correct a loop step rather than an evaluation afterthought. The current form of the setting puts one pretrained vision-language model on an unlabelled stream that simultaneously drifts, contains known classes that are supervised only by their names, and contains classes that have no text description at all, with no label read at any point. A model trained on New York street scenes, deployed on a snowy Alberta stream where a familiar category now looks different and a moose walks into frame, has to adapt, classify and discover at the same time.

Caption: four paradigms at one deployment moment. The gap is not accuracy: the three existing paradigms each break in a different way, which is why a new class has to be organised rather than merely discarded. This is a setting figure and shows no method internals, no prototypes and no performance numbers.TT-GCD

The text-only constraint is the design driver rather than a footnote. The text encoder is frozen, so the class names stay a stable semantic reference while the visual side moves; pseudo-labels are computed once at the pretrained parameters and then held fixed, so the adaptation cannot chase its own drifting outputs; and because image and text embeddings occupy separate regions, both known and novel classes are re-expressed as visual prototypes in the adapted image space, with new classes seeded geometrically instead of from text. The engineering payoff is a streaming profile with no backward pass after a one-time warm-up: roughly 60 to 76 milliseconds per batch of 128 at about 1.1 GB, against 919 to 1771 milliseconds and up to 18.1 GB for the backpropagating baselines. Two honest caveats belong next to that: the reported margins are measured against baselines the authors adapted to the new setting rather than against their own published numbers, and the absolute accuracy on fine-grained data is still low, so this is a task that is not close to solved.

One claim should be stated narrowly rather than broadly. The paper's own observation is that closed-set evaluation silently absorbs novel samples into known classes, "a failure that closed-set evaluation cannot see". That is a statement about the instrument's blindness, not a claim about dataset contaminationTT-GCD.

What the Correct step finally produces is the thing that closes the loop: a new concept, expressed as a prototype in a frozen encoder's space. That artefact is exactly what the Understand step needs in order to name something it could not name before, which is why the deck can say that what was new becomes known. The deck's own outlook puts the same requirement inside three settings: a home or warehouse robot meeting objects and layouts no dataset ever labelled, an always-on egocentric camera whose long tail is specific to one person and therefore has to be discovered continually and privately, and a world model that cannot flag novelty and will therefore simulate the unfamiliar as if it were familiar.

6. The pieces are in place: the five open problems all sit at the hand-offs between steps

The deck closes with five open problems aimed at a PhD student, and each one lands on a hand-off rather than inside a step. That is the reading worth keeping, because it converts a list of problems into a list of places to look.

Two of those rows deserve a sentence each. The open-world row is a policy gap, not a capability gap: a classifier can reject an input, but a world model will generate anyway, and nothing in the map says which behaviour is correct for a given deployment. The reliability row is where the benchmark-design result comes back, and it is deliberately not re-anchored on a newer source, because the finding that the benchmarks over-weight the easiest rung of the ladder at the time of measurement is the evidence, and the ceiling benchmarks the deck names (ZeroBench and GAMEBoT among them) tell you where a model breaks rather than whether it can be trusted.

The deck's stated target is a suite of open-world spatial-intelligence foundation models as core technology for embodied AI and autonomous driving. Read as engineering rather than aspiration, that is a commitment to a model that knows what it does not know, which pulls the Correct step out of the evaluation appendix and into the loop. The adjacent framework literature frames the same target as six grand challenges (a unified spatial representation, grounded long-horizon planning, safe deployment under uncertainty, sim-to-real transfer, scalable multi-agent coordination and efficient edge deployment) over the domains of robotics, autonomous vehicles and geospatial intelligenceSpatialAgents.

What the reader can carry away is a placement rule rather than a reading list. Reconstruction before generation before physics, because each fixes what the next can consume. Language and benchmarks alongside the geometry strand rather than on top of it, because they measure and describe the same world instead of producing it. Correction feeding back into understanding, because a new concept is exactly the kind of artefact the loop is missing at its first step. Any paper encountered later can be placed by asking two questions: which artefact does it consume, and which does it produce. One last honesty note on the material itself: the deck's summary slide groups methods under three names (GRT, iVGR and SPoT) that appear nowhere else in the deck and are never explained, so they stay as names here rather than being assigned meanings the material does not supply.

参考文献

1. Spatial intelligence in vision-language models: a comprehensive survey — https://link.springer.com/article/10.1007/s10462-026-11671-x
2. From Perception to Action: Spatial AI Agents and World Models — arXiv:2602.01644
3. DUSt3R: Geometric 3D Vision Made Easy — arXiv:2312.14132
4. 3D and 4D World Modeling: A Survey — arXiv:2509.07996
5. Generalized Category Discovery — arXiv:2201.02609
6. Test-Time Generalized Category Discovery (PACT) — arXiv:2609.33937
7. PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection — arXiv:2609.19143
8. ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits — arXiv:2609.00505