How Good Are Vision Foundation Models at Representing Spatial State? A Four-Level Probe Study Reveals Key Shortcomings

Research Background: From "Seeing" to "Continuous Perception"

An observer can only ever see part of the surrounding world, yet must stay aware of what lies beyond the field of view at all times. This means spatial perception goes far beyond recognizing a single image—it requires a continuously updated spatial state that is constantly corrected during self-motion and used to guide action.

The core question the research team posed is: do vision foundation models provide representations that support this kind of state? To answer it, they designed four progressive levels of spatial capability: spatial structure, object permanence, predictive spatial updating, and actionability. Using frozen feature probes, they evaluated ten representations under streaming egocentric observations of static indoor environments.

The advantage of this "probing" approach is that it does not rely on fine-tuning for a specific downstream task; instead, it directly tests whether decodable spatial information is contained in the representation itself.

Frozen feature probing is a standard paradigm in representation evaluation: the weights of a pretrained model are held completely fixed, and only a lightweight linear classifier or regressor is trained on top of the features output by a given layer, in order to measure how much "linearly decodable" information those features contain. Unlike fine-tuning, probing does not change the backbone's parameters, so it more purely reflects what information the representation itself carries, rather than what the model has "learned" by adapting to a new task. The limitation of this method is also clear—a probe can only show that information is present and readable; it cannot prove that the model actually uses that information in downstream reasoning. The "prefix length" mentioned in the paper refers to the number of historical frames or tokens the probe can access in streaming observation, which determines the time window over which the model can integrate information.

Four-Level Capability Probes: Design Logic and Key Findings

Spatial Structure: Scene Decodability

The first level examines the model's ability to encode the spatial structure of the visible scene. The results show that capability profiles differ markedly across network layers and prefix lengths. Some layers read out the visible scene very strongly, but this is only the starting point.

Object Permanence: Can It Remember What It Can't See?

The second level requires the model to keep tracking an object's position after it leaves the field of view. This is the core of spatial intelligence—humans do it effortlessly, but how do models perform? The study finds that strong readout of the visible scene does not guarantee accurate object permanence.

Predictive Spatial Updating: Can Future Features Be Retrieved?

The third level tests the model's ability to predict future spatial features under self-motion. Native future prediction preserves recoverable spatial structure, but a key finding is that a spatial reader trained on real features cannot be transferred as-is to predicted features. This means there are systematic differences between predicted and real features.

Actionability: Goal-Directed Action Selection

The fourth level assesses whether spatial representations can support continuous action decisions. The experiments reveal a sobering result: goal-dependent candidate selection does not establish reliable continuous action utility. Predicted actions respond to changes in the goal, but providing the correct goal does not reliably improve progress.

Candidate selection and continuous action utility are the key distinction for understanding the fourth-level probe. Candidate selection means the model picks, from a set of discrete candidate actions, the one that "looks most consistent with the goal"; such tasks usually require only scoring individual candidates, and the evaluation metric is selection accuracy. Continuous action utility, by contrast, requires the model to produce, for every continuous value in the action space, a utility estimate consistent with true progress, thereby supporting step-by-step decisions in closed-loop control. The former can be achieved through goal-conditioned similarity matching, while the latter requires the representation to implicitly encode the dynamics of "how the state would change if this action were taken." Therefore, even if a model performs well at candidate selection, it may still lack the ability to predict the consequences of actions, and thus fail to make stable progress on tasks that require continuous adjustment—this is precisely the mechanistic explanation for the paper's claim that "providing the correct goal does not reliably improve progress."

Dependencies Between Capabilities: Who Supports Whom

Using matched controls, the study separated readout precision from dependence on visual history and motion. An important structural relationship emerged: object permanence and actionability build on camera pose readout, while feature prediction builds on visible-scene readout.

This explains why some models excel at scene recognition tasks yet perform only moderately on tasks requiring continuous tracking or action planning—they rely on different underlying capabilities.

Camera pose readout refers to the ability to regress the camera's own position and orientation from model features. In egocentric streaming observation, pose is the geometric link between "past frames" and "the current frame": only by knowing how the camera has moved can the position of an object that has left the field of view be transformed from the old coordinate frame into the current one, and only then can action decisions judge what viewpoint change a given action would produce. Thus, the dependence of object permanence and actionability on pose readout is essentially a dependence on "egomotion modeling." Feature prediction, by contrast, relies more on semantic and structural readout of the visible scene, because it needs to extrapolate future visual content from the current observation. This separation of dependency structures shows that spatial intelligence is not a single capability, but is jointly supported by two relatively independent pathways: geometric self-localization and scene-content understanding.

Implications for Evaluating Vision Foundation Models

This study sends a clear signal: when evaluating spatial representations, we should not only ask "what can be decoded from the representation," but also whether that information is maintained beyond observation, whether it can be predictively updated, and whether it is actually used for goal-directed action.

Many current vision foundation models perform impressively on static image tasks, but spatial intelligence demands dynamic, continuous, actionable representations. The four-level probe framework offers the community a more comprehensive evaluation perspective and points to the gaps that future model design needs to close.

For applications such as robotics, autonomous driving, and embodied intelligence, this finding is especially important: scene understanding is only the first step; a persistent and actionable spatial state is what really matters.

One-line Summary

Four-level probes reveal that strong scene recognition in vision foundation models does not equal reliable spatial state maintenance.

Paragraph Summary

Using four progressive levels of probes—spatial structure, object permanence, predictive spatial updating, and actionability—the study systematically evaluated ten vision foundation model representations under streaming egocentric observations. The core findings are: strong readout of the visible scene does not guarantee accurate object permanence, nor does it reliably transfer to predicted features; and although predicted actions are sensitive to goal changes, providing the correct goal does not reliably improve continuous action progress. The study further reveals the dependency structure among capabilities—object permanence and actionability build on camera pose readout, while feature prediction relies on visible-scene readout. This shows that scene recognition and spatial state maintenance depend on different underlying capabilities, and that evaluating spatial intelligence must go beyond static decodability to ask whether information is continuously maintained and used for goal-directed action.