VideoLVR: Cracking the Visual Forgetting Problem in Video Understanding with Latent Visual Reasoning

Multimodal large models can already produce fluent reasoning chains for video understanding tasks, but a hidden concern persists: when the model writes out reasoning steps in natural language, it is actually gradually "forgetting" what is really happening in the frames. The paper VideoLVR: Latent Visual Reasoning over Grounded Video Evidence targets precisely this core contradiction—language reasoning traces cannot reliably make the model revisit visual evidence, leading to visual forgetting and consequently impairing fine-grained understanding capabilities.

The Dilemma of Existing Approaches: Symbolic Coordinates vs. Context Bloat

To keep the model's "attention" on the frames during reasoning, existing work has roughly taken two paths.

The first is the textual coordinate route: describing evidence locations in words, such as giving timestamps or region coordinates. The problem with this approach is that coordinates are merely symbolic position markers and do not carry the visual content itself—the model knows "where to look" but may not actually "see" anything.

The second is the re-insertion route: putting cropped regions or keyframes back into the multimodal context. Visual details are indeed preserved, but at the cost of significantly expanded context length and correspondingly higher inference cost. Each path has its trade-offs, yet neither simultaneously solves the problems of "seeing" and "seeing cheaply."

VideoLVR's Core Idea: Turning Local Evidence into Latent Visual States

VideoLVR proposes constructing latent visual evidence from local regions at relevant video moments, providing visual support for intermediate reasoning steps. Unlike textual coordinates, these latent states are encoded directly from frame content and preserve visual information; unlike re-inserting full frames or cropped images, they exist as compact latent representations that do not substantially bloat the context.

According to the paper, the model can thus switch between interleaved text tokens and latent visual states during reasoning, and integrate evidence from different moments. This is equivalent to equipping the reasoning chain with a "visual memory" that can be retrieved at any time, rather than making the model imagine the frames from textual descriptions.

Latent space and latent representations: In multimodal models, a visual encoder (such as ViT) maps image patches or video frames into a set of continuous vectors, and the dimensional space these vectors inhabit is called "latent space." Unlike human-readable text tokens, latent vectors do not directly correspond to any word or symbol; instead, they encode visual attributes such as color, texture, shape, and motion in a distributed manner. "Latent visual evidence" refers to a segment of local visual features preserved in such continuous representation space—it can be read by subsequent Transformer layers like tokens, yet does not need to be expanded into natural language descriptions. Precisely because it bypasses the "vision→text" translation step, latent representations avoid information loss during description, while their length is far smaller than re-inserting raw frames or cropped patches, striking a compromise between "fidelity" and "context economy." Understanding this is key to understanding why VideoLVR outperforms both the textual coordinate and re-insertion routes.

Unified Token Prediction Framework: Visual Reconstruction and Text Generation Share One Objective

A key design is the visual token prediction objective. VideoLVR formalizes visual token reconstruction as "prediction over a vocabulary derived from the input video," thereby unifying visual evidence reconstruction and text generation into the same token prediction framework.

This unification brings two benefits. First, supervision signals are naturally aligned: generating latent visual evidence is no longer a task separate from language modeling but shares the same mechanism with text prediction. Second, structural simplicity: the model does not need a separate decoding path for visual evidence, and interleaved reasoning can be completed within a single framework.

Visual tokens and derived vocabularies: To bring visual evidence into a language model's token prediction framework, continuous visual features usually must first be "discretized." A common approach uses vector quantizers such as VQ-VAE or VQ-GAN, maintaining a codebook that maps each visual feature vector to its nearest entry, yielding discrete visual tokens. The "vocabulary derived from the input video" mentioned by VideoLVR refers to a set of visual tokens constructed or selected for the current video, over which the model makes classification predictions in exactly the same form as predicting the next text word. In this way, the cross-entropy loss can act simultaneously on text tokens and visual tokens; visual reconstruction is no longer an independent auxiliary task but shares the same output head and training objective as language modeling—this is also the basis for the engineering simplicity of the "unified token prediction framework."

Extension to Reinforcement Learning: Optimizing Visual Tokens as Actions

The paper further extends VideoLVR to reinforcement learning scenarios: sampled visual tokens, like text tokens, are treated as actions, and the policy is jointly optimized using the reward of the final answer. This means the model is not merely "taught" to generate useful visual evidence; it also adjusts its decisions about "what to look at and when" based on task success or failure. For video tasks requiring multi-step evidence localization, this end-to-end reward-driven optimization is closer to the actual usage objective than pure supervised learning.

Reinforcement learning paradigm treating generation as actions: In sequence generation tasks, treating each generated token as an action and the complete output as a trajectory is a common practice for combining reinforcement learning with large models (e.g., policy optimization in RLHF). VideoLVR's distinctive feature is that the action space includes both text tokens and visual tokens, and the policy network must learn "when to insert visual evidence, from which moment to take it, and which region to take." Since the "quality" of visual evidence is hard to characterize with per-token supervision labels, using final-answer correctness as the reward signal for policy gradient optimization (e.g., REINFORCE-style algorithms) becomes a natural choice: the model discovers through trial and error which visual evidence-gathering behaviors genuinely help answer questions correctly, rather than merely imitating annotated trajectories in the training data.

Results and Significance: Greater Advantages on Fine-Grained Tasks

The paper validates the method on multiple video benchmarks and specifically notes that gains are more pronounced on tasks requiring fine-grained visual evidence. This is consistent with the design intuition—when reasoning steps need to repeatedly verify frame details, latent visual evidence is most valuable; for tasks that mainly rely on semantic summarization, the room for improvement is relatively limited.

From a broader perspective, VideoLVR touches on a fundamental question in multimodal reasoning: must the intermediate states of reasoning be natural language? When the evidence itself is visual, letting it participate in reasoning in visual form may be more aligned with the task's essence than forcibly translating it into text. Whether this idea can extend to longer videos and more complex multi-hop reasoning remains a direction worth watching.

Frequently Asked Questions

How is VideoLVR different from directly inserting video frames? Directly inserting frames significantly expands the context, whereas VideoLVR constructs compact latent visual evidence, preserving details while controlling context overhead.

How is latent visual evidence supervised? Through the visual token prediction objective, visual token reconstruction is formulated as prediction over a vocabulary derived from the input video, sharing the same framework as text generation.

Does it support reinforcement learning training? Yes. The paper treats sampled visual tokens as actions and jointly optimizes the policy with text tokens using the final answer reward.

One-line Summary

VideoLVR replaces textual coordinates and frame re-insertion with compact latent visual evidence, mitigating visual forgetting in video reasoning.

Paragraph Summary

The article introduces VideoLVR, a video reasoning method addressing the visual forgetting problem where multimodal large models gradually "forget" the frames when writing reasoning chains in natural language. Existing approaches each have flaws: textual coordinates provide only position symbols without visual content; re-inserting cropped images or keyframes preserves details but causes context bloat and higher inference cost. VideoLVR constructs compact latent visual evidence from local regions at relevant video moments, letting the model reason interleaved between text tokens and latent visual states and integrate evidence from different moments. The method formulates visual token reconstruction as prediction over a vocabulary derived from the input video, unifying it with text generation in the same token prediction framework, and can extend to reinforcement learning by treating visual tokens as actions and jointly optimizing with the final answer reward. Experiments show effectiveness on multiple video benchmarks, with particularly notable gains on fine-grained visual evidence tasks.