PSGAgent: Breaking Through MLLM Dynamic Visual Perception Bottlenecks with Agentic Reasoning

Panoptic Scene Graph Generation (PSG) requires models to perform holistic and fine-grained reasoning about complex relationships between objects. However, current Multimodal Large Language Models (MLLMs) applied to PSG suffer from a perception bottleneck: they act as static observers, limited to fixed-resolution views. As a result, models often overlook key relationships involving small or peripheral objects and struggle to distinguish ambiguous predicates in complex scenes. To address these limitations, researchers propose PSGAgent, an agentic framework that enhances PSG visual perception through a "Think-Act-Observe" loop.

The Perception Bottleneck of Static Observers

The fundamental problem with existing MLLMs on PSG tasks lies in their "static observer" positioning. The visual input the model receives is a fixed-resolution full image, and it cannot actively adjust its perceptual focus based on reasoning needs. In complex scenes, relationships involving small objects or objects at the image periphery often occupy very few pixels, and this relational information is easily lost under a fixed view. Meanwhile, when multiple similar objects or relationships exist in a scene, the static view lacks targeted focus, making it difficult for the model to distinguish semantically ambiguous predicates—for example, "holding" versus "touching," or "standing next to" versus "belonging to." This bottleneck is not a deficiency in the model's reasoning ability, but rather a lack of dynamic adjustment mechanisms at the perceptual level.

Panoptic Scene Graph Generation (PSG) is an extended variant of the Scene Graph Generation (SGG) task, aiming to model all objects in an image and the pairwise relationships between them as a graph structure. Unlike traditional SGG, which relies on object detection boxes, PSG uses panoptic segmentation masks as object representations, covering both "things" (countable objects) and "stuff" (background regions such as sky and road), thus more closely reflecting the completeness of real scenes. The output of PSG is a set of triples (subject, predicate, object), with predicates encompassing rich semantics such as actions, spatial relations, and interactions. Evaluation typically employs Recall@K (e.g., R@20, R@50) as well as mR@K metrics for predicate categories, measuring the model's ability to recall fine-grained relationships. Precisely because PSG requires relationship reasoning over all object pairs in the entire image, the pixel proportion of small and peripheral objects under fixed-resolution input is extremely low—making the perception bottleneck particularly pronounced.

Think-Act-Observe: The Agentic Perception Loop

The core innovation of PSGAgent lies in transforming the MLLM from a static observer into an agent capable of active perception. The framework uses a "Think-Act-Observe" loop that allows the model to dynamically refine its visual perception during reasoning. Specifically, the model first thinks based on its current understanding, determining which visual regions or relationships need further exploration; it then calls tools to execute actions; finally, it observes the visual feedback from those actions and updates its reasoning. This loop enables the model to actively "lean in for a closer look" or "focus its gaze" when uncertain—much like a human—rather than passively accepting a one-shot visual input.

Region Zoom-In Tool: Local Exploration

The Region Zoom-In Tool is designed for local exploration, enabling the model to zoom in on small objects or boundary relationships. When the model recognizes during reasoning that a key relationship involves a small-scale object or lies at the image periphery, it can proactively invoke this tool to crop the corresponding region and magnify it, obtaining a higher-resolution local view. This capability directly alleviates the problem of small-object relationships being overlooked under fixed resolution, allowing the model to access detail information it previously could not discern.

Visual Highlight Tool: Relationship Focus

The Visual Highlight Tool focuses on relationship-centric attention. Through relation-oriented segmentation, it filters out irrelevant contextual noise, enabling the model to concentrate on foreground objects and relationships relevant to the current reasoning. In complex scenes, numerous background objects and distracting information dilute the model's perception of key relationships; the highlight tool helps the model more clearly distinguish ambiguous predicates by explicitly suppressing irrelevant regions. The two tools are complementary: one magnifies details, the other eliminates distractions—together supporting dynamic perception.

Two-Stage Training: From Cold Start to Policy Optimization

To teach the model when and how to invoke tools, the researchers designed a two-stage training pipeline. The first stage constructs a high-quality, error-aware Multimodal Chain-of-Thought (M-CoT) dataset, providing robust supervision signals for cold-start learning. This dataset contains not only correct reasoning paths but also error-awareness information, helping the model understand common perception error patterns. The second stage designs a novel hierarchical reward mechanism combined with Group Sequence Policy Optimization (GSPO), enabling the model to learn strategic and adaptive tool-use policies. The hierarchical reward mechanism guides the model to select the right tool at the right time, rather than invoking tools blindly.

Group Sequence Policy Optimization (GSPO) is a method for policy optimization of large language models in reinforcement learning, which can be regarded as an improvement over GRPO (Group Relative Policy Optimization) in terms of sequence-level importance sampling. Traditional PPO-style methods compute importance ratios at the token level, which leads to high variance and unstable training for long sequences; GSPO instead computes importance weights at the entire-sequence level and normalizes by sequence length, thereby reducing variance and improving training stability for long-sequence generation. In this work, GSPO is used to optimize the agent's tool-calling policy: the model generates complete trajectories containing tool calls within the "Think-Act-Observe" loop, and GSPO performs policy updates on entire trajectories based on hierarchical rewards, enabling the model to gradually learn under which reasoning states to invoke the region zoom-in or visual highlight tools.

Multimodal Chain-of-Thought (M-CoT) refers to extending chain-of-thought reasoning to multimodal input scenarios, where the model generates explicit intermediate reasoning steps before producing a final answer, and these steps can reference visual evidence. In agentic frameworks, M-CoT typically manifests as alternating sequences of textual reasoning and tool calls/observations. The "error-aware" M-CoT dataset constructed in this paper annotates common perception error patterns (such as missing small-object relationships or misjudging ambiguous predicates) in addition to standard correct reasoning paths, so that supervision signals in the cold-start stage not only tell the model "how to do it right" but also "how it tends to go wrong"—providing a more robust initialization policy for the subsequent reinforcement learning stage.

Experimental Results and Significance

Extensive experiments on the PSG benchmark show that PSGAgent achieves SOTA performance, demonstrating strong capabilities in both holistic and fine-grained scene understanding. This result validates the effectiveness of agentic reasoning in visual perception tasks: by endowing MLLMs with dynamic perception capabilities, the model can break through the constraints of fixed views and capture relational information in scenes more comprehensively. The significance of PSGAgent extends beyond the PSG task itself—it offers a transferable paradigm for multimodal large models in scenarios requiring fine-grained visual reasoning: teaching models to perceive actively rather than receive passively.

Conclusion

PSGAgent transforms the static visual perception of MLLMs into a dynamic, active perception process through an agentic framework. The Region Zoom-In Tool and Visual Highlight Tool respectively address the overlooking of small-object relationships and contextual noise interference, while the two-stage training ensures strategic tool use. SOTA performance on the PSG benchmark proves the effectiveness of this approach and points to a new direction for perception enhancement of multimodal large models in complex visual reasoning tasks.