ROSE: Risk-Aware Orthogonal Subspace Navigation for Lifelong Knowledge Editing in Multimodal LLMs

Multimodal large language models (MLLMs) need to continuously update factual knowledge to stay current, but lifelong knowledge editing remains a tough challenge. The core obstacles are severe Locality Erosion and Catastrophic Forgetting: sequential edits inevitably interfere with pretrained capabilities and prior updates. To address this, researchers propose ROSE—a Risk-aware Orthogonal Subspace Navigation framework for Lifelong Knowledge Editing.

Two Major Pain Points of Lifelong Knowledge Editing

In real deployments, MLLMs need to continuously absorb new facts, such as correcting outdated information or adding newly emerging entities. Ideally, each edit should only change parameters related to the target fact without affecting the model's behavior on other inputs. However, existing methods often exhibit a "fix one thing, break another" phenomenon during sequential editing: after editing new knowledge, old knowledge is accidentally overwritten, or the model's original general capabilities degrade. This is the combined effect of locality erosion and catastrophic forgetting.

ROSE's starting point is clear: constrain parameter updates to the orthogonal complement of previously edited subspaces. From a geometric perspective, the orthogonal complement means new update directions are "perpendicular" to old update directions, thereby maximally avoiding mutual interference in parameter space. This idea provides a structural constraint to mitigate forgetting.

Catastrophic Forgetting stems from the classic problem of neural networks learning multiple tasks sequentially: when the model updates parameters for a new task (here, editing new facts), gradient updates overwrite or perturb weights critical to old tasks, causing a sharp drop in old-task performance. In lifelong knowledge editing, each fact update can be viewed as a new task, and edits tend to concentrate in local regions of parameter space related to the target fact. Without constraints, subsequent edits continuously erode the parameter configurations that prior edits rely on, causing loss of old knowledge. ROSE mitigates this cumulative degradation through orthogonal subspace constraints and a risk mask, explicitly protecting prior edit directions and sensitive regions during parameter updates.

Locality Erosion refers to the phenomenon where editing operations produce unintended effects on behavior beyond the target fact. Ideal knowledge editing should satisfy locality: only change outputs related to the target fact without affecting the model's behavior on other inputs. However, MLLM parameters are highly coupled, and the gradient of a single edit may diffuse into shared representations, causing unrelated facts to be accidentally modified or general capabilities to degrade. Sequential editing amplifies this erosion: the side effects of each edit accumulate in subsequent updates, gradually shifting the model away from its original behavior. ROSE's orthogonal complement constraint aims to restrict updates to a subspace perpendicular to prior edit directions, geometrically reducing interference with unrelated parameters and thus protecting locality.

ROSE's Core Mechanism: Orthogonal Subspace and Risk Mask

ROSE's framework includes two key designs. The first is orthogonal subspace navigation, which protects edited knowledge by constraining the direction of parameter updates. The second is a risk-guided mechanism that dynamically computes a risk mask to protect specific fact-parameter subspaces critical to model stability.

Relying solely on orthogonal constraints may be too rigid, because some parameter subspaces are extremely sensitive to the model's overall stability; even if the update direction is orthogonal, numerical perturbations may still affect model behavior. The role of the risk mask is to identify these sensitive regions and protect them, striking a balance between "allowing edits" and "maintaining stability." This risk-aware design makes ROSE more robust in lifelong editing scenarios.

Orthogonal Complement is a fundamental concept in linear algebra. Given a subspace ( \mathcal{S} ) in parameter space (in this scenario spanned by the update directions of prior edits), its orthogonal complement ( \mathcal{S}^\perp ) consists of all vectors orthogonal to every vector in ( \mathcal{S} ). Projecting a new parameter update onto ( \mathcal{S}^\perp ) means the update direction is perpendicular to all old update directions, so in the Euclidean geometric sense it does not change the components along old update directions. This constraint effectively prevents new edits from overwriting the parameter directions that old edits rely on, and is a common technique for mitigating forgetting in continual learning and knowledge editing. ROSE leverages this property to restrict each edit's gradient to the orthogonal complement of prior edit subspaces, achieving structural interference isolation.

Multi-Granularity Knowledge Integration at Inference

How edited knowledge is precisely activated at inference is equally critical. ROSE adopts a multi-granularity knowledge integration strategy comprising two components: prototype-anchored semantic gating and perturbation-driven subspace fusion.

Semantic gating determines whether the current query is relevant to edited knowledge, while prototype anchoring provides a stable semantic reference point to prevent the gate from misfiring on out-of-distribution inputs. Subspace fusion, driven by perturbations, injects edited knowledge more precisely into the inference process. The shared goal of both is to activate edited knowledge for relevant queries while limiting interference for irrelevant inputs. This directly echoes the core requirement of lifelong editing: "precise modification, broad preservation."

Semantic Gating is a mechanism that dynamically decides at inference time whether to activate a specific knowledge path. Its core idea: the model maintains an internal gating function that, based on the match between the current query's semantic representation and edited knowledge, outputs a weight (typically between 0 and 1) controlling the edited knowledge's contribution to the final output. In lifelong knowledge editing, the gating challenge lies in distinguishing "relevant queries" from "irrelevant inputs": if the gate is too permissive, irrelevant inputs will mistakenly trigger edited knowledge and cause interference; if too strict, relevant queries cannot activate updates. ROSE employs prototype-anchored semantic gating, maintaining a stable prototype representation as a reference point for each piece of edited knowledge to reduce the risk of misfiring on out-of-distribution inputs, thereby balancing precise activation and broad preservation.

Experimental Performance and Stability

The researchers conducted comprehensive experiments on two mainstream multimodal benchmarks. Results show that ROSE is competitive with state-of-the-art methods on five key metrics while maintaining excellent stability after more than 1,000 sequential updates. This stability data is particularly critical—the difficulty of lifelong editing lies not in the effect of a single edit but in cumulative degradation after long sequential editing. ROSE's ability to remain stable after over a thousand updates demonstrates that the combination of orthogonal constraints and risk masking is indeed effective in suppressing forgetting.

The code is open-sourced at https://github.com/sunlitsong/ROSE, providing a reproducible foundation for future research.

Implications for Multimodal Knowledge Editing

ROSE's value lies not only in proposing a new framework but also in explicitly introducing "risk" into the decision process of knowledge editing. Prior methods mostly focused on how to locate and modify knowledge, whereas ROSE reminds us that which parameter regions "must not be touched" is equally important. Orthogonal subspace navigation offers a scalable geometric constraint approach, and the risk mask provides dynamic protection for sensitive parameters. For multimodal applications requiring continuous updates, this design—balancing editing capability and model stability—points to a more sustainable technical path.

One-line Summary

ROSE uses orthogonal subspace navigation and risk masking to enable lifelong knowledge editing in multimodal LLMs, remaining stable after a thousand updates.

Paragraph Summary

Multimodal LLMs face two major pain points in lifelong knowledge editing: locality erosion and catastrophic forgetting, as sequential edits interfere with pretrained capabilities and prior updates. The ROSE framework proposes orthogonal subspace navigation, constraining parameter updates to the orthogonal complement of prior edit subspaces to geometrically avoid interference; it also introduces a risk-guided mechanism that dynamically computes a risk mask to protect parameter subspaces critical to model stability. At inference, prototype-anchored semantic gating and perturbation-driven subspace fusion precisely activate edited knowledge while limiting interference from irrelevant inputs. On five metrics across two mainstream multimodal benchmarks, ROSE is competitive with state-of-the-art methods and remains stable after more than 1,000 sequential updates. The code is open-sourced.