Safety Vulnerabilities in Vision-Language Models: Compositional Harm and Un-alignment Auditing

Large Vision-Language Models (LVLMs) face the threat of compositional harm in cross-modal reasoning: an image and a prompt are each harmless on their own, yet when the model combines the two it can produce harmful output. Recent alignment methods report high scores on compositional safety benchmarks, but LLM judges only measure observed behavior and cannot reveal whether safety mechanisms are robustly integrated into the model. A recent paper proposes that un-alignment provides a complementary auditing tool for alignment robustness. The study first shows that minimal supervised fine-tuning can match the LLM-judge scores of multi-stage reinforcement learning pipelines, so similar benchmark scores alone cannot prove robustness. Next, the research reveals that the identification and refusal of compositional harm are decoupled: nine standard direction-discovery methods detect compositional harm with AUC up to 0.99, yet ablating these directions either does not affect refusal behavior or destroys model coherence. To address this, the authors propose CMaG—a training-free un-alignment method. CMaG measures refusal intent via the log-probability margin between refusal and compliance, takes gradients with respect to intermediate-layer activations, and contrasts with partially benign pairs to isolate the cross-modal interaction term. A single backward pass over 80 samples yields a steering direction, and steering along it monotonically converts refusal into compliance. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional safety scores by up to 65 points while over 98% of outputs remain coherent, outperforming gradient-based baselines; the direction also transfers to explicit text harm and can un-align models hardened by defense training. This shows that compositional safety may appear strong behaviorally yet remains fragile to targeted internal interventions.

Compositional Harm: A Blind Spot in Alignment Evaluation

Compositional harm refers to images and prompts that are each harmless but, when combined through the model's cross-modal reasoning, produce harmful content. Such harm is difficult to capture with traditional unimodal safety benchmarks because the harm arises from interaction between modalities rather than from a single input. Existing alignment methods score highly on compositional safety benchmarks, but vulnerability may lurk behind high scores: LLM judges only check output behavior and cannot determine whether safety mechanisms are internalized. The paper points out that minimal supervised fine-tuning can achieve judge scores close to those of complex reinforcement learning pipelines, showing that benchmark scores are not equivalent to robustness. This finding challenges current safety evaluation practices that are oriented toward benchmark scores.

Compositional safety benchmarks usually refer to specially constructed evaluation sets in which images and text prompts are each harmless on their own, but requiring the model to perform joint cross-modal reasoning may elicit harmful content. Scoring for such benchmarks is generally done by an LLM judge (such as a GPT-4-class model) that classifies or scores whether the output is safe, then aggregates into a safety score. Because the judge only observes the final text, it cannot distinguish between "the model has truly internalized a refusal policy" and "the model has happened to learn on the training distribution to avoid the trigger patterns of this benchmark." Therefore, a high score may reflect surface-level fitting to the benchmark rather than a deep safety mechanism.

Minimal supervised fine-tuning refers to standard supervised fine-tuning of the model with only a small number (tens to hundreds) of safety-alignment samples, without introducing preference optimization or reinforcement learning stages. The paper uses it as a control to show that even a training pipeline far simpler than a multi-stage RLHF pipeline can achieve similar scores under an LLM judge, thereby demonstrating that benchmark scores are insensitive to the training method and cannot serve as sufficient evidence of robustness.

The Decoupling of Identification and Refusal

The research further reveals that the model's identification and refusal behaviors toward compositional harm are separate. Nine standard direction-discovery methods can detect the compositional harm direction with accuracy as high as AUC=0.99, but after ablating these directions, the model's refusal behavior either remains unchanged or the model's coherence is destroyed. This means that a direction representing harm exists inside the model, but eliminating that direction does not directly translate into unsafe output—the safety mechanism may be distributed across multiple circuits, or refusal behavior may be maintained by other factors. This decoupling phenomenon indicates that relying solely on direction discovery for safety auditing may underestimate the model's vulnerability.

Direction-discovery methods (such as linear probes, mean difference, principal component analysis, contrastive activation addition, etc.) assume that a low-dimensional subspace inside the model encodes a certain concept (here, compositional harm), and locate it by finding a linear direction in activation space that can distinguish positive from negative samples. AUC measures the ranking ability of that direction as a classifier; AUC=0.99 means near-perfect separation of activations for harmful versus harmless samples.

Ablation refers to zeroing out or projecting away the component of activations along that direction, then observing changes in model behavior. If the direction is a necessary carrier of the safety mechanism, ablation should cause a significant drop in refusal rate; if refusal is unchanged after ablation, the direction is only a correlated representation rather than a causal bottleneck; if model coherence collapses after ablation, the direction also carries general information needed for language generation and cannot be cleanly used for safety auditing. This "high detectability but low causality" is the core of the decoupling phenomenon.

CMaG: A Training-Free Un-alignment Auditing Method

To test alignment robustness more directly, the authors propose CMaG. The method is training-free: it quantifies refusal intent by computing the log-probability margin between refusal and compliance, and takes gradients with respect to intermediate-layer activations. To isolate the cross-modal interaction term, CMaG contrasts the target sample with partially benign pairs. A single backward pass over only 80 samples yields a steering direction, and steering the model along this direction monotonically converts refusal into compliance. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional safety scores by up to 65 points while over 98% of outputs remain coherent, outperforming gradient-based baseline methods. The direction also transfers to explicit text harm scenarios and can un-align models hardened by defense training.

Un-alignment refers to applying interventions directly to the model's internal activations at inference time, without retraining, to make it deviate from its aligned safety behavior. Unlike adversarial attacks, un-alignment does not optimize the input prompt but instead seeks a reusable internal steering direction, making it an auditing tool at the mechanistic level.

Log-probability margin refers to computing, for the same input, the sequence log-probabilities of the model generating a refusal template (e.g., "I cannot assist") versus a compliance template (e.g., "Sure, here is"), with the difference between the two being a continuous measure of refusal intent. Compared with discrete sampled outputs, this margin is more sensitive to the model's internal state. Taking gradients with respect to intermediate-layer activations means differentiating this margin with respect to a certain intermediate-layer activation to obtain the direction along which refusal probability decreases fastest; contrasting with partially benign pairs subtracts the gradient component caused purely by text content, thereby isolating the cross-modal interaction term. A single backward pass suffices to estimate the direction, making the method training-free with low computational overhead.

A Paradigm Shift in Safety Evaluation

This study suggests that compositional safety may appear strong at the behavioral level yet is extremely fragile to targeted internal interventions. Traditional benchmarks and LLM judges are insufficient to fully evaluate alignment robustness, and un-alignment auditing needs to be introduced as a complement. CMaG provides an efficient tool that can reveal the intrinsic fragility of model safety mechanisms and points the way for future safety alignment research: not only pursue behavioral safety, but also ensure that safety mechanisms are difficult to manipulate simply at the level of internal representations. The paper contains examples of unsafe model outputs, including self-harm references, reminding researchers to take safety precautions when reproducing.

One-line Summary

Un-alignment auditing reveals that high compositional safety scores are fragile; CMaG can sharply reduce safety scores with a single backward pass.

Paragraph Summary

Large vision-language models score highly on compositional safety benchmarks, but LLM judges only measure output behavior and cannot prove that safety mechanisms are truly internalized. The paper first shows that minimal supervised fine-tuning can achieve judge scores close to those of multi-stage reinforcement learning pipelines, indicating that benchmark scores are not equivalent to robustness. It further finds that identification and refusal are decoupled: nine direction-discovery methods detect compositional harm with AUC up to 0.99, but ablating these directions either does not affect refusal or destroys model coherence. To address this, the authors propose the training-free CMaG, which measures refusal intent via the log-probability margin between refusal and compliance, takes gradients with respect to intermediate-layer activations, and contrasts with benign pairs to isolate the cross-modal interaction term; a single backward pass yields a steering direction. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional safety scores by up to 65 points while over 98% of outputs remain coherent, outperforming gradient baselines, and the direction transfers to explicit text harm and un-aligns defense-trained models.