[KongchangAI]
· 2 min read· 1,453 words

Are New Models Increasingly Hard to Jailbreak? Abliteration Effectiveness Does Not Uniformly Decline

Are New Models Increasingly Hard to Jailbreak? Abliteration Effectiveness Does Not Uniformly Decline

Whether newer models are harder to abliterate depends on the release pair, method, and budget, not on generation.

This study systematically compared abliteration effectiveness across five model release pairs under the Heretic 2.0 framework, covering direction-based methods (difference-in-means, MPOA, SOM, J-space) and ARA. Results show that newer models retained an additional refusal rate of -1 to +37 percentage points under direction-based methods and -26 to +34 percentage points under ARA, indicating that "newer models are increasingly harder to remove refusal behavior" is not a universal rule. Effectiveness depends heavily on model family: ARA's loss was smaller on Gemma and OLMo than direction-based methods, but larger on Qwen. The study also found that evaluation set composition can reverse conclusions, and keyword counting differs from WildGuard judgment by a model-dependent 7 percentage points. The conclusion is that difficulty depends on the specific release pair, method, and search budget, requiring multi-dimensional audits rather than generational intuition.

Introduction: The Promise and Reality of Abliteration

Abliteration is a fine-tuning-free weight editing technique designed to remove refusal behavior from open-source large language models. Its core idea is to find the vector representing the refusal direction in the model's internal representations, then "excise" that direction through weight ablation. This method was once seen as a major loophole in alignment safety mechanisms—as long as you have the weights, you can cheaply remove the model's safety restrictions.

However, the developers of Arbitrary-Rank Ablation (ARA) recently reported that direction-based abliteration is increasingly prone to failure on newer-generation models. Does this claim hold up? A systematic comparison across five model release pairs provides a more nuanced answer: the difficulty of refusal removal has not uniformly increased, but depends heavily on the model family, the method used, and the search budget.

Overview of Abliteration Effectiveness Comparison Study

Refusal direction is the core assumption underlying abliteration-type methods: the model has a specific linear direction in its internal representation space, and when input triggers refusal behavior, the activation component along that direction is significantly amplified. This assumption stems from interpretability research on activation space—by comparing the mean difference in residual stream activations between harmful and harmless prompts, the direction vector can be estimated. Direction-based methods then ablate the component along that direction either at inference time or at the weight level, thereby suppressing refusal behavior. Its limitation is that refusal behavior may not be fully characterized by a single linear direction, especially on newer models trained with more complex alignment, where multi-directional or nonlinear representations may cause single-direction ablation to fail.

Experimental Design: Five Model Pairs, Multiple Methods, and Unified Evaluation

The study selected five release pairs (predecessor and successor), edited under the Heretic 2.0 framework: Qwen2.5-32B and Qwen3.8-27B, Gemma-2-27B and Gemma-4-31B, OLMo-2-32B and OLMo-3.1-32B, Mistral-Small-2409 and Mistral-Small-3.2, and Qwen2.5-7B and Qwen3.5-4B.

Each model pair was run with three to four direction-based methods (difference-in-means, MPOA, SOM, with the small model pair additionally using J-space) as well as ARA, under identical prompts, nominal search budgets, first-token KL divergence caps, and selection rules. The evaluation set consisted of 972 held-out harmful prompts, covering JailbreakBench, XSTest, StrongREJECT, HarmBench, and a self-built malicious code repository, with automated WildGuard determining whether refusal still occurred. If the new model still refused more prompts after editing, it was considered "harder to remove."

Difference-in-means, MPOA, and SOM are three representative methods for estimating the refusal direction. Difference-in-means directly takes the difference vector between the mean activations of harmful and harmless prompts as the refusal direction—simple but susceptible to prompt distribution. MPOA (Mean Projection Outlier Ablation) builds on the mean difference by introducing projection and outlier handling to more robustly locate the direction. SOM (Subspace Orthogonalization Method) no longer limits itself to a single direction but instead estimates a subspace and performs orthogonalized ablation, addressing cases where refusal behavior is distributed across multiple directions. J-space is a variant for small models, adjusting direction estimation and ablation strategies for their lower representation dimensionality. Different methods characterize the "refusal direction" at different levels of granularity, which is an important reason why different methods may yield divergent conclusions on the same model.

Key Findings: Difficulty Changes Are Not Uniformly Increasing

Direction-Based Methods: From Slight Improvement to Significant Deterioration

After direction-based removal, newer models retained -1 to +37 percentage points more refusal behavior. Negative values mean the new model was actually easier to remove, while positive values indicate greater difficulty. This wide range shows that the claim of "increasingly harder" holds only for some model families.

After incorporating run-to-run variance, the between-group heterogeneity of direction-based effects ranged from high to low, with I² from 20% to 80% according to the scale and variance models. In other words, the differences between model pairs themselves are large, and summarizing with a unified narrative of "new models are harder to abliterate" would obscure key details.

ARA: Similarly Divergent Performance

After ARA editing, newer models retained an additional refusal rate of -26 to +34 percentage points. On Gemma and OLMo, ARA's effectiveness loss was smaller than that of direction-based methods; on Qwen, the opposite was true, with ARA losing more. Each conclusion is based on only one ARA run, and some conclusions change with the choice of divergence budget or variance model.

A cautionary detail: on Gemma, the increase in refusal rate caused by direction-based methods came mainly from the malicious code repository, while the trend reversed on public evaluation sets. This shows that the composition of the evaluation set directly shapes the conclusion about whether removal is "harder."

I² (I-squared) is a commonly used metric in meta-analysis for measuring heterogeneity, representing the proportion of total variation attributable to true between-group differences (rather than sampling error or run-to-run variance), ranging from 0% to 100%. In this study, I² of 20%–80% means that effect differences across model release pairs are considerable: the low end suggests some conclusions may be driven mainly by noise, while the high end indicates significant true differences between model families. First-token KL divergence is a metric used during editing to constrain model behavior drift—it measures the difference in probability distribution over the first generated token between the edited and original models. The stricter the divergence cap, the smaller the search space and the more likely the editing effectiveness is limited. Therefore, the KL budget setting directly affects the judgment of whether removal is "harder."

Methodological Warning: A Single Audit May Yield Opposite Conclusions

The study explicitly points out that auditing only a single model pair, a single method, or a single prompt source may lead to completely opposite conclusions. For example, looking only at Gemma's performance on the malicious code repository would suggest that newer models are significantly harder to abliterate; looking only at public sets would yield the opposite conclusion.

Another counterintuitive finding is that keyword counting at the tool window did not, as predicted, on average underestimate judged refusal, but the average difference per model reached 7 percentage points. This means the bias between automated judgment and keyword heuristics is model-dependent, and keyword counting cannot simply replace WildGuard judgment.

WildGuard is an open-source automated safety judgment model used to determine whether a model response constitutes a refusal or harmful content. Compared with keyword matching, it better captures refusal expressions at the semantic level. Keyword heuristics, on the other hand, approximate judgment by counting refusal marker words such as "sorry," "cannot," and "unable" in responses. This study found that the average difference between the two reached 7 percentage points on some models, and the direction of bias varied by model, indicating that keyword counting is not an unbiased substitute for WildGuard. Sources of this bias may include: newer models tending to use more euphemistic or subtle refusal wording, or refusal expressions changing across languages/styles, rendering fixed word lists inaccurate.

Conclusion: Difficulty Depends on Release Pair, Method, and Budget, Not "Generation"

Whether newer models are harder to remove refusal behavior depends on the specific release pair, the method used, and the search budget, rather than on model generation alone. The ARA developers' observation holds for some model families but is not a universal rule. For those concerned about open-source model safety, this means: when assessing abliteration risk, multi-dimensional audits targeting specific models, specific methods, and specific evaluation sets are needed, rather than relying on a single experiment or generational intuition.

This study also reminds defenders that direction-based removal is not a master key—its effectiveness may fluctuate on newer models due to changes in architecture, training data, or alignment strategies; meanwhile, attackers will also turn to optimization methods such as ARA. The balance point in attack and defense is still shifting, but the direction of movement is not uniform.

One-line Summary

Whether newer models are harder to abliterate depends on the release pair, method, and budget, not on generation.

Paragraph Summary

This study systematically compared abliteration effectiveness across five model release pairs under the Heretic 2.0 framework, covering direction-based methods such as difference-in-means, MPOA, SOM, and J-space, as well as ARA. Results show that newer models retained an additional refusal rate of -1 to +37 percentage points under direction-based methods and -26 to +34 percentage points under ARA, indicating that "newer models are increasingly harder to remove refusal behavior" is not a universal rule. Effectiveness depends heavily on model family: ARA's loss was smaller than that of direction-based methods on Gemma and OLMo, but larger on Qwen. The study also found that evaluation set composition can reverse conclusions, and that keyword counting and WildGuard judgment differ by a model-dependent 7 percentage points. The conclusion is that difficulty depends on the specific release pair, method, and search budget, requiring multi-dimensional audits rather than reliance on generational intuition.

#Abliteration#open-source LLM safety#refusal direction#Arbitrary-Rank Ablation#model alignment#jailbreak#WildGuard
Share:

Related articles