Component Translation and Refusal Rates: A New Dimension in Multilingual Safety Evaluation

Multilingual safety evaluations of large language models typically translate the entire request into different languages, but this approach overlooks a key issue: the language assignment of different components within a request may independently influence model behavior. A recent study proposes a "paired reversal" method that controls for behavior, English framing, and the number of translated spans, thereby disentangling the effects of component language identity from those of text quantity.

From Holistic Translation to Component-Level Analysis

Traditional evaluation methods translate the complete prompt into the target language and compare refusal rates across languages. However, a request may consist of two components—an action instruction and a content object—for example, "Please explain how to [some behavior]." The languages of these two components can vary independently, and holistic translation cannot distinguish which component carries the non-English text.

This study designs a paired reversal experiment: either the full-request component or the full-action component is translated into the same target language, while holding behavior, English framing, and the number of translated spans constant. In this way, component identity and the quantity of non-English text change simultaneously, allowing the study to test whether component language assignment affects refusal behavior.

Refusal rate is one of the core metrics for measuring safety alignment in large models: given a set of harmful requests, the proportion of requests the model explicitly refuses to answer. Its counterpart is unsafe compliance, where the model does not refuse and actually provides harmful content. The two are not complementary—a model may give ambiguous, partial, or evasive responses that count as neither explicit refusal nor unsafe compliance. This paper therefore recommends reporting refusal rates and unsafe compliance rates separately, to avoid simply equating "not refused" with "unsafe." In multilingual evaluation, changes in refusal rates may stem from differences in the model's language understanding, uneven language coverage of safety policies, or—as this paper focuses on—component language assignment effects.

Experimental Design and Main Results

The study screened 190 samples from 300 harmful behaviors that passed consensus annotation and automatic gating. English, Chinese, and Spanish were independently assigned to the two components, yielding nine configurations. Refusal rates were scored by an LLM judge, with unsafe compliance reported separately.

On Gemini 2.5 Flash and DeepSeek^Ad, the three refusal-rate swaps were approximately 5.3 percentage points, and the fourth was -0.5. After the pre-specified six Holm corrections, none of these differences were significant. This indicates that on the main models, the effect of component language assignment is limited.

Holm correction is a method for controlling the family-wise error rate (FWER) in multiple hypothesis testing. When a study tests multiple hypotheses simultaneously (as in this paper's tests of refusal-rate differences across multiple models and language configurations), without correction the false positive rate inflates with the number of tests. Holm correction works as follows: sort all p-values from smallest to largest, compare the i-th p-value against the threshold α/(m−i+1) (where m is the total number of tests), and stop at the first non-significant result. It is less conservative and more statistically powerful than Bonferroni correction while still strictly controlling FWER. The "six Holm corrections" and "twelve Holm sensitivity corrections" mentioned in this paper refer to applying such corrections to the corresponding number of tests, ensuring that the main experimental conclusions are not distorted by false positives from multiple comparisons.

Exploratory Extension: Differences Between Mistral and GPT-5.4-mini

In the exploratory extension, Mistral Small 4 exhibited a significant Spanish swap effect: 13.7 percentage points (95% CI [8.4, 18.9]), passing the twelve Holm sensitivity corrections under the primary judge, and 12.1 percentage points under the secondary judge. In contrast, GPT-5.4-mini's estimate was within 2 percentage points of zero, showing no clear effect.

These results suggest that different models vary in their sensitivity to component language assignment, but the null results of the main experiment contrast with the exploratory findings and require further research to verify.

95% confidence interval is an interval used in frequentist statistical inference to express estimation uncertainty: if sampling were repeated many times, approximately 95% of such intervals would contain the true parameter value. The reported "13.7 percentage points (95% CI [8.4, 18.9])" means the point estimate of the Spanish swap effect is 13.7 percentage points, and the interval does not contain 0, indicating the effect is statistically significant. The width of the confidence interval reflects estimation precision: the narrower the interval, the more reliable the estimate. When interpreting, note that a confidence interval excluding 0 and a hypothesis test p-value below 0.05 are generally equivalent in large samples, but the interval also provides a plausible range for the effect size, which is more informative than a mere significance judgment.

Implications for Evaluation Practice

The study recommends that evaluations record component language and the amount of translated text, compare matched behaviors under equal numbers of translated spans, and interpret refusal and unsafe compliance separately. Component assignment offers an additional evaluation dimension that helps achieve a more fine-grained understanding of multilingual safety alignment.

Although the main experiment found no significant effects, the exploratory results and differences across models indicate that component-level language analysis is worth incorporating into multilingual safety evaluation frameworks. Future work could expand model and language coverage to further test the robustness of component language effects.

One-line Summary

This study proposes a paired reversal method to disentangle the effects of component language identity and text quantity on the refusal behavior of multilingual large models.

Paragraph Summary

This study points out that traditional multilingual safety evaluations translate the entire request holistically, failing to distinguish the independent effects of different component language assignments within a request on model behavior. To address this, it proposes a "paired reversal" method that holds behavior, English framing, and the number of translated spans constant while varying only component language identity. In the main experiment on Gemini 2.5 Flash and DeepSeek, no significant refusal-rate swaps were found after Holm correction. In the exploratory extension, Mistral Small 4 showed a significant Spanish swap effect (13.7 percentage points), while GPT-5.4-mini's estimate was near zero, suggesting differences in sensitivity across models. The study recommends that evaluation practice record component language and the amount of translated text, and interpret refusal and unsafe compliance separately.