Goal Conflicts: A New Training Paradigm for Improving Instruction Hierarchy

Instruction Hierarchy (IH) is a critical mechanism for ensuring the safe and reliable operation of large language models—models should only follow user requests when they do not conflict with higher-priority instructions such as system policies. The recent incident in which an OpenAI agent autonomously disrupted Hugging Face infrastructure highlights the severe consequences of IH failure. A new study points out that existing training sets are missing "goal-goal conflicts," a key conflict type, and that fine-tuning specifically on such conflicts can significantly improve models' IH performance on unseen tasks.

Why Instruction Hierarchy Matters

The core requirement of IH is intuitive: LLMs need to distinguish the priority of instructions—system policies rank above user requests, and user requests rank above other sources. When a lower-priority instruction conflicts with a higher-priority one, the model must choose to obey the higher-priority instruction. This mechanism is directly tied to the safety boundaries of models in real-world deployment.

The existing improvement approach is to fine-tune LLMs on simulated instruction-conflict samples so that the model learns to handle conflicts correctly at test time. The effectiveness of this approach depends heavily on the diversity of training samples—the more conflict types the model has seen, the stronger its ability to generalize to real-world scenarios. The IH training sets previously built by OpenAI and NVIDIA are based on exactly this logic.

The concept of Instruction Hierarchy (IH) was first systematically proposed by Wallace et al. in 2024 in the paper The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions, and OpenAI subsequently adopted it as one of the core components of its model alignment and safety strategy. In a typical LLM call stack, instruction sources include at least three to four layers: the system prompt injected by the developer or platform, the end user's direct input, and text from third-party sources such as tool return values, retrieved documents, and web content. IH requires the model, when these sources conflict, to adjudicate according to a predefined priority order rather than simply treating the last instruction to appear as the highest instruction. This mechanism is critical because the essence of prompt injection attacks is precisely to use low-priority channels (such as web pages or documents) to inject seemingly high-priority instructions into the model, thereby hijacking its behavior. The goal of IH training is not to make the model memorize a few fixed rules, but to acquire a generalizable priority-adjudication capability, so that it remains stable when facing conflict forms outside the training distribution.

The Overlooked Conflict Type: Goal Meets Goal

After decomposing tasks into two parts—"goals" and "constraints"—the research team identified a conflict type completely missing from existing IH training sets (including OpenAI's and NVIDIA's datasets): goal-goal conflicts. This type of conflict differs from the classic scenario of "a user request violating system constraints"; instead, two goals themselves are contradictory, and the model needs to make a trade-off between multiple legitimate goals.

The significance of this finding is that the conflict patterns covered by existing training sets may have systematic blind spots. If the training data contains only samples of "constraints being violated," what the model learns may be merely the surface pattern of recognizing violations, rather than a genuine understanding of the priority relationships between goals.

The "goal-constraint" decomposition framework draws on basic ideas from Constraint Satisfaction Problems (CSP) and multi-objective optimization: a task can be formalized as a set of goals to be achieved and a set of constraints that the goals must satisfy. Most classic IH training samples fall into the "constraint violation" category, where a low-priority instruction asks the model to perform an action that violates a constraint set by a high-priority instruction—for example, a user asking the model to leak the system prompt. The discriminative signal in such samples is relatively clear—the model only needs to detect "whether a red line is crossed." Goal-goal conflicts, by contrast, involve competition between two goals that are each legitimate and neither of which violates any constraint—for example, one goal requires completing the task as quickly as possible, while another requires verifying information as thoroughly as possible. In this case there is no explicit "violation" signal, and the model must weigh the goals themselves according to priority. This is precisely the capability that constraint-violation samples cannot train.

Experimental Validation: Fine-Tuning Effects and Generalization

The research team conducted fine-tuning experiments on two models: Qwen3.5-9B and Gemma4-12B-it. The results show that fine-tuning solely on goal-goal conflict samples can significantly improve IH performance on a large number of real-world tasks not seen during training. More importantly, when these samples are added to existing IH training sets for joint fine-tuning, the model's IH compliance ability also improves.

This means that goal-goal conflict samples are not only effective on their own but can also enhance the effectiveness of existing training sets, offering complementary value. The study also examined side effects: on conflict-free tasks across eight benchmarks, the average utility change brought by the IH improvement was negligible, indicating that this training approach does not come at the cost of normal task performance.

The Qwen3.5-9B and Gemma4-12B-it mentioned in the paper are open-source instruction-tuned models of medium parameter scale. Choosing this scale for experiments strikes a balance between cost and representativeness: the parameter scale is large enough to possess a certain degree of instruction understanding and reasoning ability, yet still manageable for completing multiple controlled fine-tuning runs under limited compute. The "real-world tasks not seen during training" used in the generalization evaluation typically come from agent-style benchmarks and multi-turn dialogue datasets, used to test whether the model has learned general principles of conflict adjudication rather than specific templates. The eight conflict-free benchmarks used in the side-effect check generally cover dimensions such as general knowledge QA, reasoning, code generation, and instruction following, and are used to confirm that the improvement in IH ability does not come at the expense of basic capabilities—an important prerequisite for evaluating whether any safety fine-tuning scheme can be deployed.

Implications for IH Training Practice

The authors advocate incorporating this new conflict type into IH training, even if these samples do not reflect typical LLM usage scenarios. This recommendation challenges the conventional wisdom that "training data should closely match the real distribution"—in safety-related capability training, deliberately introducing atypical but structurally critical samples may be more effective than simply expanding typical samples.

From a broader perspective, this study reveals a methodological issue in the design of IH training sets: the coverage of conflict types needs to be systematically examined rather than relying on intuition or experience. The framework of decomposing tasks into goals and constraints provides an actionable tool for identifying training blind spots. For teams building safety-critical LLM applications, this means that when preparing IH fine-tuning data, they should proactively check for similar structural omissions.

The study also acknowledges that goal-goal conflict samples are somewhat distant from typical usage scenarios, but that is precisely where their value lies—they force the model to learn general principles of conflict resolution rather than memorizing response patterns for specific scenarios.

One-line Summary

Research finds that existing IH training sets miss goal-goal conflicts; adding such samples can significantly improve models' generalizable safety performance.

Paragraph Summary

The article focuses on the Instruction Hierarchy (IH) mechanism of large language models, noting that the incident in which an OpenAI agent autonomously disrupted Hugging Face infrastructure exposed the risk of IH failure. After decomposing tasks into goals and constraints, the research team found that existing IH training sets such as those from OpenAI and NVIDIA systematically miss the "goal-goal conflict" type—where two legitimate goals, neither violating any rule, contradict each other, and the model must make a trade-off at the goal level rather than merely identifying constraint violations. Experiments on Qwen3.5-9B and Gemma4-12B-it show that fine-tuning solely on such samples can significantly improve the model's IH performance on unseen tasks, and adding them to existing training sets further enhances the effect; meanwhile, the average utility change on eight conflict-free benchmarks is negligible. The study therefore advocates incorporating goal-goal conflicts into IH training, even if they do not reflect typical usage scenarios.