Lexical Conflicts in LLM Translation: Can Prompt Definitions Override a Model's Prior Knowledge?

When "Programmer" Gets Redefined: Lexical Knowledge Conflicts in LLMs

When translating low-resource languages or domain-specific texts, large language models (LLMs) often need to draw on external lexical resources. But the definitions these resources provide may conflict with the lexical knowledge already inside the model. For example, when a common word is given an entirely new meaning in a specific context, will the model trust the definition in the prompt, or the old knowledge it learned during training?

A study titled Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation systematically investigates this question by constructing controlled Chinese-to-English translation tasks. The researchers refer to the definitions provided in the prompt as "Contextual Knowledge" (CK), and evaluate nine models across three variables: word frequency, sentence context, and prompt framing.

Experimental Design: Creating Conflicts Within and Across Semantic Categories

The research team designed an elegant controlled translation task: they selected common Chinese words and assigned them new meanings either within or across semantic categories, then asked the models to translate them into English. For example, they gave the word "programmer" an anomalous meaning like "an edible ripe fruit" and observed whether the model's output adopted the definition provided in the prompt.

The value of this design lies in isolating variables. In real translation scenarios, models may be influenced simultaneously by context, domain knowledge, and word frequency, making it difficult to determine why they rely on CK. By controlling source-word familiarity, sentence context, and prompt framing, the researchers could more clearly observe the degree of model reliance on CK.

Contextual Knowledge (CK) is the core operationalized concept in this paper, referring to the word-sense definition explicitly provided to the model in the prompt. It stands in contrast to the parametric knowledge that the model acquires from massive corpora during pre-training: the former is temporary external information injected at inference time, while the latter is a prior solidified in the weights. When the two give inconsistent definitions for the same lexical item, a lexical knowledge conflict arises. This framework originates from knowledge-conflict research, which previously focused mainly on contradictions between "retrieval augmentation" and internal model memory in factual question answering. This paper transfers it to the translation setting, using "word sense" as the carrier of conflict, so that the degree of CK reliance can be viewed as the model's tendency to comply with external instructional knowledge at inference time.

Key Findings: Effects of Low-Frequency Words, Supportive Context, and Prompt Framing

Word frequency effect: Across the nine models, the less common the source word, the stronger the model's reliance on CK. This is intuitive—the model's prior knowledge about low-frequency words is weaker, making it more easily "led astray" by the definition in the prompt.

Context effect: When the sentence context is consistent with the definition provided in the prompt, the model is more inclined to adopt CK. Context acts as an amplifier, making the new definition seem more plausible.

The role and limits of prompt framing: Prompt framing can reduce CK reliance, but the effect is limited. Among the nine models, seven still preferentially adopted CK even when explicitly instructed to "ignore the given definition." This shows that the model's compliance with definitions in the prompt is quite stubborn, and simple instruction-based negation is hard to make effective.

Effect of model scale: Within the same model family, larger models are more likely than smaller ones to use CK. This finding is intriguing—larger models are generally considered to have richer knowledge and to be better at following instructions, but in lexical conflict scenarios this capability manifests as being more easily overridden by prompt definitions.

Prompt framing refers to changing how the model interprets prompt content through wording, role setting, or instruction structure. For example, packaging a definition as an "authoritative dictionary definition" versus "someone's guess," or adding a negative instruction like "please ignore the following definition," are both framing operations. This paper finds that framing can weaken but not eliminate CK reliance, which echoes research on instruction following and sycophancy: models tend to cater to the information presented in the prompt, even when explicitly asked to resist it. The so-called "stubbornness" precisely indicates that lexical conflict is not simply a matter of attention allocation, but involves the model's default assignment of authority to the prompt.

Three Follow-up Experiments: Modal Weakening, Repetition, and Reasoning

The study also conducted three follow-up experiments, further revealing the boundary conditions of CK reliance:

• Modal weakening: Reducing the degree of certainty in the definition in the prompt (e.g., using expressions like "possibly" or "perhaps") decreases the model's reliance on CK.
• Repetition effect: The effect of repeating the same definition depends on the prompt's layout, and it is not simply "the more repetition, the better."
• Reasoning effect: The impact of reasoning steps differs across models; some models rely more on CK because of reasoning, while others do the opposite.

These results show that CK reliance is not determined by a single factor, but is the joint result of word frequency, context, prompt wording, layout, and model characteristics.

Modal weakening borrows the category of modality from linguistics: using epistemic modal markers such as "possibly," "reportedly," or "perhaps" to reduce the certainty of a proposition. In prompt engineering, this amounts to attaching an uncertainty signal to the definition so that the model no longer treats it as established fact. The results in this paper show that weakening modality can reduce CK reliance, indicating that the model's degree of adoption of prompt information is sensitive to epistemic markers; confidence cues carried by the prompt wording itself are partially decoded by the model and used to weigh old versus new knowledge. This provides a basis for a low-cost conflict-mitigation strategy: there is no need to delete the conflicting definition, only to reduce its assertive strength.

Practical Implications: Detecting and Guiding Lexical Conflicts

The significance of this study goes beyond revealing a phenomenon. It reminds us that when using external lexical resources to enhance LLM translation capabilities, we need to be alert to conflicts between the model's old knowledge and new definitions. Especially in domain-specific translation, when terms are assigned specific meanings, the model may appear to have "learned" the new word sense while actually still outputting old knowledge.

The researchers call for developing new methods to detect and evaluate lexical conflicts and to guide models toward reasonable lexical choices in translation. For practitioners who rely on LLMs for professional translation, this means they cannot blindly trust glossaries in prompts, but should design more robust prompting strategies—for example, combining modal weakening, adjusting prompt layout, or introducing explicit conflict-detection mechanisms.

As model scale continues to grow, the complexity of this problem may further increase—larger models are better at using contextual knowledge, which also means they may more stubbornly "comply" with definitions in the prompt, even when such compliance contradicts the facts.

One-line Summary

Study reveals LLMs are more likely to follow new definitions in prompts than their own prior knowledge during translation, and larger models are even more prone to this.

Paragraph Summary

Through controlled Chinese-to-English translation tasks, this study examines how LLMs choose between word senses provided in prompts (contextual knowledge, CK) and internal prior knowledge when the two conflict. Manipulating word frequency, sentence context, and prompt framing across nine models, it finds that: the lower the frequency of the source word and the more the context supports the new meaning, the more the model relies on CK; prompt framing can weaken but not eliminate this reliance, with seven models preferentially adopting CK even when instructed to ignore it; and larger models within the same family are more likely to use CK. Follow-up experiments show that modal weakening reduces CK reliance, the repetition effect depends on prompt layout, and the reasoning effect varies by model. The study calls for developing lexical conflict detection methods and designing more robust prompting strategies.