Are Typed Decision Models Really This Fragile in Subjective Text Evaluation?

When we hand text evaluation over to automated decision models, we usually assume that as long as the model's choices or scores are stable, the results are trustworthy. But a controlled audit of Jev, Clef Flash, and Perplexity Decider reveals a troubling fact: these models are extremely sensitive to how the evaluated content is presented—merely changing option position, author identity, or decision framing is enough to significantly shift evaluation results.

Audit Design: Manipulating Presentation While Holding Text Constant

The research team conducted large-scale testing on six datasets covering three subjective tasks—humor, helpfulness, and summarization—with a total of 9,000 multi-annotated entries. They collected 702,000 judgments under both native choice interfaces and scoring interfaces. Crucially, all interventions changed only the presentation, while the evaluated text itself remained exactly the same.

The manipulated dimensions included: option position, identifiers, authorship claims, endorsement information, decision framing, and scoring rules. This design allows any fluctuation in evaluation results to be attributed to presentation sensitivity rather than differences in text quality.

Position bias is one of the most common forms of presentation sensitivity in choice-based evaluation. When models are asked to choose between two or more candidate texts in pairwise comparisons or multiple-choice settings, the order in which candidates appear can affect the decision. In human subjects, this is known as the order effect; in language models, position bias typically manifests as a preference for a fixed position (such as first or second) that is unrelated to text content. The "second-position probability advantage" reported in the audit falls into this category: the model systematically has a higher probability of selecting an item when it is in second position than when it is in first position. A common method for identifying position bias is to swap the order of candidates and observe whether choices flip; if the flip rate is significantly above chance, position sensitivity is present.

Three Models, Three Kinds of Fragility

The audit found several specific and sizable sensitivity effects. Jev showed a second-position probability advantage of 6.11 percentage points on the summarization task—that is, simply placing an option in second position earned it a disproportionate selection tendency.

Perplexity Decider was highly sensitive to reader endorsement information: when reader endorsements were reversed, its selected humor text changed in 40.40% of pairs. This means nearly half of the selection results may flip solely because of how social signals are presented.

Clef Flash was affected by decision framing: using a rejection frame reduced summarization agreement with human preferences by 23.36 percentage points. The same summary, asked in a different way, saw evaluation quality drop sharply.

Additionally, scoring format changed score levels but often preserved rankings. This suggests that absolute score values may not be comparable across formats.

Decision framing effects originate from a classic finding in human judgment and decision research: the same information presented in different ways (e.g., "90% survival rate" vs. "10% mortality rate") elicits different judgments and choices. In language model evaluation, framing effects manifest as systematic shifts in model output when the question phrasing or option labels (e.g., "accept/reject" framing, "agree/oppose" framing) are changed without altering substantive content. The rejection frame mentioned in the audit is a typical negative frame that may activate the model's sensitivity to negative information, thereby changing its preference judgments. Framing sensitivity means that evaluation results are not determined solely by text quality but are entangled with the pragmatic structure of the evaluation prompt.

Reproducibility, Presentation Robustness, and Human Alignment Are Three Distinct Properties

The core contribution of this audit is distinguishing three independent properties of evaluators. Reproducibility refers to whether results are stable under the same conditions; presentation robustness refers to whether results are unaffected by irrelevant presentation changes; and human alignment refers to whether evaluation results agree with human preferences.

A model can be highly consistent across repeated tests yet extremely sensitive to presentation; it can also remain stable under presentation changes yet exhibit systematic deviations from human judgment. Conflating these three is a common misconception in current automated evaluation practice.

Reproducibility in machine learning evaluation typically refers to whether the same output can be reproduced under the same input, same model configuration, and same random seed; it measures the degree of determinism in the evaluation process. Presentation robustness, by contrast, requires that evaluation results be insensitive to presentation variables unrelated to the evaluation target (such as option order, identifiers, authorship claims, and framing wording), constituting a kind of out-of-distribution stability requirement. Human alignment measures the correlation or agreement between model judgments and human annotation preferences, typically using metrics such as accuracy and Kendall rank correlation coefficient. There is no entailment relation among the three: high reproducibility does not guarantee high presentation robustness, and high presentation robustness does not guarantee agreement with human preferences. Only by measuring them separately can we locate in what sense an evaluator is reliable and in what sense it is not.

Practical Implications for Evaluation Pipelines

The research offers straightforward recommendations: reliable use requires validating the decisions and score interpretations on which evaluation pipelines depend. Specifically, if downstream applications rely on model choice results, it is necessary to test whether those choices are stable across different presentation formats; if they rely on score thresholds, it is necessary to confirm whether score levels are comparable across scoring formats.

For teams building automated evaluation systems, this means they cannot deploy a model into production based solely on its performance on standard test sets. Presentation sensitivity audits should become a routine pre-deployment check for evaluators.

Presentation sensitivity auditing is designed similarly to perturbation testing in software engineering: hold core inputs constant, systematically perturb irrelevant dimensions, and observe whether outputs change unexpectedly. For choice-based interfaces, audits should cover at least option position swapping, identifier substitution, and framing rewording; for scoring interfaces, they should compare score distributions and ranking consistency across different rating scales (e.g., 1–5 vs. 1–100). A practical diagnostic metric is the choice flip rate: the proportion of pairwise comparisons in which the choice changes after swapping presentation. If the flip rate is significantly above zero, any downstream decision relying on those choices faces additional risk, requiring aggregation strategies (such as averaging bidirectional evaluations) or explicit control of presentation variables.

Conclusion

Typed decision models provide structured choices, scores, and probability outputs for low-cost automated evaluation, but this audit shows that the stability of these outputs is far less reliable than imagined. In subjective text evaluation settings, presentation itself is a variable that must be controlled—not noise that can be ignored.

One-line Summary

Audit reveals typed decision models are highly sensitive to presentation; reproducibility ≠ robustness ≠ human alignment.

Paragraph Summary

The article reports a controlled audit of Jev, Clef Flash, and Perplexity Decider, collecting 702,000 judgments across six datasets and 9,000 multi-annotated entries, with all interventions changing only presentation while holding text content constant. Results reveal significant sensitivity to option position, reader endorsement, decision framing, and scoring format: Jev shows a 6.11 percentage-point second-position advantage on summarization, Perplexity Decider flips 40.40% of pairwise choices when endorsements are reversed, Clef Flash loses 23.36 percentage points of human-preference agreement under a rejection frame, and scoring format changes score levels while rankings often remain unchanged. The core contribution is distinguishing reproducibility, presentation robustness, and human alignment as three independent properties, and recommending presentation sensitivity audits as a routine pre-deployment check for evaluation pipelines.