Can Jev-style System One Models Reduce Selection Bias? Structural Stability Questioned

Introduction: Selection Bias — A Stubborn Challenge for LLMs

Large language models (LLMs) continue to exhibit selection bias even when explicitly instructed to ignore the order in which options are presented, or when guided by reasoning prompts. This bias is particularly insidious in multiple-choice tasks: the model appears to give a stable answer, but when the options are rearranged, the originally correct choice may vanish. Recently, a class of models known as Jev-style System One has attempted to circumvent the biases of generative reasoning by directly predicting probabilities for candidate answers. But do such architectures truly solve the problem? A new systematic test delivers a surprising answer.

Selection Bias in the LLM context specifically refers to a model's non-content-related preference for the presentation position, order, or format of options in multiple-choice tasks—not the statistical notion of sample selection bias. Common manifestations include a tendency to choose the first option (primacy bias) or the last option (recency bias), or a systematic preference for particular letter labels (e.g., "A" or "C"). This bias is independent of whether the model "knows" the correct answer—even with sufficient internal knowledge, the output can still be swayed by arrangement. Its causes stem partly from imbalanced answer distributions in training data (e.g., textbook exercise answers often located in specific positions), and partly from positional encoding and attention patterns during autoregressive generation. The danger of selection bias is that it makes model performance appear stable while actually being fragile: in real deployment, tiny changes in input format can flip answers, and this is difficult to detect through conventional accuracy metrics.

Test Design: Shuffling Options, Tracking Individual Answers

The research team designed a simple yet rigorous test: keep the questions and candidate content unchanged, and only shuffle the presentation order of the options. They covered two Jev versions, twelve Jev replication models, and six generative LLMs, tracking each model's individual answers across eight different presentation orders and comparing them with results from repeated testing under a fixed order.

The main evaluation set contained 1,700 questions from two benchmarks. The key to this design is that it does not settle for measuring overall accuracy, but rather tracks the "fate" of the same question's answer under different arrangements—which initially correct answers fail after shuffling, and which errors persist.

Jev-style System One models refer to a class of discriminative architectures that do not rely on autoregressive generation but instead directly compute the conditional probability for each candidate answer (e.g., P(answer|question)) and take the maximum. The name borrows the metaphor of "System 1" fast, intuitive decision-making from psychology, in contrast to "System 2" generative models that require step-by-step reasoning. These models are typically based on encoder structures (e.g., BERT-style), concatenating the question with each option and performing a single forward pass to output a score for each option. Theoretically, since no word-by-word generation is involved, it should be less susceptible to interference from generation order and prompt format. However, this study reveals that even this "direct scoring" structure still produces systematic answer drift due to changes in option arrangement, suggesting that the bias may be rooted in sensitivity to positional information during the encoding stage, rather than solely originating from the generation process.

Core Findings: Stable Accuracy, but a Significantly Expanded Failure Set

The results reveal a key contradiction: Jev models' accuracy changes by less than two percentage points, appearing very stable. However, relative to the original order, their cumulative failure set expanded by 27%–29%, and the first failure persistently affects the final presentation. This means that although overall accuracy fluctuates little, a large number of originally correct answers are lost after option rearrangement.

What you may not have noticed is that this loss of initial success is far greater than performance under repeated fixed order, even for highly reproducible Jev replication models. In other words, Jev models can give answers "stably," but this stability is not equivalent to resistance to selection bias.

Comparison with Generative LLMs: Jev Has No General Advantage

The study also compared Jev models with six generative LLM baselines. In terms of retaining correct choices, Jev did not show a general advantage. That is, the Jev-style structure did not better resist biases from option order changes than traditional generative models.

This finding challenges the intuition that "structured decision-making naturally reduces selection bias." The design of Jev models, which directly predicts probabilities, should theoretically focus more on the candidate content itself, but experiments show they are still significantly affected by presentation order.

Limitations of Aggregation Strategies: Voting and Unanimous Agreement Are Both Unreliable

Faced with the instability of individual answers, a natural idea is to improve robustness through aggregation. The study tested voting mechanisms and found only modest gains for Jev models. More concerning, even when answers are completely consistent across all presentation orders (unanimous), these answers are often wrong.

This indicates that aggregation can only mitigate some random fluctuations and cannot solve systematic selection bias. Unanimously agreed-upon answers may simply be the model making the same mistake under multiple arrangements, rather than truly understanding the question.

Cumulative Failure Set is the core metric used in this study to measure robustness: it counts the proportion by which the set of questions that are wrong at least once across multiple option arrangements expands relative to the error set under the original order. Unlike overall accuracy, this metric focuses on "how the fate of the same question's answer changes across different arrangements"—which originally correct questions fail for the first time after shuffling, and that failure persistently affects the final result of all subsequent arrangements. An expansion of the cumulative failure set by 27%–29% means that although average accuracy remains almost unchanged, nearly thirty percent of originally correctly answerable questions will "permanently" lose their correct answer under some arrangement. This metric captures the hidden danger of selection bias more sensitively than accuracy: the model may "swap" correct and incorrect answers across different arrangements, so that overall accuracy masks fragility at the individual level.

Conclusion: Stable Accuracy Is Not the Same as Bias Resistance

This study exposes the gap between reproducible decision-making and maintaining correct decisions across equivalent presentations. Although Jev-style System One models provide a direct probabilistic prediction structure, they do not fundamentally solve the selection bias problem. Stable accuracy metrics may mask the model's sensitivity to option arrangement, and this sensitivity can lead to unpredictable failures in real-world deployment.

For AI application developers, this means that when choosing models or designing evaluations, one cannot rely solely on accuracy under a fixed order. Adversarial tests such as option shuffling must be introduced to truly measure the robustness of model decisions. Structural simplification (such as System One-style direct prediction) does not automatically lead to reduced bias; the root causes of selection bias may be deeper than model architecture.

Unanimous Agreement is typically regarded in aggregation strategies as a signal of high confidence: if the model gives the same answer across multiple option arrangements, that answer is presumed reliable. However, this study finds that this assumption does not hold. The reason is that selection bias is systematic rather than random—the model may make the same mistake across multiple arrangements due to the same underlying preference (e.g., a tendency toward a particular position or semantic label). For example, if the model prefers options containing a specific keyword, and that keyword happens to appear in the wrong option, then regardless of order, the model may consistently choose the wrong answer. Therefore, unanimous agreement can only rule out differences caused by random fluctuations, not common errors caused by systematic bias. The effectiveness of aggregation strategies depends on whether individual errors are independent, and selection bias precisely introduces correlation, making both voting and unanimous agreement unable to reliably improve robustness.

One-line Summary

Jev-style System One models have stable accuracy but their cumulative failure set expands by 27%–29% after option shuffling, leaving selection bias unresolved.

Paragraph Summary

A systematic test tracked the fate of individual answers from Jev-style System One models and generative LLMs across eight option arrangements. Results show that Jev models' overall accuracy changes by less than two percentage points, appearing stable, but relative to the original order, their cumulative failure set expands by 27%–29%, and the first failure persistently affects the final presentation. Compared with six generative LLM baselines, Jev has no general advantage in retaining correct choices. Voting aggregation yields only modest gains, and even when answers are consistent across all arrangements, they are often wrong. The study shows that stable accuracy cannot serve as sufficient evidence of resistance to selection bias; structural simplification does not eradicate the root causes of bias, and evaluation must introduce adversarial tests such as option shuffling.