Selection Bias
A concept in statistics and research methodology referring to the distortion that arises when the method of sample selection is non-random or exhibits a systematic tendency, causing the sample to be unrepresentative of the population and leading to skewed analytical conclusions.
Timeline (last 90 days)
In the LLM context, selection bias specifically refers to a model's content-irrelevant preference regarding the presentation position, order, or format of options in multiple-choice tasks, with common manifestations including primacy bias, recency bias, or a systematic preference for particular letter labels.
Selection bias is systematic rather than random: the model may make the same mistake under multiple orderings because of the same underlying preference (such as a tendency toward a particular position or semantic label).
METR observed a sharp rise in the number of developers who chose not to participate in the study, because they were unwilling to work without AI; 30% to 50% of participants said they would deliberately not submit certain tasks to avoid completing them without AI.
Selection bias refers to estimates systematically deviating from the true population value because the sample was not randomly drawn; it cannot be eliminated by increasing sample size and can only be mitigated by improving recruitment design or using methods such as instrumental variables.
The premise for majority voting to yield gains is that voters make mistakes on uncorrelated things, that is, errors are approximately independent, and models from different vendors are the closest thing to independent failure samples without building one's own model.
This study is a working paper that has not yet undergone peer review, the data comes from the business analytics company Jellyfish, there may be selection bias from platform users, and the authors emphasize that these are short-term effects and long-term results may differ.
'In half of cases the outcome is never known' is referred to in the machine learning field as selection bias or survivorship bias.
All Facts (7)
In the LLM context, selection bias specifically refers to a model's content-irrelevant preference regarding the presentation position, order, or format of options in multiple-choice tasks, with common manifestations including primacy bias, recency bias, or a systematic preference for particular letter labels.
50%UnverifiedSelection bias is systematic rather than random: the model may make the same mistake under multiple orderings because of the same underlying preference (such as a tendency toward a particular position or semantic label).
50%UnverifiedMETR observed a sharp rise in the number of developers who chose not to participate in the study, because they were unwilling to work without AI; 30% to 50% of participants said they would deliberately not submit certain tasks to avoid completing them without AI.
50%UnverifiedSelection bias refers to estimates systematically deviating from the true population value because the sample was not randomly drawn; it cannot be eliminated by increasing sample size and can only be mitigated by improving recruitment design or using methods such as instrumental variables.
50%UnverifiedThe premise for majority voting to yield gains is that voters make mistakes on uncorrelated things, that is, errors are approximately independent, and models from different vendors are the closest thing to independent failure samples without building one's own model.
50%UnverifiedThis study is a working paper that has not yet undergone peer review, the data comes from the business analytics company Jellyfish, there may be selection bias from platform users, and the authors emphasize that these are short-term effects and long-term results may differ.
50%Unverified'In half of cases the outcome is never known' is referred to in the machine learning field as selection bias or survivorship bias.
50%