[KongchangAI]
ConceptPosition Bias / position bias

Position Bias

A known systematic flaw in the LLM-as-a-Judge evaluation paradigm, whereby the judge model tends to assign higher scores to answers that appear earlier in a conversation or that are longer in length, irrespective of actual content quality.

Timeline (last 90 days)

Oct 19

LLM judges exhibit position bias: in pairwise comparisons, judges tend to favor the answer in a certain position, usually the first; merely swapping the order of the candidate answers can flip the ranking (Wang et al.).

Unverified50%

All Facts (1)

Unverified

LLM judges exhibit position bias: in pairwise comparisons, judges tend to favor the answer in a certain position, usually the first; merely swapping the order of the candidate answers can flip the ranking (Wang et al.).

50%

Source Articles

Is Your LLM Judge Biased? Three Calibration Steps to Make Evaluations TrustworthyOct 19Eval-Driven Development: How to Scientifically Test Your AI Skills and AgentsSep 16