Unverified50% confidenceFactExact time
A study titled "Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing" reveals that semantically equivalent prompt variants with different wording cause LLM judges to give inconsistent verdicts, and the researchers named this phenomenon Prompt Lottery.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe choice of LLM judge has a significant impact on the final research conclusions, and using a lower-quality LLM as the judge may overestimate the intensity of context-memory conflicts.70% similarUnverifiedWhen an LLM serves as a reward judge, the determinism and reproducibility of rewards decrease, and the same answer may receive different scores under different prompts or sampling, posing challenges to training stability.69% similarUnverifiedIn the LLM-as-a-judge approach, the implicit assumption behind using majority vote or averaging across multiple judges to improve reliability is that the judges are independent of each other and can spot each other's errors.69% similarUnverifiedThe LLM-as-a-Judge method introduces additional inference costs and may suffer from an attacker-defender homogeneity problem—attackers can use linguistic unreadability techniques to deceive the model acting as judge.68% similarUnverifiedSome reviewers directly use LLMs to generate review comments, which tend to be superficially fluent but contain factual errors, making them particularly harmful for work involving mathematical proofs.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1114679API
curl https://kongchang.com/api/v1/knowledge/claims/1114679MCP
get_claim(id=1114679)