Elo
A rating system for assessing the relative skill levels of players, originally developed for chess. It iteratively adjusts scores based on the deviation between match outcomes and expected outcomes, and is widely used in competitive rankings and AI model evaluation.
Timeline (last 90 days)
The Maia project trains separate models for different rating bands (such as 1100, 1500, and 1900 Elo) to capture the typical style and weaknesses of players at each level.
The Elo rating of the human world chess champion is around 2800.
Arena AI (LMArena, formerly Chatbot Arena) uses an anonymous battle format in which users see two models' answers side by side and vote for the better one, which are then aggregated into rankings via an Elo-like algorithm.
The Arena platform uses an anonymous blind testing mechanism, where users vote on the answers of two anonymous models, ultimately forming a model leaderboard through an Elo-like ranking system.
Eval score and Elo ranking are not in sync, because Eval score is generated by an automated evaluation pipeline and is good at catching hard violations but poor at quantifying whether an application truly understands user intent or whether the interaction experience is natural; Elo, on the other hand, comes from pairwise battles and is closer to real human preferences in side-by-side comparison.
Chatbot Arena uses the Elo rating system to convert a large number of voting results into model rankings, and Elo was originally used for chess.
In highly subjective tasks, pairwise comparison is gradually replacing absolute scoring as the mainstream evaluation paradigm, and its statistical results can be directly fed into the Bradley-Terry model or an Elo system to output a globally transitive ranking.
In a blind paired Elo study, PWM-WROP ranked first among continuation-style models and third overall, behind only a statistical tie between two reference video generation models.
Knowledge work benchmarks are typically Elo-based and used to measure a model's overall performance on professional tasks such as code review, technical documentation, and system design.
The ELO rating system was originally designed by physicist Arpad Elo for chess and measures ability through relative comparisons between players rather than absolute scores.
8 more timeline events
All Facts (18)
Text Arena uses an Elo-like scoring mechanism to dynamically calculate comprehensive rankings based on head-to-head model competitions
85%UnverifiedThe Maia project trains separate models for different rating bands (such as 1100, 1500, and 1900 Elo) to capture the typical style and weaknesses of players at each level.
50%UnverifiedThe Elo rating of the human world chess champion is around 2800.
50%UnverifiedArena AI (LMArena, formerly Chatbot Arena) uses an anonymous battle format in which users see two models' answers side by side and vote for the better one, which are then aggregated into rankings via an Elo-like algorithm.
50%UnverifiedThe Arena platform uses an anonymous blind testing mechanism, where users vote on the answers of two anonymous models, ultimately forming a model leaderboard through an Elo-like ranking system.
50%UnverifiedEval score and Elo ranking are not in sync, because Eval score is generated by an automated evaluation pipeline and is good at catching hard violations but poor at quantifying whether an application truly understands user intent or whether the interaction experience is natural; Elo, on the other hand, comes from pairwise battles and is closer to real human preferences in side-by-side comparison.
50%UnverifiedChatbot Arena uses the Elo rating system to convert a large number of voting results into model rankings, and Elo was originally used for chess.
50%UnverifiedIn highly subjective tasks, pairwise comparison is gradually replacing absolute scoring as the mainstream evaluation paradigm, and its statistical results can be directly fed into the Bradley-Terry model or an Elo system to output a globally transitive ranking.
50%UnverifiedKnowledge work benchmarks are typically Elo-based and used to measure a model's overall performance on professional tasks such as code review, technical documentation, and system design.
50%UnverifiedThe ELO rating system was originally designed by physicist Arpad Elo for chess and measures ability through relative comparisons between players rather than absolute scores.
50%UnverifiedCode Arena is a competitive arena focused on front-end code generation that uses blind user voting to produce Elo scores.
50%UnverifiedChatbot Arena uses real-user blind comparative evaluation, having users simultaneously converse with two anonymous models and vote, with rankings determined by an Elo scoring system.
50%UnverifiedLMSYS Chatbot Arena is an evaluation platform led by UC Berkeley that uses an Elo rating system similar to chess, allowing real users to vote in pairwise comparisons without knowing the identities of the models.
50%UnverifiedThe Elo system assumes capability is a one-dimensional scalar and cannot reflect differences in model performance across various task dimensions; LMSYS subsequently introduced a category-based ranking mechanism to address this shortcoming.
50%UnverifiedThe Elo system is based on the Bradley-Terry model, which assumes that the probability of player A defeating player B depends only on the difference between their ratings, specifically as a logistic function.
50%UnverifiedThe LMSYS team typically requires each model to accumulate at least several thousand comparative votes before considering its Elo rating statistically significant.
50%UnverifiedWhen the number of match samples is sufficiently large, Elo ratings converge to the maximum likelihood estimate.
50%UnverifiedStockfish和Leela两类引擎的ELO评分均远超3500，超过任何人类棋手
50%