Chatbot Arena
An AI model evaluation platform created by the LMSYS team at the University of California, Berkeley. It employs a blind pairwise comparison mechanism in which user preference votes generate ELO-based leaderboard rankings, making it one of the most authoritative general-purpose AI benchmarks.
Core Facts
Chatbot Arena, operated by LMSYS, ranks models through blind evaluation votes by real users on anonymous models, using an ELO rating system derived from chess.
Zheng et al. (2023) systematically proposed the LLM-as-a-Judge method in their research on MT-Bench and Chatbot Arena.
Chatbot Arena is a crowdsourced live evaluation platform operated by LMSYS, where users converse with two anonymous models simultaneously and vote for the better one.
Timeline (last 90 days)
LMArena (formerly Chatbot Arena) is a research project from UC Berkeley and other institutions, originally used to evaluate the relative capabilities of large language models in a crowdsourced way.
LLM-as-judge correlates with Arena at about 98%, costs less than 10 dollars and returns results within 3 minutes, but it has length preference and preference for its own style.
Chatbot Arena is the most trusted evaluation signal, but it is slow, expensive, and biased toward technical users.
Chatbot Arena uses the Elo rating system to convert a large number of voting results into model rankings, and Elo was originally used for chess.
Chatbot Arena (LMSYS) partially mitigates the bias limitations of LLM-as-a-judge through large-scale blind human comparisons, but its cost makes it difficult to integrate into developers' daily workflows
The LLM-as-a-Judge approach is widely adopted in benchmarks such as MT-Bench and Chatbot Arena.
LMArena's predecessor is related to Chatbot Arena.
Gemini 4.0 Pro has begun anonymous grey-scale testing (ghost testing) in the large model competition arena, Arena.
Major labs are eager to warm up new models in Arena in anonymous variant form before official release, in order to collect real user feedback and accumulate Elo scores.
Ghost testing is an anonymous grey-scale validation method used before a large model's official release, in which the model participates in blind testing in Arena under the identity of an anonymous variant.
30 more timeline events
All Facts (20)
Chatbot Arena, operated by LMSYS, ranks models through blind evaluation votes by real users on anonymous models, using an ELO rating system derived from chess.
85%VerifiedZheng et al. (2023) systematically proposed the LLM-as-a-Judge method in their research on MT-Bench and Chatbot Arena.
80%VerifiedChatbot Arena is a crowdsourced live evaluation platform operated by LMSYS, where users converse with two anonymous models simultaneously and vote for the better one.
75%VerifiedArena (Chatbot Arena) is an AI model evaluation platform created by the LMSYS team at UC Berkeley, which uses a blind battle mechanism to generate an ELO score leaderboard.
75%VerifiedLMSys Chatbot Arena was created by the LMSYS organization at UC Berkeley and has collected more than one million human votes since its launch in 2023.
75%VerifiedLM Arena (formerly Chatbot Arena) is an anonymous head-to-head large model evaluation platform developed by a team at the University of California, Berkeley.
75%VerifiedThe Chatbot Arena project was launched by the LMSYS team at UC Berkeley in 2023.
70%VerifiedThe LLM-as-a-Judge evaluation paradigm was first applied at scale by the LMSYS team in MT-Bench and Chatbot Arena.
65%VerifiedThe Chatbot Arena project was launched by the LMSYS team at UC Berkeley in 2023, accumulating human preference data through blind user voting and deriving an Elo rating leaderboard.
65%VerifiedChatbot Arena is maintained by LMSYS Org and uses the ELO rating system.
65%UnverifiedChatbot Arena was introduced by a research team at UC Berkeley in 2023 and is one of the earliest platforms to implement large-scale LLM battle evaluation
90%UnverifiedAs of 2024, Chatbot Arena has accumulated over one million human votes
85%UnverifiedWebDev Arena draws on the human preference voting mechanism of Chatbot Arena, ranking participants by ELO score through blind-test voting.
70%UnverifiedChatbot Arena uses an ELO rating mechanism to determine model rankings through large numbers of anonymous head-to-head matches.
60%UnverifiedArena-type platforms use a pairwise blind-testing mechanism in which users vote for the better response without knowing the identity of the models.
60%UnverifiedThe LLM-as-judge approach was first adopted at scale in MT-Bench and Chatbot Arena.
60%UnverifiedChatbot Arena works by having a user submit a question, the system randomly selects two anonymous models to generate responses, and the user votes for the better answer without knowing which model produced it
60%UnverifiedLMArena (formerly Chatbot Arena) is a research project from UC Berkeley and other institutions, originally used to evaluate the relative capabilities of large language models in a crowdsourced way.
50%UnverifiedChatbot Arena is the most trusted evaluation signal, but it is slow, expensive, and biased toward technical users.
50%UnverifiedLLM-as-judge correlates with Arena at about 98%, costs less than 10 dollars and returns results within 3 minutes, but it has length preference and preference for its own style.
50%