MT-Bench

Zheng et al. (2023) systematically proposed the LLM-as-a-Judge method in their research on MT-Bench and Chatbot Arena.

Mainstream large model evaluation benchmarks include MMLU, HumanEval/SWE-bench, MATH, and GPQA

The LLM-as-a-Judge evaluation paradigm was first applied at scale by the LMSYS team in MT-Bench and Chatbot Arena.

Artificial Analysis Intelligence Index整合了MMLU、HumanEval、MATH、GPQA等多个基准测试的加权结果

LLM as Judge was systematically proposed by Lianmin Zheng et al. in the MT-Bench paper (2023), with the core idea of using an independent large language model to score the output of another model as a substitute for human annotation.

MMLU, HumanEval, and MT-Bench are standardized benchmark test sets widely recognized by the AI community.

LLM-as-Judge方法由Zheng等人在2023年的MT-Bench论文中系统验证，与人类评估的一致性可达80%以上

The LLM-as-judge approach was first adopted at scale in MT-Bench and Chatbot Arena.

The experiments cover 7 English annotation tasks: HelpSteer2 and SummEval scoring, MT-Bench and Arena preferences, WMT translation quality, toxicity labeling, and OracleBench math judging.

MT-Bench is a multi-turn dialogue evaluation benchmark proposed by Zheng et al. in 2023, containing 80 multi-turn questions and 160 single-turn questions, covering eight categories including writing, role-playing, reasoning, and coding.

In 2023, the authors of MT-Bench proposed having the evaluator answer the question itself first, then using its own answer as the reference answer, which reduced the failure rate on math problems from 70% to 15%.

Current mainstream evaluation methods such as MT-Bench, AlpacaEval, and LLM-as-Judge all carry risks of circular dependency and scoring bias.

There is a distribution shift between general benchmarks such as MMLU, HumanEval, and MT-Bench and specific tasks.

The LLM-as-a-Judge approach is widely adopted in benchmarks such as MT-Bench and Chatbot Arena.

Public benchmarks (such as MMLU, HumanEval, and MT-Bench) measure a model's capabilities on academic or general tasks, but there is often a significant gap between these scores and actual business performance.

Text-based LLMs already have public benchmarks such as MMLU, HellaSwag, and MT-Bench, as well as commercial evaluation platforms like LangSmith and Braintrust, while the voice domain lacks similar standardized benchmarks.

The LLM-as-Judge paradigm was popularized by research such as MT-Bench and Chatbot Arena, and is capable of capturing semantic equivalence, logical reasoning, and factual consistency.

Commonly used comprehensive evaluation frameworks include MMLU, HumanEval, MT-Bench, and HELM, each focusing on different capability dimensions.

Evaluating actual model performance requires reference to independent benchmark results, such as MMLU, HumanEval, and MT-Bench.

Representative works of the "LLM-as-a-judge" paradigm include MT-Bench/Chatbot Arena by Zheng et al., who found that GPT-4 as a judge achieves over 80% agreement with human preferences.