ReLE Chinese LLM Benchmark: Rankings and Defect Library for 374 Models

A Continuously Updated Chinese LLM Evaluation Project

As competition in the large language model space heats up, objectively measuring how well different models perform in Chinese-language scenarios has become a shared concern among developers and researchers. The open-source GitHub project jeinlee1991/chinese-llm-benchmark offers a continuously updated answer — the ReLE benchmark.

This project focuses on evaluating Chinese AI large language models and currently covers 374 models, providing both a leaderboard and a defect library with over 2 million entries. Thanks to this positioning, the project has earned nearly 6,000 Stars and over 240 Forks on GitHub, reflecting strong community interest.

Covering Both Commercial and Open-Source Models

One of ReLE's greatest strengths is its breadth of coverage. It brings both mainstream commercial and open-source models under the same evaluation framework, making cross-model comparisons straightforward.

Commercial Models

On the closed-source commercial side, the project covers leading products from around the world, including international models like ChatGPT, GPT-5.4, Google Gemini-3.1-Pro, and Claude-4.6, as well as domestic Chinese models such as ERNIE-X1.1, ERNIE-5.0, qwen3.6-max, qwen3.6-plus, Baichuan, iFlytek Spark, and SenseChat. Placing both global and Chinese top-tier models in the same Chinese-language evaluation context provides meaningful insight into how domestic models are closing the gap.

Open-Source Models

The open-source lineup is equally impressive, featuring models such as step3.5-flash, kimi-k2.6, ernie4.5, MiniMax-M2.7, deepseek^Ad-v4, Qwen3.6, llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, gemma4, and mistral. For teams looking to deploy locally or build on top of existing models, this open-source leaderboard can help them identify the best-performing base model within their compute budget.

Beyond Rankings: The Unique Value of a 2-Million-Entry Defect Library

What sets ReLE apart from evaluation projects that simply publish score rankings is that it also makes available a defect library containing over 2 million entries.

A leaderboard tells you which model is stronger; a defect library tells you where and why a model fails. This kind of data is extremely valuable for researchers analyzing model capability boundaries, reproducing failure cases, and making targeted improvements. For teams working on fine-tuning and alignment, a systematically organized collection of failure examples is often far more actionable than a generic overall score.

From a community standpoint, making defect data publicly available means that anyone can conduct secondary research based on these real-world error cases without having to build a test set from scratch — significantly lowering the barrier to entry for LLM evaluation research.

How to Interpret Third-Party Benchmarks

It's worth maintaining a measured perspective: every benchmark comes with its own methodological assumptions and coverage limitations. Models iterate rapidly, so a leaderboard's relevance depends on continuous updates. Additionally, Chinese language capability spans multiple dimensions — comprehension, reasoning, generation, safety, and more — making it difficult for a single ranking to fully capture a model's strengths and weaknesses.

As such, ReLE is best treated as a useful starting point for model selection rather than the sole basis for decision-making. Its real advantage lies in the combination of broad coverage + continuous updates + open defect data — helping everyday users quickly understand the model landscape while giving professional researchers reusable analytical material.

For developers keeping tabs on the progress of Chinese large language models, this is an open-source resource well worth bookmarking and revisiting regularly.