Artificial Analysis Intelligence Index v5 Major Update: Open-Source vs. Closed-Source Gap Widens, Evaluation Credibility Questioned

The third-party model evaluation leaderboard Artificial Analysis has released version v5 of its Intelligence Index. The organization bills it as a "major upgrade" and claims the rankings are now more accurate. But the community discussion sparked by this update is far more complex than the rankings themselves: on one hand, the gap between open-weight and closed-source models has widened significantly; on the other, a substantial portion of the new version's evaluations are based on private test sets that are nearly impossible to reproduce, once again raising questions about the leaderboard's value as an independent reference.

What Changed in v5: A Wider Gap and Irreproducibility

The most direct change shows up in the ranking data. According to the official announcement, the open-weight gap has expanded from 3 points to 12 points. Leading the open-source camp is Kimi K3.1 Pro with a score of 46, but its weights have not yet been released. Among models with released weights, DeepSeek^Ad V4.1 Flash 1017 (max tier) and GLM-5.3 (max tier) lead the open-source side at 32 points—26 points behind the top spot, and even below Claude Haiku 5.5 (max tier, 36 points).

Another key design decision is the composition of the evaluation. In v5, roughly 60% of the index score comes from evaluations using private test sets or withheld answers, meaning external researchers cannot reproduce these results. For a leaderboard long used as a selection reference, this is a substantive methodological change, and the official description is quite brief—it only states the proportion and purpose, without detailing the verification mechanism.

Open-weight models refer to models whose parameters are publicly available, allowing anyone to download and deploy or fine-tune them locally—such as the DeepSeek, GLM, and Llama series. These stand in contrast to closed-source models offered only via API. The core difference lies in auditability and controllability: open-weight inference can be reproduced locally, facilitating security review and secondary development, whereas closed-source model behavior can only be observed through the interface. Evaluation leaderboards typically compare both types of models on the same scale, but open models can exist in an intermediate state of "weights unreleased"—the model is on the leaderboard, yet the community cannot verify it independently, creating a gap between ranking and availability.

Private test sets are question banks where the evaluator does not disclose the questions or standard answers, typically sampled from internally held data or drawn from questions contractually prohibited from disclosure. Their main purpose is to prevent model vendors from targeting public question banks (i.e., gaming the leaderboard) and to reduce inflated scores caused by training data contamination. The cost is that outsiders cannot rerun the experiments, so evaluation results depend entirely on the publisher's process discipline. When the private portion accounts for a high share of the total score, the leaderboard's verifiability drops noticeably.

What the Community Is Questioning

Two types of typical feedback have emerged in Reddit discussions. The first concerns the mismatch between real-world experience and rankings: some point to the example of GPT-Terra-6.1 ranking above Fable-5.1, arguing this contradicts their actual experience and questioning whether the index's ordering logic is truly more "accurate." Such skepticism is not isolated—nearly every major leaderboard overhaul prompts a tug-of-war between the experiential camp and the metrics camp.

The second type of criticism is more methodological. While 60% of scores coming from private test sets can help prevent leaderboard gaming and data contamination to some degree, the cost is reduced transparency. When outsiders cannot verify, the leaderboard's credibility depends heavily on the publisher's self-restraint—precisely the hardest thing for third-party evaluations to establish.

It should be noted that these discussions currently come mainly from community users' personal experiences and inferences, not verified conclusions. Whether the open-vs-closed gap has actually been "widened" or was previously underestimated—v5's answer is the latter, but it lacks reproducible evidence to support this.

Benchmark contamination refers to evaluation questions entering a model's training corpus in some form, causing the model to appear to "have seen the answer" rather than genuinely possessing the capability. As web corpora are scraped at scale, public question banks can easily be included in training sets, intentionally or not, inflating scores. Common mitigation measures in the industry include: regularly rotating question banks, using private held-out sets, and rewriting or dynamically generating questions. This is the direct motivation behind mainstream evaluation organizations' migration to private test sets, but as noted in this article, improved contamination resistance and reduced external verifiability often occur simultaneously.

Is This Leaderboard Still Worth Following?

From a practical standpoint, the Artificial Analysis index remains an effective entry point for quickly understanding models' relative positions, especially since it covers both open and closed models across different reasoning tiers (such as max) with high information density. But the changes in v5 suggest users need to adjust how they interpret it: treat the leaderboard as a directional reference rather than an absolute ordering.

Specifically, what's worth watching is the relative gap within the same camp—for example, among open models with released weights, DeepSeek V4.1 Flash and GLM-5.3 are nearly tied, and this kind of information is more valuable for selection. Cross-camp absolute score comparisons, given that 60% of evaluations are irreproducible, need to be judged alongside actual testing in your own scenarios. For teams using the leaderboard as a basis for procurement or technology selection, a safer approach is to first use the leaderboard to narrow the candidate pool, then run small-scale validation with your own task set.

This update also reflects an industry trend: mainstream evaluation organizations are migrating toward private test sets to counter data contamination and targeted optimization. While this path improves resistance to leaderboard gaming, it also weakens the space for external oversight. The "authority" of leaderboards in the future may come more from long-term consistency than from the persuasiveness of any single update.

Frequently Asked Questions

Who is the top open-source model in v5?

Kimi K3.1 Pro, with a score of 46, but its weights have not yet been released. Among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead, both at 32 points.

Why are v5's results said to be irreproducible?

Because about 60% of the index score comes from evaluations using private test sets or withheld answers. External researchers cannot access the questions and answers, so they naturally cannot rerun the verification.

What exactly is the gap between open and closed models?

According to the official announcement, the gap has widened from 3 points to 12 points, with open models that have released weights trailing the top spot by 26 points.

One-line Summary

Artificial Analysis Intelligence Index v5 widens the open-vs-closed gap, but 60% of evaluations being irreproducible raises questions.