Testing AI Models with Go: One Move Reveals Seven Core Capabilities

Why Go? Seeing Past the Scores to Real Model Performance

Leaderboard scores for large language models often look impressive — but what do those numbers actually tell us? A Bilibili content creator raised a sharp question: the test questions circulating online have very likely been seen by models during training. Looking at scores alone, you can't tell whether a model truly understands something or has simply memorized the answer.

He chose Go as his lens, with one core purpose: to observe how models respond when the situation suddenly changes. Go is naturally suited to this goal. Stones can't be taken back once played, and your opponent won't cooperate with your plans. A position that felt secure one move ago can transform completely the next. This kind of dynamic, adversarial context is exactly the blind spot that static Q&A tests fail to cover.

Building on this idea, he launched a project called CXBench, with the goal of systematically evaluating large model capabilities through Go.

---

Data contamination is one of the central challenges facing current LLM evaluation frameworks. Since mainstream training datasets are typically scraped from the internet — and most benchmark question banks come from the same public sources — models may have already "seen" the standard answers to these questions during pretraining. This means high scores don't reliably reflect reasoning ability; they may reflect nothing more than retrieval from training memory. The research community has proposed various countermeasures: private question banks, dynamically generated problems, and open-ended tasks requiring multi-step reasoning. Go falls squarely in that last category — every board position is a unique combinatorial state, and the answer must be computed rather than retrieved from a corpus.

Behind Every Move: Seven Capabilities Unpacked

Go may look like placing stones, but every decision involves multiple layers of capability. Through hands-on testing, the author gradually identified seven dimensions worth examining.

State Tracking: Does the Model's Assessment Keep Up with the Board?

The most fundamental dimension is state tracking. Suppose a black stone has only one liberty left — one more white move and it's captured. Did the model notice this change? If it continues analyzing based on the previous step, the problem is exposed: the board has changed, but the model's judgment hasn't updated.

Local vs. Global Trade-offs

But you can't just focus on wherever the fire is burning. Suppose a few white stones on the left look ripe for capturing, while a large territory on the right is waiting to be claimed. Should you take them? This tests local versus global trade-offs. The ideal approach is to survey the whole board first — don't capture two stones only to find your opponent has claimed all the big points while you weren't looking.

Long-range Planning and Branch Analysis

The author then probes further: if you play here, how will your opponent respond? What about two or three moves down the line? This tests long-range planning — focusing only on the immediate move is a recipe for being outplayed.

He then asks the model to consider an alternative line, looking for a better option — this is branch analysis. And those lines can't be built purely on wishful thinking; the opponent's counterplay must be factored in. This layer is adversarial reasoning. With a few stones on the board, it becomes immediately clear which move the model actually commits to.

Error Recovery: What Happens After a Mistake?

The author is especially interested in what happens after a bad move — error recovery. Once a stone is placed, it can't be taken back. Stubbornly sticking to the original plan at that point usually makes things worse. What he wants to see is a model that can acknowledge: "I missed that earlier — I need to switch approaches now." That's more valuable than smoothing over the mistake from start to finish.

---

Error recovery maps to a broader concept in AI evaluation: robustness and self-correction. An ideal reasoning system should not only perform consistently in favorable conditions — it should also be able to identify contradictions and adjust strategy when intermediate steps go wrong, rather than propagating and amplifying errors down the reasoning chain. A substantial body of research shows that current mainstream LLMs have notable weaknesses here. Once a model forms an incorrect premise early in its reasoning, subsequent steps tend to "rationalize" that premise for consistency rather than backtrack and correct it. This pattern is especially pronounced in tasks that require continuous state updates — like board games or mathematical proofs — because any single error contaminates all subsequent judgments that depend on that state.

Counterintuitive Findings from the Tests

The actual testing process yielded a number of surprising results.

One typical pattern: given the same position, asking the model to think two moves further ahead doesn't necessarily improve performance. Sometimes the analysis gets longer, but the final move it selects is actually worse. This suggests that "more reasoning" does not equal "better decisions."

Another revealing case was uncertainty management. In one instance, the model claimed 95% confidence in a move — but scored zero by the evaluation rubric. It not only played the wrong move, it had no idea just how uncertain it should have been. A confident tone is not the same as actually seeing the position clearly. So in addition to checking correctness, the author specifically examines whether models can acknowledge their own uncertainty.

There were even cases where a model thought at length and still couldn't commit to a single move — yet when asked to just play immediately, it gave an answer quickly. That answer wasn't great either. In the author's view, both outcomes are worth recording, because what he's waiting for isn't a lengthy analysis — it's a model that has genuinely worked out this one move.

The Design Philosophy Behind CXBench

CXBench makes several pragmatic design choices. It uses a smaller 9×9 board, reducing complexity while preserving the adversarial dynamic. The test format is flexible: sometimes the model is asked where to play, sometimes it must identify a threat from the opponent, and sometimes the position is pushed further along to see whether the model can handle a sustained sequence of moves.

To avoid the "seen-this-before" problem, positions don't necessarily come from existing game records — the author can generate novel board states and use a dedicated Go engine to help verify answer quality. This effectively reduces the inflated scores that data contamination can produce.

The core evaluation question is summarized in one sentence: did the model fail to see the danger, or did it see it but miscalculate afterward? Distinguishing between these two failure modes is far more meaningful than recording a single aggregate score.

---

The 9×9 Go board reduces the move space from 361 intersections (on a standard 19×19 board) to 81, dramatically shortening game length and branching complexity — while still preserving Go's core mechanics: captures, liberties, life-and-death reading, and strategic trade-offs that require global awareness. This format is common in teaching and introductory play, sitting at a useful balance point: complex enough to expose reasoning flaws, but not so combinatorially explosive that testing becomes unmanageable. Using a dedicated Go engine (such as the open-source KataGo or GNU Go) to help verify answer quality is also standard practice — these programs can precisely calculate win-rate changes for each move in a given position, providing objective calibration for otherwise subjective judgments.

Beyond High Scores, Ask One More Question

The author also acknowledges openly: he wouldn't conclude that a model is capable of everything just because it plays Go well. Go simply provides an intuitive vantage point — things are changing, the opponent is moving, can the model still reason through this one move clearly?

This perspective is equally useful for everyday users. The next time you see a model break a leaderboard record, it's worth asking: put it in a different situation — does it still hold up? Consistency and adaptability beyond the benchmark are what actually determine whether a model is reliable.