Self-Play AI Scores Higher, Performs Worse Against Humans? Minecraft Swordplay Experiment Reveals Negative Correlation

In the field of game AI, it is widely accepted that agents trained via self-play perform excellently against bots. But a key question remains unresolved: how much of this ability transfers to real human opponents? A developer shared preliminary experiments on Minecraft swordplay duels on Reddit, with surprising results—the ability of self-play checkpoints to judge bot matches is significantly negatively correlated with their ability to judge human matches (Spearman ρ = −0.73, p = 0.01).

Experimental Setup: From Self-Play to Human Data

The author trained swordplay agents in a vanilla Minecraft 1v1 melee environment using a real headless server. The training pipeline included PPO league self-play (snapshots, scripted styles, Elo), behavior cloning (BC) bootstrapping, then distillation into an LSTM policy with a value head and opponent prediction head. To simulate real client latency, the bot saw opponent states delayed by 200 ms and used vanilla physics.

Human data came from the author himself: 7 duels, 35 fights, about 8,800 decisions, including exact keypresses from his own client. This constitutes a "pilot-scale" dataset, and the author acknowledges the limited sample size.

PPO league self-play refers to training multiple policy variants using Proximal Policy Optimization to form a "league" where they play against each other to improve robustness. The league typically includes historical snapshots (to prevent policy forgetting), scripted styles (e.g., aggressive or conservative rule-based opponents), and Elo-based matchmaking to keep the training environment diverse. Behavior cloning bootstrapping involves first supervised learning on human or scripted data, then fine-tuning with reinforcement learning to alleviate the cold-start problem under sparse rewards. Distillation into an LSTM means compressing knowledge from multiple league policies into a long short-term memory network with memory cells, augmented with a value head (predicting win probability or health advantage) and an opponent prediction head (predicting the opponent's next action), enabling decisions in partially observable, delayed environments.

Value Head Does Not Transfer: Stronger Against Bots, Worse Against Humans

The author defines "Judgement" as the value head's explained variance minus a baseline explained variance fitted from the policy-observed health lead (at the same moment). In bot matches, the judgement of 13 checkpoints ranged from −0.34 to +0.17, roughly on par with the baseline. But in human matches, all 13 checkpoints had negative judgement, and every 90% confidence interval was below zero.

More critically, in a duel never seen by the checkpoints, the Spearman correlation between bot-match judgement and human-match judgement across 11 checkpoints was −0.73 (p=0.01). This means models that judge more accurately in bot matches are actually worse in human matches. The author also notes that these checkpoints share lineage (non-independent) and only 5 duels were shared, requiring seed-controlled experiments for further validation.

Explained variance measures the agreement between the value head's predictions and actual returns, with a maximum of 1; 0 means equivalent to the mean baseline, and negative values indicate predictions worse than simply using the mean. The author defines judgement as "value head explained variance minus a health-lead regression baseline" to remove a simple signal: the portion of win/loss predictable solely from current health lead. If the value head cannot exceed this baseline, it has not learned additional situational judgment. Spearman correlation is a non-parametric rank correlation suitable for small samples and unknown distributions; ρ=-0.73 means the bot-judgement ranking of checkpoints is almost opposite to their human-judgement ranking, and p=0.01 is significant even with only 11 checkpoints, though the author cautions about non-independent checkpoints and only 5 shared duels, risking pseudo-replication.

The Opponent Model's "Memory Gain" Is a Baseline Artifact

The author originally used "with memory vs. erased memory" to measure whether the opponent model's memory helps read the opponent. But erasing memory during training only breaks the model. Switching to a majority-keypress baseline (the author's most frequent keypresses from other duels): 83.1% movement key accuracy. Each checkpoint's opponent head accuracy ranged from 72.6% to 81.7%, none exceeding the baseline.

For checkpoints trained on bots: memory helped in bot matches (+1 to +10 percentage points) but was harmful in human matches (−1.8 to −7.8). Only in three runs also trained on human duels did memory help on unseen human duels (+1.4 to +4.8), but only just approaching the baseline.

Specialized Player Model Works, But Memory Still Doesn't Help

The author behavior-cloned a "me" model, initialized from a frozen test policy, using leave-one-duel-out cross-validation: 2.44 vs 4.20 nats/decision, compared to a frequency baseline (movement key accuracy 90.6% vs 80.6%). Adding LSTM memory actually lost 0.035 nats ([−0.054, −0.015]), with the no-memory variant being best. Frozen self-play LSTM features were even worse. This suggests that self-play recurrent features may be actively misspecified for human opponents, not merely uninformative.

nats/decision is the cross-entropy loss in natural logarithms; lower values indicate more accurate prediction of the next decision. 2.44 vs 4.20 nats/decision means the specialized player model reduces uncertainty by about 1.76 nats compared to the frequency baseline, corresponding to perplexity dropping from about 67 to about 11—a very significant improvement. Leave-one-duel-out cross-validation means holding out an entire duel as the test set each time and training on the rest, preventing adjacent decisions within the same duel from leaking into the test set, thus more rigorously evaluating generalization. Frozen self-play LSTM features means using the recurrent layer outputs of the self-play policy as fixed features for the player model without updating them; worse performance indicates these features carry abstractions for bot combat rather than human behavior patterns, and direct reuse introduces misspecification.

Next Steps and Open Questions

The author plans a pre-registered study: against frozen probe opponents with fixed hashes, pre-fixed metrics, more players and seeds, and counterfactual replay from recorded states to test behaviors the player model flags as habits.

He raises several questions: Is health-lead regression a fair baseline for value head quality in fighting games? Is the majority-keypress plus no-memory model the correct baseline combination for opponent modeling? Is the negative correlation between bot skill and human skill known in the literature (aside from human-AI collaboration work like Overcooked)? And how to design experiments to yield meaningful results when human players are scarce?

Implications: Evaluation Design Matters More Than Training

Although small, this pilot study exposes a blind spot in self-play evaluation: metrics optimized in bot matches may systematically diverge from human match performance. A negative correlation means that selecting models by bot-match performance could pick agents that are least friendly to humans. The author's emphasis on "correct baselines" is equally critical—the majority-keypress baseline is simple yet strong, reminding us that many "gains" may just be artifacts of poorly chosen baselines.

For game AI and human-AI interaction researchers, the value of this work lies not in the certainty of its conclusions but in how it rigorously questions its own evaluation methods. The author's proactive sharing of negative results and uncertainties is a hallmark of healthy scientific practice.

(Note: This article is based on a Reddit post; the experiment is pilot-scale and conclusions require larger-sample validation.)

One-line Summary

Minecraft self-play swordplay experiment finds bot judgement significantly negatively correlated with human judgement.

Paragraph Summary

A developer trained self-play swordplay agents in vanilla Minecraft 1v1 melee and evaluated them on pilot-scale human data from 7 duels and about 8,800 decisions. All 13 checkpoints had negative value-head judgement in human matches, and the Spearman correlation between bot-match and human-match judgement was -0.73 (p=0.01)—better at judging bots means worse at judging humans. The opponent model's memory gain proved to be a baseline artifact: majority-keypress baseline accuracy reached 83.1%, and no checkpoint exceeded it. A specialized behavior-cloned player model worked (2.44 vs 4.20 nats/decision), but adding LSTM memory lost 0.035 nats. The author emphasizes correct baselines and pre-registered design, warning that selecting models by bot performance may pick the worst agents for humans.