McNemar's Test
A statistical test method for paired binary classification data, suitable for comparing whether the performance difference between two models on the same batch of samples is significant.
Timeline (last 90 days)
The McNemar test p-values for the comparison of EMBER and ORBIT with the LLM baseline are 1.3e-9 and 1.5e-6, respectively, which is statistically significant.
Under a paired design, any routing jitter, temperature sampling or re-running of unstable endpoints degrades paired data into unpaired data, invalidating the premises of the McNemar test.
The comparison used Wilson confidence intervals and a paired McNemar test with Holm-Bonferroni correction.
After decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
Determining whether a model truly outperforms the baseline usually requires larger samples (hundreds to thousands), paired tests (such as the McNemar test), and repeated experiments under multiple random seeds.
All Facts (5)
The McNemar test p-values for the comparison of EMBER and ORBIT with the LLM baseline are 1.3e-9 and 1.5e-6, respectively, which is statistically significant.
50%UnverifiedUnder a paired design, any routing jitter, temperature sampling or re-running of unstable endpoints degrades paired data into unpaired data, invalidating the premises of the McNemar test.
50%UnverifiedThe comparison used Wilson confidence intervals and a paired McNemar test with Holm-Bonferroni correction.
50%UnverifiedAfter decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
50%UnverifiedDetermining whether a model truly outperforms the baseline usually requires larger samples (hundreds to thousands), paired tests (such as the McNemar test), and repeated experiments under multiple random seeds.
50%