pass@1
A standard scoring convention in code generation evaluation in which the model produces a single answer per task; a passing result scores 1 and a failing result scores 0, and the final score is the average across all tasks, measuring single-attempt delivery success rate.
Core Facts
Timeline (last 90 days)
In the P2NPCBench evaluation, 12 proprietary and open-source models achieved an average pass@1 of 35.4% on NPC problems and 63.8% on P twins, a gap of nearly 28 percentage points.
In both static prediction and dynamic tasks, the framework's pass@1 advantage over stronger interventions (such as model scaling or increased inference compute) may reverse under a larger attempt budget.
Comparing the merits of different interventions based on pass@1 alone may be misleading; a comprehensive judgment should combine performance under multiple attempt budgets.
Leaderboards usually use single-attempt (pass@1) as the standard, whereas an error-repair loop amounts to multiple sampling or self-correction; reporting repaired results mixed with single-attempt results overstates model capability and misleads technology selection.
pass@1 measures the proportion of first responses that conform to the schema, pass@repair measures the cumulative usable proportion after one repair, and reject rate measures the proportion that should be rejected.
After GRPO reinforcement learning, checkpoints using TailSFT achieved a final pass@1 improvement of up to 3.9 points.
If pass@16 is much higher than pass@1, it indicates that the model's probability distribution contains the correct answer but does not place it at the highest probability position, which is precisely the space where RL can exert its effect through reward signals.
All Facts (8)
The pass@1 metric was originally borrowed from the code generation evaluation domain (such as the HumanEval benchmark) to measure the probability of a model generating correct code on the first attempt.
75%UnverifiedIn the P2NPCBench evaluation, 12 proprietary and open-source models achieved an average pass@1 of 35.4% on NPC problems and 63.8% on P twins, a gap of nearly 28 percentage points.
50%UnverifiedIn both static prediction and dynamic tasks, the framework's pass@1 advantage over stronger interventions (such as model scaling or increased inference compute) may reverse under a larger attempt budget.
50%UnverifiedComparing the merits of different interventions based on pass@1 alone may be misleading; a comprehensive judgment should combine performance under multiple attempt budgets.
50%UnverifiedLeaderboards usually use single-attempt (pass@1) as the standard, whereas an error-repair loop amounts to multiple sampling or self-correction; reporting repaired results mixed with single-attempt results overstates model capability and misleads technology selection.
50%Unverifiedpass@1 measures the proportion of first responses that conform to the schema, pass@repair measures the cumulative usable proportion after one repair, and reject rate measures the proportion that should be rejected.
50%UnverifiedIf pass@16 is much higher than pass@1, it indicates that the model's probability distribution contains the correct answer but does not place it at the highest probability position, which is precisely the space where RL can exert its effect through reward signals.
50%UnverifiedAfter GRPO reinforcement learning, checkpoints using TailSFT achieved a final pass@1 improvement of up to 3.9 points.
50%