JevBench
A benchmark designed to evaluate the performance of System One-class reasoning models, measuring the reasoning and planning capabilities of large language models.
Core Facts
Timeline (last 90 days)
On the public JevBench benchmark (231 downloadable questions), the unmodified 4B version scored 170/231 overall, with 48/48 correct in the Easy group, 69/72 in the Original group, and only 53/111 in the Hard group, with 58 of the 61 errors coming from the Hard group.
Broken down by task family in JevBench, the accuracy was 0/15 for temporal and numerical questions and 4/19 for long-horizon strategic judgment.
Microsoft reported that Decision-1 achieved the highest accuracy in a comparison across 36 benchmarks covering nearly 150,000 questions, and claimed its P50 latency is approximately 35 times faster than GPT-6 Sol.
Winnow is the top-ranked open-source System One model on the JevBench benchmark.
A benchmark called JevBench has emerged, specifically designed to compare Jev-class decision models.
All Facts (4)
Microsoft reported that Decision-1 achieved the highest accuracy in a comparison across 36 benchmarks covering nearly 150,000 questions, and claimed its P50 latency is approximately 35 times faster than GPT-6 Sol.
65%UnverifiedOn the public JevBench benchmark (231 downloadable questions), the unmodified 4B version scored 170/231 overall, with 48/48 correct in the Easy group, 69/72 in the Original group, and only 53/111 in the Hard group, with 58 of the 61 errors coming from the Hard group.
50%UnverifiedBroken down by task family in JevBench, the accuracy was 0/15 for temporal and numerical questions and 4/19 for long-horizon strategic judgment.
50%UnverifiedA benchmark called JevBench has emerged, specifically designed to compare Jev-class decision models.
50%