Gauntlet 7.2
An agentic coding benchmark developed by LeanZero, based on tasks involving the construction of payment applications. It is used to evaluate the agentic programming capabilities of AI models.
Core Facts
Timeline (last 90 days)
The dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering the three leaderboards Gauntlet 7.2, Forge 1.0, and Forge 2.0.
The Gauntlet 7.2 benchmark requires models to build a payments app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.
Each row of data on the Gauntlet and Forge 1.0 leaderboards comes from only a single run, run-to-run noise genuinely exists, and the author suggests not over-interpreting when scores differ by only a few percent.
The dataset's cost field is incomplete: only one row in Gauntlet records cost; the Forge 1.0 figures are lower bounds; most Forge 2.0 figures are complete.
DeepSeek-V4.1-Flash-1017 scores 77.0 on DeepSWE v1.1, surpassing Claude Opus 5's 74.0.
On the Gauntlet 7.2 benchmark, Haiku scored 0.661 and Luna scored 0.4797.
All Facts (5)
DeepSeek-V4.1-Flash-1017 scores 77.0 on DeepSWE v1.1, surpassing Claude Opus 5's 74.0.
70%UnverifiedThe dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering the three leaderboards Gauntlet 7.2, Forge 1.0, and Forge 2.0.
50%UnverifiedThe Gauntlet 7.2 benchmark requires models to build a payments app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.
50%UnverifiedEach row of data on the Gauntlet and Forge 1.0 leaderboards comes from only a single run, run-to-run noise genuinely exists, and the author suggests not over-interpreting when scores differ by only a few percent.
50%UnverifiedThe dataset's cost field is incomplete: only one row in Gauntlet records cost; the Forge 1.0 figures are lower bounds; most Forge 2.0 figures are complete.
50%