Forge 1.0
An agentic coding benchmark developed by LeanZero, based on tasks that involve building Atlassian Forge applications in offline environments. It is used to evaluate the agentic programming capabilities of AI models.
Timeline (last 90 days)
The dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering the three leaderboards Gauntlet 7.2, Forge 1.0, and Forge 2.0.
The Gauntlet 7.2 benchmark requires models to build a payments app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.
Each row of data on the Gauntlet and Forge 1.0 leaderboards comes from only a single run, run-to-run noise genuinely exists, and the author suggests not over-interpreting when scores differ by only a few percent.
The dataset's cost field is incomplete: only one row in Gauntlet records cost; the Forge 1.0 figures are lower bounds; most Forge 2.0 figures are complete.
On the Forge 1.0 benchmark, Haiku scored 0.9661 and Luna scored 0.395, with Haiku trailing Claude Opus 5.5 by only 0.0106.
Luna never imported historical changes during the Forge test—none of the 62 historical changes were included—triggering a backfill check labeled as critical, which multiplied the entire round's score by 0.6, changing it from 0.6583 to 0.395.
All Facts (4)
The dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering the three leaderboards Gauntlet 7.2, Forge 1.0, and Forge 2.0.
50%UnverifiedThe Gauntlet 7.2 benchmark requires models to build a payments app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.
50%UnverifiedEach row of data on the Gauntlet and Forge 1.0 leaderboards comes from only a single run, run-to-run noise genuinely exists, and the author suggests not over-interpreting when scores differ by only a few percent.
50%UnverifiedThe dataset's cost field is incomplete: only one row in Gauntlet records cost; the Forge 1.0 figures are lower bounds; most Forge 2.0 figures are complete.
50%