BashArena
A conventional control evaluation setting used to predict AI deployment safety, but with relatively large error when extrapolated to real deployments.
Timeline (last 90 days)
In the LinuxArena experiments, the conventional control setup BashArena predicted deployment safety with a root mean square error (RMSE) of 10.1 percentage points (at a 5% audit budget and under five monitor conditions).
Simulated control evaluations (SCEs) predicted safety with a root mean square error of 3.6 percentage points (at a 5% audit budget and under five monitor conditions), compared with 10.1 percentage points for BashArena.
The KL divergence between SCEs and real execution is more than ten times lower than that of BashArena, indicating that the simulated environment closely matches the real environment in behavior distribution.
Attack strategies discovered through simulation also improve monitor performance on held-out LinuxArena attacks, outperforming BashArena and roughly matching the improvements brought by real execution strategies.
All Facts (4)
In the LinuxArena experiments, the conventional control setup BashArena predicted deployment safety with a root mean square error (RMSE) of 10.1 percentage points (at a 5% audit budget and under five monitor conditions).
50%UnverifiedSimulated control evaluations (SCEs) predicted safety with a root mean square error of 3.6 percentage points (at a 5% audit budget and under five monitor conditions), compared with 10.1 percentage points for BashArena.
50%UnverifiedThe KL divergence between SCEs and real execution is more than ten times lower than that of BashArena, indicating that the simulated environment closely matches the real environment in behavior distribution.
50%UnverifiedAttack strategies discovered through simulation also improve monitor performance on held-out LinuxArena attacks, outperforming BashArena and roughly matching the improvements brought by real execution strategies.
50%