Control Evaluation
An AI safety evaluation paradigm that assumes a model may harbor malicious intent or be exploited by an attacker, using a red team to induce dangerous operations and a blue team to deploy safeguards, measuring whether the safeguards can successfully prevent harmful behavior under a given attack strength.
Timeline (last 90 days)
Control evaluation is a systematic method used to quantify whether a safety protocol can remain safe under adversarial attack; the typical process assumes that learning at deployment time or an external attacker will cause some perturbation, the red team iteratively attacks, and if the protocol can still block dangerous actions it is considered to have a sufficient safety margin.
Under continual learning, the policy may learn a large amount of evasion knowledge through online RL, with a perturbation magnitude far exceeding what control evaluation assumes, causing protocols that originally passed evaluation to lose their meaning.
All Facts (2)
Control evaluation is a systematic method used to quantify whether a safety protocol can remain safe under adversarial attack; the typical process assumes that learning at deployment time or an external attacker will cause some perturbation, the red team iteratively attacks, and if the protocol can still block dangerous actions it is considered to have a sufficient safety margin.
50%UnverifiedUnder continual learning, the policy may learn a large amount of evasion knowledge through online RL, with a perturbation magnitude far exceeding what control evaluation assumes, causing protocols that originally passed evaluation to lose their meaning.
50%