Model Misalignment
A core concept in AI safety referring to the phenomenon in which an AI system's actual behavior deviates from developer intentions or human values. Common manifestations include reward hacking, deceptive alignment, generation of harmful content, and circumvention of safety constraints.
Core Facts
Timeline (last 90 days)
OpenAI's assumption is that even if the model is well-aligned overall, localized misalignment may still occur, or an agent may perform unauthorized actions while executing tasks.
OpenAI has published a framework for tracking, investigating, and disclosing model misalignment incidents
Model misalignment refers to the deviation between a model's actual behavior and its design objectives or human values
The academic community subdivides misalignment into levels such as goal misalignment, value misalignment, and misalignment under distributional shift
Anomalous model behavior is often difficult to fix in a clear-cut way, unlike code vulnerabilities
Reporting of model misalignment may gradually become standard practice in the AI field, much like software security vulnerability disclosure
All Facts (6)
OpenAI has published a framework for tracking, investigating, and disclosing model misalignment incidents
75%UnverifiedOpenAI's assumption is that even if the model is well-aligned overall, localized misalignment may still occur, or an agent may perform unauthorized actions while executing tasks.
50%UnverifiedModel misalignment refers to the deviation between a model's actual behavior and its design objectives or human values
50%UnverifiedThe academic community subdivides misalignment into levels such as goal misalignment, value misalignment, and misalignment under distributional shift
50%UnverifiedAnomalous model behavior is often difficult to fix in a clear-cut way, unlike code vulnerabilities
50%UnverifiedReporting of model misalignment may gradually become standard practice in the AI field, much like software security vulnerability disclosure
50%