Alignment Failure
An alignment problem in which model outputs deviate from human intent, typically occurring during the training phase. It differs from runtime behavioral drift, in which an agent during inference and deployment produces behavior due to sparse rewards or ambiguous instructions.
Timeline (last 90 days)
The response to the risk of alignment failure lies in interpretability research, value alignment techniques, and systematic red-teaming.
The "capability risk" camp, represented by Geoffrey Hinton and Yoshua Bengio, believes that current large models have already exhibited emergent capabilities that are difficult to fully explain, and that engineering measures alone cannot eliminate the fundamental risk of alignment failure.
All Facts (2)
The response to the risk of alignment failure lies in interpretability research, value alignment techniques, and systematic red-teaming.
50%UnverifiedThe "capability risk" camp, represented by Geoffrey Hinton and Yoshua Bengio, believes that current large models have already exhibited emergent capabilities that are difficult to fully explain, and that engineering measures alone cannot eliminate the fundamental risk of alignment failure.
50%