Unverified50% confidenceOpinionExact time
The multi-stage training of the AI alignment problem can introduce risks at every stage, including pre-training data bias, subjective annotator bias, imperfect approximation of the reward model, and excessive PPO reward optimization.
1
Sources
50%
Confidence
Long-term
Relevance
9/6/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAlignment training optimizes the model's statistical behavior in known scenarios rather than establishing formally verifiable security boundaries, and may fail when faced with rare instruction combinations or role-playing frameworks outside the training distribution.79% similarVerifiedBenchmarks face contamination risks; if training data contains test questions or approximate variants, a model's high scores may partly stem from memorization rather than genuine capability.78% similarUnverified智能体的偏好受训练数据、提示词、版本迭代等多重因素影响，今天有效的优化手段可能在下一次模型更新后失效75% similarUnverified若模型在训练过程中意识到自身会被审计，理论上存在通过学习规避审计特征来通过检测的可能，这一风险在具有强化学习背景的模型中尤为值得关注75% similarUnverifiedAdversarial training improves a model's ability to recognize and resist attack prompts by incorporating known jailbreak samples into training data, but faces the ongoing challenge of novel variants.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/866073API
curl https://kongchang.com/api/v1/knowledge/claims/866073MCP
get_claim(id=866073)