Unverified50% confidenceTradeoffExact time
RLHF carries the risk of reward over-optimization, where models may learn to satisfy the surface preferences of human annotators rather than genuinely improving accuracy or safety.
1
Sources
50%
Confidence
Long-term
Relevance
9/17/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedTraditional RLHF methods have a fundamental limitation of 'reward hacking,' where models may learn to perform well in evaluation scenarios but fail in out-of-distribution scenarios75% similarUnverified经过RLHF训练的模型在面对精心设计的越狱提示时往往能够绕过安全防护输出有害内容72% similarUnverified不同厂商在RLHF阶段对安全性与有用性的权重取舍不同，过于强调规避风险会使模型给出保守、模糊、附带免责声明的回答70% similarUnverifiedThe main challenges of RLHF include high human annotation costs, the possibility of the reward model being fooled (reward hacking), and a complex process that is difficult to reproduce.70% similarVerifiedSome models carry the risk of over-optimization toward test sets, causing leaderboard scores to deviate from real-world performance (benchmark saturation).66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/929138API
curl https://kongchang.com/api/v1/knowledge/claims/929138MCP
get_claim(id=929138)