In RLHF reward model training, annotators tend to penalize candid responses when dealing with emotionally sensitive topics, even when those responses are more accurate at the factual level.