DPO
Direct Preference Optimization (DPO) is a training algorithm for aligning large language models with human preferences. Its core idea is to implicitly incorporate a reward model into the policy optimization objective, enabling direct optimization of the language model policy on human preference data without explicitly training a separate reward model or requiring reinforcement learning sampling. It relies solely on the policy model and a reference model, serving as a simplified alternative to the RLHF framework.
Core Facts
DPO (Direct Preference Optimization) is an alignment method proposed in 2023 that can skip reward model training and directly optimize the policy model from human preference data.
RLHF trains a reward model by collecting human preference ratings, then fine-tunes the language model using reinforcement learning algorithms such as PPO; however, the reward model can be deceived by adversarial inputs and cannot enumerate all variants of harmful request expressions.
"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).
Timeline (last 90 days)
The key difference between reinforcement learning on preference data (such as RLHF and DPO) and ordinary supervised fine-tuning lies in the form of the training signal: supervised fine-tuning tells the model the correct answer, while preference learning only tells the model which output humans prefer
JOLT is not the first work to attempt to compress post-training costs; previous directions include rejection sampling fine-tuning, the direct preference optimization (DPO) family of methods, and various self-rewarding mechanisms.
RLHF, DPO, and GRPO all belong to the family of reinforcement learning (RL) post-training methods.
DPO skips the explicit reward model and directly constructs the loss function from preference pairs, making it simpler to implement and more stable to train; it is currently a lightweight alternative to RLHF.
The Recipes provided by Together cover post-training paradigms such as supervised fine-tuning, DPO, and GRPO, and come with default hyperparameters and sample data.
LLaMA-Factory integrates training algorithms such as incremental pre-training, instruction-supervised fine-tuning (SFT), reward model training (Reward Model), PPO, DPO, KTO, and ORPO.
In the DPO setting, there is only a weak correlation between the hyperparameters that minimize validation loss and the optimal hyperparameters on downstream benchmarks.
DPO hyperparameters do not transfer across different training datasets and need to be tuned separately for the target model, dataset, and evaluation objective.
The coefficient β in the DPO loss controls how far the model deviates from the reference policy; a larger β is more conservative and a smaller β is more aggressive.
A systematic study, through dense grid experiments, found that DPO's optimal learning rate shifts as model scale changes.
40 more timeline events
All Facts (20)
DPO (Direct Preference Optimization) is an alignment method proposed in 2023 that can skip reward model training and directly optimize the policy model from human preference data.
90%VerifiedRLHF trains a reward model by collecting human preference ratings, then fine-tunes the language model using reinforcement learning algorithms such as PPO; however, the reward model can be deceived by adversarial inputs and cannot enumerate all variants of harmful request expressions.
80%Verified"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).
80%VerifiedSupervised Fine-Tuning (SFT) trains the model using 'prompt-response' example pairs, teaching the model to respond in a conversational format, thereby making outputs more coherent.
75%VerifiedLanguage model alignment techniques include Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), aimed at making model behavior conform to human intentions and values
75%VerifiedLarge model training goes through three core stages: pre-training, supervised fine-tuning (SFT), and preference alignment.
75%VerifiedDPO (Direct Preference Optimization) is a simpler alternative to RLHF that has been gaining traction in recent years.
75%VerifiedAlignment algorithms such as DPO (Direct Preference Optimization) reduce training costs while improving the consistency of model performance on open-ended instructions.
70%VerifiedSafety alignment refers to the ability to train a model to refuse to cooperate with harmful requests, with common techniques including RLHF and Constitutional AI.
65%VerifiedLLM safety guardrails typically encompass three layers: RLHF during the training phase, content filters during the inference phase, and system-level prompts.
65%VerifiedRLHF is essentially a behavioral-level constraint that does not alter the model's underlying capabilities; models trained with RLHF still retain latent knowledge of the restricted behaviors.
65%VerifiedCurrent mainstream AI alignment methods include RLHF, Constitutional AI, and DPO, which internalize safety constraints into model weights during the model training phase.
65%UnverifiedEcommerce-Agent的技术路线分为三步：LoRA做领域适配、DPO做行为对齐、ReAct做能力扩展
90%UnverifiedEcommerce-Agent使用Python开发，结合了LoRA微调、DPO对齐和ReAct推理框架三项技术
90%Unverified训练SWE-bench智能体的先进方法包括拒绝采样（Rejection Sampling）、DPO（Direct Preference Optimization）以及过程奖励模型（Process Reward Model）
85%UnverifiedHermes模型使用DPO（直接偏好优化）等对齐技术进行训练
80%UnverifiedInception Labs 的创始团队由斯坦福、UCLA 和康奈尔大学的教授组成，曾主导过 Flash Attention、决策 Transformer 及直接偏好优化（DPO）等研究
80%Unverifiednpcpy的模型微调使用Beta为0.1的DPO，并用SFT迁移了Llama 3的学术风格用于科学写作场景
80%Partially Verified从预训练检查点到可商用产品需经历监督式指令微调（SFT）和基于人类反馈的强化学习（RLHF）两个后训练阶段
65%UnverifiedModel alignment uses techniques such as RLHF and DPO to make model outputs conform to human expectations, and alignment adjustments affect the output length and style of the model.
60%