KTO
An alignment method variant in the DPO family that, drawing on prospect theory, relaxes the requirement for paired preference data and can directly use unpaired feedback data.
Timeline (last 90 days)
LLaMA-Factory integrates training algorithms such as incremental pre-training, instruction-supervised fine-tuning (SFT), reward model training (Reward Model), PPO, DPO, KTO, and ORPO.
KTO only requires labeling "good/bad" rather than pairwise comparisons, so the data threshold is lower.
"Do Factuality Methods Make LLMs More Factual?" completed the first independent reproduction of five factuality alignment methods (PPO, GRPO, Dr. GRPO, DPO, KTO), using a unified benchmark and evaluator.
None of the factuality alignment methods is fully reproducible, and none beats the base model; the reward mechanism ultimately optimizes format or length rather than factuality itself.
DPO variants such as KTO, ORPO and SimPO further relax the requirements for paired data or reference models.
The structured feedback produced by CriticGen can be used to automatically generate preference pairs in alignment training methods such as DPO or KTO, using the original answer as the rejected sample and the revised answer as the chosen sample.
All Facts (6)
LLaMA-Factory integrates training algorithms such as incremental pre-training, instruction-supervised fine-tuning (SFT), reward model training (Reward Model), PPO, DPO, KTO, and ORPO.
50%UnverifiedKTO only requires labeling "good/bad" rather than pairwise comparisons, so the data threshold is lower.
50%Unverified"Do Factuality Methods Make LLMs More Factual?" completed the first independent reproduction of five factuality alignment methods (PPO, GRPO, Dr. GRPO, DPO, KTO), using a unified benchmark and evaluator.
50%UnverifiedNone of the factuality alignment methods is fully reproducible, and none beats the base model; the reward mechanism ultimately optimizes format or length rather than factuality itself.
50%UnverifiedDPO variants such as KTO, ORPO and SimPO further relax the requirements for paired data or reference models.
50%UnverifiedThe structured feedback produced by CriticGen can be used to automatically generate preference pairs in alignment training methods such as DPO or KTO, using the original answer as the rejected sample and the revised answer as the chosen sample.
50%