SDFT
Self-Distillation Fine-Tuning, which introduces a self-distillation objective during the fine-tuning stage so that the student model fits both the task data and the teacher's soft labels, using an EMA teacher to provide a stable supervision signal.
Timeline (last 90 days)
SDFT refers to Self-Distillation Fine-Tuning, that is, introducing a self-distillation objective during the fine-tuning stage, with the teacher and student evolving synchronously, which falls under the category of online self-distillation.
The three main methods of self-distillation are Self-Distillation Fine-Tuning (SDFT), Self-Distillation Preference Optimization (SDPO), and on-policy self-distillation.
All Facts (2)
SDFT refers to Self-Distillation Fine-Tuning, that is, introducing a self-distillation objective during the fine-tuning stage, with the teacher and student evolving synchronously, which falls under the category of online self-distillation.
50%UnverifiedThe three main methods of self-distillation are Self-Distillation Fine-Tuning (SDFT), Self-Distillation Preference Optimization (SDPO), and on-policy self-distillation.
50%