SFT
Supervised Fine-Tuning (SFT) is a machine learning technique that involves training a pre-trained model on labeled instruction-response pair data in a supervised manner, enabling the model to follow instructions and generate expected outputs. In the alignment pipeline for large language models, SFT is a key step for equipping models with conversational and instruction-following capabilities. During training, loss is typically computed only on the model's response portion to guide the model toward correct response patterns.
Core Facts
Modern large language models typically go through three training stages: pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF).
The RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.
当代前沿模型的训练方法包括RLHF（基于人类反馈的强化学习）及其后继者RLAIF和过程奖励模型（PRM）
Timeline (last 90 days)
Supervised fine-tuning (SFT) is suited to narrowly defined tasks such as classification, extraction and summarization.
There are plenty of hosted services to choose from for SFT, whereas RL currently lacks a genuinely excellent hosted solution.
SFT forces the model to incorporate incorrect pseudo-labels into the policy's support set, destroying reward variance and causing the model to lose the ability to distinguish right from wrong.
GRPO's advantage over SFT holds consistently across four Whisper model sizes, four teacher model families, three domains, and four languages.
GRPO passed through only 27% of the teacher's errors, whereas SFT reproduced the teacher's errors almost 100% verbatim.
Using pseudo-labels as reward references for GRPO rather than as SFT training targets can significantly improve ASR models' robustness to label noise.
When using pseudo-labels generated by Whisper in the banking domain, GRPO achieved relative WER improvements of 29.3% to 56.2% compared with SFT.
When building a semi-supervised ASR pipeline, one can keep the existing teacher model and pseudo-label generation process and simply replace the student model's training objective from the SFT loss to the GRPO reward.
In engineering practice, SFT is usually used for cold start first, followed by RL fine-tuning, which is the practical reason Together offers both paths.
SFT is usually much cheaper than pretraining but is extremely sensitive to data quality; data ratios, the number of epochs, and the learning rate all significantly affect results.
40 more timeline events
All Facts (20)
Modern large language models typically go through three training stages: pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF).
90%VerifiedThe RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.
90%Verified当代前沿模型的训练方法包括RLHF（基于人类反馈的强化学习）及其后继者RLAIF和过程奖励模型（PRM）
80%Verified"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).
80%VerifiedSupervised Fine-Tuning (SFT) trains the model using 'prompt-response' example pairs, teaching the model to respond in a conversational format, thereby making outputs more coherent.
75%VerifiedLarge model training goes through three core stages: pre-training, supervised fine-tuning (SFT), and preference alignment.
75%VerifiedDomain fine-tuning typically first injects domain knowledge through supervised fine-tuning (SFT), then aligns with user preferences through reinforcement learning from human feedback (RLHF).
65%VerifiedDuring SFT training, the model only computes loss on the assistant portion (loss masking technique), meaning it only learns how to answer, not how to ask questions.
65%Unverified大模型核心原理包括Transformer架构、预训练、SFT监督微调、RLHF人类反馈强化学习等关键概念
95%Unverified继续预训练（CPT）后通常还需要再做一轮SFT才能恢复模型的对话能力，因为CPT可能会削弱模型在SFT阶段学到的指令遵循能力
85%UnverifiedSpecialized models trained specifically on cybersecurity data have the potential to surpass general-purpose models in vulnerability identification accuracy and false positive rates.
70%UnverifiedModels trained with reinforcement learning (RL) outperform models trained solely with SFT on Agent tasks.
70%Partially Verified从预训练检查点到可商用产品需经历监督式指令微调（SFT）和基于人类反馈的强化学习（RLHF）两个后训练阶段
65%UnverifiedSFT fine-tuning is effective but causes a decline in general knowledge in small models—i.e., the phenomenon of catastrophic forgetting—due to the prominent capacity bottleneck of small models.
60%UnverifiedA typical post-training pipeline is: pre-training → supervised fine-tuning (SFT) → preference optimization (RLHF / DPO / GRPO).
60%UnverifiedThe emergence of BERT and GPT in 2018 initiated the pre-training + fine-tuning paradigm
60%UnverifiedInstruction fine-tuning and RLHF reinforce the model's tendency to follow instructions, making it more susceptible to being misled by external instructions while improving task capabilities.
60%UnverifiedThe learning rate during the SFT stage is typically 1–2 orders of magnitude lower than during pre-training, to avoid disrupting the general knowledge learned during the pre-training stage.
60%UnverifiedSupervised fine-tuning (SFT) is suited to narrowly defined tasks such as classification, extraction and summarization.
50%UnverifiedThere are plenty of hosted services to choose from for SFT, whereas RL currently lacks a genuinely excellent hosted solution.
50%