Knowledge Distillation
A model compression technique that transfers knowledge from a large teacher model to a lightweight student model. Soft labels convey inter-class semantic structure. Methods are categorized into logits-based, feature-based, and relation-based approaches.
Core Facts
Knowledge distillation transfers the full probability distribution information output by the teacher model via soft labels, producing an implicit data augmentation effect.
Knowledge distillation was proposed by Hinton et al. in 2015, originally referring to training a small model using the soft label outputs of a large model.
Knowledge distillation was first proposed by Hinton et al. in 2015.
Timeline (last 90 days)
In language model inference optimization, common approaches fall roughly into two categories: one reduces the computation and memory overhead of a single inference (such as quantization, knowledge distillation, sparsification, and pruning), and the other reduces redundant computation and improves throughput (such as KV cache reuse, prefix caching, continuous batching, and speculative decoding).
The community often compresses large models to a level runnable on consumer-grade graphics cards through 4-bit quantization, knowledge distillation, and other methods, but this loses some precision.
The capability retention rate of distillation typically fluctuates between 70% and 90%.
Nathan Lambert believes there is some basis for viewing distillation as a primary means by which Chinese open-source models catch up, but that this is oversimplified; he notes that post-training has now shifted to being mainly about reinforcement learning environments, and that simply using the strongest model to generate completions for SFT and watching scores rise is hard to reproduce in the open literature.
The experiment used knowledge distillation, with Llama 3 8B (token-based) as the teacher model and the byte model as the student model, requiring the design of an alignment mapping between tokens and byte sequences.
After the fix, the student network no longer drifts away from the teacher network in knowledge distillation, and the distillation process becomes more stable.
In knowledge distillation, when the student logits exactly equal the teacher logits, the standard softmax implementation produces non-zero spurious gradients.
Traditional distillation or teacher-student frameworks require two separate models: the teacher generates high-quality signals, and the student learns from them.
In RoCo-PRM, the teacher is a strong MoE model whose soft labels come from step scores computed from routing statistics rather than output logits; the student is an independent PRM.
In NLP, knowledge distillation can be divided into representation-level (aligning hidden layer states) and task-level (aligning output logits).
40 more timeline events
All Facts (20)
Knowledge distillation transfers the full probability distribution information output by the teacher model via soft labels, producing an implicit data augmentation effect.
90%VerifiedKnowledge distillation was proposed by Hinton et al. in 2015, originally referring to training a small model using the soft label outputs of a large model.
90%VerifiedKnowledge distillation was first proposed by Hinton et al. in 2015.
90%VerifiedKnowledge Distillation is a technique for transferring the knowledge of a large model to a smaller, lightweight model, capable of retaining most of the performance while reducing the parameter count by tens of times.
80%Verified知识蒸馏由Hinton等人在2015年提出，核心思想是让小型学生模型模仿大型教师模型的输出分布
80%Verified模型蒸馏是将大模型的能力压缩到小模型中以降低推理成本的技术
80%VerifiedThis experiment adopts trajectory-level distillation, which belongs to the category of behavioral cloning, directly having the student model imitate the external behavioral sequences of the teacher model.
75%VerifiedKnowledge distillation was first proposed by Hinton, Vinyals, and Dean at Google in 2015, with the core idea of having a small student model learn from the soft labels output by a large teacher model.
75%VerifiedDistilBERT reduces BERT's parameter count by 40% through distillation, increases inference speed by 60%, and incurs only approximately 3% performance loss
75%VerifiedKnowledge distillation was systematized by Hinton et al. in 2015; its core idea is to transfer the dark knowledge of a large model to a small model, using soft probability distributions as supervision signals.
75%VerifiedThe teacher model outputs in distillation training rarely include self-correction examples, causing student models to form an implicit belief that "strong models don't make mistakes," leading them to repeat rather than correct when verification fails.
75%VerifiedFlash series models acquire capabilities through knowledge distillation from Pro or larger-scale Ultra models.
75%VerifiedModel distillation transfers the knowledge of a large model to a smaller student model, resulting in significant speed improvements but with some capability loss.
75%VerifiedKnowledge distillation, proposed by Hinton et al., has student models inherit implicit knowledge by learning from the soft probability distributions output by teacher models.
70%VerifiedDistillation can be viewed as an implicit, data-constrained form of model merging—distilling a student using the averaged outputs of multiple teacher models is effectively performing a soft merge at the level of output distributions.
65%VerifiedIf the dataset is small while the model has too many parameters, the model may memorize the training data prematurely and degenerate; this can be mitigated by adding more data, applying dropout regularization, or reducing model size.
65%VerifiedWhen compression techniques such as quantization, knowledge distillation, and operator fusion are used in combination, inference speed typically improves by 2–4×.
65%VerifiedQuantization is a model compression technique that reduces model parameters from high-precision floating-point numbers to integer representations of lower bit-width
65%VerifiedOn-device AI refers to completing the inference process of machine learning models directly on the end device rather than uploading to cloud servers for processing, requiring model compression and hardware acceleration to compensate for the computational gap.
65%VerifiedLightweight models are compressed in scale through knowledge distillation and model pruning techniques; knowledge distillation has the small model mimic the output probability distribution of the large model, while pruning removes network weights that contribute least to the output.
65%