Distillation

Distillation is a technique for training a small model to mimic the behavior of a large model.

Inference throughput optimization relies on technologies such as KV Cache management and Continuous Batching, which directly determine response speed and service cost under concurrent user loads.

Flash lightweight models reduce inference costs and improve response speed through smaller parameter sizes, quantization, or distillation techniques.

Model Distillation in machine learning refers to training a small student model using the outputs of a large teacher model, in order to compress model size while retaining capability.

量化会不可避免地引入精度损失，在逻辑推理等对准确性敏感的任务上可能出现性能下降

The teacher model in distillation does not have to be larger than the student model; it can be a domain expert model of comparable size.

Model acceleration aims to reduce the number of sampling steps, and typical methods include DDIM and distillation methods.

The distilled version of DeepSeek-R1 is a successful example of distillation technology; quantized or distilled versions can be deployed on consumer-grade GPUs or even laptops.

Distillation is not simple copying: the student model has limited capacity, so trade-offs must be made in data ratios, temperature parameters, and loss functions, while also being alert to the teacher model's errors being inherited along with everything else.

In the context of fine-tuning, "sequence dataset" usually refers to converting raw corpus into input-output pairs that the model can consume, with the format determined by the training objective: instruction fine-tuning requires constructing instruction-response pairs, distillation requires the teacher model's output as labels, and embedding model fine-tuning requires positive and negative sample pairs.

The three scenarios in the second stage of the course are: embedding model fine-tuning in the RAG knowledge base scenario, distillation, and intelligent analysis (asking questions in natural language to generate analysis code).

Enterprises often have limited computing resources when implementing, and need small models that match their business capabilities; distillation is a common solution.

Teacher Li Jin suggested an advanced FDE technical path: first do AI agent development, then go deeper into model fine-tuning, distillation and pretraining.

In terms of technical difficulty, the technical threshold of fine-tuning is lower than that of distillation, and distillation is lower than that of pretraining.

Lightweight policy models often use distillation, quantization or specialized architectures to achieve millisecond-level response under limited compute, but may sacrifice generalization ability.

Even with optimizations such as quantization, distillation and caching, the marginal cost of inference remains hard to bring close to zero.

Technological advances such as distillation, sparsification, and inference-side scaling may enable low-compute models to achieve capabilities close to high-compute models, so compute thresholds need periodic calibration to remain effective.

Distilled models often show degradation in out-of-distribution generalization and complex reasoning, making it hard to fully replace the original model's capability ceiling.

Migrating a model from cloud inference to local operation typically requires compression steps such as quantization, pruning and distillation.

Against the backdrop of compute scarcity, the priority of model compression techniques such as quantization and distillation has been significantly moved forward.