Distilled Model
A lightweight model obtained by transferring the capabilities of a large teacher model to a small student model through knowledge distillation; inference cost is low, but open-domain reasoning capability is limited.
Core Facts
Timeline (last 90 days)
Distilled models further improve alignment by having the draft model fit the target model's output distribution, thereby raising the acceptance rate.
A distilled model is a model with fewer parameters and lower inference cost trained using the outputs of a large model as the training signal.
In the ticket classification example, with fifty thousand tickets per month, the leaderboard champion model and the distilled "flash" model produced identical labels for obvious tickets.
Distilled models are sufficient to cover most traffic in routine customer-service scenarios, but degrade noticeably when open-domain reasoning or long-chain planning is required.
In a real-time code review tool built for a client, using the largest and smartest model gave an average response of 8.7 seconds; after switching to a smaller distilled model, the response dropped to 2.3 seconds, with review quality falling by only about 5%.
The model names labeled in API documentation (such as gpt-4, claude-3) may correspond to cheaper distilled versions, quantized versions, or even previous-generation models.
Distillation can be viewed as an implicit, data-constrained form of model merging—distilling a student using the averaged outputs of multiple teacher models is effectively performing a soft merge at the level of output distributions.
All Facts (8)
Distillation can be viewed as an implicit, data-constrained form of model merging—distilling a student using the averaged outputs of multiple teacher models is effectively performing a soft merge at the level of output distributions.
65%UnverifiedAI服务商在算力受限时可能采用减少beam search宽度、压缩KV Cache、动态切换蒸馏模型等手段降低资源消耗
85%UnverifiedDistilled models further improve alignment by having the draft model fit the target model's output distribution, thereby raising the acceptance rate.
50%UnverifiedA distilled model is a model with fewer parameters and lower inference cost trained using the outputs of a large model as the training signal.
50%UnverifiedIn the ticket classification example, with fifty thousand tickets per month, the leaderboard champion model and the distilled "flash" model produced identical labels for obvious tickets.
50%UnverifiedDistilled models are sufficient to cover most traffic in routine customer-service scenarios, but degrade noticeably when open-domain reasoning or long-chain planning is required.
50%UnverifiedThe model names labeled in API documentation (such as gpt-4, claude-3) may correspond to cheaper distilled versions, quantized versions, or even previous-generation models.
50%UnverifiedIn a real-time code review tool built for a client, using the largest and smartest model gave an average response of 8.7 seconds; after switching to a smaller distilled model, the response dropped to 2.3 seconds, with review quality falling by only about 5%.
50%