Soft Labels
Probability distribution labels output by a teacher model in knowledge distillation. Compared to hard labels, soft labels carry richer inter-class relational information and are typically smoothed using a temperature parameter during the distillation process.
Timeline (last 90 days)
Generative recommendation is gradually shifting from autoregressive generation to parallel generation to improve efficiency, and how to fully exploit the soft-label knowledge of traditional recommendation models under this new structure has become an urgent problem to be solved.
Traditional sequential recommendation models (such as Softmax-based models) can output a full item probability distribution, and such soft labels contain rich negative-sample information and relative relationships among items.
The second-stage soft-label generation process does not require manual annotation of the teacher's own training data.
In soft-label distillation the teacher output carries confidence information, so the student can perceive uncertainty; whereas text-level distillation treats teacher output as deterministic ground truth, so the student inherits the teacher's systematic errors without reservation.
Traditional distillation uses the teacher's output soft labels (i.e., temperature-scaled class probability distributions) as the supervision signal, whereas current mainstream large language model distillation mostly adopts sequence-level supervision in which the teacher generates responses and the student imitates them.
Soft labels are the probability distribution output by the teacher model after softmax normalization, carrying similarity information between classes, and their information content far exceeds that of a single hard label.
Soft labels refer to the probability distribution of outputs rather than simple 0/1 classification results, containing implicit knowledge such as inter-class similarity, and are richer in information than hard labels.
All Facts (7)
Traditional sequential recommendation models (such as Softmax-based models) can output a full item probability distribution, and such soft labels contain rich negative-sample information and relative relationships among items.
50%UnverifiedThe second-stage soft-label generation process does not require manual annotation of the teacher's own training data.
50%UnverifiedIn soft-label distillation the teacher output carries confidence information, so the student can perceive uncertainty; whereas text-level distillation treats teacher output as deterministic ground truth, so the student inherits the teacher's systematic errors without reservation.
50%UnverifiedTraditional distillation uses the teacher's output soft labels (i.e., temperature-scaled class probability distributions) as the supervision signal, whereas current mainstream large language model distillation mostly adopts sequence-level supervision in which the teacher generates responses and the student imitates them.
50%UnverifiedSoft labels are the probability distribution output by the teacher model after softmax normalization, carrying similarity information between classes, and their information content far exceeds that of a single hard label.
50%UnverifiedSoft labels refer to the probability distribution of outputs rather than simple 0/1 classification results, containing implicit knowledge such as inter-class similarity, and are richer in information than hard labels.
50%UnverifiedGenerative recommendation is gradually shifting from autoregressive generation to parallel generation to improve efficiency, and how to fully exploit the soft-label knowledge of traditional recommendation models under this new structure has become an urgent problem to be solved.
50%