Cross-Item and Cross-Encoding-Space Knowledge Distillation: A New Approach for Parallel Generative Recommendation

Generative recommendation systems are gradually becoming the mainstream paradigm for sequential recommendation, in which Semantic IDs represent items as discrete code sequences and generate recommendations by directly predicting these codes. However, existing Semantic ID recommendation models are typically trained only with hard labels of the observed next item. Although this supervisory signal can indicate the target item, it cannot provide distributional information about other candidate items under the same user history. Soft-label knowledge from sequential recommendation models can fill this gap, but how to transfer item-space teacher knowledge to a code-space student model remains a key challenge.

This article introduces a novel knowledge distillation framework for parallel Semantic ID recommenders. Through two complementary distillation objectives—Item Distribution Matching (IDM) and Code Marginal Matching (CMM)—it effectively transfers the teacher model's item-level predictive knowledge to the student model while preserving its parallel prediction structure. Experiments show that the method significantly outperforms existing state-of-the-art approaches on multiple benchmark datasets.

The Challenge of Integrating Semantic ID Recommendation and Knowledge Distillation

The core idea of Semantic ID recommendation is to transform item semantic representations (such as text descriptions and image features) into discrete code sequences through quantization or clustering. For example, an item may be represented as a code combination like [12, 45, 78]. Generative models recommend items by predicting these codes, avoiding the large item-ID embedding table overhead in traditional recommendation.

However, current training methods have clear limitations: the model is supervised only by the hard label of the next item, i.e., it is only told "which code sequence should be predicted," but not provided with distributional information about "how likely other items are to be selected." In contrast, traditional sequential recommendation models (such as Softmax-based models) can output a complete item probability distribution. Such soft labels contain rich negative-sample information and relative relationships among items, which are valuable for improving recommendation quality.

The difficulty is that the teacher model outputs a probability distribution in item space, while the student model performs parallel prediction in code space—the two spaces are inconsistent and cannot be directly aligned. In addition, the parallel prediction structure means that the student model predicts all code positions simultaneously rather than generating them autoregressively position by position, which further increases the complexity of knowledge transfer.

Generation and Quantization of Semantic IDs

Semantic IDs are usually obtained by quantizing dense semantic representations of items (such as text embeddings or visual features). Common approaches include using clustering algorithms such as K-means to discretize continuous representations into indices in multiple codebooks, or using techniques such as residual quantization (RQ) and product quantization (PQ) to decompose high-dimensional vectors into code combinations of multiple low-dimensional subvectors. For example, an item's Semantic ID may be formed by concatenating indices from multiple codebooks, such as [12, 45, 78], where each index corresponds to a cluster center in a codebook. This discretization process preserves semantic similarity among items while mapping items into a compact code space, making it convenient for generative models to directly predict code sequences. Understanding this generation mechanism helps grasp the meaning of the "fixed item-to-code mapping" in IDM and CMM, namely that each item corresponds to a deterministic code tuple, thereby supporting the reconstruction of item probabilities from code probabilities.

Dual-Path Distillation: Design Ideas of IDM and CMM

To solve the above challenges, the researchers proposed two complementary knowledge distillation objectives that achieve cross-space knowledge transfer from different perspectives.

Item Distribution Matching (IDM)

The core idea of IDM is to reconstruct a catalog-level item distribution from the student model's parallel code predictions, and then align it with the teacher model's item distribution. Specifically, the student model outputs a probability distribution for each code position. IDM combines the code probabilities at each position through the fixed item-to-code mapping to compute the joint probability of each candidate item. This reconstruction process preserves the student model's parallel prediction structure while converting predictions in code space into a distribution in item space. IDM then uses measures such as KL divergence to minimize the difference between the student's reconstructed distribution and the teacher's distribution.

The advantage of this method is that it aligns directly in item space and fully leverages the complete soft-label information provided by the teacher model. The reconstructed item distribution not only contains the probability of the target item but also preserves the relative probabilities of all candidate items, thereby providing a richer supervisory signal for the student model.

KL Divergence and Soft-Label Distillation

KL divergence (Kullback-Leibler divergence) is a common metric for measuring the difference between two probability distributions and is often used as a loss function in knowledge distillation. Given a teacher distribution p and a student reconstructed distribution q, KL divergence is defined as Σ p(x) log(p(x)/q(x)), which measures the amount of information lost when using q to approximate p. In IDM, minimizing KL divergence is equivalent to making the student's reconstructed item distribution as close as possible to the teacher's soft-label distribution. Soft labels not only contain the high probability of the target item but also preserve the relative probabilities of other candidate items, providing a richer supervisory signal than hard labels. This soft-label distillation has been shown in traditional recommendation models to effectively improve generalization ability, and IDM introduces it into Semantic ID recommendation, solving the challenge of cross-space alignment.

Code Marginal Matching (CMM)

CMM approaches the problem from another angle: it projects the teacher model's item probabilities onto each code position to obtain code marginal distributions, and then matches them with the outputs of the corresponding code heads of the student model. Given a fixed item-to-code assignment, the teacher's probability for each item can be accumulated onto each code position to form the marginal probability distribution at that position. For example, the probability that code position 1 takes value 12 equals the sum of the teacher probabilities of all items containing code 12.

CMM decomposes teacher knowledge onto each code position so that each code head of the student model can receive soft supervision from the teacher model. This method acts more directly on code space, is naturally compatible with the parallel prediction structure, and is computationally efficient.

The two objectives complement each other: IDM performs global alignment in item space, while CMM performs local alignment in code space, together forming a complete distillation framework.

Marginal Distributions and Parallel Prediction

A marginal distribution is the distribution of a single variable obtained from a joint distribution by ignoring the other variables. In CMM, the teacher model outputs a joint probability distribution in item space, while the student model predicts each code position in parallel. Through the fixed item-to-code mapping, CMM accumulates the teacher's probability for each item onto each value of each code position to obtain code marginal distributions. For example, the marginal probability that code position 1 takes value 12 equals the sum of the teacher probabilities of all items containing code 12. In this way, teacher knowledge is decomposed onto each code position, and each code head of the student model can independently align with the corresponding marginal distribution. This method is naturally compatible with the parallel prediction structure, because parallel prediction outputs the distributions of all positions simultaneously without autoregressive dependencies, resulting in high computational efficiency and easy implementation.

Experimental Validation: Significantly Outperforming Existing Methods

The researchers conducted experiments on multiple benchmark datasets, comparing IDM and CMM with existing state-of-the-art Semantic ID recommendation methods. The results show that both distillation methods consistently outperform the baselines.

Specifically, IDM improves Recall@10 by an average of 15.39%, while CMM improves NDCG@10 by an average of 8.28%. This result indicates that distillation in either the reconstructed item space or the code marginal space can significantly enhance the effect of hard-label supervision. You may not have noticed that the two objectives each have advantages on different metrics: IDM is better at improving recall, indicating that reconstructing the item distribution helps the model cover more relevant items; CMM performs better on ranking quality (NDCG), indicating that code marginal matching can optimize ranking more finely.

The experiments also verified the generality of the distillation framework: the method can be applied to different parallel Semantic ID recommender architectures and does not change the student model's original parallel prediction structure, making it convenient for practical deployment.

Implications for the Field of Generative Recommendation

The value of this work lies not only in proposing two effective distillation objectives but also in providing new ideas for knowledge transfer in generative recommendation. As generative recommendation gradually shifts from autoregressive generation to parallel generation to improve efficiency, how to make full use of the soft-label knowledge of traditional recommendation models under this new structure has become an urgent problem to solve. IDM and CMM provide answers from item space and code space, respectively, demonstrating the feasibility of cross-space distillation.

In the future, this framework is expected to be extended to more modalities of Semantic ID generation scenarios, such as multimodal item representations and cross-domain recommendation. At the same time, how to adaptively combine IDM and CMM or design more efficient distillation strategies is also worth further exploration. For industrial recommendation systems, this method provides a low-cost way to improve the performance of existing parallel generative models and has high practical value.

One-line Summary

Proposes a dual-path distillation framework with IDM and CMM to transfer teacher soft labels to parallel Semantic ID recommenders.

Paragraph Summary

To address the insufficiency of hard-label supervision in parallel Semantic ID recommenders, this article proposes a knowledge distillation framework that transfers the soft-label knowledge of traditional sequential recommendation models in item space to a code-space student model through two complementary objectives: Item Distribution Matching (IDM) and Code Marginal Matching (CMM). IDM reconstructs the item distribution from the student's parallel code predictions and aligns it with the teacher, while CMM projects teacher item probabilities into code marginal distributions and matches them with the student's code heads. Experiments show that IDM improves Recall@10 by an average of 15.39%, and CMM improves NDCG@10 by an average of 8.28%, both outperforming existing state-of-the-art methods while preserving the student model's parallel structure for easy deployment.