LLaCO: Competition-Aware Scheduling Revolutionizes Continuous Diffusion Language Models

Continuous diffusion models, as a promising paradigm for non-autoregressive text generation, show potential in continuous embedding space. However, the misalignment between their geometric trajectory and temporal progression leads to a severe temporal compression problem. A recent paper proposes the LLaCO framework to address this core contradiction through competition-aware semantic scheduling.

The Temporal Compression Dilemma of Continuous Diffusion Models

Continuous diffusion models generate text in continuous embedding space. Along a continuous trajectory, the identity of the residual token directly affects the denoising objective. But the paper points out that the geometric trajectory and temporal progression are not aligned, causing severe temporal compression. Specifically, the decay of embedding norms causes Euclidean distance to produce spurious transcendence, meaning token competition must be evaluated directionally.

In this space, a candidate token only transcends the source token when the effective noise magnitude exceeds the semantic separation. The authors formalize this pairwise dynamic as Competition Threshold Theory (CTT). The key finding is that transcendence in high-dimensional space requires extreme noise levels, causing standard trajectories to concentrate identity decay within a narrow window near the noise endpoint.

This theory reveals a fundamental flaw in existing continuous diffusion models: token identity is determined either too early or too late during denoising, failing to match the natural rhythm of semantic evolution.

Continuous Diffusion Models are a class of generative models that model text generation as a gradual denoising process in continuous embedding space, distinct from traditional discrete diffusion models (such as D3PM) that operate directly on a discrete vocabulary. The basic idea is to first map discrete tokens to continuous vectors, then add Gaussian noise in vector space and train a network to reverse this process, ultimately restoring text through nearest-neighbor or rounding. Since the embedding space has geometric structure, distances, directions, and norms between tokens all influence denoising decisions. This enables continuous diffusion models to theoretically model semantic evolution more smoothly, but also introduces unique challenges such as norm decay and geometric-temporal misalignment.

Transcendence in Competition Threshold Theory (CTT) refers to: at a certain moment in the denoising trajectory, the distance between a candidate token and the source token exceeds the distance between the source token and the target token, thereby "replacing" the source token as the current prediction in the nearest-neighbor sense. In embedding space, this comparison is affected by both directional differences and norm magnitudes. When embedding norms decay over time, Euclidean distances are globally compressed, making tokens with originally large directional differences appear numerically closer, producing spurious transcendence. CTT points out that for genuine semantic transcendence to occur, the effective noise magnitude must exceed the semantic separation between the candidate token and the source token; in high-dimensional space, this condition is often only satisfied near the noise endpoint, so identity decay is compressed into a narrow window—this is the geometric root of "temporal compression."

Semantic Retention Scheduling: Aligning Time with Identity Decay

To solve the temporal compression problem, the paper proposes Semantic Retention Scheduling (SRS), which aims to align time allocation with the process of identity decay. The authors also prove that existing token identity metrics are essentially statistical aggregations of these pairwise crossings.

The core idea of SRS is to proportionally redistribute the denoising trajectory according to log-rank decay. This means the model allocates different time budgets for retaining and updating token identity at different stages of denoising, thereby avoiding excessive concentration of identity decay in a narrow interval. This scheduling strategy directly targets the dynamics revealed by CTT, making time allocation better match the actual needs of semantic evolution.

Log-rank decay borrows a concept from order statistics: tokens are assigned decreasing weights or time budgets according to their competitive ranking in the candidate distribution. Tokens ranked higher (closer to becoming the prediction) receive more retention time, while lower-ranked tokens decay faster. In SRS, this decay curve is used to proportionally redistribute the time steps of the entire denoising trajectory, so that the rate of identity decay matches the natural rhythm of semantic evolution, rather than having all token identities determined collectively near the noise endpoint. Compared to linear or exponential decay, log-rank decay can retain more candidates in early stages and converge quickly in later stages, thereby alleviating temporal compression.

The LLaCO Framework: Direction Adaptation and Spherical Noising

Based on the above theory, the authors propose the LLaCO generation framework. LLaCO adapts a pretrained backbone network to continuous representations through an embedding direction adapter and timestep-conditioned modulation. To operate in direction space, LLaCO employs a spherical noising process, formalized as geodesic interpolation toward a mask-biased endpoint, thereby eliminating norm decay while preserving accurate token competition.

Guided by SRS, LLaCO redistributes the denoising trajectory according to log-rank decay proportions. As an 8B-parameter model instance, LLaCO surpasses previous continuous diffusion baselines in experiments and is comparable to discrete diffusion models of the same scale.

Spherical noising is a variant that defines the diffusion process on a unit sphere: each noising step does not simply add Gaussian noise to the embedding vector, but interpolates the current vector along a spherical geodesic toward a mask-biased endpoint. A geodesic is the shortest path between two points on a sphere, equivalent to a great-circle arc. Since all intermediate states remain on the unit sphere, the vector norm is always 1, norm decay is artificially eliminated, and comparison between tokens reduces to purely directional angular comparison. This bypasses the problem of spurious Euclidean distance transcendence, allows competition relationships to more accurately reflect semantic directional differences, and provides stable input geometry for the direction adapter.

Implications for the Text Generation Paradigm

LLaCO's work provides a new theoretical perspective for continuous diffusion language models: token competition is not a simple distance comparison but a directional dynamic process. Competition Threshold Theory explains why standard continuous diffusion models struggle to precisely control generation timing, while Semantic Retention Scheduling offers a principled solution.

From a broader perspective, this research highlights the alignment problem between geometry and time in continuous diffusion models. The design of spherical noising and direction adapters provides a technical path that subsequent research can build upon. Although LLaCO has currently only been validated at the 8B scale, its theoretical framework is general and may inspire further exploration of generation scheduling under continuous representations.

The paper's limitation is that experiments mainly compare against continuous diffusion baselines, and competition with discrete diffusion models is only "comparable" rather than superior. Future work may need to further validate LLaCO's performance on larger-scale models and more diverse tasks.

One-line Summary

LLaCO solves the temporal compression problem in continuous diffusion text generation through Competition Threshold Theory and Semantic Retention Scheduling.

Paragraph Summary

This article reviews the continuous diffusion text generation framework LLaCO. The article notes that continuous diffusion models suffer from temporal compression caused by misalignment between geometric trajectory and temporal progression: embedding norm decay produces spurious transcendence in Euclidean distance, and token identity decay is compressed into a narrow window near the noise endpoint. The authors' Competition Threshold Theory (CTT) formalizes pairwise token dynamics as directional competition, revealing the fundamental flaw that transcendence in high-dimensional space requires extreme noise levels. To address this, LLaCO introduces Semantic Retention Scheduling (SRS), which redistributes denoising time according to log-rank decay, and employs spherical noising and direction adapters to eliminate norm decay and preserve accurate token competition. The 8B-parameter instance surpasses continuous diffusion baselines and is comparable to discrete diffusion models of the same scale, providing a general theoretical framework for generation scheduling under continuous representations.