Spherical Brownian Diffusion: A New Geometric Paradigm for Continuous Diffusion Language Models

The design of continuous diffusion language models has long revolved around two coupled questions: how to inject noise into continuous token representations, and how to reverse this process during generation. Recent work has begun moving language generation onto the unit sphere, modeling it with geodesic or von Mises–Fisher (vMF) paths, but this raises a follow-up question: which forward process should be used on this geometric structure, and how important is this choice relative to the inference stage? A new paper proposes "Spherical Brownian Diffusion" in an attempt to answer this question systematically.

Canonical Diffusion on the Sphere: Brownian Motion as the Forward Process

The core claim of Spherical Brownian Diffusion is that Brownian motion is the most natural isotropic diffusion process on the sphere. Unlike artificially designed geodesic or vMF paths, the heat semigroup of Brownian motion provides a principled forward process while bringing three key properties: an additive, dimension-independent noise clock, and an exact ancestral bridge. In addition, the clean-token posterior learned via cross-entropy can support a variety of samplers without being tied to a single inference strategy.

The significance of this design lies in decoupling "how to add noise" from "how to sample." In prior work, the geometric choice of the forward process was often entangled with the sampling strategy at inference time, making it difficult to determine which part actually drove performance gains. By providing a forward process that is analytically cleaner, Brownian diffusion creates the conditions for this separation.

Heat semigroup and noise clock: On the sphere, the heat semigroup of Brownian motion is generated by the Laplace–Beltrami operator, and its kernel can be expanded in spherical harmonics, yielding an analytical transition density. This allows defining a dimension-independent "noise clock"—that is, the cumulative amount of noise depends only on the time parameter and does not vary with data dimension, facilitating comparisons across scales. Ancestral bridge: Given the endpoint, the Brownian bridge describes the exact conditional distribution of the reverse process and is the theoretical foundation for reverse sampling in diffusion models; on the sphere, thanks to the symmetry of Brownian motion, the ancestral bridge can be expressed exactly without approximation. Together, these properties make forward corruption and reverse sampling mathematically separable, laying the groundwork for subsequent controlled experiments.

Separating Corruption from Inference: The Approximate Relationship Between Brownian and vMF

To quantitatively distinguish the contributions of the corruption process and the inference process, the authors derive the approximation rate between Brownian corruption and moment-matched vMF corruption in high-dimensional space, and use this relationship to conduct controlled substitution experiments. In simple terms, they identify the conditions under which the two corruption processes can approximate each other, enabling them to switch the corruption type while holding other conditions fixed and observe changes in performance.

This experimental design directly addresses a gap in the field: previously, many improvements in spherical language modeling lacked a clean control for determining whether they came from the geometric choice or from the sampler. The paper provides an operational substitution criterion via the approximation rate, allowing the "corruption" variable to be controlled in isolation.

Moment-matched vMF corruption: The von Mises–Fisher (vMF) distribution is an exponential family distribution on the sphere, often used as a noise model for directional data. Moment matching refers to adjusting the concentration parameter of the vMF so that its first moment (mean direction) matches the distribution after Brownian corruption. In the high-dimensional limit, the short-time diffusion of Brownian motion can be approximated by a vMF, with the approximation rate determined jointly by dimension and time scale. This approximate relationship allows researchers to replace Brownian corruption with vMF corruption while holding other conditions fixed, thereby evaluating the impact of corruption geometry in isolation.

Key Finding: The Choice of Sampler Matters Far More Than the Corruption Process

On the Sudoku task, Brownian corruption and moment-matched vMF corruption produced nearly identical results across four samplers and five budgets. This means that, at least under controlled conditions, the specific geometric form of the forward process has limited impact on final performance.

In contrast, when the corruption process was fixed, the choice of sampler produced a 25.0 percentage point performance gap at a budget of 180 calls. This comparison clearly shows that the sampling strategy at inference time is far more important than the geometric choice of forward corruption.

Furthermore, the authors use the ancestral bridge to decompose the gains from the sampler into two effects: a "memory effect" present on both Sudoku and GSM8K (arising from re-noising states), and a "token-selection effect" that appears on GSM8K and is most pronounced at small budgets. This decomposition provides a mechanistic explanation for why different samplers are effective.

Memory effect and token-selection effect: The memory effect refers to a sampler's tendency, when re-noising states, to retain information about already generated tokens, thereby reducing repeated errors; the token-selection effect refers to the impact on final sequence quality when a sampler makes deterministic or stochastic choices among candidate tokens, which is especially significant at small budgets due to insufficient exploration. The exactness of the ancestral bridge allows the sampler gains to be decomposed into these two parts, providing a basis for designing more efficient inference strategies.

Task Type Determines the Optimal Strategy: The Divide Between Reasoning Tasks and Open-Ended Generation

The paper also reveals how task nature influences strategy preferences. On reasoning tasks (such as GSM8K), categorical refinement can improve performance; in unconditional text generation, updates that preserve randomness are more conducive to maintaining diversity. This finding suggests that the optimal inference configuration for continuous diffusion language models is not universal but should be adjusted according to task goals: when pursuing precise reasoning, deterministic refinement is preferred; when pursuing generation diversity, randomness must be preserved.

From a broader perspective, the value of this work lies not only in proposing a new spherical diffusion process but also in providing an analytical framework—through approximation rates and controlled substitution, it decomposes the design space of diffusion language models into independently evaluable components. For future research, this means that when improving spherical language models, one should prioritize examining the configuration of the sampler and inference budget rather than over-focusing on the geometric details of forward corruption.

Categorical refinement: In continuous diffusion language models, categorical refinement refers to projecting the continuous representation onto the token probability simplex at each step of the reverse process and adopting deterministic or low-randomness update rules to enhance accuracy on reasoning tasks. In contrast, updates that preserve randomness (such as sampling from the predicted distribution) help maintain generation diversity. This distinction echoes the classic trade-off between "deterministic sampling vs. stochastic sampling" in image generation with diffusion models, but in language tasks it must be adjusted according to specific goals.

Key Takeaways

• Brownian motion, as the canonical isotropic diffusion on the sphere, offers analytical advantages such as the heat semigroup, a dimension-free noise clock, and an exact ancestral bridge.
• Brownian and moment-matched vMF corruption perform almost identically on Sudoku, while the choice of sampler creates a 25.0 percentage point gap at the same budget.
• Sampler gains can be decomposed into a memory effect (Sudoku, GSM8K) and a token-selection effect (GSM8K, largest at small budgets).
• Reasoning tasks benefit from categorical refinement, while unconditional generation requires stochastic updates to maintain diversity.

One-line Summary

Spherical Brownian Diffusion decouples forward corruption from inference sampling and finds that the choice of sampler matters more than the corruption geometry.

Paragraph Summary

This paper proposes Spherical Brownian Diffusion, using Brownian motion as the most natural isotropic forward process on the sphere, featuring the heat semigroup, a dimension-free noise clock, and an exact ancestral bridge. By deriving the approximation rate between Brownian and moment-matched vMF corruption, the authors conduct controlled substitution experiments and find that on the Sudoku task the two corruption methods yield nearly identical results, while the choice of sampler creates a 25.0 percentage point performance gap at a budget of 180 calls. Sampler gains can be decomposed into a memory effect (Sudoku, GSM8K) and a token-selection effect (most pronounced on GSM8K at small budgets). Task type determines the optimal strategy: reasoning tasks benefit from categorical refinement, while unconditional generation requires stochastic updates to maintain diversity.