Animated Breakdown: How a Language Model Picks the Next Token

When a language model generates text, it has to pick one token out of tens of thousands of candidates for every word it outputs. This process looks simple, but it hides a whole interplay of mechanisms: softmax, temperature, top-k, top-p, and more. One creator built a 7-minute animation using real probability data from Qwen2.5-0.5B-Instruct, walking through the entire decision chain.

A Single Real Case Runs Through It All

The author gave Qwen2.5-0.5B-Instruct the same prompt, "Write one sentence about the moon.", and ran it twice at temperature 1 with top-k and top-p disabled, changing only the random seed. Both responses began with "The", then diverged: one went toward "The moon, a satellite of the Earth…", the other became "The lunar South Equatorial region…". This fork is exactly the best window into understanding the sampling mechanism.

After "The", the model scores all 151,936 vocabulary tokens one by one. Once softmax turns these raw scores into a probability distribution, "moon" gets 78.04%, "full" 4.46%, "vast" 2.24%, "Moon" (capitalized) 1.51%, and "lunar" 1.50%. These numbers directly show the "concentrated head, long scattered tail" character of the probability distribution—the top candidate holds nearly 80% of the weight, yet a string of non-negligible low-probability options still trails behind it.

The softmax function is a key step in the decoding pipeline, converting the model's raw output scores (logits) into a probability distribution. Specifically, for each token's logit ( z_i ), softmax computes ( p_i = \frac{e^{z_i}}{\sum_j e^{z_j}} ), ensuring all probabilities are non-negative and sum to 1. In language models, the vocabulary typically contains tens of thousands of tokens, so softmax must handle outputs of that dimensionality, making it computationally expensive. Moreover, softmax is sensitive to the scale of the logits: small differences in logits can be exponentially amplified, producing a sharp distribution; conversely, if the logit differences are small, the distribution is flat. The temperature parameter adjusts this sharpness precisely by scaling the logits, thereby affecting the diversity of generation.

The Fundamental Divergence Between Greedy and Sampling

Greedy decoding always picks the highest-probability token, so after "The" it must be followed by "moon". Sampling, by contrast, draws randomly according to weight—"lunar" gets picked roughly once every 67 tries, and the second run happened to draw it. From there the model enters a loop: append the chosen token to the end of the sequence, run the model again, and pick the next one.

The author tallied the rank of the chosen token in subsequent steps: most of the time the model picked the top-ranked one, but the second run also drew rank 219 "South" and rank 1,098 "empty", while the first run picked rank 536 "green". These "surprises" aren't model errors but normal sampling behavior—low-probability events will eventually happen given enough attempts.

How Temperature Reshapes the Probability Distribution

The temperature parameter divides the raw scores before softmax. The author demonstrated on the same post-"The" step: at temperature as low as 0.2, "moon"'s probability is pushed to near 100%; at temperature 2.0, "moon" has only 6.3% left. Low temperature sharpens the distribution and makes the model more conservative; high temperature flattens the weights and gives unpopular tokens more chances.

There's an easily overlooked detail here: temperature does not change the ranking among candidates, only the relative gaps between them. So at T=0.2, "moon" still ranks first—its lead is just magnified to the point of being nearly unassailable.

The temperature parameter acts as a scaling factor in softmax: given logits ( z_i ) and temperature ( T ), the adjusted probability is ( p_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}} ). When ( T < 1 ), the logits are magnified, the probability distribution is sharper, and the advantage of high-probability tokens is more pronounced; when ( T > 1 ), the logits are shrunk, the distribution is flatter, and low-probability tokens get more chances. Temperature does not change the relative order of the logits, so the ranking of candidates stays the same, but it changes the probability gaps between them. In extreme cases, as ( T \to 0 ) it degenerates to greedy decoding (always picking the highest-probability token), and as ( T \to \infty ) it approaches a uniform distribution. In practice, temperature is often used as a knob for tuning the creativity and consistency of generation.

top-k and top-p: Two Approaches to Truncation

The author compared two truncation strategies on the same decision step. top-k 3 keeps only the top three; after renormalization, "moon"'s probability is raised from 78.04% to 92.1%. top-p 0.9, by contrast, keeps the smallest set of tokens whose cumulative probability reaches 0.9—8 tokens at this step; but at some later step where "of" accounts for 97.9%, top-p 0.9 keeps only 1 token—because the first token's cumulative probability already exceeds the threshold.

This is precisely the core argument of the 2019 paper by Holtzman et al.: top-k uses a fixed-size candidate set that cannot adapt to changes in the shape of the distribution; top-p lets the candidate set size expand and contract dynamically with the distribution. In this example, the model's default top-p 0.8 would directly cut off "lunar", meaning the "lunar South Equatorial region" path from the second run would never be taken at all.

Top-k sampling and top-p (nucleus) sampling are two common truncation strategies. Top-k keeps the k highest-probability tokens, sets the probabilities of the remaining tokens to zero, and then renormalizes. k is a fixed value that does not change with the shape of the distribution, so when the distribution is sharp it may keep too many low-probability tokens, and when the distribution is flat it may truncate useful candidates. Top-p (also called nucleus sampling), proposed by Holtzman et al. in 2019, keeps the smallest set of tokens whose cumulative probability reaches the threshold p, with the set size changing dynamically: when the distribution is sharp, the cumulative probability of a few tokens quickly exceeds p and the set shrinks; when the distribution is flat, more tokens are needed to reach p and the set grows. This dynamic adaptability makes top-p more robust across different contexts and the default sampling strategy for many models.

Temperature 0 Doesn't Guarantee Determinism Either

A counterintuitive fact: even if you set temperature to 0, outputs on a server can still differ. The author cites an experiment from Thinking Machines—running Qwen3-235B 1,000 times produced 80 different answers, rooted in the fact that batch size affects the order of floating-point addition. This means "deterministic decoding" is never absolute at the engineering implementation level.

All data were obtained by the author actually running transformers on CPU in Colab, and the video narration is AI reading a script written by the author. For anyone who wants an intuitive understanding of decoding strategies, this approach of walking the entire chain with real probabilities is more convincing than abstract formulas.

The nondeterminism of floating-point arithmetic stems from the limited precision of computers. In deep learning, operations such as matrix multiplication and reduction involve large numbers of floating-point additions, and the order of addition affects rounding errors. For example, ((a+b)+c) and (a+(b+c)) may produce tiny differences. During GPU parallel computation, factors such as batch size and thread scheduling change the reduction order, leading to slightly different results. This difference can be amplified in model inference: even with temperature set to 0 (greedy decoding), outputs can still differ across batches or hardware. Therefore, in engineering practice, reproducibility is often pursued by fixing random seeds, disabling parallel reductions, or using deterministic algorithms—but absolute determinism remains difficult to guarantee.

Key Takeaways

• After "The", the model scores 151,936 tokens; "moon" holds 78.04%, while "lunar" has only 1.50% yet can still be sampled
• Temperature divides and scales the scores before softmax: at 0.2 "moon" is near 100%, at 2.0 it drops to 6.3%
• top-k keeps a fixed top-k, while top-p truncates dynamically by cumulative probability; the latter adapts to changes in distribution shape
• The default top-p 0.8 directly excludes "lunar", changing the model's actual generation path
• Even with temperature set to 0, differences in floating-point addition order caused by batching can still produce different outputs

One-line Summary

Using real probabilities from Qwen2.5-0.5B, an animation visually demonstrates how softmax, temperature, top-k, and top-p interact in language model decoding.

Paragraph Summary

The article uses two runs of Qwen2.5-0.5B-Instruct on the prompt "Write one sentence about the moon." to show the full token-by-token generation process. The model scores 151,936 vocabulary tokens, softmax converts the logits into a probability distribution, "moon" tops the list at 78.04%, and "lunar" has only 1.50% yet can still be sampled. The temperature parameter scales the logits before softmax, changing the sharpness of the distribution without changing the ranking; top-k keeps a fixed top-k tokens, while top-p truncates dynamically by cumulative probability, adapting better to changes in distribution shape. The default top-p 0.8 directly excludes "lunar", changing the generation path. Even with temperature set to 0, the nondeterminism of floating-point arithmetic can still lead to different outputs. The article uses real data to intuitively reveal how these mechanisms interact.