Dense Model

Frontier models commonly adopt a Mixture of Experts (MoE) architecture that activates only a subset of parameters during inference to balance speed and quality

Dense models have simpler inference logic and better hardware compatibility in local deployment scenarios, whereas MoE models, despite activating fewer parameters during inference, still require all weights to be loaded into memory, placing even higher demands on hardware.

In dense models, the computation of the verification window is approximately linear in the number of tokens, because all tokens share the same set of weights.

27B is a medium-to-large dense model scale, between lightweight 7B–14B models and large models of 70B and above.

The advantages of dense models lie in deterministic inference paths, memory usage that corresponds roughly linearly to parameter count, and relatively simple deployment and quantization; the drawback is that under equal compute it is hard to scale parameters to the level of MoE.

KV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.

Traditional dense models use all parameters for every inference, so the complete weights must reside in memory or VRAM, otherwise every step of computation triggers a disk read.

The advantage of a dense architecture lies in more predictable behavior, more straightforward fine-tuning, and easier domain adaptation to achieve stable performance, at the cost of a relatively fixed inference computation load that cannot save computing power through sparse activation.

Traditional dense models activate all parameters when processing every token.

In dense models, all parameters participate in computation for every token, and the amount of computation is strictly tied to the parameter scale.

Compared to MoE architectures, dense models have more uniform gradient flow during training, are less prone to expert collapse, and exhibit more deterministic inference behavior.