Activated Parameter Count
The number of parameters that actually participate in computation during a single forward pass of a model, decoupled from the total parameter count and determining the computational cost per token.
Core Facts
Timeline (last 90 days)
In a mixture-of-experts (MoE) architecture, the total parameter count is the sum of all experts, and the activated parameters are the part actually involved in computation in a single forward pass; VRAM must be able to hold all the weights, but the compute cost is only comparable to the activated parameters.
When evaluating architectures like reViT, one must distinguish three metrics: total parameter count, activated parameter count, and inference FLOPs, otherwise it is easy to reach misleading conclusions.
All Facts (2)
In a mixture-of-experts (MoE) architecture, the total parameter count is the sum of all experts, and the activated parameters are the part actually involved in computation in a single forward pass; VRAM must be able to hold all the weights, but the compute cost is only comparable to the activated parameters.
65%UnverifiedWhen evaluating architectures like reViT, one must distinguish three metrics: total parameter count, activated parameter count, and inference FLOPs, otherwise it is easy to reach misleading conclusions.
50%