Decode Stage
The decode stage is the second stage of large-model inference, in which the model generates output tokens one by one, with each token requiring a forward computation. Its speed depends on compute capacity and model size.
Timeline (last 90 days)
The prefill phase of large model inference is compute-intensive (matrix-matrix multiplication) and bound by compute; the decode phase is memory-intensive (matrix-vector multiplication) and bound by memory bandwidth.
Transformer inference is divided into two phases: a parallel compute-intensive prefill phase and a sequential memory-bandwidth-intensive decode phase.
All Facts (2)
The prefill phase of large model inference is compute-intensive (matrix-matrix multiplication) and bound by compute; the decode phase is memory-intensive (matrix-vector multiplication) and bound by memory bandwidth.
50%UnverifiedTransformer inference is divided into two phases: a parallel compute-intensive prefill phase and a sequential memory-bandwidth-intensive decode phase.
50%