Parallel Prefix Sum
Parallel prefix sum is a classical algorithm that parallelizes recurrence operations satisfying the associativity property. It reduces O(n) serial computation steps to O(log n) steps and is the theoretical basis of chunked scan kernels.
Timeline (last 90 days)
Because the recurrence has a strict temporal dependency, selective scan cannot compute all positions in parallel at once as attention does, and must use blocked scan or parallel prefix sum algorithms to break the serial chain into two stages: parallelism within blocks and transfer between blocks.
Parallel prefix sum (parallel prefix sum / scan) is a classic algorithm for parallelizing associative recurrence operations, and algorithms such as the Blelloch scan can complete accumulation in O(log n) steps.
All Facts (2)
Because the recurrence has a strict temporal dependency, selective scan cannot compute all positions in parallel at once as attention does, and must use blocked scan or parallel prefix sum algorithms to break the serial chain into two stages: parallelism within blocks and transfer between blocks.
50%UnverifiedParallel prefix sum (parallel prefix sum / scan) is a classic algorithm for parallelizing associative recurrence operations, and algorithms such as the Blelloch scan can complete accumulation in O(log n) steps.
50%