PACE

The conventional approach assumes that the state needs to be updated precisely at every token, but deferred rounding combined with structure-aware encoding can greatly reduce memory requirements with almost no loss in accuracy.

The PACE method achieves post-hoc 4-bit recurrent state quantization through periodic rounding and channel-aligned encoding, and can be applied without fine-tuning.

PACE defers the rounding operation to once every few generated tokens, with intermediate tokens temporarily stored in a buffer, to reduce the accumulation of rounding error.

PACE shares the quantization scale for each column rather than quantizing each element independently, which better matches the state distribution and reduces quantization error.

PACE uses 8-bit precision for attention heads whose rounding errors persist longer, forming a mixed-precision scheme.

PACE with mixed 4-bit and 8-bit heads performs comparably to 8-bit DAMP on three larger models while using only about two-thirds of its memory.

PACE using 4-bit heads throughout scores higher than Q-Mamba at the same memory.

The strategy of rounding once every few tokens preserves the decoding speedup brought by the buffer.

PACE was evaluated on four hybrid models, with parameter sizes ranging from 4B to 35B.

PACE 测量到成功率强烈依赖任务，且对视界非单调，于是给出相位感知调度。