BitMTP
An LLM acceleration method that jointly designs multi-token prediction (MTP) with an ultra-low-bit quantized backbone. A single model supports both autoregressive generation and block-causal drafting, and combined with accepted-prefix alignment and CUDA Graph reuse it enables accelerated speculative decoding.
Timeline (last 90 days)
BitMTP was proposed by the PolyU-VCLab research team, and its core mechanism is a joint next-token prediction and multi-token prediction (MTP) objective that equips a single model with both autoregressive generation and block-causal drafting capabilities, without maintaining a separate draft model.
BitMTP offers three optional decoding modes: confidence-driven decoding, speculative decoding, and hybrid decoding, corresponding to different quality-throughput trade-off points.
BitMTP introduces an accepted-prefix alignment mechanism that requires the committed prefix, the materialized causal KV boundary, and the CUDA Graph replay state to remain consistent, in order to handle the partially accepted intermediate states in speculative decoding.
BitMTP implements width-specialized low-bit execution, selecting appropriate low-bit kernels for different batch or sequence widths and reusing captured computation graphs on top of shared KV storage.
Paper evaluations show that, with hybrid decoding on NVIDIA H20, BitMTP achieves a 2.2x five-task average speedup on BitNet b1.58-2B and 2.1x to 2.4x on the 1.7B to 4B Bonsai model series, with comparable task quality.
Paper evaluations show that, on RTX 3090 and Jetson Orin, BitMTP achieves up to 2.0x five-task average throughput improvement, with the MBPP+ single task reaching 3.3x.
BitMTP's source code has been open-sourced in the PolyU-VCLab/BitMTP repository on GitHub.
The speedup ratios given in the paper were all measured under specific model and task combinations, and actual business benefits need to be evaluated in light of one's own workload.
All Facts (8)
The speedup ratios given in the paper were all measured under specific model and task combinations, and actual business benefits need to be evaluated in light of one's own workload.
50%UnverifiedBitMTP offers three optional decoding modes: confidence-driven decoding, speculative decoding, and hybrid decoding, corresponding to different quality-throughput trade-off points.
50%UnverifiedBitMTP introduces an accepted-prefix alignment mechanism that requires the committed prefix, the materialized causal KV boundary, and the CUDA Graph replay state to remain consistent, in order to handle the partially accepted intermediate states in speculative decoding.
50%UnverifiedBitMTP implements width-specialized low-bit execution, selecting appropriate low-bit kernels for different batch or sequence widths and reusing captured computation graphs on top of shared KV storage.
50%UnverifiedPaper evaluations show that, with hybrid decoding on NVIDIA H20, BitMTP achieves a 2.2x five-task average speedup on BitNet b1.58-2B and 2.1x to 2.4x on the 1.7B to 4B Bonsai model series, with comparable task quality.
50%UnverifiedPaper evaluations show that, on RTX 3090 and Jetson Orin, BitMTP achieves up to 2.0x five-task average throughput improvement, with the MBPP+ single task reaching 3.3x.
50%UnverifiedBitMTP was proposed by the PolyU-VCLab research team, and its core mechanism is a joint next-token prediction and multi-token prediction (MTP) objective that equips a single model with both autoregressive generation and block-causal drafting capabilities, without maintaining a separate draft model.
50%UnverifiedBitMTP's source code has been open-sourced in the PolyU-VCLab/BitMTP repository on GitHub.
50%