vLLM v0.31.0rc4 Released: Fix for MTP Acceptance Collapse Under FULL CUDA Graph Mode

vLLM Releases v0.31.0rc4 Release Candidate

vLLM has recently tagged v0.31.0rc4 as a Release Candidate on GitHub. As one of the most popular large model inference engines available today, vLLM has accumulated over 93,000 stars and approximately 22,900 forks on GitHub, with a consistently active community. This rc4 release was tagged by maintainer LucasWilkinson, continuing the project's characteristic high-frequency iteration pace.

By naming convention, "rc" (Release Candidate) indicates the version is close to a stable release, with efforts focused on stability fixes and regression resolution rather than introducing major new features. For users who want to test or preview new functionality early, release candidates provide a relatively stable validation window.

Core Fix: MTP Acceptance Collapse Under FULL Graph Mode

The commit message for this release explicitly references a critical fix: [Bugfix][HiSparse] Fix MTP acceptance collapse under FULL graphs. This relates to MTP (Multi-Token Prediction), a mechanism tied to Speculative Decoding.

MTP has emerged as an important technique for accelerating large model inference. By predicting multiple future tokens in a single forward pass and then having the main model verify and accept them, it can significantly boost generation throughput. "Acceptance collapse" refers to a scenario where the proportion of MTP-predicted tokens that get accepted drops sharply under certain configurations, causing the acceleration benefit to vanish or even resulting in performance regression.

This fix specifically targets the acceptance collapse issue occurring in FULL graphs mode (full CUDA graph capture). CUDA Graph is a common technique for reducing inference kernel launch overhead, but when combined with complex decoding logic like MTP, it can introduce edge cases in state management. The HiSparse tag suggests this fix may also be related to sparse computation code paths.

The basic principle of Speculative Decoding is: a smaller "draft model" rapidly generates several candidate tokens, which are then verified in parallel by the large (main) model. Candidates that align with the main model's distribution are "accepted"; those that don't are "rejected" and re-sampled. Ideally, most tokens are accepted, significantly boosting throughput without sacrificing output quality. MTP is a variant of this approach that doesn't rely on an independent draft model. Instead, it attaches multi-head prediction layers directly to the main model's architecture to predict multiple future token positions at once. Models like DeepSeek^Ad natively support MTP during training, allowing seamless integration with speculative decoding at inference time without the overhead of deploying a separate draft model.

CUDA Graph is a mechanism provided by NVIDIA that pre-records a series of GPU operations (kernel launches, memory copies, etc.) into a static computation graph for replay as a whole. In the traditional per-kernel scheduling model, every operation incurs CPU-side scheduling overhead. CUDA Graph amortizes this into a one-time "graph launch," significantly reducing latency in small-batch, high-concurrency inference scenarios. vLLM's FULL graphs mode captures as much of the forward computation as possible into a CUDA Graph. However, MTP's verification logic involves dynamic token accept/reject decisions. If the relevant state updates are not correctly reflected in the input tensors of the static graph, the verification results become invalid — manifesting as a sudden drop in acceptance rate. The fix in rc4 directly addresses this state synchronization edge case.

The Significance of This Release Candidate and Upgrade Recommendations

For production users, release candidates are generally not recommended for direct deployment, but they serve as a critical testing stage before an official release. If your deployment uses MTP speculative decoding with FULL CUDA graph mode enabled, and you've previously experienced an unexplained drop in acceleration performance, rc4 is worth validating in a test environment first.

How to Obtain This Version

Users can download the corresponding tag from the Releases page on GitHub, or watch for the pre-release version being pushed to PyPI. Release candidates typically undergo a period of community validation before being merged into the official v0.31.0 release.

Looking Ahead to the Official Release

The rc4 designation implies that v0.31.0 has already gone through multiple rounds of candidate testing. Teams with high stability requirements are advised to wait for the official release, and to carefully review the complete Release Notes and Breaking Changes to assess the impact of upgrading on existing services.

Summary

vLLM v0.31.0rc4 is a release candidate focused primarily on bug fixes, with the core change resolving the MTP acceptance collapse issue under FULL CUDA graph mode. While fixes like this are less flashy than new features, they directly affect the usability and stability of inference acceleration techniques in real-world scenarios — reflecting vLLM's ongoing commitment to engineering excellence in high-performance inference. Since publicly available information on this release is limited, readers are encouraged to consult the official complete release notes for a comprehensive view of all changes in this version.