DFlash2

Change #19149 adds DFlash2 support to MLX, an expansion of that backend's support matrix.

The test is based on the Qwen3.8-27B model, using EXL3 4.0bpw quantization, a DFlash2 block drafter (EXL3 4bpw, 7-token draft), greedy decoding, running on 2 T4 GPUs.

On the same benchmark with the Qwen3.8-27B model, DFlash2 achieves a 2.80x speedup, DSpark 2.57x, and MTP 2.22x.

Beam combined with DFlash2 speculative decoding achieves 1.72x to 3.03x generation speedup depending on the task and hardware configuration, while preserving the full model's exact output distribution.

Draft-flash-style schemes (such as DFlash2) let the draft phase reuse part of the main model's computation, making them more sensitive to KV cache and VRAM contention under long contexts, which is the cause of the 20k-context crash in the article.

HyperQwen is an Apache-2.0-licensed custom vLLM build released by GitHub user syv-ai, tuned for the Qwen3.8-27B model in the W4A16 quantisation format, with weights of about 19.5GB and DFlash2 speculative decoding.

The base model's 1M context is achieved by stretching its native 262,144 positions via YARN override, and the DFlash2 draft model cannot run simultaneously with the 1M context, so one must choose between 1M without a draft model and 256K with a draft model.

Below 64k, the phone cannot accelerate generation speed; generation speedup is achieved by custom kernels in the fork, including SME2 instructions on the M4 CPU, Metal fused operators, and DFlash2 speculative decoding, pushing generation from the original llama.cpp's 11.3 tok/s to approximately 25 tok/s at around 30k context.

vLLM v0.28.0 introduces DFlash2 and DSpark confidence-scheduled verification in the speculative decoding domain.

Injecting concept representation into the draft model of DFlash2 can improve the mean acceptance length by 4.17%.

When the cache hit boundary is not aligned with the block boundary, if the hit endpoint falls in the middle of a block, it triggers recomputation of the entire block, even though most tokens could have been reused.

DFlash2 packs consecutive tokens into cache blocks of a fixed size (typically 16 tokens) under a mixed-layout scheme.

In actual testing, the draft acceptance rate was only approximately 41%, with the draft model outputting 1,055 tokens but only approximately 438 being accepted.

At approximately 52 concurrent connections, the system is essentially 100% saturated, generating a maximum of approximately 704 tokens per second, but with latency spiking to 64 seconds.

Methods for improving the draft acceptance rate include using early checkpoints of the main model as the draft model, fine-tuning the draft model on domain-specific data, and using a stronger small model as the draft model.

The official figure states that DFlash2 achieves an average of 6 accepted tokens, but in actual testing the higher end was approximately 58%–60%, generally between 40%–50% (with 3–4 tokens actually accepted out of every 7).

Under DFlash2's hybrid layout, the draft model generates a block of 16 tokens at a time, and during verification the entire 16-token cache block must be deleted and recomputed, rather than discarding only a single token.

Reflection AI released the companion draft model DFlash2, with 1.93 billion parameters, designed specifically for speculative decoding.

Beam ships with DFlash2, a speculative decoding drafter optimized for agent latency.

vLLM v0.28.0rc2 introduces a new speculative decoding feature called DFlash2, which combines two major technical components: local convolution and a candidate selector (PR #52816).