DeepInfer
A high-performance, large-scale large-model inference framework open-sourced by DeepSeek, aimed at deployment and inference optimization for DeepSeek models. It involves engineering optimizations such as KV Cache management and batching strategies.
Core Facts
DeepSeek is developed by DeepSeek (深度求索) company.
The header of DeepInfer's LLM class file states 'Derived from vLLM, Modified by DeepSeek', indicating it is built on vLLM and also references SGLang and TensorRT-LLM.
DeepSeek V4.1 Flash scores lower than the previous version on GPQA, HLE, and other benchmarks.
Timeline (last 90 days)
vLLM used a DEP32 parallelism strategy in the comparison, while DeepInfer adopted an EP64DP64 configuration.
Across batch sizes from 8 to 256, vLLM's decoding performance is on par with DeepInfer.
The open-source version of DeepInfer covers only one model (DeepSeek V4.1 Flash) and two types of hardware (NVIDIA SM100 and Ascend 950), and supports only offline inference; KV Cache storage, PD disaggregation, load balancing, and online serving are all not included.
DeepInfer will not replace vLLM and SGLang, because it does not do prefix caching or online serving, but it can serve as a reference showing which state must be stored and which can be recomputed.
DeepInfer sets three expectations for hardware: expert weights in FP4, activations and KV in FP8, and scaling factors in the low-precision UE8M0 format; KV Cache per token under 1KB; and the bottleneck shifting to large-scale high-speed interconnect and device-side communication.
In DeepInfer, Ascend and NVIDIA are first-class citizens, with 30 TileLang operators on each side corresponding one-to-one, plus Ascend versions of DeepGEMM and DeepEP.
DeepInfer integrates mechanisms such as DeepSpark speculative decoding with consistency scheduling verification, SWA bounded replay, asynchronous zero-overhead scheduling, and structured output.
DeepInfer is currently an offline engine that does not provide out-of-the-box online service capability, and is closer to an inference kernel and scheduling reference implementation than a finished product that can directly replace vLLM production deployments.
DeepSig promises to continuously update the inference code for new models into the DeepInfer repository.
According to the video author's view, the significance of open-sourcing DeepInfer lies in letting the community see the internal implementation of a production-grade inference engine, which helps reduce compute costs and energy consumption.
40 more timeline events
All Facts (20)
DeepSeek is developed by DeepSeek (深度求索) company.
80%VerifiedThe header of DeepInfer's LLM class file states 'Derived from vLLM, Modified by DeepSeek', indicating it is built on vLLM and also references SGLang and TensorRT-LLM.
75%VerifiedDeepSeek V4.1 Flash scores lower than the previous version on GPQA, HLE, and other benchmarks.
75%VerifiedThe open-source version of DeepInfer is not a complete production system and does not include parts coupled with internal infrastructure, such as KV buffer management, cross-machine prefill-decode disaggregation, and load balancing across replicas.
70%VerifiedDeepInfer supports two types of hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
70%VerifiedThe vLLM team integrated the new kernels open-sourced by DeepSeek into the inference engine, further improving underlying computational efficiency.
65%VerifiedIn its July changelog, DeepSeek credited its internal Harness (the "minimal mode") as the behind-the-scenes contributor to V4 Flash's programming benchmark scores.
65%VerifiedDeepSeek adopts a relatively permissive license that allows commercial secondary development.
65%VerifiedDeepSeek open-sourced the inference framework DeepInfer on October 20, providing optimized operators, multi-card parallelism and asynchronous scheduling capabilities for V4.1 Flash, under the Apache 2.0 license.
80%Partially VerifiedDeepInfer is an open-source state-of-the-art (SOTA) inference system.
70%UnverifiedA single DeepSeek cluster processes 155 trillion tokens per day, equivalent to generating about 1.8 billion words per second.
60%UnverifiedWhat DeepInfer open-sourced is the offline inference core; the complete online service chain still requires filling in the caching layer and load balancing yourself.
60%UnverifiedAccording to the video information, DeepInfer is compatible with both the NVIDIA and Huawei Ascend platforms, a single cluster can reach a daily throughput of 155 trillion tokens, and each GPU can earn $38 per hour.
60%Partially VerifiedDeepSeek has officially open-sourced its inference framework DeepInfer, with the news spreading from Bilibili.
65%UnverifiedAcross batch sizes from 8 to 256, vLLM's decoding performance is on par with DeepInfer.
50%UnverifiedvLLM used a DEP32 parallelism strategy in the comparison, while DeepInfer adopted an EP64DP64 configuration.
50%UnverifiedDeepInfer will not replace vLLM and SGLang, because it does not do prefix caching or online serving, but it can serve as a reference showing which state must be stored and which can be recomputed.
50%UnverifiedDeepInfer sets three expectations for hardware: expert weights in FP4, activations and KV in FP8, and scaling factors in the low-precision UE8M0 format; KV Cache per token under 1KB; and the bottleneck shifting to large-scale high-speed interconnect and device-side communication.
50%UnverifiedThe open-source version of DeepInfer covers only one model (DeepSeek V4.1 Flash) and two types of hardware (NVIDIA SM100 and Ascend 950), and supports only offline inference; KV Cache storage, PD disaggregation, load balancing, and online serving are all not included.
50%UnverifiedIn DeepInfer, Ascend and NVIDIA are first-class citizens, with 30 TileLang operators on each side corresponding one-to-one, plus Ascend versions of DeepGEMM and DeepEP.
50%