DeepSeek-Infer
An internal inference framework open-sourced by DeepSeek, deeply customized for its models and hardware characteristics. It supports large-scale online serving and includes optimization techniques such as speculative decoding and one-step scheduling.
Timeline (last 90 days)
DeepSeek-Infer specifically optimizes model execution, sampling, and operators, and takes multi-GPU work partitioning and expert load balancing into account.
DeepSeek-Infer employs Speculative Decoding technology.
DeepSeek-Infer supports limited replay with a sliding window, replaying only a small recent stretch of tokens when restoring state to reduce redundant computation.
DeepSeek-Infer uses One-step Scheduling, overlapping scheduling decisions with GPU computation to hide scheduling overhead.
DeepSeek-Infer has batch invariance, ensuring that the model's result for the same request remains consistent no matter which batch it is computed in.
DeepSeek-Infer supports structured output, for example requiring the model to reply in a specified JSON format.
What DeepSeek-Infer has open-sourced this time is the offline inference part.
The DeepSeek-Infer open-source repository does not include a complete KVCache system, vision encoding and separated Prefill/Decode deployment, or request load balancing across multiple replicas.
Currently the hardware supported by DeepSeek-Infer is NVIDIA SM100 GPUs and Huawei Ascend 950.
All Facts (9)
DeepSeek-Infer specifically optimizes model execution, sampling, and operators, and takes multi-GPU work partitioning and expert load balancing into account.
50%UnverifiedDeepSeek-Infer employs Speculative Decoding technology.
50%UnverifiedDeepSeek-Infer supports limited replay with a sliding window, replaying only a small recent stretch of tokens when restoring state to reduce redundant computation.
50%UnverifiedDeepSeek-Infer uses One-step Scheduling, overlapping scheduling decisions with GPU computation to hide scheduling overhead.
50%UnverifiedDeepSeek-Infer has batch invariance, ensuring that the model's result for the same request remains consistent no matter which batch it is computed in.
50%UnverifiedDeepSeek-Infer supports structured output, for example requiring the model to reply in a specified JSON format.
50%UnverifiedWhat DeepSeek-Infer has open-sourced this time is the offline inference part.
50%UnverifiedThe DeepSeek-Infer open-source repository does not include a complete KVCache system, vision encoding and separated Prefill/Decode deployment, or request load balancing across multiple replicas.
50%UnverifiedCurrently the hardware supported by DeepSeek-Infer is NVIDIA SM100 GPUs and Huawei Ascend 950.
50%