#推理优化
87 related articles
The "Uneven Performance" of Speculative Decoding: 7 Tokens Accepted per Step on Code, Only 2.5 on General Prose — Ceiling or Optimizable?
Reddit tests show a block drafter accepts ~7 tokens/step on code but only 2.5 on prose. Multi-source validation points to drafter characteristics, not runtime issues.
Stanford & Berkeley TokenGate: Up to 11.3× Faster Long-Context Prefill
Stanford and UC Berkeley release TokenGate, a learned token pruning module that speeds up long-context LLM prefill TTFT by 1.4–11.3× and cuts peak memory by 1.5–3.4× with no accuracy loss. This article explains its mechanism and practical value.
DeepSeek Open-Sources DeepInfer: Inference Costs Slashed and Commercial Licensing Fully Open
DeepSeek open-sources DeepInfer, a permissively licensed inference framework for commercial use. Speculative decoding cuts payback from 10 to 4 months.
DeepSeek Open-Sources Inference Engine: 80% Gross Margin Crushes Silicon Valley, but Copycats Find the Assembly Line Missing
DeepSeek open-sources its inference engine DeepInfer and Rust gateway DeepSeek Recipe, revealing 155T tokens/day at 80% gross margin. A breakdown of causal encoder-decoder architecture, wide expert parallelism, EPLB, dSpark, bounded replay, and Ascend 950 support—plus the three closed-source barriers that remain.
Understanding vLLM and SGLang from First Principles: Deconstructing the Core Technologies of Inference Engines
Deconstructing vLLM and SGLang inference engines from first principles: how PagedAttention reduces memory fragmentation, how continuous batching boosts throughput, how RadixAttention and DSL scheduling optimize prefix reuse and complex generation workflows, plus selection advice.
TensorRT Global Performance Tuner: Breaking Past Single-Build-Route Performance Optimization
An analysis of NVIDIA TensorRT Global Performance Tuner: why single build routes can't reach hardware limits, how global tuning turns compilation from manual trial and error into configurable search, and its impact on inference deployment pipelines.
Kimi K3.1 Pro Dual Versions Debut: Highspeed Edition Steals the Spotlight
Kimi K3.1 Pro and K3.1 Pro Highspeed spark Hacker News debate. We break down the standard vs. highspeed trade-offs, community concerns over transparency and pricing, and practical guidance on model selection and architecture routing.
DeepSeek Open-Sources Inference Engine DeepInfer: All the Cost-Saving Tricks Laid Bare
DeepSeek open-sources inference engine DeepInfer and Recipe. We break down PD disaggregation, KV cache, SWA bounded replay, speculative decoding, and what $38 unit economics really means.
DeepInfer Goes Open Source: $330K Annual Revenue Per GPU — How a Inference Framework Is Disrupting AI Compute Economics
DeepSea open-sources DeepInfer, disclosing over $330K annual revenue per GPU and gross margins up to 96.94%. We break down how prefill/decode disaggregation and KV Cache disk offloading cut inference costs by 96%, and compare revenue certainty against mining.
DeepSeek Open-Sources Inference Engine DeepInfer: The AI Cost War and the Contest Over National Destiny
DeepSeek open-sources DeepInfer under Apache 2.0, boosting prefill throughput 25x and supporting NVIDIA SM100 and Huawei Ascend 950. A deep dive into its optimizations, open-source strategy, and the AI cost war.
DeepSeek Open-Sources Inference Framework Deepinflow: The AI Price War Moves into the Back Kitchen
DeepSeek open-sources its internal inference framework Deepinflow, revealing optimization experience in high-performance operators, parallelism strategies, and speculative decoding—challenging the high-priced API model of OpenAI and other giants by open-sourcing deployment efficiency to disrupt AI pricing logic and push large model inference from luxury good to public infrastructure.
DeepSeek's Open-Source Infra Map: What Ordinary Programmers Should Understand
A visual guide to DeepSeek's open-source infrastructure: Flash MLA, DeepGEMM operators, DeepEP communication, 3FS storage, and DeepInfer inference—what each library solves, how they fit together, and what ordinary programmers can take away.
DeepSeek Open-Sources DeepInfer Inference Framework: A Revolution in Local LLM Deployment Efficiency
DeepSeek open-sources DeepInfer, its internal inference framework integrating speculative decoding, MoE async scheduling, and high-performance operators to boost local LLM deployment efficiency.
DeepSeek Open-Sources DeepInfer: How a 72-Hour Context Cache Rewrites AI Inference Costs
DeepSeek releases open-source inference framework DeepInfer, using 72-hour context caching to slash AI inference costs—up to $38 revenue per GPU per hour. We break down the tech, hardware strategy, and the open vs. closed source battle.
DeepSeek Open-Sources DeepInfer Inference Engine — Why Did It Trigger a US Ban?
DeepSeek open-sources its inference engine DeepInfer for commercial use with impressive performance, yet it was banned by US federal civilian agencies. This article analyzes DeepInfer's technical features, the significance of open sourcing, and the game behind the ban.
DeepSeek Open-Sources Inference Framework DeepInfer: The Secret to Halving AI Inference Costs
DeepSeek open-sources DeepInfer inference framework, revealing online inference engineering secrets: PD separation, KV Cache, MoE load balancing, and speculative decoding—how it earns $38/GPU/hour, avoids recomputing 96% of input, plus open-source boundaries and industry impact.
4 DGX Sparks Test DeepSeek V4.1 Flash: How Good Is Local Inference Productivity?
Four DGX Sparks in a 200G DAC ring tested running DeepSeek V4.1 Flash locally: code generation 119.8 tokens/s, 369.9 tokens/s across 16 channels, ~5 min first token at 1M context, 0.4s after cache hit. Includes four agent task tests and deployment workflow.
GLOBE: Geometry-Guided Early Stopping for Reasoning — Halving Output Without Losing Accuracy
GLOBE is a training-free online early-stopping algorithm that uses unit-sphere semantic trajectories to decide when reasoning can stop, cutting output tokens by 49.7% and latency by 51.0% across five benchmarks while matching full CoT accuracy.
Build Your Own Inference Stack: A Practical Guide to Running Production-Grade AI Agents on Dedicated GPUs
Based on an Akamai workshop recording, a detailed guide to building a production-grade AI agent inference stack on dedicated GPUs: vLLM deployment, VRAM budgeting, real-world FP8 quantization gains, speculative decoding pitfalls, and concurrency tuning.
Deepgram Voice AI Optimization: Porting Nova-3 to AWS Trainium2 in Practice
Deepgram ported Nova-3 to AWS Trainium2 using the Neuron stack, kernel-level tuning, and a Rust data-parallel runtime to boost batch inference throughput and cut costs.