#推理优化
85 related articles
StepBack: Step-Level Error-Localized Resampling Cuts Reasoning Tokens by 41%
StepBack introduces step-level error-localized resampling: locate the first erroneous step in draft trajectories, resample that step and its suffix, matching self-consistency accuracy on seven math reasoning and multi-hop QA benchmarks while cutting token usage by up to 41.26%. It can also be combined with adaptive reasoning methods such as Adaptive Consistency.
LooM: Loop-Invariant Memory Makes Loop Transformer Inference Budget Adjustable
LooM proposes loop-invariant memory to solve three loop Transformer issues: memory growing with loops, coarse computation granularity, and hard-to-change loop counts. Looping only in CED decoder layers keeps the KV cache from growing with loops, supporting a continuous budget of 3 to 21 layer executions per token with a single model covering all budgets and no router.
MoE Routing Reveals Reasoning Behavior, Cutting Test-Time Compute by 30% Without Training
New research uses MoE routers for training-free control of reasoning behavior, cutting test-time compute by 30.4% while accuracy rises. Learn how routing-score editing suppresses overthinking and optimizes AI inference costs.
DeepSeek Open-Sources Inference Engine: The Inside Story of Tripling GPU Value
DeepSeek open-sources DeepInfer, its production inference engine, using pre-fill/decoding separation, wide expert parallelism, and 96% KV cache hit rate to lift revenue per GPU-hour from $12.91 to $38 with 84%-92% gross margin.
Polymorphic Inference: Making Inference Engines Self-Adaptive Like Living Organisms
Polymorphic Inference offers self-tuning inference in response to constant changes in models, traffic, and chips. Together Compiler uses AI agents to keep production inference tuned. This article analyzes its approach and its significance for the open-source inference ecosystem.
Together Compiler: Continuous Compilation Optimization for AI Inference
Together Compiler uses agents and a persistent engine for continuous AI inference compilation, claiming open model optimization within 24 hours and 1.3x to 3x throughput gains.
Adversarial Speculative Reasoning: A New Decoding Approach for More Reliable Long-Horizon Agents
Speculative decoding isn't just an accelerator. Numerical noise in the verification window alters long-horizon agent outputs; training the draft model to be actively rejected at key reasoning points boosts end-to-end success without changing the target model.
KVCache Meetup Korea 2026: PyTorchKR and XCENA Explore KV Cache Optimization Together
KVCache Meetup Korea 2026, co-hosted by PyTorchKR and XCENA, focused on KV cache optimization and inference acceleration. This article covers the event background, KV cache technology trends, and practical takeaways for developers.
Deep Dive into LLM Mechanisms: Tokens, Attention, and KV Cache Dynamics
An in-depth analysis of LLM core mechanisms: how tokenization affects cost and context windows, the complexity bottlenecks and optimization directions of attention computation, and the memory trade-offs of KV caching in inference acceleration. Based on an in-depth Reddit community discussion, it outlines LLM underlying logic and engineering practice insights.
ACCT: A New Method for Cross-LLM KV Cache Transfer
ACCT proposes a cross-LLM context transfer method using Cartographer, Allocator, and Compiler to convert source model KV cache into sparse KV for direct consumption by the target model, reducing KV cache by 75% while maintaining performance, with KV conversion 6.5-21.2x faster than prefill at 19K context.
TraceSpec: A Speculative Decoding Optimization Framework for LLM Agents
TraceSpec improves speculative decoding for LLM agents via trajectory evaluation and adaptive draft training, achieving 12.5%-22.6% throughput gains and revealing local acceptance declines hidden by aggregate metrics.
Running a 68GB Large Model on a 12GB GPU: Betelgeuse Engine Benchmarked at 22 tok/s
Developer releases Betelgeuse inference engine, running a 68GB Qwen 3.8 Flash Next MoE model at 22 tok/s sustained (48 tok/s peak) on an RTX 3060 12GB + 16GB DDR4 PC with 100% bit-exact output.
DeepSeek Open-Sources DeepInfer: The Cost-Saving Tricks Are There, the Scheduling Isn't
DeepSeek open-sources production inference engine DeepInfer, disclosing PD disaggregation, KV cache, and Dspark speculative decoding—but cluster scheduling stays closed. We break down the cost logic behind the 4x revenue-to-rent ratio and the open-source boundary.
Altman Says You're Wasting DRAM: The Truth Behind AI's Memory Bottleneck
Altman's "you're wasting DRAM" remark sparked HN debate. This article breaks down common sources of DRAM waste in AI inference, the memory wall's impact on cost, and actionable memory optimization ideas for developers.
vLLM Singapore Meetup: Focus on High-Throughput Inference and Agent Infrastructure
vLLM Singapore Meetup on November 13 at SQ Collective features engineers from inferact, NVIDIA, Alibaba Cloud, and vLLM-Omni discussing high-throughput prefill, RL, Blackwell, Vera Rubin, Agent infrastructure, and interactive world models.
What Should MoE Verification Actually Count? The Math Behind Four Tokens and Eight Experts
In MoE speculative decoding, verifying multiple draft tokens can touch different expert weights—four tokens and eight experts may involve up to 32 routed experts. This article explains why unique expert count should complement accepted token count, and how profilers can expose expert overlap at the routing layer.
vLLM Catches Up to DeepSeek's In-House Engine DeepInfer: The Technical Dynamics Behind Decoding Performance
vLLM matches DeepSeek's in-house DeepInfer engine under the DeepSeek-V4.1-Flash decoding config, using only DEP32 against EP64DP64. This article analyzes the technical background, practical implications for developers, and how open-source collaboration is closing the inference engine gap.
vLLM Achieves Best Result on AgentX: Breaking DeepSeek Inference Record
vLLM sets the best DeepSeek-V4.1-Flash result on AgentX with real agentic workloads, achieving 954K tokens/s/GPU throughput and $38.96/GPU-hour revenue, matching DeepInfer.
Jev in Practice: A Decision Model That Replaces Token-by-Token Decoding with a Single Prefill
Breaking down Jev's inference paradigm: skip token-by-token decoding and use a single prefill forward pass to output choices, ordinal scores, and booleans. Comparing encoder and decoder paths, multi-question mask design, duplicate token pitfalls, and RLCD with GRPO for probability calibration training.
v0.40.12 Pre-release: MLX Inference Speedup and Gemma Vision Detail Optimization
v0.40.12 pre-release breakdown: MLX backend boosts prompt speed for gemma/flint/laguna, improves small-image detail in Gemma 4 compact vision tower, adds DFlash2, and syncs llama.cpp to b11901.