#推理优化
87 related articles
One Machine Becomes Two: LLM Inference Disaggregation and the Engineering Logic Behind a 75% Throughput Gain
Explaining PD disaggregation for LLM inference: from Splitwise to Prime Inference's GB200 tuning, 75% more requests on the same GPUs, with KV transfer at 3-4% of TTFT.
vLLM Introduces Push-Based KV Transfer: 3x TTFT Improvement in Disaggregated Inference
vLLM introduces a push-based NIXL connector to optimize KV cache transfer in disaggregated inference. On Amazon EC2 P5en with AWS EFA, TTFT improves up to 3x and TPOT by 30%.
How KV Cache Reshapes AI Agent Cost Structures
Analyzing how KV Cache affects AI Agent operating costs: shrinking cache saves memory, reusing prefixes cuts redundant compute, nearby storage reduces transmission latency — each saves different resources.
vLLM P/D Disaggregation: The Calculation Rule Where KV Transfer Cost Determines Benefit
Is vLLM prefill/decode disaggregation worth it? We break down KV cache transfer bytes and bandwidth ratios, showing overhead for 70B models from 100GbE to 400Gb links, with a p99 ITL measurement script and a decision table.
New Breakthrough in Peak Memory Scheduling: Exact Implementation and Provable Optimality
This article analyzes the latest research on peak memory scheduling: achieving reliable peak memory reduction through exact modeling and statement permutation, providing provable optimality certificates, and exploring the theoretical boundaries of selective recomputation, offering new ideas for edge AI inference optimization.
DeepSeek V4.1 Flash 1017 Technical Breakdown: How the MoE Architecture Breaks Through the Throughput Bottleneck
A deep technical breakdown of DeepSeek V4.1 Flash 1017 (Max): how MoE dynamic routing solves expert starvation, how MLA compresses the KV cache to break the memory wall, and the kernel-level tuning and deployment constraints behind TTFT and throughput gains.
Veloce-Engine: The Technical Inside Story of Real-Time AI Inference Orchestration
A deep dive into the Veloce-Engine real-time AI inference orchestration layer: adaptive routing, semantic caching that cuts redundant calls by 70%, rate-limit retries, and streaming responses. Learn about its architecture, use cases, and pricing.
GPUStack vLLM Production Deployment: KV Cache Pinning and OOM Avoidance in Practice
Real-world ops lessons from a four-node mixed GPU cluster: pinning KV Cache in GPUStack + vLLM production deployments, avoiding 70B model OOM, and diagnosing TTFT and RSS decay.
DuckDB and Arrow Dominate AI Data Preparation—Why Do They Hit a Wall in Inference Serving?
Columnar engines like DuckDB and Arrow excel at AI data preparation but hit bottlenecks in inference serving. This article analyzes the structural reasons and presents an architecture that separates offline columnar storage from online row-based storage, emphasizing Arrow's true value as a zero-copy exchange format.
Implementing GPT-2 KV Cache from Scratch: Correctness, Latency, and Memory Benchmarks
Implementing GPT-2 small KV Cache from scratch with benchmarks: 6.1e-5 logit error, only 6x single-token speedup at context 1000 (not 1000x), 72KiB/token formula plus 8.5MB cuBLAS workspace offset, and a multi-token decoding failure case.
llama.cpp MoE Cache Benchmark: Doubling 30B Model Decode Speed on a T4
llama.cpp's new MoE expert cache doubles Qwen3-30B-A3B decode speed on a free Tesla T4, from 22.6 to 43.2 tokens/s. Cache mechanics, config script, benchmarks, and concurrency pitfalls.
NVFP4 vs FP8 vs GGUF: 2.3x Faster Decoding on 16GB Ada GPUs
Same model Nemotron Dory 1.0 4B compared on RTX 4070 Ti SUPER across NVFP4, FP8, MXFP8, GGUF, and BF16: NVFP4 decodes 2.3x faster, matches BF16 on GSM8K, FP8 leads long-prompt prefill, MXFP8 best token-level fidelity.
Full-Stack Optimization for LLM Inference: GPU, CPU, and Frontend Acceleration
Explore full-stack optimization for LLM inference: GPU-layer wide-EP with EPLB for load balancing, CPU All-Reduce tuning, and NVIDIA Dynamo with a Rust frontend for KV-aware routing.
AMD MI355X Beats B300 in Inference Performance: 24.4% Higher Throughput on 8 GPUs
AMD MI355X shows inference strength in SemiAnalysis AgentX: 7,567 tok/s per GPU on 8 GPUs, 24.4% above NVIDIA HGX B300 and 87.4% more tokens per dollar. We break down the vLLM optimizations and AMD kernels that narrowed the gap.
Cloud T4 Falls Short of Spec: Gemma 4 Decoding Slows 7% Due to Heat
Measured thermal drift on a single Google Cloud Tesla T4 with Gemma 4 E2B inference: 115.3 tokens/s decoding at a cold 45°C, falling to 106.8 at a hot 77°C — a sustained 7.4% slowdown. Thermal throttling never triggers; the power cap and leakage current are the real culprits.
SSR Framework: Liberating Small-Model Multimodal Search Agents from Lengthy Reasoning
SSR framework proposes a selection-over-generation multimodal reasoning paradigm, freeing small-model search agents from lengthy ineffective reasoning and dramatically cutting inference costs for resource-constrained scenarios.
Delta Tokenizer: How Session Deltas and Multi-Core Parallelism Speed Up Long-Prompt Tokenization by 40x
Delta Tokenizer is a CPU tokenization frontend that uses session-delta encoding and multi-core segmented parallelism to speed up single-turn tokenization by ~40x, cutting median TTFT by 59% at 100K tokens in SGLang and P99 TTFT by 9.5x under load.
GeoThinker-F: A New Framework That Frees Multimodal LLMs from External 3D Encoders for Spatial Intelligence
GeoThinker-F proposes an internally predicted world representation framework that frees multimodal LLMs from external 3D geometry encoders through two-stage decoupled training, achieving 3.6x faster inference and reducing latency from 1.45s to 0.40s while maintaining spatial reasoning performance.
DoubtLess: Training-Free Contrastive Decoding That Helps Reasoning Models Think Less and Answer Right
DoubtLess is a training-free contrastive decoding method that uses a small reasoning model to guide large models toward shorter chain-of-thought. On seven benchmarks, it cuts Qwen3-32B's CoT by 31% with unchanged accuracy, reduces thinking tokens per turn by 20% in coding agent scenarios, and generalizes across model families.
DeepSeek V4.1 Flash: How Read-Write Separation Reshapes Long-Conversation Costs
DeepSeek V4.1 Flash uses read-write separation—shorter input paths, shared attention memory, and selective reconstruction—to restructure inference costs in long conversations.