#推理优化
87 related articles
Inside the LLMD Project: How Open Source Inference Achieves Production-Grade Scale
IBM Distinguished Engineer Carlos Costa shares LLMD project updates: how open source inference achieves production-scale with agentic serving performance, multi-tenant QoS, and zero-downtime ops, to be showcased with Red Hat and Google at PyTorch Conference.
Multi-Node LLM Inference Orchestration Topologies: Performance Bottlenecks and Architectural Trade-offs
A deep dive into multi-node LLM inference orchestration topology evaluation: the synchronization latency of cross-node tensor parallelism over standard Ethernet, and the predictability advantages and architectural trade-offs of disaggregated prefill/decode with KV-aware routing in high-concurrency multi-turn scenarios.
From Silicon to Token Economics: Full-Stack Optimization of AI Inference Infrastructure
Dylan Patel of SemiAnalysis joins Fireworks Forge to discuss full-stack AI inference optimization and token economics, from chip architecture to the application layer.
SGLang-Diffusion: A High-Performance Serving Framework for Diffusion Models
SGLang-Diffusion is a high-performance inference engine and serving stack for diffusion models, supporting offline and real-time workloads with parallelization and execution path optimizations. Debuting at PyTorch Conference 2026.
KVFetch: Filling the Missing Temporal Recall Channel in KV Cache Compression
New arXiv paper KVFetch identifies sequential forgetting in KV cache compression and proposes a training-free temporal recall channel, restoring verbatim copy scores from 0.8 to 78.4 on RULER-16K.
AttSVD: Attention-Guided SVD Compresses KV Cache to Half the Memory Without Accuracy Loss
AttSVD compresses KV cache via attention-guided SVD along the feature axis — no training required, 50% memory, full accuracy on LongBench and agentic benchmarks.
The GPT-6.1 Sol "Leak" Mystery: Looped and Nested Model Architectures Come Into Focus
A Reddit post reverse-engineers GPT-6/6.1 Sol, Astra, and Luna architectures from inference speed data, revealing looped transformers and nested models as likely strategies for large-scale, low-cost deployment.
The Hidden Cost of Inference Latency: Are You Overlooking Model Load Time?
Tests show model loading can take 13.9s vs 1.6s for inference itself — a 10x gap. This article breaks down model availability latency, state observability, and three production deployment strategies.
GLM-5.3 Hits 469 tok/s on 8×MI355X: A Deep Dive into vLLM + TileRT Disaggregated Inference
TileRT and AMD achieve 469 tok/s single-user decode for GLM-5.3 on 8×MI355X using a disaggregated architecture where vLLM handles prefill and TileRT handles latency-critical decode.
Canary Rollouts: A Practical Guide to Zero-Downtime AI Model Upgrades in Production
Canary rollouts enable zero-downtime AI model upgrades in production. Learn how phased traffic ramping, metric gates, and automatic rollback reduce risk in dedicated inference deployments.
The Rise of Overfit Inference Engines: How Specialization Is Reshaping Local AI Deployment
Overfit inference engines like Strata, ninfer, and DwarfStar are abandoning generality to maximize performance on specific hardware. Here's what that means for local AI.
Why Real-Time AI Inference Costs 56x More Than It Should — And How Persistent Kernels Fix It
Why does serving real-time AI cost 56x more than it should? We break down continuous vs. request-based inference, and how persistent GPU kernels, synchronous batching, and compile-time scheduling fix it.
Fireworks Launches Ember-1: Shorter Reasoning, 40% Fewer Tokens
Fireworks Research releases Ember-1, a reasoning model built on Kimi K3 that cuts token usage by ~40% while maintaining top-tier quality. Here's why it matters.
TPU Megakernel Achieves 709 tokens/s on Kimi K3, Outpacing GB200
The @inferact team open-sources a TPU megakernel for Kimi K3 achieving 709 tokens/s, beating GB200's 450 tokens/s via 92-layer MoE fusion and cross-layer weight prefetching in a single Pallas kernel.
Tech FrontiersOpenVINO vs Standard CPU Inference: Real-World Comparison on i5 and GTX 1650 Ti
Reddit test: OpenVINO matches standard CPU EP on i5-10300H, while DirectML significantly accelerates inference on mobile GTX 1650 Ti. Analysis of results, causes, and deployment takeaways.
16 FPS on a Single RTX 5090: Optimizing LingBot-World 2.0 for Real-Time Inference
A developer optimizes LingBot-World 2.0 1.3B to 16 FPS on a single RTX 5090 — 2.5× faster than SGLang and 1.9× faster than NVIDIA FlashDreams — using low-precision ops, SageAttention, and custom CUDA kernels.
vLLM Deployment of Kimi K3 Achieves 2.2–2.8x Throughput Gain: A Performance Optimization Breakdown
Kimi K3 on vLLM achieves 2.2–2.8x throughput gains on B300 benchmarks over v0.27.1, driven by optimizations in scheduling, KDA state handling, and MoE kernels.
Inference Networks: Optimizing LLM Inference Costs with Threshold Policies
A new arXiv paper proposes the Inference Networks graph framework for optimizing LLM inference costs, proving that optimal model activation follows a threshold structure for significant cost reduction.
Jev: A New Frontier Model Claiming 40–400x Lower Cost and 20–200x Faster Inference
New frontier model Jev claims 40–400x lower cost and 20–200x faster inference. We break down the efficiency claims, likely technical approaches, and how to evaluate them.
Understanding KV Cache: The Space-Time Tradeoff Behind LLM Inference Optimization
A deep dive into KV Cache for LLM inference: how it eliminates redundant Key/Value recomputation in autoregressive generation, with complexity analysis and memory estimation formulas.