llm-d

On heterogeneous hardware (mixed GPU and TPU clusters), a unified API does not equal unified performance, and performance tuning still requires empirical verification.

llm-d supports running across vLLM, SGLang, and heterogeneous TPU/GPU hardware.

llm-d 1.0 introduces cache-aware routing, which incorporates cache state into routing decisions and directs requests to backend instances that already hold the relevant context.

llm-d 1.0 adds production-oriented engineering capabilities such as multi-tenant flow control, zero-downtime upgrades, and agent-aware routing.

Multi-tenant flow control lets different teams or applications sharing the same inference cluster obtain isolated quotas and priorities, preventing a single high-concurrency task from dragging down the whole system.

Zero-downtime upgrades handle canary rollout and draining at the scheduling layer, reducing the risk of version iteration.

Agent-aware routing reduces cross-node state synchronization overhead in multi-step agent workflows by being aware of the call graph.

llm-d extends the Kubernetes API in the form of custom resource definitions (CRDs) and controllers, reusing its declarative configuration, service discovery, health checking, and elastic scaling mechanisms.

llm-d introduces semantics similar to the Gateway API Inference Extension.

llm-d is positioned as an inference service mesh handling east-west traffic governance between services (routing, retries, circuit breaking, tenant isolation), rather than a north-south ingress gateway.

Teams already running Kubernetes inference workloads at multi-node scale are llm-d's most direct target users; at the single-GPU or single-node stage, the marginal benefit of introducing a scheduling layer is limited.

Google, Red Hat, and IBM jointly released llm-d 1.0, a Kubernetes-native open-source distributed inference stack.