NVIDIA Dynamo
A distributed inference framework from NVIDIA designed for efficient serving of large language models across multi-GPU and multi-node clusters. Core capabilities include cross-node KV cache scheduling and reuse, dynamic request routing, and integration with orchestration systems such as Kubernetes.
Core Facts
Timeline (last 90 days)
Teams can start deployment directly from a validated DGD manifest; after applying it with kubectl, the Dynamo Operator reconciles the inference graph into actually running components that serve traffic.
DGDR (DynamoGraph Deployment Request) is an intent-driven deployment path: users only need to describe the model, backend, workload, hardware, and optional SLA targets, and Dynamo analyzes these inputs and generates a DynamoGraph Deployment (DGD).
For teams migrating a small existing endpoint, it is recommended to start with DGDR to get end-to-end running, verify that the OpenAI-compatible endpoint works, and then adjust the deployment according to capacity needs, rather than refactoring the entire stack.
The five-step DGDR workflow is: define the DGDR → apply it with kubectl → wait for the Operator to create and reconcile the DGD → send requests to the API endpoint → verify that responses come from the Dynamo endpoint.
NVIDIA Dynamo adopts the Operator pattern on Kubernetes, with the Operator continuously reconciling so that the cluster's actual state stays consistent with the user's declared desired state.
The Dynamo Operator is usually installed via the Dynamo platform Helm chart, which is the recommended way.
The prerequisites for deploying inference services on Kubernetes with Dynamo are: a Kubernetes cluster with access to the required GPUs, and an installed Dynamo Operator.
Dynamo's Workers can run the SGLang, TensorRT-LLM, or vLLM inference engines, depending on the deployment configuration.
The Dynamo frontend exposes an OpenAI-compatible API, clients call it using familiar routes, and the frontend routes requests to backend Workers.
NVIDIA Dynamo, combined with a Sidecar on vLLM's native Rust frontend, removes Python output processing from the critical path and implements KV-aware routing.
13 more timeline events
All Facts (20)
NVIDIA Dynamo is a distributed inference framework launched by NVIDIA, designed for efficient serving of large language models on multi-GPU, multi-node clusters.
75%UnverifiedTeams can start deployment directly from a validated DGD manifest; after applying it with kubectl, the Dynamo Operator reconciles the inference graph into actually running components that serve traffic.
50%UnverifiedFor teams migrating a small existing endpoint, it is recommended to start with DGDR to get end-to-end running, verify that the OpenAI-compatible endpoint works, and then adjust the deployment according to capacity needs, rather than refactoring the entire stack.
50%UnverifiedDynamo's Workers can run the SGLang, TensorRT-LLM, or vLLM inference engines, depending on the deployment configuration.
50%UnverifiedNVIDIA Dynamo adopts the Operator pattern on Kubernetes, with the Operator continuously reconciling so that the cluster's actual state stays consistent with the user's declared desired state.
50%UnverifiedThe Dynamo Operator is usually installed via the Dynamo platform Helm chart, which is the recommended way.
50%UnverifiedThe prerequisites for deploying inference services on Kubernetes with Dynamo are: a Kubernetes cluster with access to the required GPUs, and an installed Dynamo Operator.
50%UnverifiedDGDR (DynamoGraph Deployment Request) is an intent-driven deployment path: users only need to describe the model, backend, workload, hardware, and optional SLA targets, and Dynamo analyzes these inputs and generates a DynamoGraph Deployment (DGD).
50%UnverifiedThe Dynamo frontend exposes an OpenAI-compatible API, clients call it using familiar routes, and the frontend routes requests to backend Workers.
50%UnverifiedBefore configuring intelligent routing, aggregated serving requires 2 GPUs, while disaggregated serving requires 4 GPUs.
50%UnverifiedThe Dynamo container runs as UID 1000 and needs write permission to download model weights.
50%UnverifiedAccording to official data, compared with traditional round-robin load balancing, KV-aware routing can reduce time to first token (TTFT) by up to 40% and increase overall throughput by 25-30%.
50%UnverifiedDynamo's KV Router relies on etcd for worker discovery and registration, and NATS is responsible for real-time propagation of KV cache events.
50%Unverifiedetcd is a strongly consistent distributed key-value store, often used as a service discovery and configuration center; in Dynamo, each worker registers its own metadata with etcd at startup and periodically renews its lease.
50%UnverifiedDynamo's KV Router intelligently distributes inference requests by evaluating the computational cost of each worker.
50%UnverifiedThe industry usually compares LLMD with projects such as NVIDIA's Dynamo and KServe in the same track, with the common focus on how to simultaneously do well in GPU utilization and end-to-end latency under multi-node, multi-model, and multi-tenant conditions.
50%UnverifiedNVIDIA Dynamo's core capabilities include cross-node scheduling and reuse of KV Cache, dynamic routing of requests, and integration with orchestration systems such as Kubernetes.
50%UnverifiedTopograph can be combined with NVIDIA Dynamo to route inference requests to the GPU node physically closest to the cached KV data, reducing the communication overhead caused by cache migration.
50%UnverifiedNVIDIA Dynamo implements distributed request routing and service discovery based on NATS and etcd, supports efficient inter-GPU data transfer based on NIXL, and is compatible with inference backends such as TensorRT-LLM and vLLM.
50%UnverifiedThe five-step DGDR workflow is: define the DGDR → apply it with kubectl → wait for the Operator to create and reconcile the DGD → send requests to the API endpoint → verify that responses come from the Dynamo endpoint.
50%