NIXL
A low-latency KV-cache transfer library provided by NVIDIA, designed for disaggregated LLM inference scenarios. It efficiently transfers KV caches from prefill instances to decode instances after prefill computation is complete.
Timeline (last 90 days)
NIXL (NVIDIA Inference Xfer Library) is a high-performance data transfer library designed specifically for AI inference scenarios, supporting GPUDirect, RDMA, and multiple network backends.
In push mode, the prefill instance proactively pushes the KV cache to the decode instance after completing KV cache generation, and the decode instance only needs to wait for the data to arrive, so transfer coordination no longer blocks the decode instance's token generation loop.
NIXL provides low-latency, high-throughput asynchronous transfer primitives and allows applications to customize transfer coordination logic.
The vLLM team introduced a push-based NIXL connector that moves transfer coordination off the critical path.
In vLLM, NIXL is commonly used to implement cross-node KV cache transfer under a Prefill/Decode disaggregated architecture
The key change in v0.30.0rc2 is a fix for notification-only requests in the NIXL transfer component to prevent them from triggering unnecessary receive reports
NIXL (NVIDIA Inference Xfer Library) is a library used to accelerate inference data transfers
NIXL typically leverages high-speed interconnect technologies such as NVLink, InfiniBand, or RDMA at the underlying level to transfer data directly between GPU memory, bypassing the CPU
NIXL is a low-latency KV cache transfer library provided by NVIDIA, designed specifically for disaggregated inference scenarios.
The validation environment uses eight NVIDIA A40 GPUs, each running one vLLM engine, with KV caches transferred between resource pools via NIXL.
1 more timeline events
All Facts (12)
NIXL是SGLang用于Prefill-Decode分离部署的关键通信组件，负责在Prefill节点和Decode节点之间传输KV Cache状态
85%UnverifiedNIXL provides low-latency, high-throughput asynchronous transfer primitives and allows applications to customize transfer coordination logic.
50%UnverifiedIn push mode, the prefill instance proactively pushes the KV cache to the decode instance after completing KV cache generation, and the decode instance only needs to wait for the data to arrive, so transfer coordination no longer blocks the decode instance's token generation loop.
50%UnverifiedNIXL (NVIDIA Inference Xfer Library) is a high-performance data transfer library designed specifically for AI inference scenarios, supporting GPUDirect, RDMA, and multiple network backends.
50%UnverifiedThe key change in v0.30.0rc2 is a fix for notification-only requests in the NIXL transfer component to prevent them from triggering unnecessary receive reports
50%UnverifiedNIXL (NVIDIA Inference Xfer Library) is a library used to accelerate inference data transfers
50%UnverifiedIn vLLM, NIXL is commonly used to implement cross-node KV cache transfer under a Prefill/Decode disaggregated architecture
50%UnverifiedNIXL typically leverages high-speed interconnect technologies such as NVLink, InfiniBand, or RDMA at the underlying level to transfer data directly between GPU memory, bypassing the CPU
50%UnverifiedNIXL is a low-latency KV cache transfer library provided by NVIDIA, designed specifically for disaggregated inference scenarios.
50%UnverifiedThe validation environment uses eight NVIDIA A40 GPUs, each running one vLLM engine, with KV caches transferred between resource pools via NIXL.
50%UnverifiedNVIDIA Dynamo implements distributed request routing and service discovery based on NATS and etcd, supports efficient inter-GPU data transfer based on NIXL, and is compatible with inference backends such as TensorRT-LLM and vLLM.
50%UnverifiedThe vLLM team introduced a push-based NIXL connector that moves transfer coordination off the critical path.
50%