PCIe

The bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.

NVLink-C2C technology enables high-speed interconnection between CPU and GPU.

The CPU offloading strategy keeps the weights of inactive experts resident in system memory and only temporarily loads them into GPU VRAM when selected by the router, at the cost of introducing PCIe bus transfer latency.

CXL is an open interconnect standard based on the PCIe physical layer that allows high-speed, low-latency cache-coherent communication between CPUs and various accelerators and memory expansion devices.

The core goal of NVLink is to break through the bandwidth limitations of the traditional PCIe bus and allow multiple GPUs to communicate directly at extremely high speeds.

In traditional servers, x86 CPUs connect to discrete GPUs via the PCIe bus.

The RTX 5060 Laptop uses PCIe Gen 4 x8, with a theoretical bandwidth of about 16 GB/s, far below VRAM bandwidth, making it a hidden bottleneck for expert offloading schemes.

Xtier exploits the sparsity of MoE by leaving inactive expert parameters in CPU memory and moving only active experts into GPU memory, at the cost that expert switching requires moving data over the PCIe bus, which may become an inference bottleneck.

The core idea of GDS is to establish a direct DMA channel between storage and GPU memory, with data flowing from NVMe devices through PCIe switches directly into GPU memory.

Estimating a single-token KV Cache at about 1 MB, moving the cache for 8K tokens over PCIe takes several hundred milliseconds, enough to double the first-token latency.

NVIDIA's Xid numbering system covers GPU hardware and firmware errors, with each number corresponding to a specific class of failure root cause, for example Xid 79 indicates the GPU has fallen off the PCIe bus, Xid 48 corresponds to an uncorrectable double-bit ECC memory error, Xid 74 involves an NVLink link failure, and Xid 119/120 relate to GSP firmware timeouts.

Traditional PCs' discrete VRAM offers higher bandwidth, but capacity is limited by the graphics card model, and once a model exceeds VRAM it must be moved over the PCIe bus, drastically reducing speed.

In heterogeneous backends, PCIe bandwidth is typically far lower than NVLink and will become the bottleneck for cross-device scheduling; placing the draft model on a device under the same PCIe root complex as the target model, or directly on the CPU, is often more worthwhile than moving data across NUMA nodes.

Swapping KV cache in and out depends on the interconnect bandwidth between host memory and GPU VRAM; taking PCIe 4.0 x16 as an example, the one-way effective bandwidth is on the order of about 25 GB/s, while the HBM bandwidth of a single data center GPU can reach 2–3 TB/s, a difference of about two orders of magnitude.

以隐藏维度 8192 的 70B 模型、batch size = 1 为例，每生成一个 token 穿越 PCIe 的总数据量只有几 MB。

PCIe Gen4 x16 bandwidth is approximately 64GB/s, which often becomes a hard bottleneck in memory offloading scenarios.

EPYC平台通常提供业界领先的内存通道数和PCIe通道数

System-bound means non-GPU factors such as CPU scheduling, PCIe transfer, and Python interpreter overhead become the bottleneck, requiring system-level measures such as asynchronous execution and reducing synchronization points.

Vendors have already released PCIe-to-ISA adapter cards based on FPGAs, enabling legacy ISA industrial cards to be inserted into modern server motherboards and operated.

By routing inter-GPU data exchange through NVLink and bypassing the host PCIe channel, reliance on host bandwidth in inference scenarios can be reduced.