InfiniBand
A network communication standard for high-performance computing, characterized by microsecond-level ultra-low latency and bandwidths exceeding 400 Gb/s. Widely deployed in supercomputing centers and large-scale AI training clusters for node interconnection, it is one of the mainstream high-performance interconnect solutions.
Core Facts
NVLink 4.0 offers a bidirectional bandwidth of up to 900 GB/s, while cross-node InfiniBand HDR offers only approximately 50 GB/s, a gap of nearly 20 times.
AI training typically requires thousands of high-end GPUs working in concert for weeks or even months, with extremely high demands on bandwidth, latency, and computational consistency, making it currently almost exclusively feasible in centralized data centers.
Timeline (last 90 days)
Spectrum-X's core selling point is adaptive routing and congestion control, used to approach InfiniBand's tail latency performance in RoCEv2 environments.
Commonly used high-speed interconnect solutions in the industry include InfiniBand and RoCE; the former offers better performance but higher cost.
In actual deployments, tensor parallelism and pipeline parallelism are often combined, along with cross-node unified memory addressing or high-speed interconnects (such as NVLink, InfiniBand or high-speed Ethernet).
An enterprise private cluster stack needs to span from the application orchestration layer, dynamic model serving (vLLM/TensorRT-LLM), and Kubernetes and GPU Operator/Slurm orchestration all the way to 800Gbps NDR InfiniBand/RoCEv2 network fabric and high-density liquid-cooled hardware.
Private deployment requires dedicated RoCEv2 or NVIDIA Quantum-2 InfiniBand switches, paired with NVMe-oF storage nodes, with sequential read speeds of at least 50GB/s.
GPU clusters rely on NVLink and InfiniBand for high-bandwidth, low-latency communication, with the NCCL library deeply optimized for this; the interconnect topologies of NPU devices are often different, such as Huawei Ascend's HCCL or the TPU's ICI.
Communication optimization methods include compressing gradients to low precision (such as FP16 or INT8), splitting All-Reduce into Reduce-Scatter and All-Gather and overlapping them with backward computation, and adopting hierarchical communication (intra-node NVLink aggregation, cross-node InfiniBand switching).
RoCE和InfiniBand是常见的RDMA实现。
InfiniBand是一种高吞吐、低延迟的网络互连标准，广泛应用于超算和AI训练集群。
AMD's ROCm ecosystem achieves RDMA capability by supporting RDMA over Converged Ethernet (RoCE), competing with InfiniBand-based solutions.
20 more timeline events
All Facts (20)
NVLink 4.0 offers a bidirectional bandwidth of up to 900 GB/s, while cross-node InfiniBand HDR offers only approximately 50 GB/s, a gap of nearly 20 times.
80%VerifiedAI training typically requires thousands of high-end GPUs working in concert for weeks or even months, with extremely high demands on bandwidth, latency, and computational consistency, making it currently almost exclusively feasible in centralized data centers.
65%Unverified传统DGX集群中，8块GPU组成一个节点，节点内走NVLink，节点间走InfiniBand
90%Unverified当前主流的NDR InfiniBand提供400Gbps的端口速率
90%Unverified大语言模型训练背后需要精密的数据中心电源分配单元、液冷或风冷散热系统、高速互联网络（NVLink、InfiniBand）以及UPS不间断电源保障
90%Unverified在大规模训练集群中，节点内通信（NVLink）与节点间通信（InfiniBand或RoCE）的性能差异可达一个数量级
80%UnverifiedGPU interconnect bandwidth within data centers is typically on the order of 100–400 Gbps (via InfiniBand HDR/NDR or NVSwitch), while wide-area network bandwidth between data centers may be only 1–10 Gbps.
70%UnverifiedNVIDIA acquired InfiniBand technology through its acquisition of Mellanox.
70%UnverifiedSpectrum-X's core selling point is adaptive routing and congestion control, used to approach InfiniBand's tail latency performance in RoCEv2 environments.
50%UnverifiedCommonly used high-speed interconnect solutions in the industry include InfiniBand and RoCE; the former offers better performance but higher cost.
50%UnverifiedIn actual deployments, tensor parallelism and pipeline parallelism are often combined, along with cross-node unified memory addressing or high-speed interconnects (such as NVLink, InfiniBand or high-speed Ethernet).
50%UnverifiedAn enterprise private cluster stack needs to span from the application orchestration layer, dynamic model serving (vLLM/TensorRT-LLM), and Kubernetes and GPU Operator/Slurm orchestration all the way to 800Gbps NDR InfiniBand/RoCEv2 network fabric and high-density liquid-cooled hardware.
50%UnverifiedPrivate deployment requires dedicated RoCEv2 or NVIDIA Quantum-2 InfiniBand switches, paired with NVMe-oF storage nodes, with sequential read speeds of at least 50GB/s.
50%UnverifiedGPU clusters rely on NVLink and InfiniBand for high-bandwidth, low-latency communication, with the NCCL library deeply optimized for this; the interconnect topologies of NPU devices are often different, such as Huawei Ascend's HCCL or the TPU's ICI.
50%UnverifiedCommunication optimization methods include compressing gradients to low precision (such as FP16 or INT8), splitting All-Reduce into Reduce-Scatter and All-Gather and overlapping them with backward computation, and adopting hierarchical communication (intra-node NVLink aggregation, cross-node InfiniBand switching).
50%UnverifiedRoCE和InfiniBand是常见的RDMA实现。
50%UnverifiedInfiniBand是一种高吞吐、低延迟的网络互连标准，广泛应用于超算和AI训练集群。
50%UnverifiedAMD's ROCm ecosystem achieves RDMA capability by supporting RDMA over Converged Ethernet (RoCE), competing with InfiniBand-based solutions.
50%UnverifiedGPUs transmit gradient data via InfiniBand or RoCE protocols, with bandwidth reaching 400 Gbps per card or higher.
50%UnverifiedTensor parallelism is typically used only between NVLink-connected GPUs within the same node; cross-node deployment severely slows training throughput due to insufficient InfiniBand bandwidth
50%