NCCL
NCCL (NVIDIA Collective Communications Library) is a multi-GPU collective communication library developed by NVIDIA, designed to accelerate inter-GPU communication in deep learning training across multiple GPUs and nodes. It provides standard collective communication primitives including AllReduce, Broadcast, Reduce, and AllGather, and fully utilizes high-speed interconnects such as NVLink and InfiniBand to achieve low-latency, high-bandwidth data exchange between GPUs. NCCL is widely integrated into major deep learning frameworks including TensorFlow and PyTorch, serving as a core communication component for large-scale distributed training.
Core Facts
Since its introduction in 2006, CUDA has become the de facto industry standard in the AI/deep learning field, with mainstream frameworks such as PyTorch and TensorFlow using CUDA as their default backend.
CUDA是NVIDIA于2006年推出的并行计算平台和编程模型
The CUDA Toolkit includes the compiler nvcc, debuggers, performance analysis tools, and math libraries such as cuBLAS and cuFFT.
Timeline (last 90 days)
PyTorch 2.15 is officially released, and the NCCL backend has been completely rewritten and made the default option.
When upgrading to PyTorch 2.15 in multi-GPU scenarios, it is recommended to first verify communication-related timeout and exception logs on non-critical tasks, confirm normal operation with existing code and driver versions, and then roll it out gradually.
"Hangs" in distributed training are mostly caused by mismatched collective communication calls, such as inconsistent order or number of collectives entered by each rank, or a rank failing to participate due to OOM or an abnormal exit.
The underlying NCCL version is still determined by the user's environment, and when upgrading one needs to check driver and NCCL version compatibility.
"Backend rewrite" usually means that on the PyTorch side, the wrapping logic of ProcessGroupNCCL, stream management, error handling and interaction with CUDA streams have been refactored, rather than the NCCL library itself being replaced.
PyTorch's torch.distributed wraps NCCL through ProcessGroupNCCL.
PyTorch's distributed data parallel (DDP) and FSDP are both built on NCCL.
The rewritten NCCL backend introduces a flight recorder mechanism to leave on-site information usable for tracing when training hangs or crashes.
To judge whether a training library supports horizontal scaling, the key is whether it has built-in wrappers for distributed communication backends (such as NCCL and Gloo) and whether it supports sharded data loaders and gradient accumulation.
Distributed training refers to splitting a training task across multiple GPUs and machines for parallel execution when a single GPU cannot hold the model or data, with common strategies including data parallelism, tensor parallelism, and pipeline parallelism.
22 more timeline events
All Facts (20)
Since its introduction in 2006, CUDA has become the de facto industry standard in the AI/deep learning field, with mainstream frameworks such as PyTorch and TensorFlow using CUDA as their default backend.
80%VerifiedCUDA是NVIDIA于2006年推出的并行计算平台和编程模型
80%VerifiedThe CUDA Toolkit includes the compiler nvcc, debuggers, performance analysis tools, and math libraries such as cuBLAS and cuFFT.
75%VerifiedThe barriers of the CUDA ecosystem are deeply rooted in operator libraries (cuBLAS, cuDNN), compilation optimization toolchains (NVCC, TensorRT), and distributed training frameworks (NCCL), forming NVIDIA's deep moat.
65%VerifiedSince its release in 2007, CUDA has accumulated nearly two decades of development, with its moat reflected in low-level optimization libraries such as cuDNN, TensorRT, and NCCL.
65%UnverifiedNCCL通信库已经针对NVLink拓扑做了深度优化
90%UnverifiedNCCL内部实现了Ring、Tree、Direct等多种集合通信算法以适应不同的网络拓扑和消息大小
90%UnverifiedNCCL Inspector直接嵌入NCCL运行时环境，以极低的性能开销采集每一次集合通信操作的关键指标
85%UnverifiedNCCL is a library designed by NVIDIA specifically for multi-GPU communication, supporting collective communication primitives such as AllReduce, Broadcast, and AllGather
70%UnverifiedPyTorch's distributed data parallel (DDP) and FSDP are both built on NCCL.
50%Unverified"Backend rewrite" usually means that on the PyTorch side, the wrapping logic of ProcessGroupNCCL, stream management, error handling and interaction with CUDA streams have been refactored, rather than the NCCL library itself being replaced.
50%UnverifiedThe underlying NCCL version is still determined by the user's environment, and when upgrading one needs to check driver and NCCL version compatibility.
50%Unverified"Hangs" in distributed training are mostly caused by mismatched collective communication calls, such as inconsistent order or number of collectives entered by each rank, or a rank failing to participate due to OOM or an abnormal exit.
50%UnverifiedThe rewritten NCCL backend introduces a flight recorder mechanism to leave on-site information usable for tracing when training hangs or crashes.
50%UnverifiedWhen upgrading to PyTorch 2.15 in multi-GPU scenarios, it is recommended to first verify communication-related timeout and exception logs on non-critical tasks, confirm normal operation with existing code and driver versions, and then roll it out gradually.
50%UnverifiedPyTorch's torch.distributed wraps NCCL through ProcessGroupNCCL.
50%UnverifiedTo judge whether a training library supports horizontal scaling, the key is whether it has built-in wrappers for distributed communication backends (such as NCCL and Gloo) and whether it supports sharded data loaders and gradient accumulation.
50%UnverifiedDistributed training refers to splitting a training task across multiple GPUs and machines for parallel execution when a single GPU cannot hold the model or data, with common strategies including data parallelism, tensor parallelism, and pipeline parallelism.
50%UnverifiedNVIDIA's NCCL is a communication library for large-model training that handles efficient data exchange across multiple GPUs and machines and is key to distributed training performance.
50%UnverifiedCUDA's moat includes optimized libraries such as cuBLAS, cuDNN, and NCCL, the Nsight series of profilers and debuggers, and more than a decade of accumulated tutorials, Q&A, and third-party framework adaptations.
50%