AllReduce

At the thousand-GPU scale, the latency of a single AllReduce operation can account for more than 30% of the per-step training time.

Large-scale training relies on collective communication (such as AllReduce) to synchronize gradients across hundreds to thousands of GPUs, and any anomaly in a link, switch port, or network interface on the fabric may cause the training job to hang or throughput to plummet rather than exiting directly with an error.

An intermittent straggler refers to performance degradation in a certain GPU or network link that slows down collective communication (such as AllReduce) as a whole and is difficult to locate from single-point logs.

在 batch size = 1 的小消息交换场景下，AllReduce 的性能由延迟主导，而非带宽，因此 x16 相对 x8 并不能显著降低延迟。

AllReduce is essentially distributed floating-point summation, whose result depends on the response order of participating nodes and is inherently nondeterministic in asynchronous network environments.

Tensor parallelism splits a single matrix operation across multiple GPUs working together, requiring an AllReduce communication to aggregate results after each operation.

常见的集合通信操作包括AllReduce、AllGather和ReduceScatter，其中AllReduce将所有节点的梯度求和后广播回每个节点

AllReduce在每个训练迭代反向传播结束后要求所有GPU同步梯度，这一步骤在数千卡规模下可能消耗数十毫秒甚至更长时间

In large-scale distributed training, forward propagation, backward propagation, and AllReduce/AllGather communication operations are interleaved; if a communication operation is not complete, downstream compute operators must wait, creating bubbles.

In collective communication operations during AI training (such as AllReduce gradient aggregation), overall performance is determined by the slowest node.

Communication primitives in distributed training (such as AllReduce gradient aggregation) can introduce non-deterministic accumulated errors due to differences in parallelism configuration and network topology.

Unlike AllReduce, the traffic pattern of All-to-All is highly irregular and difficult to predict, and load-balancing imbalances can create hotspots.

In tensor-parallel inference, AllReduce operations occur at every layer, and for large models with tens or even hundreds of layers, the cumulative volume of communicated data is considerable.

Data parallelism shards training data across multiple GPUs, where each GPU holds a complete model replica and independently computes gradients, then synchronizes gradients through collective communication operations such as AllReduce.