Tensor Parallelism

扩散模型的核心原理分为前向扩散过程（逐步添加高斯噪声）和反向去噪过程（从噪声中还原图像）两个阶段

Modern large model training typically employs a combination of multiple parallelism strategies simultaneously—including data parallelism, pipeline parallelism, tensor parallelism, and expert parallelism—forming a 3D or 4D parallel architecture.

块调度建议将张量并行（TP）限制在NVLink带宽最高的GPU子集内（同一个最小块），数据并行（DP）跨块分布，流水线并行（PP）沿拓扑层次展开

Tensor parallelism splits the computation of a single layer (such as matrix multiplication) across multiple GPUs for parallel execution; it has a high communication frequency and typically requires high-bandwidth interconnects such as NVLink between GPUs.

Each token generated by a large model requires a large number of matrix multiplication operations on the GPU, which is the core computational step of the attention mechanism in the Transformer architecture.

When large-parameter models do not fit on a single card, tensor parallelism is required, and multi-card communication overhead lowers throughput, so actual cost cannot be estimated linearly from parameter count alone.

Parallel strategies (data parallelism, tensor parallelism, pipeline parallelism and their combinations) need to match the hardware topology, otherwise communication bottlenecks will occur.

In actual deployments, tensor parallelism and pipeline parallelism are often combined, along with cross-node unified memory addressing or high-speed interconnects (such as NVLink, InfiniBand or high-speed Ethernet).

An 8-node cluster can split a model across different nodes via tensor parallelism or pipeline parallelism, breaking through the memory capacity limit of a single machine.

Tensor parallelism splits a single weight matrix by row or by column across different devices, and the computation of each layer requires All-Reduce synchronization between nodes, making it extremely sensitive to interconnect bandwidth.

Tensor parallelism splits a single tensor operation across multiple GPUs for joint completion, suiting models with huge parameters in a single layer but introducing frequent communication.

Distributed inference parallelism strategies include tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP), and PD disaggregation.

Tensor parallelism, pipeline parallelism and data parallelism are the three basic strategies for distributed training of large models, and they are often used in combination (i.e. 3D parallelism).

In distributed inference, pipeline parallelism (splitting by layer) has lower network bandwidth requirements but uneven device utilization; tensor parallelism (splitting the matrix operations of a single layer) communicates frequently and is latency-sensitive.

Tensor parallelism splits the weight matrix of the same layer and places it across multiple machines, requiring one All-Reduce communication per layer computed, is extremely sensitive to bandwidth and latency, and is usually only practical under high-speed interconnects such as NVLink.

张量并行的通信瓶颈不是单次传输的数据量，而是同步次数乘以每次同步的固有延迟。

在张量并行模式下，模型权重在推理开始前已分片常驻各 GPU 显存，跨 PCIe 传输的只有前向传播中的中间激活张量，而非权重。

MoE models in distributed training typically require the simultaneous deployment of four strategies: data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism.

Tensor parallelism splits a single matrix operation across multiple GPUs working together, requiring an AllReduce communication to aggregate results after each operation.

Megatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.