Pipeline Parallelism

扩散模型的核心原理分为前向扩散过程（逐步添加高斯噪声）和反向去噪过程（从噪声中还原图像）两个阶段

Modern large model training typically employs a combination of multiple parallelism strategies simultaneously—including data parallelism, pipeline parallelism, tensor parallelism, and expert parallelism—forming a 3D or 4D parallel architecture.

块调度建议将张量并行（TP）限制在NVLink带宽最高的GPU子集内（同一个最小块），数据并行（DP）跨块分布，流水线并行（PP）沿拓扑层次展开

Non-deterministic GPU operations are designed to prioritize performance over determinism; forcing a fixed execution order introduces numerous synchronization barriers that significantly reduce parallel efficiency.

Parallel strategies (data parallelism, tensor parallelism, pipeline parallelism and their combinations) need to match the hardware topology, otherwise communication bottlenecks will occur.

In actual deployments, tensor parallelism and pipeline parallelism are often combined, along with cross-node unified memory addressing or high-speed interconnects (such as NVLink, InfiniBand or high-speed Ethernet).

An 8-node cluster can split a model across different nodes via tensor parallelism or pipeline parallelism, breaking through the memory capacity limit of a single machine.

Distributed inference parallelism strategies include tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP), and PD disaggregation.

Tensor parallelism, pipeline parallelism and data parallelism are the three basic strategies for distributed training of large models, and they are often used in combination (i.e. 3D parallelism).

In distributed inference, pipeline parallelism (splitting by layer) has lower network bandwidth requirements but uneven device utilization; tensor parallelism (splitting the matrix operations of a single layer) communicates frequently and is latency-sensitive.

MoE models in distributed training typically require the simultaneous deployment of four strategies: data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism.

Megatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.

The solution splits the Transformer by layer, with the Mac handling layers 1–40 of each 256-token batch, then streaming the activations to the phone via USB-C, while the phone uses its own GPU to run layers 41–64, achieving pipeline parallelism.

The trade-off of pipeline parallelism is that processing each token requires a full end-to-end wait, with inference latency accumulating linearly with the number of nodes

The most common distribution strategy in microcontroller clusters is Pipeline Parallelism, which splits the Transformer model layer by layer

In topology-aware scheduling practice, GPUs in a tensor-parallel group should be preferentially placed within the same node, pipeline stages can span nodes but should ideally be on the same rack, and data-parallel replicas can be distributed flexibly.

The deployment uses a TP2 PP4 parallelism scheme: GPUs are paired in groups of two for tensor parallelism, forming four groups, with pipeline parallelism applied between groups.

Pipeline parallelism vertically partitions by model layers, with far less communication volume than tensor parallelism, but suffers from pipeline bubble issues that cause GPU idling.

Parallelization strategies for multi-GPU inference include tensor parallelism (splitting individual mathematical operations across multiple GPUs), pipeline parallelism (assigning different neural network layers to different GPUs), and data parallelism (replicating the entire model to process multiple batches of data).

This test most likely deploys multiple GPUs using pipeline parallelism, with the first half of the model's layers on the first card and the second half on the second card.