Data parallelism
A parallelism strategy in distributed training that distributes different data batches across multiple GPUs, with each GPU holding a complete model replica and synchronizing only gradients. It is simple to implement but incurs high memory overhead.
Core Facts
Timeline (last 90 days)
Data parallelism (DP) distributes identical model replicas across multiple devices, each processing a different input batch, and is suitable for improving throughput with a simple implementation.
Parallel strategies (data parallelism, tensor parallelism, pipeline parallelism and their combinations) need to match the hardware topology, otherwise communication bottlenecks will occur.
Data parallelism distributes different data batches across multiple GPUs, with each GPU holding a full model replica and only gradients needing synchronization; it is simple to implement but incurs high memory overhead.
Tensor parallelism, pipeline parallelism and data parallelism are the three basic strategies for distributed training of large models, and they are often used in combination (i.e. 3D parallelism).
Megatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.
Data parallelism requires gradient synchronization across all nodes, with communication volume proportional to the scale of model parameters.
In topology-aware scheduling practice, GPUs in a tensor-parallel group should be preferentially placed within the same node, pipeline stages can span nodes but should ideally be on the same rack, and data-parallel replicas can be distributed flexibly.
Parallelization strategies for multi-GPU inference include tensor parallelism (splitting individual mathematical operations across multiple GPUs), pipeline parallelism (assigning different neural network layers to different GPUs), and data parallelism (replicating the entire model to process multiple batches of data).
Data parallelism replicates the same model to multiple devices, each processing different batches of data before synchronizing gradients; model parallelism distributes different layers or modules of the model to different devices.
All Facts (10)
Data parallelism replicates the same model to multiple devices, each processing different batches of data before synchronizing gradients; model parallelism distributes different layers or modules of the model to different devices.
75%Unverified块调度建议将张量并行（TP）限制在NVLink带宽最高的GPU子集内（同一个最小块），数据并行（DP）跨块分布，流水线并行（PP）沿拓扑层次展开
85%UnverifiedData parallelism (DP) distributes identical model replicas across multiple devices, each processing a different input batch, and is suitable for improving throughput with a simple implementation.
50%UnverifiedParallel strategies (data parallelism, tensor parallelism, pipeline parallelism and their combinations) need to match the hardware topology, otherwise communication bottlenecks will occur.
50%UnverifiedData parallelism distributes different data batches across multiple GPUs, with each GPU holding a full model replica and only gradients needing synchronization; it is simple to implement but incurs high memory overhead.
50%UnverifiedTensor parallelism, pipeline parallelism and data parallelism are the three basic strategies for distributed training of large models, and they are often used in combination (i.e. 3D parallelism).
50%UnverifiedMegatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.
50%UnverifiedIn topology-aware scheduling practice, GPUs in a tensor-parallel group should be preferentially placed within the same node, pipeline stages can span nodes but should ideally be on the same rack, and data-parallel replicas can be distributed flexibly.
50%UnverifiedData parallelism requires gradient synchronization across all nodes, with communication volume proportional to the scale of model parameters.
50%UnverifiedParallelization strategies for multi-GPU inference include tensor parallelism (splitting individual mathematical operations across multiple GPUs), pipeline parallelism (assigning different neural network layers to different GPUs), and data parallelism (replicating the entire model to process multiple batches of data).
50%