FSDP
Fully Sharded Data Parallel, PyTorch's official large-scale distributed training solution. It shards model parameters, gradients, and optimizer states across multiple GPU devices, substantially reducing per-device memory consumption and enabling training of very large models on limited hardware resources.
Core Facts
Timeline (last 90 days)
PyTorch's distributed data parallel (DDP) and FSDP are both built on NCCL.
For teams that already have mature training frameworks (such as PyTorch + FSDP/DeepSpeed), keeping the training loop in their own code is better for debugging and migration.
The Olmo-core 3 implementation switches from FSDP to DDP: each rank holds a model partition across the entire accumulation window, gradients are synchronized only once per optimizer step, and the distributed optimizer shards the FP32 master weights and optimizer states; expert parallelism distributes routed experts across ranks, writing only the selected token rows directly into the target rank's buffer via NVSHMEM.
Ray Train provides wrappers for both DDP and FSDP strategies, and can automatically handle process group initialization, fault recovery, and checkpoint saving
FSDP shards model parameters, gradients, and optimizer states across multiple GPUs, and is the mainstream solution for training models with hundreds of billions of parameters
PyTorch provides native distributed training solutions such as DDP (DistributedDataParallel) and FSDP (Fully Sharded Data Parallel).
Teams using FSDP or tensor parallelism need to verify that the logit output for the same sequence is deterministic under different degrees of parallelism, as the floating-point accumulation order of communication operators may change with the parallel configuration.
FSDP reduces single-GPU memory usage by sharding model parameters, gradients, and optimizer states across multiple GPUs, and is one of the mainstream approaches for training models with hundreds of billions of parameters.
In large model training scenarios, the collaborative optimization of torch.compile and FSDP (Fully Sharded Data Parallel) is one of the most active research directions in the community.
Compared to DDP, FSDP is more aggressive in overlapping communication with computation, but introduces higher debugging complexity.
3 more timeline events
All Facts (14)
FSDP (Fully Sharded Data Parallel) is PyTorch's native distributed training solution, contributed by Meta.
65%UnverifiedPyTorch provides native distributed training solutions such as DDP (DistributedDataParallel) and FSDP (Fully Sharded Data Parallel).
60%UnverifiedPyTorch's distributed data parallel (DDP) and FSDP are both built on NCCL.
50%UnverifiedFor teams that already have mature training frameworks (such as PyTorch + FSDP/DeepSpeed), keeping the training loop in their own code is better for debugging and migration.
50%UnverifiedThe Olmo-core 3 implementation switches from FSDP to DDP: each rank holds a model partition across the entire accumulation window, gradients are synchronized only once per optimizer step, and the distributed optimizer shards the FP32 master weights and optimizer states; expert parallelism distributes routed experts across ranks, writing only the selected token rows directly into the target rank's buffer via NVSHMEM.
50%UnverifiedRay Train provides wrappers for both DDP and FSDP strategies, and can automatically handle process group initialization, fault recovery, and checkpoint saving
50%UnverifiedFSDP shards model parameters, gradients, and optimizer states across multiple GPUs, and is the mainstream solution for training models with hundreds of billions of parameters
50%UnverifiedTeams using FSDP or tensor parallelism need to verify that the logit output for the same sequence is deterministic under different degrees of parallelism, as the floating-point accumulation order of communication operators may change with the parallel configuration.
50%UnverifiedFSDP reduces single-GPU memory usage by sharding model parameters, gradients, and optimizer states across multiple GPUs, and is one of the mainstream approaches for training models with hundreds of billions of parameters.
50%UnverifiedIn large model training scenarios, the collaborative optimization of torch.compile and FSDP (Fully Sharded Data Parallel) is one of the most active research directions in the community.
50%UnverifiedFSDP is the officially released fully sharded data parallelism solution from PyTorch, which reduces per-device memory usage from O(N) to O(N/world_size) by sharding model parameters, gradients, and optimizer states across different devices.
50%UnverifiedCompared to DDP, FSDP is more aggressive in overlapping communication with computation, but introduces higher debugging complexity.
50%UnverifiedPyTorch's FSDP (Fully Sharded Data Parallel) and DeepSpeed ZeRO optimize memory efficiency, enabling models with tens of billions of parameters to be trained on a single machine.
50%UnverifiedDistributed training strategies such as DeepSpeed ZeRO, FSDP, tensor parallelism, and pipeline parallelism are necessary tools for training large video models.
50%