FSDP2
PyTorch's fully sharded data parallel training scheme (Fully Sharded Data Parallel v2), which shards model parameters, gradients, and optimizer states across multiple GPUs to reduce per-GPU memory usage.
Timeline (last 90 days)
Unsloth supports distributing LoRA, QLoRA and full fine-tuning across multiple GPUs in Python using FSDP2, and can split long sequences via context_parallel_size.
Version 5.18.0 unified device_mesh initialization to support pipeline-parallel plus tensor-parallel inference, and combined FSDP2 with expert parallelism via a two-dimensional (fsdp, tp) device mesh.
All Facts (2)
Unsloth supports distributing LoRA, QLoRA and full fine-tuning across multiple GPUs in Python using FSDP2, and can split long sequences via context_parallel_size.
50%UnverifiedVersion 5.18.0 unified device_mesh initialization to support pipeline-parallel plus tensor-parallel inference, and combined FSDP2 with expert parallelism via a two-dimensional (fsdp, tp) device mesh.
50%