DeepSpeed
An open-source large-scale distributed training optimization framework developed by Microsoft and designed for training large models. It supports multiple parallelism strategies, including data parallelism, tensor parallelism, and pipeline parallelism.
Core Facts
Cube Studio supports distributed acceleration frameworks such as DeepSpeed, ColossalAI, and Horovod.
ZeRO was proposed by the DeepSpeed team in 2020 and eliminates redundancy in data parallelism through three stages: ZeRO-1 shards optimizer states, ZeRO-2 shards gradients, and ZeRO-3 shards model parameters.
Timeline (last 90 days)
This open-source project provides a DeepSpeed configuration that fixes memory spikes for GPU memory-constrained scenarios.
For teams that already have mature training frameworks (such as PyTorch + FSDP/DeepSpeed), keeping the training loop in their own code is better for debugging and migration.
Unsloth adds DeepSpeed ZeRO-2 and ZeRO-3 support, including 4-bit QLoRA, full fine-tuning, DPO and CPU offload; the DDP + gradient checkpointing LoRA training issue that previously caused crashes has been fixed.
During the CAD phase, the team forked Megatron-LM and Microsoft DeepSpeed and rewrote kernels, laying the foundation for the inference capability behind Cursor's more than 500 million model calls per day.
ZeRO (Zero Redundancy Optimizer) reduces GPU memory usage by partitioning optimizer states, gradients, and parameters
DeepSpeed recently updated to add tensor parallelism, sequence parallelism, and expert parallelism capabilities
Presenter Masahiro stated that with these updates, DeepSpeed can rival other state-of-the-art frameworks in training the latest MoE models
PyTorch's torch.save/torch.load, Hugging Face Transformers' Trainer class, DeepSpeed, and Megatron-LM all have built-in periodic checkpoint support.
Large model training involves multiple stages including data cleaning, annotation, alignment, and safety fine-tuning
The numerical inconsistency problem is especially pronounced in heterogeneous architectures that use vLLM or TensorRT-LLM for rollout generation and PyTorch/DeepSpeed for training.
4 more timeline events
All Facts (19)
Cube Studio supports distributed acceleration frameworks such as DeepSpeed, ColossalAI, and Horovod.
80%VerifiedZeRO was proposed by the DeepSpeed team in 2020 and eliminates redundancy in data parallelism through three stages: ZeRO-1 shards optimizer states, ZeRO-2 shards gradients, and ZeRO-3 shards model parameters.
75%UnverifiedFlashInfer 是专为 LLM 推理设计的高性能 CUDA 算子库
90%UnverifiedLarge model training involves multiple stages including data cleaning, annotation, alignment, and safety fine-tuning
60%UnverifiedMainstream training frameworks such as Megatron-LM and DeepSpeed generate detailed checkpoints and data-loading records during distributed training.
60%UnverifiedTransformers与DeepSpeed、Accelerate等分布式训练工具无缝集成
60%UnverifiedFor teams that already have mature training frameworks (such as PyTorch + FSDP/DeepSpeed), keeping the training loop in their own code is better for debugging and migration.
50%UnverifiedDuring the CAD phase, the team forked Megatron-LM and Microsoft DeepSpeed and rewrote kernels, laying the foundation for the inference capability behind Cursor's more than 500 million model calls per day.
50%UnverifiedZeRO (Zero Redundancy Optimizer) reduces GPU memory usage by partitioning optimizer states, gradients, and parameters
50%UnverifiedPresenter Masahiro stated that with these updates, DeepSpeed can rival other state-of-the-art frameworks in training the latest MoE models
50%UnverifiedDeepSpeed recently updated to add tensor parallelism, sequence parallelism, and expert parallelism capabilities
50%UnverifiedPyTorch's torch.save/torch.load, Hugging Face Transformers' Trainer class, DeepSpeed, and Megatron-LM all have built-in periodic checkpoint support.
50%UnverifiedThe numerical inconsistency problem is especially pronounced in heterogeneous architectures that use vLLM or TensorRT-LLM for rollout generation and PyTorch/DeepSpeed for training.
50%UnverifiedDeepSpeed's ZeRO technology has three stages: Stage 1 shards optimizer states (saving approximately 4× memory), Stage 2 additionally shards gradients (saving approximately 8×), and Stage 3 also shards model parameters.
50%UnverifiedMegatron-LM implements a 3D parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism, while DeepSpeed distributes optimizer states across multiple nodes via ZeRO technology.
50%UnverifiedDeepSpeed推出了ZeRO-Infinity，通过利用NVMe SSD的存储空间进一步扩展了可训练模型的规模上限
50%Unverified微软的 DeepSpeed 库与 ORT Training 有深度集成，两者结合可在千卡规模集群上训练百亿参数级别模型
50%UnverifiedThis open-source project provides a DeepSpeed configuration that fixes memory spikes for GPU memory-constrained scenarios.
50%UnverifiedUnsloth adds DeepSpeed ZeRO-2 and ZeRO-3 support, including 4-bit QLoRA, full fine-tuning, DPO and CPU offload; the DDP + gradient checkpointing LoRA training issue that previously caused crashes has been fixed.
50%