Model Parallelism
A parallelism strategy in distributed training that splits different parts of a model across multiple devices, used for large models that do not fit in the memory of a single accelerator.
Core Facts
Timeline (last 90 days)
Memory-bound typically refers to insufficient GPU memory capacity, which requires offloading or model parallelism to address.
Data parallelism replicates the same model to multiple devices, each processing different batches of data before synchronizing gradients; model parallelism distributes different layers or modules of the model to different devices.
All Facts (2)
Data parallelism replicates the same model to multiple devices, each processing different batches of data before synchronizing gradients; model parallelism distributes different layers or modules of the model to different devices.
75%UnverifiedMemory-bound typically refers to insufficient GPU memory capacity, which requires offloading or model parallelism to address.
50%