From Single-GPU Bottlenecks to Hundreds of Billions of Parameters: 6 Key Questions in Large Model Distributed Parallelism

When private deployment and model training become essential needs, the memory and compute bottlenecks of single-GPU setups can no longer be avoided. This Bilibili tutorial starts from the very basics—"how does a large model generate a token"—and breaks the core issues of distributed parallelism into six understandable modules. This article provides a systematic overview based on the video content to help you build a cognitive framework for scaling from a single GPU to hundreds of billions of parameters.

The Computational Essence of Large Models: X times W equals Y

To understand distribution, we must first return to what happens on a single GPU. According to the Bilibili tutorial, a large model is essentially a neural network, and the current mainstream architecture is decoder-only—it originates from Transformer but retains only the decoder part. The GPT series is a typical example.

The model operates by generating tokens one at a time: given the input "the dog", the first step outputs "sat", then "sat" is fed back as input to produce the next output, and so on until a complete sentence is formed. Mathematically, this process can be simplified as a matrix operation: data X multiplied by model parameters W yields result Y.

The key point is that both X and W must be loaded into the GPU to perform the computation. GPUs are the primary hardware for neural network acceleration today, but the tutorial also mentions that NPUs and some domestic hardware can also handle computation. The problem then arises: when the size of W reaches hundreds of billions of parameters, a single GPU's memory cannot hold it, making distribution the only way out.

Decoder-only architecture is a variant of Transformer that removes the encoder part of the original Transformer and retains only the decoder stack. Each decoder layer contains self-attention and a feed-forward network (FFN), and maintains training stability through residual connections and layer normalization. Unlike encoder-decoder architectures, decoder-only uses a causal mask during generation, ensuring that each token can only attend to its left-side context, thereby achieving autoregressive token-by-token generation. Mainstream large models such as the GPT series and LLaMA all adopt this structure. Its advantages include structural simplicity, ease of scaling, and strong emergent capabilities in large-scale pretraining. Understanding the intra-layer computation of decoder-only (especially the matrix multiplications in attention and FFN) is the foundation for subsequent analysis of tensor parallelism split points.

KV Cache is a key optimization technique in autoregressive inference. During token-by-token generation, the model needs to cache the Key and Value matrices of historical tokens to avoid redundant computation, reducing attention complexity from O(n²) to O(n). However, KV Cache memory usage grows linearly with sequence length and batch size. For long-context scenarios (e.g., 32K tokens), a single GPU's memory can be heavily occupied by KV Cache. In distributed deployment, the partitioning strategy for KV Cache (e.g., splitting by head or by layer) directly affects memory balance and communication overhead, making it an engineering detail that must be considered in inference parallelism.

Six Key Questions Connect the Full Picture of Distributed Parallelism

The video title highlights "6 key questions". Although the original material only elaborates on the basics of the first two questions, the tutorial's thread clearly reveals the core framework of distributed parallelism. These six questions typically correspond to the complete scaling path from a single GPU to multiple GPUs:

Question 1: Why must models be distributed? A single GPU's memory cannot hold hundreds of billions of parameters, and single-GPU compute cannot sustain training throughput. This is the most direct motivation.

Question 2: How are data and parameters partitioned? This leads to the fundamental distinction between data parallelism and model parallelism—data parallelism partitions input X, while model parallelism partitions parameters W.

Question 3: How does tensor parallelism decompose single-layer computation? Matrix multiplication is split across multiple GPUs, parallelizing within a layer. Communication is frequent but per-GPU load is reduced.

Question 4: How does pipeline parallelism split across layers? The model is divided into multiple stages by layer, with different GPUs handling different layers, requiring handling of pipeline bubbles.

Question 5: How are expert parallelism and hybrid parallelism combined? Targeting MoE architectures and ultra-large-scale scenarios, multiple parallelism strategies are stacked.

Question 6: How are communication and memory optimized in distributed training? This includes engineering details such as gradient synchronization, communication overlap, and memory fragmentation management.

The tutorial starts from token-by-token generation in decoder-only architecture, which actually lays the groundwork for subsequent parallel partitioning: only by understanding where each computation step occurs can you understand where to choose split points.

Tensor Parallelism splits matrix multiplications within a single layer along specific dimensions across multiple GPUs. Taking the FFN layer of Transformer as an example, the weight matrix of the first linear layer can be split by column, and the second linear layer by row, thereby reducing per-GPU memory usage while maintaining mathematical equivalence. However, tensor parallelism requires frequent All-Reduce communication to synchronize intermediate results, so it is typically restricted to intra-node usage (NVLink interconnect). Pipeline Parallelism, on the other hand, divides the model into multiple stages by layer, with different GPUs responsible for different layers, hiding pipeline bubbles through micro-batch scheduling. The two are often used together: tensor parallelism handles large matrices within layers, pipeline parallelism handles inter-layer division, jointly supporting distributed training and inference of hundred-billion-parameter models.

Practical Significance in Private Deployment Scenarios

The tutorial opens with a very practical background: whether for agent development or private deployment, a large model is called behind the scenes to complete interactions. Online APIs are one option, but private deployment is often preferred by enterprises for data security and cost control.

Once private deployment is chosen, distributed deployment becomes an unavoidable engineering problem. Hundred-billion-parameter models require far more memory than a single GPU can provide, and KV Cache in inference scenarios further strains memory. Distributed parallelism is not an academic concept but an engineering question that must be answered during deployment.

Understanding parallel strategies from single-GPU bottlenecks is much more intuitive than diving directly into distributed training papers—once you know where each computation originally occurs, you understand the communication overhead and load-balancing costs of splitting it across multiple GPUs.

Data Parallelism is the most basic distributed strategy: each GPU holds a complete model replica but processes different data batches, synchronizing parameter updates through gradient All-Reduce. Its advantages are simple implementation and regular communication patterns, but it requires a single GPU to accommodate the full model. When model size exceeds single-GPU capacity, Model Parallelism must be introduced, including the aforementioned tensor parallelism and pipeline parallelism. Expert Parallelism targets MoE (Mixture of Experts) architectures, distributing different expert networks across different GPUs and dynamically assigning tokens through routing mechanisms. In actual deployment, hybrid strategies combining data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism (such as 3D parallelism) are often adopted to balance memory, compute, and communication.

Suggested Learning Path

For developers new to this field, it is recommended to progress in the order of "single-GPU inference → data parallelism → tensor parallelism → pipeline parallelism → hybrid parallelism". First, get the model generation pipeline running on a single GPU, understand the loading and computation of X and W, then gradually introduce partitioning strategies. The decoder-only structure and token-by-token generation mechanism repeatedly emphasized in the tutorial are the cornerstone for understanding all parallel schemes.

The value of this Bilibili tutorial lies in breaking down the seemingly massive topic of distribution into a progressive path starting from computational fundamentals, making it suitable as an entry point for systematic learning.