Stanford CS336: The Hardcore Philosophy of Building Large Models from Scratch

In an era where AI coding assistants can generate language models zero-shot, why would anyone still want to build a large model from scratch by hand? Stanford's CS336 course offers the answer: abstractions leak, and true understanding comes from building. Taught by professors including Percy Liang and Tatsu Hashimoto, the course is now in its third iteration, yet its core philosophy—starting from scratch—remains unchanged.

Why Building from Scratch Still Matters

The course team observed that AI researchers are becoming disconnected from underlying technology. A decade ago, researchers implemented and trained models themselves; eight years ago, people would still download BERT and fine-tune it; today, much can be accomplished by simply prompting a model. However, abstractions leak. When you are limited by a model's capabilities and cannot implement a specific feature, without foundational knowledge you are stuck.

"If you're genuinely interested in foundational research, relying solely on prompting models drastically limits the design space you can explore." — Percy Liang

The industrialization of frontier models has made training a GPT-4-level model from scratch a pipe dream. As mentioned in the course, GPT-4's training cost is reportedly as high as $100 million, and today's frontier models may cost on the order of $1 billion. Large labs have enormous GPU counts, and their construction details are kept secret. The GPT-4 paper explicitly states that, for competitive and safety reasons, no construction details will be shared.

Therefore, the course chooses to build small language models, but emphasizes that small models may not represent the behavior of frontier models. For example, at smaller scales, MLP layers account for about 44% of computation, but when scaled to 175B parameters, this rises to 80%. Optimization priorities shift with scale. Moreover, emergent abilities only suddenly appear at critical scales.

Three Types of Transferable Knowledge

The course categorizes knowledge into three types:

• Mechanics: What a Transformer is, how model parallelism works. This part teaches best.
• Mindset: How to squeeze maximum performance from hardware, taking scaling laws seriously. This is equally transferable.
• Intuitions: Which modeling and data decisions lead to good performance. This transfers worst across scales and must be gained through large-scale experiments.

The course emphasizes that efficiency is crucial. Model accuracy can be viewed as the product of efficiency and resources. In small-scale experiments, doubling runtime might just mean waiting a bit longer; but at scale, costs can reach hundreds of millions of dollars, and even a 5% efficiency gain is significant. OpenAI's 2020 paper showed that in image quality, algorithmic efficiency improved 44-fold between 2012 and 2019, with hardware advances multiplied by algorithmic improvements bringing huge leaps in efficiency and accuracy.

Course Structure and Core Themes

CS336 is divided into five parts, corresponding to five intensive assignments. All course materials are publicly available, and lecture videos will be posted on YouTube for self-learners to follow.

Foundations: From Tokenization to Training

The goal of the first week is to train a language model. The course starts with tokenizers, diving deep into the BPE (Byte Pair Encoding) algorithm. BPE builds a vocabulary by iteratively merging the most frequent adjacent byte pairs, making common sequences single tokens and rare sequences split into multiple tokens, avoiding the unknown word (UNK) problem.

In terms of model architecture, the course covers activation functions, positional encodings, the evolution of normalization layers, and various methods to reduce quadratic Attention computation, including state space models (like Mamba) and hybrid architectures such as Gated Delta Networks. Mixture of Experts (MoE) has become the mainstream paradigm for building compute-efficient Transformers, and the course will specifically discuss its training techniques.

The training section involves loss functions (from next-token prediction to multi-token prediction), optimizers (from Adam to Muon), initialization, learning rate scheduling, and more. The course emphasizes that these seemingly mundane hyperparameters, in the context of scale, setting them in a principled way is the key to distinguishing training collapse from reaching SOTA.

Assignment 1 requires students to implement a BPE tokenizer, Transformer, loss function, and optimizer, and to perform resource accounting to understand where FLOPs go.

BPE (Byte Pair Encoding) is a data-driven subword tokenization algorithm, originally used for data compression and later introduced to neural machine translation and language models. Its core idea: starting from a character-level or byte-level vocabulary, repeatedly count the frequency of adjacent symbol pairs in the corpus, merge the most frequent symbol pair into a new symbol, until the vocabulary reaches a preset size. This process turns high-frequency words or roots into single tokens, while rare words are split into multiple subword tokens, thereby eliminating the out-of-vocabulary (UNK) problem while keeping the vocabulary size manageable. For open-vocabulary scenarios such as multilingual text or code, BPE's byte-level variant (BBPE) can cover any Unicode character, becoming the standard tokenization scheme for models like the GPT series.

Mixture of Experts (MoE) is a sparsely activated architecture: replacing the feed-forward network (FFN) layers in a Transformer with multiple parallel "expert" sub-networks, and using a learnable router to selectively activate a few experts (e.g., Top-2) for each token. This way, the total parameter count can be enormous, but the actual computation per token is proportional only to the number of activated experts, thus expanding model capacity with roughly unchanged inference cost. MoE's challenges lie in load balancing (preventing expert collapse), training stability, and communication overhead from expert parallelism—topics that the course's systems section will delve into.

Systems: Squeezing Hardware Performance

Assignment 2 focuses on system optimization. The core principle is minimizing data movement. The course uses B200 as an example: 2.25 PFLOPS of BF16 compute, 8 TB/s of memory bandwidth. Compute units are separate from memory, and data movement is often the bottleneck. Operator fusion and tiling are key techniques to reduce data movement.

Multi-GPU training requires understanding collective communication operations (gather, reduce, all-reduce), as well as sharding strategies for parameters, activations, gradients, and optimizer states (data parallelism, model parallelism, pipeline parallelism, sequence parallelism, expert parallelism). The inference section discusses the prefill and decode phases, as well as acceleration techniques like quantization, distillation, and speculative decoding.

Assignment 2 involves writing kernels in Triton and parallel training.

Operator fusion refers to combining multiple fine-grained computational operations into a single kernel execution, to avoid intermediate results being repeatedly moved between GPU high-bandwidth memory (HBM) and on-chip cache. For example, fusing the mean and variance computation of LayerNorm with the subsequent scaling and shifting, or fusing the softmax in Attention with matrix multiplication, can significantly reduce memory read/write counts. Since modern GPU compute speed far outpaces memory bandwidth growth (the "memory wall"), data movement is often more expensive than computation itself, making fusion a key means to improve actual throughput.

Tiling is another technique to reduce data movement: splitting large matrix operations into small blocks that fit into on-chip shared memory (SRAM), loading and computing block by block, so that data is reused as much as possible before being evicted. FlashAttention precisely uses tiling and recomputation strategies to reduce Attention's memory complexity from O(n²) to O(n), while greatly reducing HBM accesses. These techniques, combined with kernel programming languages like Triton, are the foundation for students to implement high-performance operators by hand in Assignment 2.

Scaling Laws: Predicting Large-Scale Performance

Assignment 3 confronts the core challenge of large-scale training: given 1E25 FLOPs (tens of millions of dollars), what model should you train? The key mindset shift is: don't just think about training a single model, but design a scaling recipe—a mapping from FLOPs budget to hyperparameter configuration. By fitting scaling laws from small-scale experiments, predict the loss at the target scale.

"Predictability is at least as important as optimality."

The course emphasizes that scaling laws are not natural laws, but are "willed" into existence through carefully constructed scaling recipes. Hyperparameter transfer is crucial: if the learning rate jumps erratically across scales, reliable prediction is impossible. The classic Chinchilla scaling law states that for each FLOPs budget, scan different model sizes and choose the optimal one, fitting the relationship between parameter count and budget. Rule of thumb: training tokens ≈ 20× parameters. But inference costs change this calculation, and many models choose smaller sizes but train on more tokens.

The course team practiced preregistration in the Marin project: first fitting scaling laws to predict large-scale loss, then actually training to verify.

Assignment 3 provides a training API that simulates small-scale experiments; students must fit scaling laws and predict model performance under a given budget, simulating high-stakes decision scenarios.

Hyperparameter transfer refers to scaling hyperparameters (especially learning rate and initialization scale) with model size according to specific rules, so that configurations tuned on small models can be directly transferred to large models without re-searching. Its theoretical basis is work such as "μP (Maximal Update Parametrization)": by adjusting the parameterization, the optimal learning rate remains stable when scaling width, thereby supporting zero-shot or few-shot transfer from small proxy models to large targets. Without this transferability, scaling laws fitted from small-scale experiments cannot be reliably extrapolated to the target scale, and the predictive power of the scaling recipe is lost.

Data: The Definer of Model Capabilities

Assignment 4 deals with data. Data quality determines model capabilities. The course distinguishes internal evaluation (e.g., perplexity, good for model development, requiring smoothness across scales) and external evaluation (ecological validity, for users and reviewers). Data sources are diverse (web, books, code, papers), involving legal issues of copyright and licensing. Data processing includes transformation, filtering, deduplication, mixing, and synthetic data generation. Pretraining, mid-training (high-quality data, long context), and post-training data each serve different purposes.

Assignment 4 requires students to start from raw web crawling data and complete the full pipeline of filtering, deduplication, and cleaning—the course acknowledges this is "dirty work," but the complete from-scratch experience is indispensable.

Alignment: The Power of Weak Supervision

Assignment 5 focuses on alignment. After sufficient supervised training on next-token prediction, the model already has basic capabilities, but can be further improved through weak supervision—criticism is often easier than generation. The basic template is: generate responses from the model, have humans, verifiers, or AI judge and score them, then update the model to prefer better responses. This can be done via RL algorithms like PPO, GRPO, or simpler DPO.

RL's challenges lie in instability and difficult tuning; large-scale RL also faces system challenges: inference servers and training servers must coordinate, with rollout generation, code execution environments, and off-policy issues intertwined. The specific content of Assignment 5 is still being finalized; last year it was implementing DPO and GRPO and running them on math benchmarks.

The Efficiency Perspective Throughout

The course uses efficiency as a unifying framework. Tokenization improves computational efficiency; architecture improvements fine-tune memory and FLOPs; data filtering avoids wasting gradient updates on redundant bad data; scaling laws are about efficiently tuning with small models. Even if we enter a data-limited era in the future, the efficiency mindset still applies.

The course also adopts an "executable lecture" format: the lecture notes themselves are Python programs, allowing code to be stepped through, intuitively demonstrating hierarchical structures.

Teaching Innovation and AI Policy

The course this year introduced an AI teaching assistant policy: providing a prompt (agents.md) with a pedagogical inclination that students must follow when using AI. This prompt will answer code questions and clarify concepts, but will not directly generate code when students need to implement a Transformer. This is a first attempt, and the course team welcomes feedback.

The five assignments are extremely intensive. According to course reviews, the first assignment alone is equivalent to the sum of CS224N's five assignments. The course explicitly lists situations where it is not suitable: wanting to produce research results this quarter, only wanting to learn the latest hot technologies (the course does not delve deeply into multimodal and Agents), or wanting quick results in applied areas. For the latter, the course suggests trying prompting and fine-tuning first, and considering pretraining only as a last resort.

Conclusion

The value of CS336 lies not in chasing the latest tech trends, but in building a deep understanding of the full stack of language models. From tokenization, architecture, training, systems, to data and alignment, this course forges researchers' engineering muscles and foundational intuitions through intensive practice. In today's world of increasingly powerful AI, this "hardcore" from-scratch training is precisely the key to maintaining technical autonomy and depth of innovation.

One-line Summary

Stanford CS336 advocates building small language models from scratch to master mechanics, mindset, and the efficiency perspective.

Paragraph Summary

Stanford CS336 is now in its third iteration, with the core philosophy of building language models from scratch. The course team believes that AI researchers are becoming disconnected from underlying technology, and abstractions leak; lacking foundational knowledge will limit research design space. The course categorizes knowledge into mechanics, mindset, and intuitions; the first two are transferable across scales, while intuitions require large-scale experiments. The course uses efficiency as a unifying framework, divided into five parts corresponding to five intensive assignments: foundations (tokenization, architecture, training), systems (minimizing data movement, multi-GPU parallelism), scaling laws (designing scaling recipes, hyperparameter transfer), data (filtering, deduplication, mixing), and alignment (weak supervision, RL). The course also introduces an AI teaching assistant policy and emphasizes that predictability is as important as optimality.