#LLaMa

Training LLaMa from Scratch: A Full Record of 11-Hour Training for a 125M Model

A developer trains a LLaMa model from scratch: manually implements RoPE, GQA, RMSNorm, SwiGLU, and KV cache, training a 125M model in 11 hours on dual A5000 GPUs.