Peak Memory

The maximum level of memory usage during a single training step, typically determined jointly by activations, gradients and optimizer states. For Transformer-type models, the longer the sequence and the more tokens involved in computation, the larger the activation footprint.