SWA
A sparse attention variant in which each token attends only to neighboring tokens within a fixed preceding window. It reduces the computational complexity of attention from quadratic to approximately linear, supporting very long contexts and lowering inference cost.
Timeline (last 90 days)
V4.1 adopts SWA bounded replay: it recomputes only the last 128 tokens, and attention is also confined to within these 128 tokens, yielding an approximate window cache; the recomputation cost becomes constant, and the Decoder's 20 layers, which originally had to process the last 2,560 tokens, now only need to process 128, reducing total prefill computation by nearly half.
The SWA (Sliding Window Attention) bounded replay feature introduced in DeepSeek models has been fully implemented by vLLM, and official data shows TTFT drops by about 30%.
SWA (Sliding Window Attention) is a sparse attention variant that lets each token attend only to its neighbors within a fixed preceding window, reducing complexity to approximately linear.
The five design dimensions of SADL include: a baseline loss function combining Dice and top-K losses, strong data augmentation, reweighting by lesion volume, Stochastic Weight Averaging (SWA), and the pairing of cohort-balanced sampling with a residual encoder backbone.
SWA (Stochastic Weight Averaging) averages multiple checkpoints sampled at a high learning rate during the late stage of training, guiding the model to converge to a flatter loss basin and thereby improving generalization.
All Facts (5)
V4.1 adopts SWA bounded replay: it recomputes only the last 128 tokens, and attention is also confined to within these 128 tokens, yielding an approximate window cache; the recomputation cost becomes constant, and the Decoder's 20 layers, which originally had to process the last 2,560 tokens, now only need to process 128, reducing total prefill computation by nearly half.
50%UnverifiedThe SWA (Sliding Window Attention) bounded replay feature introduced in DeepSeek models has been fully implemented by vLLM, and official data shows TTFT drops by about 30%.
50%UnverifiedSWA (Sliding Window Attention) is a sparse attention variant that lets each token attend only to its neighbors within a fixed preceding window, reducing complexity to approximately linear.
50%UnverifiedThe five design dimensions of SADL include: a baseline loss function combining Dice and top-K losses, strong data augmentation, reweighting by lesion volume, Stochastic Weight Averaging (SWA), and the pairing of cohort-balanced sampling with a residual encoder backbone.
50%UnverifiedSWA (Stochastic Weight Averaging) averages multiple checkpoints sampled at a high learning rate during the late stage of training, guiding the model to converge to a flatter loss basin and thereby improving generalization.
50%