Unverified50% confidenceFactExact time
Pretraining significantly reduced the number of updates needed to surpass the fixed configuration (roughly halved).
1
Sources
50%
Confidence
Long-term
Relevance
10/27/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe 6.5x reduction reported by JOLT is measured in tokens; if it requires an additional teacher training phase, more training steps, or multiple forward passes per step (teacher evaluation + student generation), the total FLOPs savings may be noticeably smaller than 6.5x, or may even invert when the model is small.71% similarUnverified仅调整 num_workers、pin_memory、prefetch_factor 三个 DataLoader 参数，相同 2000 步训练耗时从 633 秒降至 358 秒，运行时间缩短 43%69% similarUnverifiedA single frontier pretraining run often lasts weeks or even months, during which any hardware failure or software anomaly can interrupt training, causing enormous waste of compute.69% similarUnverifiedThe learning rate during the SFT stage is typically 1–2 orders of magnitude lower than during pre-training, to avoid disrupting the general knowledge learned during the pre-training stage.69% similarUnverifiedAt the thousand-GPU scale, the latency of a single AllReduce operation can account for more than 30% of the per-step training time.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1102013API
curl https://kongchang.com/api/v1/knowledge/claims/1102013MCP
get_claim(id=1102013)