NVMe
A high-speed solid-state storage interface protocol that provides low-latency, high-throughput data access, commonly used for local high-speed disks and training data caches.
Timeline (last 90 days)
The MicroSD Express standard is based on PCIe 3.0 and the NVMe protocol, with a theoretical maximum transfer speed of about 985 MB/s.
GDS relies on local NVMe devices that support direct I/O, or on certified networked storage solutions (such as RDMA-based parallel file systems).
The core idea of GDS is to establish a direct DMA channel between storage and GPU memory, with data flowing from NVMe devices through PCIe switches directly into GPU memory.
Running this engine requires a Blackwell GPU (the kernels use the NVFP4 hardware path), about 70GB of space on fast NVMe, and 32GB of RAM.
The real bottleneck is NVMe disk reads (RAM cache misses) at 50-55ms, not Python launch overhead.
Using GPUDirect Storage, adapter weights can be streamed from NVMe back to HBM in under 40 milliseconds.
After fixing the non-compute overhead (configuring persistent storage, migrating data to local NVMe, caching the base model and tokenizer, and increasing num_workers), the single-run time dropped from 9.5 hours to 2 hours 40 minutes, nearly a 4x speedup compared with the RTX 4070's 11 hours.
After NVMe SSDs became fast enough to respond at the microsecond level, the storage industry adopted polling instead of interrupts, having the CPU actively and frequently check the completion queue, sacrificing idle-time energy efficiency for determinism and throughput at low latency.
Writing to memory is a one-time cost, as the first processing of text still requires full KV state computation; the KV states for 50 million tokens occupy several terabytes of local NVMe disk space.
The open-source tool galahad-kv saves the KV states of each approximately 16,000-token chunk to an encrypted local NVMe disk, then reloads them byte-accurately, completely skipping recomputation.
4 more timeline events
All Facts (14)
The MicroSD Express standard is based on PCIe 3.0 and the NVMe protocol, with a theoretical maximum transfer speed of about 985 MB/s.
50%UnverifiedGDS relies on local NVMe devices that support direct I/O, or on certified networked storage solutions (such as RDMA-based parallel file systems).
50%UnverifiedThe core idea of GDS is to establish a direct DMA channel between storage and GPU memory, with data flowing from NVMe devices through PCIe switches directly into GPU memory.
50%UnverifiedThe real bottleneck is NVMe disk reads (RAM cache misses) at 50-55ms, not Python launch overhead.
50%UnverifiedRunning this engine requires a Blackwell GPU (the kernels use the NVFP4 hardware path), about 70GB of space on fast NVMe, and 32GB of RAM.
50%UnverifiedAfter fixing the non-compute overhead (configuring persistent storage, migrating data to local NVMe, caching the base model and tokenizer, and increasing num_workers), the single-run time dropped from 9.5 hours to 2 hours 40 minutes, nearly a 4x speedup compared with the RTX 4070's 11 hours.
50%UnverifiedAfter NVMe SSDs became fast enough to respond at the microsecond level, the storage industry adopted polling instead of interrupts, having the CPU actively and frequently check the completion queue, sacrificing idle-time energy efficiency for determinism and throughput at low latency.
50%UnverifiedThe open-source tool galahad-kv saves the KV states of each approximately 16,000-token chunk to an encrypted local NVMe disk, then reloads them byte-accurately, completely skipping recomputation.
50%UnverifiedWriting to memory is a one-time cost, as the first processing of text still requires full KV state computation; the KV states for 50 million tokens occupy several terabytes of local NVMe disk space.
50%UnverifiedFor NVMe SSDs, each unit in the Data Units Written field of the smartctl output corresponds to 1,000 512-byte sectors, or approximately 512 KB.
50%UnverifiedNVMe is a communication protocol designed specifically for flash storage that communicates directly with the CPU via the PCIe bus, with latency as low as the microsecond level.
50%UnverifiedTRIM is a command in the ATA instruction set, corresponding to Deallocate in the NVMe protocol, allowing the operating system to proactively inform the SSD which logical blocks are no longer in use so that the controller can perform garbage collection.
50%Unverified现代存储金字塔中GPU显存(HBM)带宽可达数TB/s但消费级容量通常8-24GB，CPU内存(DDR5)带宽约50-100GB/s，NVMe固态硬盘带宽约5-14GB/s
50%UnverifiedUsing GPUDirect Storage, adapter weights can be streamed from NVMe back to HBM in under 40 milliseconds.
50%