H100
Core Facts
The NVIDIA A100's HBM2e capacity is 80GB, the H100 has 80GB HBM3, and the H200 has 141GB.
The HBM3 memory used by NVIDIA's H100 has a bandwidth of up to 3.35 TB/s.
Since 2022, the United States has imposed advanced semiconductor export controls on China, restricting the export of high-end AI training chips such as the NVIDIA A100/H100 and their successor products.
Timeline (last 90 days)
Nvidia's AI chips (such as H100 and A100) have been placed on the export control list by the US Bureau of Industry and Security (BIS) since 2022, under the Export Administration Regulations (EAR).
The power supply and cooling capacity of hyperscale data centers are scarce resources, and the economic return from freeing up 300 racks of space to deploy H100/H200 clusters is far higher than keeping A100s.
As demand for large-model training and inference continues to swell while professional-grade accelerator cards (such as the H100 and B200 series) have limited production capacity and high prices, buyers turn their attention to consumer-grade flagship cards.
The H100 is based on the Hopper architecture and is equipped with 80GB of HBM3 memory.
Products such as the H100 and B200 typically have unit prices in the tens of thousands of dollars, and due to production capacity and export controls, lead times are long and channel markups are severe.
The author said that if someone is willing to donate 8x H100 compute, he will try to use KS3 to go after the world record for nanogpt training speed, and claims that on proxy hardware he has already found a configuration that can beat the existing record.
CADRE can run on a single H100 and does not require multi-GPU parallelism or a cluster.
According to MarkTechPost, a world model built on CADRE can run at 20 fps on a single H100.
Different GPU architectures have different SM counts, Tensor Core generations, memory bandwidth, and L2 cache sizes, so the relative merits of the same tactic on A100 and H100 may be completely opposite.
When the same H100 GPU executes the same inference request and outputs the same number of tokens, the power consumption is basically the same.
40 more timeline events
All Facts (20)
The NVIDIA A100's HBM2e capacity is 80GB, the H100 has 80GB HBM3, and the H200 has 141GB.
90%VerifiedThe HBM3 memory used by NVIDIA's H100 has a bandwidth of up to 3.35 TB/s.
90%VerifiedSince 2022, the United States has imposed advanced semiconductor export controls on China, restricting the export of high-end AI training chips such as the NVIDIA A100/H100 and their successor products.
90%VerifiedxAI built a supercomputing cluster called Colossus, reportedly equipped with more than 100,000 NVIDIA H100 GPUs.
90%VerifiedThe pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
90%VerifiedTraining a model at the level of GPT-4 requires tens of thousands of A100/H100 GPUs working in concert for several months.
90%VerifiedThe retail price of a single NVIDIA H100/H200 GPU is as high as $30,000–$40,000.
90%VerifiedNVIDIA's H100/H200 series GPUs hold more than 80% of the AI training and inference market.
90%VerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.
80%VerifiedCerebras' latest-generation WSE-3 chip has an area of approximately 46,225 square millimeters, integrates 4 trillion transistors and 900,000 AI-optimized cores, and is more than 50 times the area of NVIDIA's H100 GPU.
80%VerifiedNVIDIA's MIG technology enables hardware-level resource isolation on high-end GPUs such as the A100/H100.
80%VerifiedCore innovations of the Blackwell architecture include the second-generation Transformer Engine (supporting FP4 precision), the NVLink Switch system (supporting interconnection of 576 GPUs), 20 petaFLOPS of FP4 compute, and 192 GB HBM3e memory (with bandwidth reaching 8 TB/s).
80%VerifiedBoth the NVIDIA H100/H200 and the AMD MI300X integrate HBM.
75%VerifiedBlackwell's second-generation Transformer Engine adds FP4 precision support, which in theory can improve Transformer model training throughput by 2–4x compared to the H100.
75%VerifiedA 35B model requires approximately 70GB of VRAM at FP16 precision, far exceeding the RTX 5090's 32GB capacity, so single-GPU deployment requires quantization.
75%VerifiedHBM is a core component of Nvidia's AI training chips including the A100, H100, and H200.
75%VerifiedEven after quantization, 70B-level models often require more than 40 GB of VRAM or a multi-GPU configuration.
75%VerifiedMainstream NVMe SSDs can achieve sequential read speeds of 3500–7000 MB/s.
75%VerifiedThe KV Cache of a 70B-parameter model with 128K context support can require hundreds of gigabytes of VRAM at full load, exceeding the 80GB VRAM capacity of a single H100.
70%VerifiedNVLink 4.0 (used in the H100) can achieve a unidirectional bandwidth of 900 GB/s.
70%