Verified65% confidenceBenchmarkExact time
If bandwidth reached 2.4 TB/s, the theoretical speed ceiling for a 100 GB model could reach 24 tokens/s, a 60 GB model 40 tokens/s, and a 35 GB model close to 69 tokens/s.
3
Sources
65%
Confidence
Medium-term (~90 days)
Relevance
9/23/2026
First Seen
Valid until: 12/22/2026
Sources
Related Entities
Related Claims
UnverifiedTaking a 70B BF16 model as an example, generating one token requires reading approximately 140 GB of weights, and when fully utilizing the H100's 3.35 TB/s memory bandwidth, the per-token latency is at least 42 milliseconds, with a theoretical TPS upper limit of approximately 24 tokens/second.78% similarUnverified对于70B模型（序列长度2048、隐藏层维度8192），每层激活值约32MB，在InfiniBand(400Gbps)中约需0.6毫秒，而在50Mbps家庭宽带下需超过5秒78% similarUnverifiedOfficial estimates suggest a 24GB GPU can run approximately 100–140 tokens/second.78% similarUnverifiedOn NVIDIA L4, the bf16 model (6.32 GiB) has a theoretical memory-bandwidth ceiling of about 47 tokens/s, while the int4 model (2.85 GiB) has a theoretical ceiling of about 105 tokens/s, with measured peak reaching 131.9 tokens/s.76% similarUnverifiedWhen running at 1-bit precision on an M2 Max equipped with 96GB of VRAM, generation speed for simple tasks is consistently around 28–30 tokens/second75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/945144API
curl https://kongchang.com/api/v1/knowledge/claims/945144MCP
get_claim(id=945144)