Unverified50% confidenceBenchmarkExact time
On a consumer-grade GPU with 8GB of VRAM, ActionAssist achieves an inference speed of 122 tokens/second using Q6_K_L quantization with a 32k context window and 8-bit KV cache quantization.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
9/12/2026
First Seen
Valid until: 12/11/2026
Sources
Related Entities
Related Claims
UnverifiedFor a typical 7B model (32 layers, 32 heads, 128 dimensions) processing a 2048-token sequence at FP16 precision with a batch size of 8, the KV Cache consumes approximately 4 GB of VRAM.73% similarUnverifiedA GPU with 8 GB of VRAM can expand the context window to 8,192–16,384 tokens when running a Q4-quantized 9B model.72% similarUnverified在12GB显存的显卡上运行9B模型时，通常采用4-bit量化（如Q4_K_M格式）以降低显存占用72% similarVerified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB71% similarUnverifiedWhen running at 1-bit precision on an M2 Max equipped with 96GB of VRAM, generation speed for simple tasks is consistently around 28–30 tokens/second70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/912378API
curl https://kongchang.com/api/v1/knowledge/claims/912378MCP
get_claim(id=912378)