bitsandbytes
An open-source library for quantization and acceleration of deep neural networks. It supports 8-bit and 4-bit quantization such as NF4 and is commonly used for large-model inference under low GPU memory.
Timeline (last 90 days)
Unsloth's 3bit quantization is not a standard bitsandbytes configuration; it usually refers to its own asymmetric quantization scheme, which can compress 7B-class model weights to about 3GB.
Different quantization formats (GGUF, AWQ, GPTQ, bitsandbytes) have different metadata overhead and layouts, and peak memory during loading may also differ.
According to the Transformers bitsandbytes documentation, quantized linear layers and other modules are handled separately, and weight storage and compute dtype are two different things.
On the Python side, single-prompt model.generate speeds up Llama and Qwen3 models by up to 2.7x at 32K context; with bitsandbytes 4-bit models, generating 2 to 4 prompts at a time speeds up by up to 2.8x on some NVIDIA GPUs.
In the same test, the bf16 baseline exceeded the self-developed W4 kernel (25.5 tok/s) and bitsandbytes NF4 (25.7 tok/s) at 31.3 tok/s, reaching 29.8% of the theoretical roofline.
On an RTX 3060, the developer's hand-written Triton fused W4A16 GEMV kernel targeting batch-1 decoding runs at about 25.5 tok/s, matching the roughly 25.7 tok/s of the bitsandbytes NF4 quantization scheme.
The self-developed W4 kernel's perplexity (probe ppl) is 15.76, better than bitsandbytes NF4's 18.98.
Pascal架构对应CUDA计算能力6.x，而vLLM、bitsandbytes等现代推理框架已将最低支持版本提升至7.0，部分功能要求8.0以上
Mainstream quantization toolchains include llama.cpp (GGUF format) and bitsandbytes.
常见的量化工具包括GPTQ、AWQ和bitsandbytes
1 more timeline events
All Facts (9)
Transformers框架原生支持bitsandbytes、GPTQ、AWQ等主流量化方案
90%UnverifiedTransformers库集成了GPTQ、AWQ、bitsandbytes等量化技术和Flash Attention加速方案
90%UnverifiedUnsloth's 3bit quantization is not a standard bitsandbytes configuration; it usually refers to its own asymmetric quantization scheme, which can compress 7B-class model weights to about 3GB.
50%UnverifiedAccording to the Transformers bitsandbytes documentation, quantized linear layers and other modules are handled separately, and weight storage and compute dtype are two different things.
50%UnverifiedDifferent quantization formats (GGUF, AWQ, GPTQ, bitsandbytes) have different metadata overhead and layouts, and peak memory during loading may also differ.
50%UnverifiedPascal架构对应CUDA计算能力6.x，而vLLM、bitsandbytes等现代推理框架已将最低支持版本提升至7.0，部分功能要求8.0以上
50%UnverifiedMainstream quantization toolchains include llama.cpp (GGUF format) and bitsandbytes.
50%Unverified常见的量化工具包括GPTQ、AWQ和bitsandbytes
50%Unverifiedflash-attention、xformers、bitsandbytes等高性能加速库通常只提供Linux预编译wheel，在Windows上需要从源码编译
50%