GGUF
GGUF (GPT-Generated Unified Format) is a binary file format for storing and distributing large language models, designed and promoted by the llama.cpp project. It encapsulates model weights, metadata, and quantization parameters in a single file and supports multiple quantization precision levels to reduce model size and memory footprint. GGUF is widely used in local inference scenarios and is compatible with multiple inference frameworks and hardware platforms.
Core Facts
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
The GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
Timeline (last 90 days)
LLaMA.cpp itself is an inference engine responsible for running GGUF-format models, with all control in the user's hands.
The 1-bit GGUF quantized version of MiniMax-M3.1-Flash requires at least 130GB of combined memory and VRAM.
Ollama provides local inference capability in the form of a command-line service plus REST API, with models stored locally as GGUF quantized files.
Most current SBOM toolchains do not parse model weight formats (such as safetensors and GGUF), and they lack a standard process for comparing weight hashes against official release versions, which lets abliteration-processed model variants easily bypass routine dependency audits.
Ollama model files are stored in the GGUF quantized format, and their size is much smaller than the original weights.
The plugin marks GGUF files with their corresponding roles: main model, shard, draft model, MTP file, and multimodal projector.
The plugin's model library indexes public GGUF repositories and supports searching by name; downloaded models appear in the local list and the main model is automatically identified.
llama.cpp's tokenizer implementation follows the two mainstream schemes of SentencePiece and BPE, and provides a unified abstraction for the vocabulary embedded in GGUF, covering details such as special tokens, byte fallback and normalization rules.
A Reddit user tried to hand-build an inference engine from scratch to run a GGUF Q4 model in the browser via WebGL2, and ended up going through a great deal of debugging difficulty.
Quantized GGUF weights are stored in blocks, each block containing a scale factor and several low-bit integers, and inference requires dequantization before participating in matrix multiplication.
40 more timeline events
All Facts (20)
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedA 7B-parameter model requires only approximately 4 GB of VRAM to run after 4-bit quantization.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedCommon model quantization methods include formats such as GPTQ, AWQ, and GGUF.
90%Verified4-bit quantization techniques (such as GPTQ, AWQ, and the GGUF format) can reduce model VRAM requirements to 1/4 of the original, but come with a certain degree of accuracy loss
90%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedIn GGUF quantization naming, Q4_K_M denotes a 4-bit quantization, Medium-precision variant using the K-Quant grouped quantization method
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%VerifiedGGUF packages model weights, quantization parameters, vocabulary, and hyperparameters into a single file and supports memory-mapped (mmap) loading.
80%Verified4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.
80%VerifiedThe GGUF format uses a key-value metadata header design with built-in runtime information such as the tokenizer vocabulary, allowing a single file to fully describe a runnable model.
80%VerifiedThe mainstream quantization formats in the community include GGUF, AWQ, and GPTQ, among which GGUF was popularized by the llama.cpp project and supports CPU inference.
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedModel quantization techniques (such as 4-bit/8-bit quantization in GGUF format) enable large models that originally required tens of GB of VRAM to run on consumer-grade GPUs or even CPUs.
80%