llama.cpp
llama.cpp is an open-source large language model inference framework initiated by Georgi Gerganov, implemented entirely in C/C++ without dependency on external deep learning libraries. The project supports running large language models on diverse hardware including CPUs, GPUs, and Apple Silicon, offers multiple model quantization schemes to reduce memory and compute requirements, and is compatible with several mainstream open-source model formats, making it broadly applicable to local LLM deployment scenarios.
Core Facts
Muse Spark Open can be called on the day of release through platforms such as Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI.
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
Timeline (last 90 days)
llama.cpp supports about 150 model architectures, while Strata supports only one.
llama.cpp processes prompts in batches of 512 tokens by default, while Strata chunks prefill at 6144 tokens, reducing the frequency of experts crossing PCIe.
Strata directly reuses llama.cpp's MMQ kernel for expert matrix multiplication, and its performance advantage comes mainly from data scheduling rather than computational instruction optimization.
Strata's default output is not byte-for-byte reproducible, because the same expert computed by GPU or CPU produces slight numerical differences; but the output distribution difference is even smaller than llama.cpp's own difference between CPU and GPU.
The model is now available on Hugging Face, and the blog provides access methods via Transformers, llama.cpp, vLLM, SGLang, as well as Mistral's own platform and its coding agent Le Vibe.
llama.cpp released version v0.40.13-rc1.
The core change in v0.40.13-rc1 is a temporary compatibility patch for the Metal backend, used to fix crashes caused by failure to compile the unary library on older versions of macOS.
The commit title uses the word workaround, and the patch is explicitly labeled as a temporary patch; the maintainers clearly know this is not the final fix.
Through the Metal backend of the ggml tensor computation library, llama.cpp offloads inference workloads such as matrix operations to the Mac's GPU for execution.
After upgrading to v0.40.13-rc1, Metal inference on older versions of macOS should return to normal.
40 more timeline events
All Facts (20)
Muse Spark Open can be called on the day of release through platforms such as Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI.
90%Verifiedllama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%Verifiedllama.cpp was initiated by Georgi Gerganov in March 2023.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedMLX is specifically optimized for Apple Silicon's unified memory architecture.
85%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedOllama provides a Docker-like one-click model management experience, vLLM achieves high-throughput batch inference via PagedAttention, and LM Studio provides a graphical interface for desktop users.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%VerifiedThe mainstream quantization formats in the community include GGUF, AWQ, and GPTQ, among which GGUF was popularized by the llama.cpp project and supports CPU inference.
80%Verifiedllama.cpp supports hybrid CPU and GPU inference (layer offloading), allowing partial model layers to be loaded into GPU VRAM while the remaining layers are kept in memory
80%Verifiedllama-server is a local HTTP inference service provided by llama.cpp that implements an interface specification highly compatible with the OpenAI Chat Completions API.
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedOllama simplifies local model deployment to a single command-line instruction and provides a local API interface compatible with OpenAI.
80%Verifiedllama.cpp supports GPU acceleration backends including Metal, CUDA, Vulkan, and SYCL.
80%