llama.cpp
llama.cpp is an open-source large language model inference framework initiated by Georgi Gerganov, implemented entirely in C/C++ without dependency on external deep learning libraries. The project supports running large language models on diverse hardware including CPUs, GPUs, and Apple Silicon, offers multiple model quantization schemes to reduce memory and compute requirements, and is compatible with several mainstream open-source model formats, making it broadly applicable to local LLM deployment scenarios.
Core Facts
Muse Spark Open can be called on the day of release through platforms such as Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI.
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
Timeline (last 90 days)
Through the Metal backend of the ggml tensor computation library, llama.cpp offloads inference workloads such as matrix operations to the Mac's GPU for execution.
The commit title uses the word workaround, and the patch is explicitly labeled as a temporary patch; the maintainers clearly know this is not the final fix.
The core change in v0.40.13-rc1 is a temporary compatibility patch for the Metal backend, used to fix crashes caused by failure to compile the unary library on older versions of macOS.
llama.cpp released version v0.40.13-rc1.
After upgrading to v0.40.13-rc1, Metal inference on older versions of macOS should return to normal.
Ollama itself does not rewrite the underlying operators but compiles llama.cpp into the distribution package as a dependency, so the llama.cpp version number effectively represents the upper limit of hardware acceleration capability Ollama can obtain.
Ollama treats MLX as an inference backend separate from llama.cpp.
Generic backends (such as the CPU-based llama.cpp path) cannot directly reuse MLX's dedicated tensor layouts and quantization scheme adaptations.
Running AI models on-device typically uses techniques such as quantization and pruning to compress the model and then executes inference on the CPU, GPU or NPU of a personal computer; common frameworks include llama.cpp and ONNX Runtime.
The Messages/Chat Completions interfaces of vendors such as Anthropic and OpenAI all support prefilling parameters, and local inference frameworks (vLLM, llama.cpp, Transformers) can also implement prefilling by directly concatenating token sequences.
40 more timeline events
All Facts (20)
Muse Spark Open can be called on the day of release through platforms such as Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI.
90%Verifiedllama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%Verifiedllama.cpp was initiated by Georgi Gerganov in March 2023.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedMLX is specifically optimized for Apple Silicon's unified memory architecture.
85%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedOllama provides a Docker-like one-click model management experience, vLLM achieves high-throughput batch inference via PagedAttention, and LM Studio provides a graphical interface for desktop users.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%VerifiedThe mainstream quantization formats in the community include GGUF, AWQ, and GPTQ, among which GGUF was popularized by the llama.cpp project and supports CPU inference.
80%Verifiedllama.cpp supports hybrid CPU and GPU inference (layer offloading), allowing partial model layers to be loaded into GPU VRAM while the remaining layers are kept in memory
80%Verifiedllama-server is a local HTTP inference service provided by llama.cpp that implements an interface specification highly compatible with the OpenAI Chat Completions API.
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedOllama simplifies local model deployment to a single command-line instruction and provides a local API interface compatible with OpenAI.
80%Verifiedllama.cpp supports GPU acceleration backends including Metal, CUDA, Vulkan, and SYCL.
80%