Magnitude
An open-source local large model inference server that automatically selects the most suitable local model based on the user's hardware configuration. It integrates with mainstream AI agent tools via an OpenAI-compatible API, supporting coding assistants such as Claude Code and Cline.
Timeline (last 90 days)
On an M2 Max, using Llama-3.1-8B-Instruct (Q4_K_M quantization), Magnitude's throughput is 38.7 tok/s, and Ollama's is 18.2 tok/s.
Magnitude's GitHub repository address is github.com/magnitudedev/magnitude.
Magnitude provides a Modelfile-like declarative configuration containing directives such as FROM and PARAMETER, and supports parameters such as magnitude.cache_kernels and magnitude.tune_on_load.
Scenarios with high-frequency model switching are not suitable for Magnitude, because the accumulated tuning overhead eats up the per-request gains.
In virtual machines with fluctuating CPU quotas or Kubernetes MIG shared GPU slice environments, Magnitude's tuned kernels may misjudge the actual resources and perform worse than Ollama.
Magnitude's first load requires 45 to 90 seconds of kernel tuning, and subsequent startups use cached kernels, with cold-start time on par with Ollama.
Magnitude's CUDA kernel autotuning activates the RTX 4090's Tensor Cores for INT4 operations, which is one of the keys to its doubled throughput.
Magnitude fuses KV-cache updates with attention computation through kernel fusion, cutting cudaMemcpy overhead from 22% to 4%.
Magnitude uses a JIT approach to tune on the spot for the current hardware and compile specialized kernels when a model is first loaded, rather than using precompiled generic kernels.
Magnitude is an inference engine released under the Apache-2.0 open-source license.
24 more timeline events
All Facts (20)
Magnitude first analyzes the user's hardware configuration, then recommends a suitable model accordingly, and automatically handles the download and configuration process.
60%UnverifiedMagnitude's GitHub repository address is github.com/magnitudedev/magnitude.
50%UnverifiedMagnitude provides a Modelfile-like declarative configuration containing directives such as FROM and PARAMETER, and supports parameters such as magnitude.cache_kernels and magnitude.tune_on_load.
50%UnverifiedScenarios with high-frequency model switching are not suitable for Magnitude, because the accumulated tuning overhead eats up the per-request gains.
50%UnverifiedIn virtual machines with fluctuating CPU quotas or Kubernetes MIG shared GPU slice environments, Magnitude's tuned kernels may misjudge the actual resources and perform worse than Ollama.
50%UnverifiedMagnitude's first load requires 45 to 90 seconds of kernel tuning, and subsequent startups use cached kernels, with cold-start time on par with Ollama.
50%UnverifiedMagnitude's CUDA kernel autotuning activates the RTX 4090's Tensor Cores for INT4 operations, which is one of the keys to its doubled throughput.
50%UnverifiedMagnitude fuses KV-cache updates with attention computation through kernel fusion, cutting cudaMemcpy overhead from 22% to 4%.
50%UnverifiedOn an M2 Max, using Llama-3.1-8B-Instruct (Q4_K_M quantization), Magnitude's throughput is 38.7 tok/s, and Ollama's is 18.2 tok/s.
50%UnverifiedMagnitude uses a JIT approach to tune on the spot for the current hardware and compile specialized kernels when a model is first loaded, rather than using precompiled generic kernels.
50%UnverifiedMagnitude is capable of compiling and automatically tuning compute kernels on the user's own hardware.
50%UnverifiedMagnitude is an inference engine released under the Apache-2.0 open-source license.
50%UnverifiedOfficial claims state that Magnitude's decoding speed can reach up to twice that of llama.cpp.
50%UnverifiedMagnitude is positioned to become the local inference layer backend for AI agent runtimes.
50%UnverifiedThe first launch of self-tuning kernel compilation requires a period of time for compilation and benchmarking overhead.
50%UnverifiedThe more non-mainstream the hardware, the greater the performance headroom that self-tuning can extract, because hand-written optimizations in llama.cpp are already quite thorough on mainstream hardware.
50%UnverifiedMagnitude is still in an early stage, and its stability, model coverage, and handling of edge cases remain to be refined.
50%UnverifiedBenchmark performance data is typically highly correlated with the specific model, quantization method, and hardware, and actual gains vary depending on the environment.
50%UnverifiedMagnitude enables tools to connect to local models without code modifications—only configuration changes—by locally exposing an HTTP interface compliant with the OpenAI Chat Completions API standard
50%UnverifiedMagnitude is written in TypeScript
50%