Strata
An open-source semantic middleware layer for large language models that inserts a controllable semantic layer between data and models. It intercepts non-compliant, unsafe, or unauthorized LLM requests through rule-based constraints, and supports auditable allow/deny policy expressions to govern model interactions.
Core Facts
该项目已在GitHub开源
Strata uses low-bit quantization; the minimum configuration requires approximately 38 GB of combined memory and VRAM, the highest-quality configuration requires approximately 55 GB, and 64 GB of system memory is the ideal target.
In an RTX 5070 (12GB VRAM) configuration, the local Qwen3 9B model's output speed reached approximately 70–76 tokens/second.
Timeline (last 90 days)
About two-thirds of Strata's commits are co-signed by an AI coding agent.
Strata's default output is not byte-for-byte reproducible, because the same expert computed by GPU or CPU produces slight numerical differences; but the output distribution difference is even smaller than llama.cpp's own difference between CPU and GPU.
Strata serves only one request at a time and has no batching capability, making it unsuitable for multi-user concurrency or running multiple agents in parallel.
Strata runs best on Linux; on Windows there is a cap on multi-GPU memory locking, and under WSL even less memory can be locked.
Strata's experimental speed projection setting is off by default; when enabled, about 1 in every 10 positions has a top token change, and code perplexity rises by 15%.
Strata is an inference engine that supports only a single model architecture, under the MIT license, with about 9000 stars, and most of the code was written by one developer in two weeks.
Strata uses a separate thread to DMA the next layer's experts into a GPU ring buffer, overlapping computation and loading.
Strata's IQ3 quantized version locks about 40GB of fixed memory while running, and other programs cannot use this memory during that time.
Strata directly reuses llama.cpp's MMQ kernel for expert matrix multiplication, and its performance advantage comes mainly from data scheduling rather than computational instruction optimization.
llama.cpp supports about 150 model architectures, while Strata supports only one.
40 more timeline events
All Facts (20)
该项目已在GitHub开源
80%VerifiedStrata uses low-bit quantization; the minimum configuration requires approximately 38 GB of combined memory and VRAM, the highest-quality configuration requires approximately 55 GB, and 64 GB of system memory is the ideal target.
70%VerifiedIn an RTX 5070 (12GB VRAM) configuration, the local Qwen3 9B model's output speed reached approximately 70–76 tokens/second.
70%VerifiedStrata is an open-source project that serves as both an inference engine and a one-click packaged installer, providing a pre-selected quantized version by default along with an OpenAI-compatible local API.
65%VerifiedDeveloper Primitive reported a generation speed of 142.6 tokens/s for Qwen3 Flash Next running locally.
65%Partially VerifiedStrata's core approach is to have the CPU compute directly in memory in-place when an expert is not on the GPU, while the GPU processes cached experts in parallel, making memory a second compute domain rather than slow virtual VRAM.
65%UnverifiedOfficial estimates suggest a 24GB GPU can run approximately 100–140 tokens/second.
60%UnverifiedIf the quantized model weights are 150 GB, reading the weights once at a bandwidth of 273 GB/s takes approximately 0.55 seconds, giving a theoretical single-token generation speed of approximately 1.8 tokens/s (excluding computation latency and cross-node communication overhead).
60%UnverifiedStrata's default output is not byte-for-byte reproducible, because the same expert computed by GPU or CPU produces slight numerical differences; but the output distribution difference is even smaller than llama.cpp's own difference between CPU and GPU.
50%Unverifiedllama.cpp supports about 150 model architectures, while Strata supports only one.
50%UnverifiedStrata's experimental speed projection setting is off by default; when enabled, about 1 in every 10 positions has a top token change, and code perplexity rises by 15%.
50%UnverifiedStrata serves only one request at a time and has no batching capability, making it unsuitable for multi-user concurrency or running multiple agents in parallel.
50%UnverifiedStrata directly reuses llama.cpp's MMQ kernel for expert matrix multiplication, and its performance advantage comes mainly from data scheduling rather than computational instruction optimization.
50%UnverifiedStrata is an inference engine that supports only a single model architecture, under the MIT license, with about 9000 stars, and most of the code was written by one developer in two weeks.
50%UnverifiedStrata's IQ3 quantized version locks about 40GB of fixed memory while running, and other programs cannot use this memory during that time.
50%UnverifiedStrata uses a separate thread to DMA the next layer's experts into a GPU ring buffer, overlapping computation and loading.
50%Unverifiedllama.cpp processes prompts in batches of 512 tokens by default, while Strata chunks prefill at 6144 tokens, reducing the frequency of experts crossing PCIe.
50%UnverifiedAbout two-thirds of Strata's commits are co-signed by an AI coding agent.
50%UnverifiedStrata runs best on Linux; on Windows there is a cap on multi-GPU memory locking, and under WSL even less memory can be locked.
50%UnverifiedWhen using the Strata inference engine, the engine randomly dropped space tokens in the code, causing about 13% of tool calls to fail.
50%