llama-box
llama-box is an inference engine wrapper built on llama.cpp, used to run large language models in GGUF format on platforms such as GPUStack.
Timeline (last 90 days)
GPUStack is a placement and lifecycle management layer, not an inference engine; when deploying models it selects worker nodes, launches vLLM, SGLang, or llama-box underneath, and proxies requests.
On heterogeneous nodes where TP cannot be used, GPUStack falls back to llama-box with layer offloading; num_gpu 60 (out of 80 layers) is the honest upper limit on 48 GB of VRAM, and setting it to 80 crashes on the first 4k prompt.
All Facts (2)
GPUStack is a placement and lifecycle management layer, not an inference engine; when deploying models it selects worker nodes, launches vLLM, SGLang, or llama-box underneath, and proxies requests.
50%UnverifiedOn heterogeneous nodes where TP cannot be used, GPUStack falls back to llama-box with layer offloading; num_gpu 60 (out of 80 layers) is the honest upper limit on 48 GB of VRAM, and setting it to 80 crashes on the first 4k prompt.
50%