GPUStack
An open-source GPU cluster management and inference platform that supports inference backends such as SGLang by containerizing inference backends and providing visual configuration. It offers a web-based interface and enables production-level deployment on the day a model is released.
Timeline (last 90 days)
GPUStack is a placement and lifecycle management layer, not an inference engine; when deploying models it selects worker nodes, launches vLLM, SGLang, or llama-box underneath, and proxies requests.
GPUStack's VRAM estimation is based on weight volume plus the declared context length, does not account for activation peaks or CUDA graph capture overhead, and has no knowledge of the actual request distribution.
When deploying Llama-3.3-70B (bf16) with TP=4 on four cards and keeping the default max-model-len, worker nodes OOM under real long-request traffic of 6k to 8k tokens, and GPUStack promptly restarts them.
On heterogeneous nodes where TP cannot be used, GPUStack falls back to llama-box with layer offloading; num_gpu 60 (out of 80 layers) is the honest upper limit on 48 GB of VRAM, and setting it to 80 crashes on the first 4k prompt.
The original team's four-node mixed-GPU cluster throughput decayed by about 18% over 36 hours while GPU utilization remained steady; the cause was diagnosed as KV block fragmentation leading to monotonically rising RSS and drifting TTFT p95.
The original team solved the throughput decay caused by KV block fragmentation by fixing --max-num-seqs 32 and recycling worker nodes based on an RSS slope threshold (rather than a fixed cron).
In testing, all nine inputs of varying difficulty were routed through demo-smart, and the actually selected models matched expectations in every case.
GPUStack v2.3.0rc2 previews advanced routing capabilities that select the appropriate generative model for requests via the Jev decision service.
The three tasks of translation, interval merging, and system design were each repeated three times in non-streaming mode with stable model selection, and the streaming versions also selected the same models.
Key names in the capability description must match the actual candidate model names exactly; routing aliases or provider names cannot be used, otherwise the mapping to candidates will fail.
4 more timeline events
All Facts (14)
The original team solved the throughput decay caused by KV block fragmentation by fixing --max-num-seqs 32 and recycling worker nodes based on an RSS slope threshold (rather than a fixed cron).
50%UnverifiedGPUStack is a placement and lifecycle management layer, not an inference engine; when deploying models it selects worker nodes, launches vLLM, SGLang, or llama-box underneath, and proxies requests.
50%UnverifiedGPUStack's VRAM estimation is based on weight volume plus the declared context length, does not account for activation peaks or CUDA graph capture overhead, and has no knowledge of the actual request distribution.
50%UnverifiedOn heterogeneous nodes where TP cannot be used, GPUStack falls back to llama-box with layer offloading; num_gpu 60 (out of 80 layers) is the honest upper limit on 48 GB of VRAM, and setting it to 80 crashes on the first 4k prompt.
50%UnverifiedActual request forwarding, plugin execution (including calls to the Jev decision service), and load balancing all occur on the Higress side, while GPUStack is responsible for managing model deployments and routing policy configuration and synchronizing them to Higress.
50%UnverifiedThe only reliable means of verifying routing selection is through log tracing, as the HTTP response itself carries no information about which upstream model generated it.
50%UnverifiedIn testing, all nine inputs of varying difficulty were routed through demo-smart, and the actually selected models matched expectations in every case.
50%UnverifiedThe three tasks of translation, interval merging, and system design were each repeated three times in non-streaming mode with stable model selection, and the streaming versions also selected the same models.
50%UnverifiedKey names in the capability description must match the actual candidate model names exactly; routing aliases or provider names cannot be used, otherwise the mapping to candidates will fail.
50%UnverifiedThe decision contribution weight is the relative influence on the score, not a traffic percentage.
50%UnverifiedHigress is a cloud-native API gateway open-sourced by Alibaba Cloud, built on Envoy, and GPUStack uses it as the gateway layer for handling AI inference traffic.
50%UnverifiedWhen deploying Llama-3.3-70B (bf16) with TP=4 on four cards and keeping the default max-model-len, worker nodes OOM under real long-request traffic of 6k to 8k tokens, and GPUStack promptly restarts them.
50%UnverifiedThe original team's four-node mixed-GPU cluster throughput decayed by about 18% over 36 hours while GPU utilization remained steady; the cause was diagnosed as KV block fragmentation leading to monotonically rising RSS and drifting TTFT p95.
50%UnverifiedGPUStack v2.3.0rc2 previews advanced routing capabilities that select the appropriate generative model for requests via the Jev decision service.
50%