llama-swap
A model memory scheduling tool designed for use with llama.cpp, supporting on-demand dynamic loading and automatic unloading of models to enable efficient memory management across multiple models.
Timeline (last 90 days)
llama-swap is a lightweight model-switching proxy used to load and unload different models on demand on the llama.cpp server.
Berth supports native switching among multiple engines such as Ninfer-4090 (llama-swap), Strata, ComfyUI, and OpenWebUI.
1664x960 occupies 21.6GB in 24GB of VRAM (1.2GB baseline) and can only run when the GPU is truly idle; multiple rendering failures occurred because llama-swap and a resident engine slipped back onto the GPU mid-render.
This scheme locks each backend by host and port, using cross-process flock, with the lock granularity being a single turn: acquired at the start of the turn and released at the end.
The final configuration is model.single_session_lock: true, and the gate locks per backend rather than per model, because the GPU loads only one model at a time; when a second GPU channel appears, the key will become model-scoped.
systemctl stop strata will also stop llama-swap, because llama-swap carries Requires=strata.service; and llama-swap cannot load GGUF members while Strata is running, because Strata always occupies VRAM; the HyperQwen window can only bypass the proxy and connect directly to :18020, at which point :8080 is offline.
Frequent agent cron jobs must be paused, because llama-swap burns a 420-second health check timeout per request for dead members, and the cron agent will error out mid-run.
llama-swap is a lightweight HTTP proxy built on top of llama.cpp, responsible for the dynamic loading and unloading scheduling of multiple models.
In actual testing, llama.cpp combined with llama-swap can achieve 75–80 tokens per second, and the author considers it faster and easier to use than Ollama.
All Facts (9)
llama-swap is a lightweight model-switching proxy used to load and unload different models on demand on the llama.cpp server.
50%UnverifiedBerth supports native switching among multiple engines such as Ninfer-4090 (llama-swap), Strata, ComfyUI, and OpenWebUI.
50%Unverified1664x960 occupies 21.6GB in 24GB of VRAM (1.2GB baseline) and can only run when the GPU is truly idle; multiple rendering failures occurred because llama-swap and a resident engine slipped back onto the GPU mid-render.
50%UnverifiedThis scheme locks each backend by host and port, using cross-process flock, with the lock granularity being a single turn: acquired at the start of the turn and released at the end.
50%UnverifiedThe final configuration is model.single_session_lock: true, and the gate locks per backend rather than per model, because the GPU loads only one model at a time; when a second GPU channel appears, the key will become model-scoped.
50%Unverifiedsystemctl stop strata will also stop llama-swap, because llama-swap carries Requires=strata.service; and llama-swap cannot load GGUF members while Strata is running, because Strata always occupies VRAM; the HyperQwen window can only bypass the proxy and connect directly to :18020, at which point :8080 is offline.
50%UnverifiedFrequent agent cron jobs must be paused, because llama-swap burns a 420-second health check timeout per request for dead members, and the cron agent will error out mid-run.
50%Unverifiedllama-swap is a lightweight HTTP proxy built on top of llama.cpp, responsible for the dynamic loading and unloading scheduling of multiple models.
50%UnverifiedIn actual testing, llama.cpp combined with llama-swap can achieve 75–80 tokens per second, and the author considers it faster and easier to use than Ollama.
50%