llama-swap

llama-swap is a proxy for running multiple generative AI models on one machine and switching between them on demand. It sits in front of any OpenAI- or Anthropic-API-compatible inference server, reads the `model` field from each incoming request, and starts or replaces the upstream server needed to serve it. The project is written in Go, ships as one binary plus one configuration file, and reports no external dependencies. Supported upstreams include llama.cpp and its forks, vllm, stable-diffusion.cpp, whisper.cpp, audio.cpp and ComfyUI. Because the proxy is protocol-level rather than tied to a specific engine, the README notes that inference servers can be upgraded independently. API surface - OpenAI-style endpoints: `v1/completions`, `v1/chat/completions`, `v1/responses`, `v1/embeddings`, `v1/models`, `v1/audio/speech`, `v1/audio/transcriptions`, `v1/audio/voices`, `v1/images/generations` and `v1/images/edits`. - Anthropic-style endpoints: `v1/messages` and `v1/messages/count_tokens`. - llama-server extras: `v1/rerank`, `v1/reranking`, `/rerank`, `/infill`, `/completion`, `/models` and `/props`. - SDAPI endpoints from stable-diffusion.cpp's server, plus audio.cpp task endpoints and a `/comfyui/` custom endpoint. - Management endpoints: `/ui`, `/upstream/:model_id`, `/running`, `POST /api/models/unload` (all models) and `POST /api/models/unload/:model_id`, profile listing and activation via `/api/profiles`, `/health`, and `/metrics` for Prometheus system and GPU metrics. - Log endpoints: `/logs` for buffered plain-text logs, `/logs/stream` for live streaming, with `/logs/stream/proxy`, `/logs/stream/upstream` and `/logs/stream/{model_id}` variants; `?no-history` streams only new lines. Configuration and features A minimal config declares a `models` map where each entry has an ID and a `cmd`; `${PORT}` is substituted with an automatically assigned port. Optional settings include `ttl` for automatic unloading after inactivity, `unloadTimeout`, `aliases` for familiar model names, `env` variables, `cmdStop` for graceful Docker/Podman shutdown, `useModelName`, request `filters` (`stripParams`, `setParams`, `setParamsByID`), `hooks` for preloading on startup, `macros`, and a `matrix` DSL for running concurrent models with custom swap logic. API keys can be defined to restrict endpoint access, and profiles allow switching model-ID routing at runtime. The bundled web UI provides a playground, token metrics, request/response inspection, manual model load and unload, and real-time log streaming. A Help page runs a local tool-capable model against llama-swap's own documentation, and the same tools are exposed as an MCP endpoint at `/api/mcp`. Installation Options listed are Docker, Homebrew, MacPorts, WinGet, release binaries (Linux, macOS, Windows, FreeBSD) and building from source with Go and Node.js. Nightly Docker images come in two families: unified images bundling llama-server, ik-llama-server, stable-diffusion.cpp, whisper.cpp, audio.cpp and llama-swap (CUDA 12, CUDA 13 and Vulkan variants, recommended), and a legacy image based on llama.cpp's own `llama-server` container. Unified images can be configured through `LLAMA_SWAP_*` environment variables mapping to command-line flags. Operational notes The README advises disabling response buffering when placing llama-swap behind nginx, since buffering breaks SSE and streaming chat completions; llama-swap also sets `X-Accel-Buffering: no` on SSE responses. For Python-based servers such as vllm or tabbyAPI, running them under Podman or Docker is recommended for environment isolation and correct `SIGTERM` handling.