llm-gw

The LLM gateway is a lightweight, self-contained proxy/router designed for personal single-user use with local hardware. It dynamically manages multiple Llama.cpp-compatible engines, automatically selecting and launching the appropriate model variant based on incoming request parameters — especially context size requirements. The system estimates token usage via standalone tokenizer or running engine endpoints, restarts engines with optimized configurations when needed, and shuts down idle instances to conserve memory. It supports configuration through Lua scripts (e.g., example.cfg.lua), defining server settings and model variants with different context sizes, binary paths, and startup arguments. Users can configure multiple variants per model (e.g., low VRAM vs high performance) for dynamic switching. The gateway listens on configurable IPv4/IPv6 addresses and exposes OpenAI API-compatible endpoints like /v1/chat/completions and /v1/models. Only one request is processed at a time due to internal locking. It currently implements only the chat completions endpoint and supports only llama.cpp engines, though the architecture allows extension. Setup involves installing Python dependencies via init.bat/sh, editing config files, and launching with run.bat/sh or uv run main.py. WARNING: Improper configuration may cause memory exhaustion or system freeze. Configuration is validated on load using embedded Lua scripts. This is experimental software intended for research/testing; not recommended for production use.