council-router

Council Router is an intelligent proxy server designed to unify access to multiple large language model (LLM) backends — including Ollama, OpenAI, Anthropic, Gemini, and over 100 others via LiteLLM — through a single OpenAI-compatible API endpoint. It enables seamless routing across heterogeneous LLM servers with features like priority-based selection, round-robin load balancing, and automatic model fallback chains. The system intelligently prefers backends where the requested model is already loaded in VRAM to minimize latency. A standout feature is the "LLM Council" mode, which allows multiple models or personas to deliberate on a user’s query in parallel before synthesizing a final answer — useful for complex decision-making or code review scenarios. Built-in templates include diverse analysis, code review, and debate modes, with support for custom YAML-defined councils. All responses are optionally stored in compressed SQLite for offline analysis, including intermediate thinking tokens. Prometheus metrics track request rates, latency, token usage, and backend health, with a pre-configured Grafana dashboard available via Docker Compose. The service also exposes native Ollama endpoints alongside OpenAI ones, making it compatible with tools like Open WebUI. Security-wise, inference endpoints are unauthenticated by default (intended for trusted networks), while admin endpoints require a Bearer token. Response storage is enabled by default but not encrypted, so sensitive prompts should be handled with care. The project is built with FastAPI, runs on Python 3.12+, and includes full test coverage (127 passing tests). Docker Compose provides a ready-to-run stack with monitoring. Configuration is managed via YAML, supporting environment variable injection. Development tools include linting, formatting, and testing scripts.