llm-router

LLM Router is a proxy server and Python library that consolidates multiple Large Language Model (LLM) providers into a single endpoint. It allows users to interact with various models without changing API keys or codebases by managing configurations via a YAML file. Key capabilities include: - **Multi-Provider Support**: Out-of-the-box integration with 10 providers, including Google, Cohere, Mistral, OpenRouter, Cerebras, Groq, Hugging Face, Kilo Code, llm7.io, and NVIDIA NIM. - **Intelligent Routing**: Automatically selects models based on capabilities (vision, tool use, etc.), context window size, and quality modes (quality, fast, cheap, or code). - **Reliability Features**: Implements exponential backoff retries, automatic fallback to alternative models upon failure, and key rotation with cooldowns for rate-limited keys. - **OpenAI Compatibility**: Includes a local server that exposes an OpenAI-compatible API, making it compatible with editors like Void and Continue. - **Observability**: Logs all requests and responses to a local SQLite database for history and auditing. - **Flexible Usage**: Can be used as a standalone local server or integrated directly into Python projects as a library.