llm-gateway

LLM Gateway is a unified proxy server designed to streamline interactions with multiple Large Language Model (LLM) providers. It provides a single, OpenAI-compatible endpoint that routes requests to various backends including OpenAI, OpenRouter, Google Gemini, and local instances like Ollama. Key capabilities include an automatic fallback mechanism that attempts alternative providers if the primary one fails, ensuring high availability. The gateway also implements rate limiting per API key to prevent overuse and provides a comprehensive metrics endpoint for tracking total requests, success rates, and token consumption across different providers. It is designed for easy deployment via Docker and can serve as a drop-in replacement for existing OpenAI client configurations by simply updating the base URL. Supported features include chat completions, embeddings, and model listing, with configurable fallback chains and requests-per-minute limits.