llm-router

LLM Router is a FastAPI application designed to act as a unified proxy for multiple AI backends while maintaining full compatibility with the OpenAI Chat Completions API. It allows developers to route requests across various providers—including OpenAI, Cerebras, DeepInfra, and local Ollama instances—using a priority-based system with automatic failover. Key capabilities include: - Multi-Backend Routing: Seamlessly switch between different AI services based on availability and priority. - OpenAI Compatibility: Functions as a drop-in replacement for existing OpenAI-based applications by simply changing the base URL. - Resilience Features: Implements automatic failover to secondary backends if the primary service fails or hits rate limits. - Traffic Management: Features a built-in rate limiting system using a sliding window algorithm and supports multiple API keys for authentication. - Monitoring & Stats: Provides dedicated endpoints for health checks, backend status, and usage metrics. - Flexible Configuration: Supports environment-based setup or advanced JSON-based routing rules. The tool is particularly useful for developers looking to increase the reliability of their AI-powered applications by diversifying their model providers without changing their codebase.