llm-failover-proxy

llm-failover-proxy provides a single local endpoint (OpenAI-compatible) that manages a prioritized chain of LLM providers. If a primary model fails due to rate limits, errors, or timeouts, the proxy invisibly switches to the next available model in the chain without interrupting the application. Key capabilities include: - Parallel Racing: If a preferred model is slow to respond (default 5 seconds), the proxy requests the next model in parallel, accepting the first usable answer to minimize latency. - Model Lists: Supports multiple named lists for different purposes, such as separate chains for chat and embeddings. - Provider Management: Includes a terminal UI for configuring API keys, managing model priority, and monitoring performance stats. - Health Monitoring: Provides endpoints (/health, /stats) to check service liveness and track success/failure rates per model. - Background Operation: Can be installed as a background service that starts automatically on system login across Windows, macOS, and Linux. - Compatibility: Supports streaming, tool/function calling, vision, and JSON mode by passing them directly through to the providers.