llm-apipool

llm-apipool is a self-hosted tool that unifies access to over 40 free-tier large language model (LLM) APIs — including Groq, Mistral, Cerebras, Google Gemini, OpenRouter, Hugging Face, GitHub, and more — behind a single OpenAI-compatible endpoint (e.g., http://localhost:8000/v1). It eliminates the need to manage multiple API keys, SDKs, or rate limits manually. The system automatically rotates keys per provider, applies cooldowns when rate-limited, and falls back to alternative providers or tiers. It includes a CLI for setup and management, an interactive Textual-based TUI for real-time key and audit log monitoring, and a React dashboard with analytics and settings. Supports streaming (SSE for compatible providers), unified effort configuration across models (low/medium/high reasoning), subscriber tracking, connection pooling, and Docker deployment with health checks. Integrates seamlessly with LangChain via AggregatorChat, and works with any OpenAI-compatible tool (Hermes Agent, Cursor, Continue.dev, etc.) by simply changing the base URL. Built for individual developers and small teams aiming to avoid API costs while maintaining production-grade reliability. Includes 522 passing tests, Alembic migrations, and documented provider guides. Roadmap includes plugin support, WebSocket streaming, multi-user auth, and distributed clustering.