ai-router

ai-router is a comprehensive Python library for managing Large Language Model (LLM) API interactions. It provides a unified interface to route requests across various providers such as OpenAI, Anthropic, DeepSeek, and Google Gemini. Key capabilities include: - Smart Routing: Implements strategies like round-robin, weighted distribution, and cost/latency optimization. It includes automatic retries, circuit breakers, and semantic caching to improve reliability and performance. - RAG Pipeline: Offers tools for document chunking, embedding generation, hybrid retrieval (BM25 + vector), and reranking. - Agent Framework: Supports ReAct agents, tool registration, multi-agent orchestration (sequential, parallel, debate), and memory management. - Evaluation & Monitoring: Includes benchmarking tools for latency and throughput, along with scoring metrics like BLEU, ROUGE, and semantic similarity. - API Server: Features a FastAPI-based server that is compatible with the OpenAI /v1/chat/completions endpoint, facilitating seamless provider switching. The toolkit is designed for production environments, offering rate limiting, comprehensive metrics tracking, and easy deployment via Docker or CLI.