smart-slm-router

Smart SLM Router is an asynchronous local proxy designed to reduce API costs by intelligently routing LLM requests. It is fully compatible with the OpenAI API specification, allowing it to serve as a drop-in replacement for existing OpenAI SDK applications by updating the base URL. Key capabilities include: - Local Intent Classification: Uses length heuristics, token regex patterns, and tool-call indicators to decide routing in under 15ms without incurring network latency. - Dynamic Routing: Directs simple queries to SLMs (e.g., via Ollama or vLLM) and escalates complex, technical, or multi-step queries to Tier-1 frontier models (e.g., GPT-4o, Claude). - Transparent Fallback: Automatically retries and escalates requests to a frontier model if the SLM returns a connection failure, model refusal, or JSON parse error. - Full API Support: Supports both non-streaming and streaming (SSE) responses. - Observability: Provides a `/metrics` endpoint to track real-time USD cost savings, request counts, and decision latency. The system is built with Python 3.12+ and FastAPI, utilizing environment variables for configuration of backend URLs, model names, and token thresholds.