llm-router

llm-router is a lightweight, single-binary reverse proxy designed to optimize costs when using LLM APIs like Claude. It sits between your application and LLM providers, intelligently routing each request to the cheapest model capable of handling it — such as Haiku for simple queries, Sonnet for moderate tasks, or Opus for complex reasoning. It includes semantic caching (using Jaccard similarity without external embeddings), deduplication of identical concurrent requests, and execution of multi-step agent workflows defined as DAGs with dependency tracking and parallel execution. The system tracks backend health and load in real time, automatically avoiding degraded or overloaded models. It supports ML-assisted routing with feedback calibration and provides detailed cost tracking and savings reports. Configurable via YAML, it works with Anthropic, Ollama, vLLM, or any OpenAI-compatible backend. Built from techniques in 7 research papers and open-source systems, it requires no external dependencies and includes 57 tests. Ideal for reducing API spend while maintaining performance and resilience in production environments.