llm-router

LLM Router is a FastAPI-based backend service designed to dynamically route user prompts to the most suitable Large Language Model (LLM) among OpenAI, Google Gemini, and Mistral. It evaluates models using a scoring system that balances usefulness, cost, and reliability — not just raw performance. The system supports three routing modes: full auto (system selects best model), semi-auto (user picks provider, system picks model), and manual (user specifies both). It includes Retrieval-Augmented Generation (RAG) powered by Mistral embeddings and cosine similarity retrieval, supporting .txt, .md, and .pdf files. Users can upload documents via /upload and clear data via /clear-data. Each user maintains a 2-turn conversation memory for context-aware responses. The architecture separates decision-making from execution: routing logic chooses the model, while client modules handle actual API calls. Models are disqualified if they exceed token limits. Responses include metadata such as provider, model, confidence, tokens used, and cost estimate. WebSocket support enables real-time streaming. Future plans include feedback-driven tuning, local LLM integration, and A/B testing. The project is actively under development, with core features like intent-aware routing, cost optimization, and RAG already implemented.