frugal-llm

Frugal LLM is a lightweight proxy server written in Go that provides an OpenAI-compatible unified API interface. Its primary purpose is to optimize API budget and response quality by automatically routing incoming prompts to specialized model architectures based on the task domain (e.g., routing casual chat to high-speed engines and complex reasoning to flagship models). Key capabilities include: - Dynamic Routing: Uses a template-driven LLM classifier engine to match workloads to the ideal provider and model. - Broad Provider Support: Integrates with OpenAI, Google Gemini, Anthropic Claude, DeepSeek, Groq, Together AI, HuggingFace, and local servers like Ollama, LM Studio, and vLLM. - OpenAI Specification Compatibility: Works with standard OpenAI SDKs, LangChain, LlamaIndex, and coding assistants like Cursor and Cline without code refactoring. - Technical Features: Supports SSE streaming, provider retries with exponential backoff, and flexible payload unmarshaling for structured content. - Configuration: Supports zero-edit setup via environment variables or advanced customization through YAML files.