llm-gateway

LLM Gateway is a caching reverse proxy for the OpenAI chat completions API that reduces costs by serving repeat or semantically similar questions from Redis. It supports two cache tiers: exact match (normalizing prompts) and semantic match (using vector search via Redis Stack). Follow-up questions are rewritten into standalone forms before embedding to enable cross-conversation matching. Built with FastAPI and Redis Stack, it adds X-Cache headers to responses so users can verify which tier served each request. Local setup requires Python 3.12+, Docker, and an OpenAI key. It includes a demo playground, load testing, and metrics showing exact hits at ~2ms, semantic hits at ~250ms, and misses at 1–4s. Designed for deployment as a stateless container with Redis Stack, it supports budget controls, rate limits, and moderation when enabled for public demos. Cache entries expire after one hour; no streaming support or authentication layer is included.