llm-cache-server

LLM Cache Proxy is a caching layer for the OpenAI API built with FastAPI. It intercepts requests to the OpenAI chat completion endpoint and serves cached responses for identical requests, which helps in reducing API costs and improving response latency during development. Key capabilities include: - Caching of OpenAI API responses with support for both streaming and non-streaming modes. - OpenAI API compatibility for chat completion endpoints. - Persistent cache storage to maintain data across restarts. - Deployment options via Python or Docker. - Two distinct endpoints: `/cache/chat/completions` for cached requests and `/chat/completions` for uncached requests.