llm-context-squeezer

LLM Context Squeezer is a local API proxy built with FastAPI that acts as an intermediary between applications and OpenAI-compatible LLM providers. It aims to reduce token consumption and API costs through several automated optimization techniques: - **Dynamic Context Compression**: When conversation history exceeds a set threshold, the proxy uses a cheaper model (e.g., gpt-4o-mini) to summarize the oldest portion of the chat, injecting a dense summary into the system prompt. - **Redundancy Stripping**: A two-pass engine uses SHA-256 hashing for exact matches of code blocks and Jaccard similarity for fuzzy matching of near-duplicate text, replacing repetitions with compact markers. - **Semantic Caching**: Utilizes SQLite in WAL mode to provide exact and fuzzy prompt matching, returning cached responses to eliminate redundant API calls. - **Drop-in Integration**: Implements the OpenAI /v1/chat/completions contract, allowing it to work with the OpenAI Python SDK, LangChain, AutoGen, and CrewAI by simply changing the base URL. The proxy provides observability via injected response fields such as `x-squeezer-tokens-saved` and `x-squeezer-cache-hit`, and includes endpoints for health checks and runtime statistics.