tap-token-analytics-proxy

TAP (Token Analytics Proxy) is a self-hosted middleware designed to sit between an application and LLM providers (specifically OpenAI). It allows developers to monitor and manage API traffic without modifying their application's core logic. Key capabilities include: - **Traffic Analytics**: Records every call in a PostgreSQL database, tracking model usage, token counts, costs, latency (including TTFT for streaming), and status codes. - **Caching**: Uses Redis to store request/response pairs, allowing identical requests to be served from cache to reduce costs and latency. - **Rate Limiting**: Implements request-per-minute budgets per API key using Redis. - **API Key Management**: Allows the issuance of custom TAP keys to identify different apps or teams. - **Monitoring Dashboard**: A React-based frontend providing live charts for volume, cost per model, speed, cache hit rates, and error rates. The system supports streaming responses and uses an asynchronous stack featuring FastAPI, SQLAlchemy 2.0, and Redis. It includes a CLI for administrative tasks and a mock upstream server for development without requiring a live API key.