zai-proxy

zai-proxy is a Go-based reverse proxy that sits in front of the Z.AI Claude API, adding observability and reliability features that bare API access lacks. It consists of two main components: a proxy service and a monitoring dashboard. The proxy listens on port 8080, forwards requests to the upstream LLM API, and exports Prometheus metrics at /metrics. Key capabilities include token tracking (counting input and output tokens per request using API-reported usage or tiktoken estimation as a fallback), adaptive rate limiting (tracking upstream 429 responses with an EWMA ceiling estimator that automatically holds just below the limit and periodically probes for ceiling increases), and SSE streaming pass-through without buffering. It also supports blue/green and canary deployments via a DEPLOYMENT_VARIANT label for comparing proxy versions or model configurations side-by-side. The dashboard is a Go backend with a React frontend that scrapes the proxy's Prometheus endpoint every 5 seconds, maintains dual-resolution snapshots in bounded in-memory rings (5-second resolution for 24 hours, 1-minute resolution for 7 days), and streams updates to the browser via SSE. It displays request throughput, latency percentiles, token usage totals, error rates, and per-variant comparison panels. The proxy exposes a comprehensive set of Prometheus metrics including request counts, latency histograms, token rates, error rates, rate-limit wait times, retry attempts, worker utilization, and build information, all labelled by model and deployment variant. Configuration is handled through environment variables covering API keys, target URLs, rate limiting parameters, quota polling, and retry behavior.