llm-api-gateway

This repository provides a provider-neutral FastAPI gateway designed to abstract away direct dependencies on specific LLM providers like Gemini or OpenAI. Instead of hardcoding model names such as 'gemini-3.5-flash' or 'gpt-5.6-terra', applications request capabilities via stable aliases like 'fast', 'reasoning', 'cheap', or 'embedding'. The gateway handles provider selection, retry logic with exponential backoff and jitter, fallback routing, timeout enforcement, error normalization, token usage tracking, cost estimation, and persistent storage of usage events in SQLite. Key features include: - Model alias-based routing with ordered fallback chains - Provider adapters for Gemini and OpenAI - Structured JSON output using JSON Schema constraints - Text embeddings with normalized metadata - Request ID propagation and latency headers - Configurable pricing catalog for cost estimation - Usage analytics endpoint with breakdown by provider/model - Async tests, Docker support, and GitHub Actions CI/CD - Security controls including API key authentication in production The architecture enforces a clean separation between application logic and LLM provider complexity. It includes middleware for request context, typed Pydantic responses, and detailed logging. Failed attempts are persisted alongside successful ones to maintain operational transparency. The system is extensible but intentionally avoids advanced features like JWT auth, streaming, or distributed rate limiting to focus on core abstraction and reliability. Setup involves cloning the repo, creating a virtual environment, installing dependencies, configuring .env with API keys, and running via uvicorn or Docker. Swagger documentation is available at /docs. The project uses Ruff, pytest, CodeQL, and Dependabot for code quality and security. A detailed NOTES.md explains design decisions and learning objectives for AI engineering projects.