About this project

This project demonstrates building an LLM Gateway using LiteLLM and LangChain, enabling unified access to multiple AI providers (OpenAI, Groq, etc.) through a single OpenAI-compatible API. It includes intelligent model routing, automatic failover, load balancing, cost tracking, caching, observability, and input-side guardrails like PII redaction and prompt injection blocking. The implementation is delivered as a runnable Jupyter notebook that walks through each concept step-by-step, from basic unified API calls to production-ready proxy server setup using config.yaml. It integrates seamlessly with LangChain chains and supports deployment behind autoscaling infrastructure. Best practices include Redis caching, per-user rate limits, logging to observability backends, and version-controlled configuration. The gateway abstracts provider-specific SDKs, allowing teams to switch models or providers without code changes, while centralizing cost, latency, and security controls.