About this project

AISIX AI Gateway is an open-source, Rust-native gateway that provides a single, OpenAI-compatible API in front of multiple LLM providers, including OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, DeepSeek, and any OpenAI-compatible endpoint. It is designed for platform teams to route, govern, secure, and observe LLM traffic from one control point. Key features include: - **Unified API**: Speak the OpenAI or Anthropic wire format; the gateway translates to the target provider. - **Routing & failover**: Six strategies (round_robin, consistent_hash, failover, least_cost, least_latency, least_busy), priority tiers, retry budgets, and cooldowns. - **Ensemble models**: Fan out requests to multiple models and synthesize answers with a judge model. - **Semantic routing**: Route based on prompt meaning using embeddings. - **Rate limiting & concurrency**: RPS/RPM/RPH/RPD, TPM/TPD, and concurrency caps, with Redis backend for distributed limits. - **Guardrails**: Content-policy enforcement (keyword/regex, PII detection, Presidio, Lakera, OpenAI Moderation, AWS Bedrock Guardrails, Azure AI Content Safety). - **Caching**: Exact-match response cache with TTL, memory/Redis backends, and automatic prompt caching for Anthropic. - **MCP gateway**: Front registered MCP servers at /mcp with gateway-held credentials and tool namespaces. - **A2A agent gateway**: Front A2A agents at /a2a/:agent. - **Inbound authentication**: Caller API keys (SHA-256 hashed) or OIDC/JWT bearer tokens. - **Observability**: Prometheus metrics, structured access logs, OTLP/GenAI span export, Datadog and Aliyun SLS exporters, object-storage telemetry. - **Declarative configuration**: One resources.yaml file, validated against JSON Schemas, with atomic SIGHUP reloads. It runs as a single static binary with low overhead, supports hot configuration reloads, and can use etcd for multi-replica clusters. The project is Apache-2.0 licensed and built by the creators of Apache APISIX. A commercial cloud offering (AISIX Cloud) adds a control plane with budgets, audit, and a dashboard.