server

Hyperstrate Server is a self-hosted AI gateway designed for teams requiring granular control over LLM infrastructure. It acts as a centralized layer between applications and various AI providers, supporting hosted services (OpenAI, Anthropic, Gemini, Mistral, Azure OpenAI, Bedrock, Groq, Cohere), self-hosted models (Ollama, vLLM, LocalAI), and custom HTTP providers. Key capabilities include: - **Routing & Orchestration**: Supports multiple routing strategies (round-robin, weighted, failover, latency-based) and pipeline features such as semantic caching, context trimming, retries, and MCP tool integration. - **Governance & Security**: Implements request caps, cost budgets, rate limits, virtual keys, and OIDC group mappings to manage multi-tenant access. - **Observability**: Provides detailed inference logs, pipeline traces, Prometheus metrics, and analytics to track cost, latency, and token usage. - **Evaluation**: Includes tools for managing versioned prompt templates and running evaluation sets with LLM-judged scoring to test router regressions. - **Developer Experience**: Offers SDK-compatible APIs (OpenAI and Anthropic compatible) allowing existing clients to integrate by simply changing the base URL. Technically, the server is built with Go 1.25 and Gin, supporting SQLite for local development and PostgreSQL for production. It can be deployed as a standalone binary or via AWS SAM using Lambda and SQS for asynchronous job processing.