mcp-server

Iris is an open-source MCP (Model Context Protocol) server designed to evaluate AI agent outputs for quality, safety, and cost. It runs entirely on your machine, stores traces in a local SQLite database, and requires no account, no SDK, and no telemetry. The project is MIT-licensed and requires Node.js 20 or later. Core capabilities include trace logging with hierarchical span trees, per-tool-call latency, token usage, and cost tracking in USD. Output evaluation uses 20 built-in deterministic rules across four categories: completeness, relevance, safety, and cost. Safety rules detect PII (19 patterns including SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API keys, passport, and cloud provider tokens), prompt injection (37 patterns), hallucination markers (25 context-grounded fabrication signals), and trajectory violations such as unacknowledged failed tool calls, repeated calls, schema-rejected arguments, and cited sources the agent never read. An optional LLM-as-judge feature provides semantic scoring via Anthropic or OpenAI with a hard per-eval cost cap (default $0.25). Iris registers twelve MCP tools that any MCP-compatible agent can invoke: log_trace, evaluate_output, get_traces, list_rules, deploy_rule, delete_rule, delete_trace, evaluate_with_llm_judge, verify_citations, compare_runs, compare_traces, and evaluate_runs. The verify_citations tool extracts citations from output, fetches sources behind an SSRF-guarded resolver, and uses an LLM judge to check whether each source supports the cited claim. A web dashboard serves at http://localhost:6920, landing on failures sorted worst and newest first. It provides trace visualization, eval results, cost breakdowns, and a command palette (Cmd+K) for searching rules, traces, and evals. The dashboard also exposes an HTTP ingest endpoint (POST /api/v1/traces) for capturing traces without a model in the loop, and a capabilities endpoint (GET /api/v1/capabilities) that describes what the server can judge. For CI/CD integration, the ingest CLI command reads JSON or NDJSON traces from stdin or a file, evaluates them, prints one JSON line per trace with the verdict, and exits with code 1 when a verdict matches a --fail-on filter. This enables gating agent deployments on evaluation results. Custom rules can be authored inline (up to 10 per evaluate_output call) or deployed persistently via deploy_rule. Rule types include regex_match, regex_no_match, min_length, max_length, contains_keywords, excludes_keywords, json_schema, and cost_threshold. Deployed rules persist in custom-rules.json under the Iris home directory and fire on every future evaluate_output of their category. Severity levels (low, medium, high, critical) determine whether a rule failure hard-fails the evaluation or only affects the score. The passed verdict is decided by a composer that reads each rule by the kind of claim it makes: configured policies gate, critical detectors veto, critical checks that could not answer produce an unknown verdict, and remaining detectors combine into a probability weighed against a configurable loss ratio. Genuine safety violations (no_pii, no_injection_patterns, no_blocklist_words) hard-fail by default. Every built-in rule has published precision, recall, and F1 scores with 95% confidence intervals, measured on a labelled corpus in the repository. CI re-runs the measurement on every pull request and fails if committed numbers differ from what the code produces. Iris is local-first: everything lives in SQLite on disk. Outbound HTTP happens only where you opt in — your own LLM-judge key, citation fetching, or an OpenTelemetry exporter you configure. The project supports Docker deployment with two ports (3000 for MCP HTTP transport, 6920 for dashboard and ingest), and integrates with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI, and any other MCP-compatible agent.