tracely-ai

Tracely is a self-hostable trace-native CI/CD system for AI agents. It ingests agent traces over OTLP/HTTP, grades them with structural checks and LLM judges, clusters failures into issues, and promotes failing traces into hermetic regression cases with recorded tool and LLM fixtures. These cases replay in CI at zero model cost and can block pull requests via a GitHub Action. The platform includes a dashboard with trace timelines, conversation replay and Fleet views, evaluator columns, failure clusters, scenario authoring, CI gates, a visual alert flow builder with Slack/email/webhook/LLM steps, trends and cross-metric analysis, judge calibration, an in-app assistant, and an MCP server. It ships with Docker Compose for local use, a Railway one-click deployment, and a Python SDK that auto-instruments OpenAI, Anthropic, Google, Mistral, and LangChain calls. The stack uses FastAPI, Celery, Next.js, Postgres, ClickHouse, Redis, and MinIO.