seismograph

SEISMOGRAPH is an open-source tool designed to detect when an LLM API starts answering differently, even when the model name, endpoint, speed, and uptime remain unchanged. It addresses the problem of silent semantic drift in third-party LLM APIs, which conventional monitoring (latency, error rate, uptime) misses. The tool works by running a fixed, content-addressed canary suite (up to 200 prompts) against any OpenAI-compatible endpoint at temperature 0. Each response is reduced to privacy-preserving features: SHA-256 hashes and Laplace differential-privacy-noised aggregates (epsilon=2.0). Raw prompts and outputs never leave the probe perimeter. Every batch is Ed25519-signed. The gateway ingests probe batches and applies Page-CUSUM change-point detection per (model_tuple, metric_name) on distributional metrics like average output length and JSON success rate, not on exact-match hashes. A federated quorum system ensures that a public drift alert is only promoted when multiple independent organizations (minimum 2) agree, preventing false alarms from single noisy probes or Sybil attacks. The repository includes a live dashboard ("Model Weather") showing STABLE/DRIFTING/STALE status per model, a reproducible drift-floor measurement demonstrating that a newer model (gemini-3.5-flash-lite) reproduced itself only ~60% of the time at temperature 0, while an older model (gemini-3.1-flash-lite) reproduced exactly 100%, and a seeded synthetic backtest of an Anthropic incident showing a 38-day lead time over the official postmortem. Key components: a Python probe SDK, FastAPI ingestion gateway, SQLAlchemy storage, CUSUM detector, quorum correlation engine, and a vanilla JS dashboard. The project is Apache-2.0 licensed, with CI (ruff, pytest, CodeQL) and a test suite covering adversarial scenarios like single-org noise blocking and quorum-triggered alerts.