About this project

## Overview Traffic Analytics is a web‑analytics proxy designed for Ghost sites. It intercepts the `POST /api/v1/page_hit` calls made by Ghost's `ghost-stats.js` script, enriches the payload (user‑agent parsing, referrer analysis, user signature generation) and forwards the data to Tinybird's `/v0/events` API, where it is stored in a ClickHouse database. ## Architecture & Run Modes - **Batch mode (default)** – The ingest service validates requests, filters bots, and publishes raw events to a Google Cloud Pub/Sub topic. A separate worker consumes the subscription, enriches each event, batches them, and forwards to Tinybird. This decouples request handling from ingestion and improves throughput. - **Proxy mode (synchronous)** – When no Pub/Sub topic is configured, the ingest service performs enrichment inline and proxies the request directly to Tinybird in the same HTTP cycle. The mode is selected via the `WORKER_MODE` environment variable and the presence of `PUBSUB_TOPIC_PAGE_HITS_RAW`. ## Key Features - User‑agent parsing for OS, browser and device detection. - Referrer URL parsing and categorisation. - Privacy‑preserving user signatures with daily‑rotating salts. - Optional `x-ghost-bot-detected: true` header for filtered bot traffic. ## Configuration Copy `.env.example` to `.env` and adjust values. Important variables include: - `WORKER_MODE` – `worker` or `ingest`. - `PUBSUB_TOPIC_PAGE_HITS_RAW` – defines batch mode. - `ENABLE_BOT_DETECTION_HEADER` – toggles the bot‑detection response header. ## Development Workflow 1. **Prerequisites** – Docker (Desktop or Orbstack) and Docker Compose. 2. Clone the repository and run `pnpm dev` to start all services; the analytics API will be reachable at `http://localhost:3000`. 3. For local integration with a Ghost checkout, run `pnpm dev:ghost` in this repo and `pnpm dev:analytics:local` in the Ghost repo. This wires the two containers together via a shared Docker network. ### Multi‑Worktree Support The project can run multiple Git worktrees simultaneously. Each worktree uses its own `.env` file to set unique ports, Docker compose project names, and isolated volumes, allowing parallel development without port conflicts. ## Testing & Linting - `pnpm test:types` – TypeScript type checks. - `pnpm test:unit` – Unit tests. - `pnpm test:integration` – Integration tests. - `pnpm test:e2e` – End‑to‑end tests with WireMock. - `pnpm lint` – ESLint linting. All test commands run inside Docker containers for environment consistency. ## Deployment Pipeline - **Branch workflow** – Open a PR, optionally label `deploy-staging` to trigger a staging deployment. - **Merge actions** – Automatic patch version bump, Git tag creation, Docker Hub image publish, Cloud Run deployments to staging and production, health checks, and Slack notifications. - **Manual trigger** – Use the GitHub Actions UI to run the "Deploy" workflow via `workflow_dispatch`. Full CI/CD details are in `docs/deployment.md`. ## Documentation - `docs/architecture.md` – Detailed diagrams of batch vs. proxy modes, Pub/Sub pipeline, OpenTelemetry, and worker design. - `docs/deployment.md` – CI/CD pipeline, staging/production flow, and rollback procedures. ## License MIT © Ghost Foundation (2013‑2026).