aethergrid

AetherGrid is a distributed AI workload orchestrator that schedules workloads across compute nodes based on resource requirements and constraints, managing the full job lifecycle from queued through scheduled, running, completed, failed, retried, and cancelled. It is built around strict Domain-Driven Design, with a decoupled domain core that has no infrastructure dependencies, an event-driven reconciliation loop, and lease-based execution ownership. The project's stated motivation is to address the hard problems of scheduling at scale: exclusive execution ownership under failure, reconciliation after partial failures, and enforced resource limits. The domain layer (Job, Node, Worker, Lease, Event, ApiKey aggregates) contains the scheduling algorithm and lifecycle state machine as plain Python, with no imports from FastAPI or psycopg. Application services coordinate domain objects and repositories, while infrastructure provides PostgreSQL implementations for every repository using raw psycopg rather than an ORM, plus SQLite implementations for some repositories in local development. The presentation layer is FastAPI, and the frontend is React with TypeScript. Key capabilities described include API key authentication gating every route (including the key-issuing endpoint, with the first key minted only via a local script with direct database access), explicit job state transitions including cancellation of a running job via a CANCELLING state, per-job lifecycle history pages, a constraint-aware best-fit allocator, node draining, worker registration and heartbeats, a standalone worker agent process that polls over HTTP and executes jobs as real subprocesses, lease-based execution ownership with continuous renewal, explicit execution-start confirmation, real subprocess execution with two-stage shutdown (SIGTERM then SIGKILL), node liveness tracking, reconciliation with bounded retries, domain event recording, a live cluster-wide event feed, worker visibility, and a multi-page dashboard. The README emphasizes engineering decision records: 39 ADRs are documented in /docs/adr, covering topics such as the reconciliation loop, job reclaim and reconciliation repair, real job execution, continuous lease renewal, API key authentication, domain ownership of scheduling policy, the standalone worker agent, cancellation via lease renewal, idempotent worker registration, reclaiming jobs abandoned mid-cancellation, moving cluster tick execution off the event loop, persisting RUNNING transitions, and several lease-fencing fixes by lease identity rather than worker identity. The README states 347 tests across domain, application, infrastructure, and API layers, including contract tests proving repository implementations behave identically. The project includes a live demo deployed on Render with real Postgres, real auth, and API-key-gated endpoints. It is licensed under MIT and uses Python 3.12, FastAPI, PostgreSQL, React/TypeScript, and Docker Compose.