gemaal

# gemaal **gemaal** (Dutch for pumping station) is an orchestration tool designed to maintain the health of shared Kubernetes test clusters. Just as a physical pumping station keeps a polder dry by continuously removing water, gemaal keeps a test cluster usable by automatically detecting and cleaning up stale, ephemeral installations. It ensures that temporary test tenants do not accumulate and degrade the environment for other users. ## Core Philosophy * **Non-Blocking Cleanup**: gemaal never installs anything. Clients run `helm upgrade --install` themselves. The service only uninstalls and sweeps resources that are no longer in use. If the service goes down, cleanup is delayed, but it never blocks the installation loop for clients. * **Shadow Mode**: By default, the service runs in shadow mode (`dryRun: true`). It plans, reports, and logs deletion actions but does not execute them until explicitly enabled. This provides safety during deployment and allows operators to verify cleanup logic before it impacts live resources. * **Level-Triggered Watcher**: The service operates on a cron-like loop that re-derives the state of the cluster from scratch every tick. It uses `helm list` per namespace to see exactly what an operator sees, grouping releases into ring pairs (app + infra) and applying garbage collection rules based on time-to-live (TTL) labels. ## Three Faces 1. **`gemaal` (Service)**: An in-cluster watcher responsible for: * TTL housekeeping over test tenants. * Ring-pair-aware teardown (ensuring infrastructure is torn down after applications). * Orphaned-artifact sweeps (e.g., S3 subtrees). * Exposing six ConnectRPC RPCs: Plan, ListTenants, Checkout, Extend, Sweep, Resolve. * Providing a web console for monitoring tenants, ages, tiers, and sweep history. 2. **`gemaalctl` (CLI)**: A command-line interface for: * Checking identity evidence chains and resolved tenants (`whoami`). * Managing client-side Helm installs/uninstalls with ledger labels stamped. * Interacting with the service via ConnectRPC for planning, checking out, or extending tenant lifetimes. 3. **Go Library**: Imported by test harnesses to: * Resolve standing tenants. * Bracket test suite phases (build, deploy, setup, teardown). * Manage identity resolution and configuration loading. ## Key Features * **Tenant Isolation & Identity**: Uses tier labels (e.g., `tenancy.truvity.io/tier`) to identify reachable namespaces. Ignores system namespaces like `gemaal-system`. Identity is resolved through a chain involving email, kubectl groups, and AWS SSO sessions. * **Garbage Collection Rules**: * Uniform TTL based on last activity, configurable per tenant or tier. * `keep-until` precedence for specific retention needs. * Ring-ordered teardown (application before infrastructure). * Collection of orphaned artifacts past a grace period. * **Authentication & Authorization**: * Mutations authenticate via TokenReview against the Kubernetes API (for workloads) or OIDC JWT (for humans). * Checkout/Extend require owner or admin rights. * Sweep operations are admin-only. * **Web Console**: A Vite/React/MUI single-page application embedded in the binary, providing a fleet console stack for visualizing tenant status and sweep history. ## Test Harness Integration Projects can integrate gemaal into their Go integration tests using the `pkg/harness` library. The harness resolves a standing tenant once in `TestMain`, allowing tests to run inside a dedicated namespace. The service handles cleanup via labels stamped on Helm releases, so the test suite itself creates and deletes nothing directly (except in interim teardown hooks). Environment variables like `GEMAAL_TEST_SKIP_BUILD`, `GEMAAL_TEST_SKIP_DEPLOY`, and `GEMAAL_TEST_KEEP` allow fine-grained control over the test lifecycle in CI/CD pipelines. ## AWS Access The service supports standard AWS credential chains, including EKS Pod Identity and IRSA (IAM Roles for Service Accounts). It interacts with AWS Systems Manager (SSM) for artifact storage and requires appropriate permissions for STS and pod identity endpoints. ## Development * **Toolchain**: Uses [devbox](https://www.jetify.com/devbox/) and [just](https://just.systems/) for task management. * **Commands**: * `just check`: Runs build, test, lint, and vulnerability checks. * `just generate`: Regenerates code from Protobuf definitions. * `just run`: Runs the service skeleton against example configuration. ## Status The project is in early development (G4 phase). The design, proto surface, client library, CLI, and service are in place. The service has not yet seen a production deployment; initial deployments will be in shadow mode. The client API is in real use but may change between 0.x minor versions. ## License MIT License