substrate

Agent Substrate is a secure-by-default agent execution runtime engineered to run millions of sandboxes with 10x higher density than standard container runtimes. Purpose-built for autonomous agents, it delivers sub-500ms resume operations at over 500 suspend/resume activations per second with native zero-trust kernel and network isolation. It supports multiple sandbox technologies including microVMs and gVisor, enabling consistent lifecycle operations for all sandbox types. At its core, Agent Substrate maps a larger set of 'actors' (applications such as agents) onto a smaller set of ready 'workers', relying on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing. It provides functionality to manage an actor's lifecycle (e.g., create/destroy, suspend/resume), assign actors to workers in real time, and route incoming traffic to them. Agent Substrate is intended to be a low-opinion system. The workloads it manages don't have to be literal AI agents, but those are the best example of the kind of applications it is designed for. It is not an SDK for building agents, but rather a system for running them at scale. Agent Substrate leverages Kubernetes for infrastructure provisioning and worker lifecycle management (Kubernetes Pods). It builds on top of Kubernetes features like Pods and Pod autoscaling, while Agent Substrate provides agent-specific scheduling and control to achieve lower latency. Using Kubernetes as the underlying system enables consistent infrastructure management across all workload types required for end-to-end agentic deployments and allows holistic infrastructure optimizations for RL scenarios that span agentic, inference, and training cycles. ## Demo A demo video shows the Agent Substrate cluster multiplexing ~250 stateful actors across just 8 physical pods. Key capabilities demonstrated include: 1. **Actor Teleport:** High-performance suspend and resume of actors onto any available worker in the pool with sub-second activation. 2. **State Persistence:** Persistent working memory (volatile RAM) and filesystem state preserved perfectly across hibernation cycles via full-state snapshots. 3. **Agent Multiplexing:** Demonstrates 30x+ oversubscription by 'juggling' a large registry of stateful actors onto a small pool of shared physical pods. ## Framework Agnostic & Compatibility Agent Substrate is designed to be framework and agent harness agnostic. Because it manages standard OCI containers at the kernel level (via gVisor), it can host agents built on any stack. - **Agent Development Kit (ADK):** Support for ADK agents with session state preservation across invocations as actor state. - **LangChain:** Ideal execution environment for LangChain agents and tool calls. - **Claude Code, CodeX, and Antigravity:** Support for high-density, stateful coding environments that preserve system state and filesystem state across sessions. - **Model Context Protocol (MCP):** Support for deploying secure, sandboxed MCP servers as Substrate Actors to provide durable tools for any model. ## Ecosystem & Examples - **Agent Executor (google/ax):** A distributed agent runtime demonstrating building a secure, hyper-scalable agent harness on Agent Substrate. - **kagent:** A CNCF Sandbox project and Kubernetes-native framework for building, deploying, and managing AI agents that uses Agent Substrate to run sandboxed, stateful agent workloads. ## Status and compatibility Agent Substrate is currently in early development. It is not ready for production use, and the APIs are almost guaranteed to change. No backward compatibility guarantees are made at this stage. ### Supported Kubernetes Releases Currently aims to support the latest stable release of Kubernetes and the previous minor release. ## Community Join the ate-dev Google Group for announcements and discussions. Weekly community meetings are held every Thursday from 10:00am - 11:00am PST. There are also CNCF Slack channels: #substrate-users and #substrate-dev. ## Quickstart (Development) To set up the complete environment locally: 1. Ensure Go, kubectl, and docker are installed. 2. Run the following steps: - `hack/create-kind-cluster.sh` to create cluster and local registry. - `hack/install-ate-kind.sh --deploy-ate-system` to install ate, PostgreSQL, rustfs. - `hack/install-ate-kind.sh --deploy-demo-counter` to install counter demo. - `go install ./cmd/kubectl-ate` to install kubectl-ate. - Create a counter actor: `kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter`. - Port-forward the network router: `kubectl port-forward -n ate-system svc/atenet-router 8000:80`. 3. In a separate terminal, send an HTTP request to increment the counter: ```shell curl -X POST -H "ate-target-actor: ate-demo-counter/my-counter-1" -i http://localhost:8000/ ``` Worker capacity is versioned: the dataplane schedules only on nodes that carry the `ate.dev/substrate-version` label, and the install stamps it on every node that exists when it runs. ### GKE Quickstart (Development) 1. Copy and edit the environment file: `cp hack/ate-dev-env.sh.example .ate-dev-env.sh`. 2. Enable application-default credentials: `gcloud auth application-default login --project=${PROJECT_ID}`. 3. Provision GCP resources: `go run ./tools/setup-gcp bootstrap`. 4. Deploy the system: `./hack/install-ate.sh --deploy-ate-system`. 5. Deploy sample apps: `./hack/install-ate.sh --deploy-demo-counter`. Custom setup and teardown scripts are available for individual steps. ## Demos - **Counter Demo:** A stateful Go HTTP server demonstrating state preservation across suspends/resumes. - **Sandbox Demo (Antigravity):** A secure, sandboxed execution environment (Alpine Linux) allowing arbitrary shell execution while preserving filesystem state. - **Claude Code Multiplex:** Demonstrates oversubscribing physical hardware by multiplexing multiple Claude Code agents onto a limited pool of workers. - **Multi-Template:** Two ActorTemplates running different binaries share one WorkerPool. - **Request Parking:** An oversubscribed pool where the router holds inbound requests until a worker frees up. - **Autoscaled WorkerPool:** Scales a WorkerPool on its assigned-worker count with an HPA fed by prometheus-adapter. ## Documentation & Guides - Architecture: How the control plane, node supervisor, and networking stack fit together. - API Configuration Guide: Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes. - Full CLI Documentation: Installation and usage for kubectl-ate. - Glossary: Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateom). - Integration Repositories: Where integrations live and how fixes flow back to core. - Observability Guide: Actor logging, metrics, and distributed tracing. - Authentication Guide: Configure trusted JWT providers and human credentials. - Egress Traffic: Which protocols an Actor may reach the outside world with and which are blocked. - Enabling MITM interception for Actor Egress policy. - Request Parking: How the router parks requests through transient worker-pool saturation. - Rolling Upgrade Runbook: Upgrade a running substrate node by node without losing actor state. - Threat Model: Trust boundaries, assumptions, and known risks. - Roadmap: Current limitations and what is planned next. - Benchmarking Guide: Locust-based load tests, monitoring stack, and orchestrated benchmark harness. ## Tour Key commands and components: - `cmd/ateapi`: Core control plane API server exposing gRPC endpoints. - `cmd/atelet`: Node-level DaemonSet supervising physical worker pods. - `cmd/atecontroller`: Kubernetes controller reconciling WorkerPool custom resources. - `cmd/atenet`: Combined networking controller providing Envoy routing and proxy sidecars. - `cmd/ateom-gvisor`: Interior-pod helper executing runsc checkpoint and restore commands. - `cmd/ateom-microvm`: Micro-VM peer running actors as cloud-hypervisor VMs. - `cmd/podcertcontroller`: Polyfill providing Pod Certificate signers. - `cmd/kubectl-ate`: CLI tool for managing Agent Substrate resources. - `cmd/benchmarking`: Synthetic workloads for load tests. - `tools/setup-gcp`: Provisioning utility for GCP infrastructure. - `demos/`: Sample applications demonstrating capabilities.