Agent Substrate: How Suspended Agents Share a Warm Pool
If you run agents in containers today, most of the compute you reserve sits waiting on a model reply, a tool, or a person. Agent Substrate changes the unit you schedule: an agent becomes an actor that lives in a snapshot, and it borrows a warm worker pod only for the moments it is doing work. This part covers how that swap works, what a snapshot holds, and how to size the pool.
Agent Substrate is an open-source runtime published under its own GitHub organization, agent-substrate/substrate, under Apache-2.0. Its README carries the line "This is not an officially supported Google product," while its community meetings, mailing list, and meeting notes run on Google tooling, and its network egress doc says the certificate extension used for actor identity "will change after the CNCF donation is complete." Google Cloud announced it on May 20, 2026 in the same post that introduced Agent Executor, and on September 15 made it available on GKE for non-production workloads, with production support "available via allowlist." The repository itself still says it is in early development and that "the APIs are almost guaranteed to change," so treat everything below as a reading of the design and docs as of today. Nothing here was run on a cluster: every manifest and command is quoted from the project's docs and demos.
Where it sits in the stack
This series reads one vertical slice of the agent stack, bottom to top. Kubernetes owns nodes and pods. Agent Substrate runs on top of it and owns the fast part: which sandboxed actor is live on which pod right now. Google's AX, short for Agent Executor, sits above that and gives you tasks, workspaces, gateways, and models; part 2 covers it. Above AX sits the harness that runs the agent loop (the Substrate README names Claude Code, Codex, and Antigravity as workloads it can host), and above the harness, whatever framework authored the agent, such as ADK or LangChain. MCP, A2A, and AP2 do not live in one layer; they are how any layer reaches tools, other agents, and payments.
One naming note, since it comes up in every thread about this stack: "AX" is also the abbreviation Netlify co-founder Mathias Biilmann uses for agent experience, a design discipline in the spirit of UX and DX. That usage is unrelated to Google. In this series AX always means the Agent Executor runtime.
The Substrate README is explicit about scope: "It is not an SDK for building agents, but rather a system for running them at scale." It does not know what a model is, and it does not care whether the workload is an agent at all. It cares about one property of the workload.
The property it is built around: agents mostly wait
The architecture doc states the premise in one line: "Agents and 'agent-like' workloads are generally very bursty, spending most of their time waiting for input or events." They also run untrusted code, so each one gets its own sandbox, and there are a great many of them. Put those together and you get the usual failure: thousands of single-tenant pods, each holding CPU and memory for an agent that is idle almost all the time.
Substrate's answer is to separate two things a pod normally fuses. An actor is one instance of a workload, identified by an atespace and a name. A worker is a pre-started pod waiting to receive an actor's state. A worker hosts at most one actor at a time, and the control plane maps a large population of mostly suspended actors onto a small pool of workers. The README demo shows about 250 stateful actors juggled across 8 pods, which the project describes as "30x+ oversubscription."
Two sets of numbers circulate for this, and they measure different things. The architecture doc lists north star targets: 100 ms activation latency at the 95th percentile, 1 billion actors per cluster, and 1000 wakeup events per second. The README and the GKE post state shipped figures: "sub-500ms resume operations at over 500 suspend/resume activations per second" and "10x higher density than standard container runtimes." The first set is where the project is aiming; the second is what Google claims today. Neither was measured here.
Each square is an actor. Blue ones are running right now; the rest are suspended in object storage and cost no pod. Set how many actors you have, what share of the time each one is busy, and how much headroom you want, and see how many workers the pool needs.
Actor population
Worker pool
Illustrative arithmetic, not a Substrate tool: running at once is population times busy share, and workers are that times one plus headroom, rounded up. Real traffic is bursty and correlated, so size the pool for measured peaks. The 250 actors on 8 pods default mirrors the README demo.
Two kinds of object, two kinds of storage
The worker side is ordinary Kubernetes. A WorkerPool is a custom resource in the ate.dev/v1alpha1 group that the atecontroller turns into a Deployment of worker pods. The counter demo's pool, from the repo:
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
name: counter
namespace: ate-demo-counter
labels:
workload: counter
spec:
replicas: 3
workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor
template:
nodeSelector:
ate.dev/substrate-version: "${SUBSTRATE_VERSION}"
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 250m
memory: 1GiFrom the substrate docs (demos/counter), not run here.
Two details in that file are worth copying. The comments explain that capacity is read from the limits while the Kubernetes scheduler packs nodes by the requests, so the CPU request sits below the limit to keep the pool placeable. And the nodeSelector on ate.dev/substrate-version is not decoration: the README warns that "a node added later hosts no workers until you label it with the installed version."
The actor side is deliberately not Kubernetes. An ActorTemplate is "not a CRD"; it lives in the Substrate control plane under an atespace and is created through the ate API with the kubectl ate plugin. Actors and workers are records in a PostgreSQL-backed store, because, in the architecture doc's words, "The Kubernetes API server is not designed to handle millions of resources," and actor state can change many times a second. The counter demo's template, trimmed to the fields discussed here:
metadata:
atespace: ate-demo-counter
name: counter
workerSelector:
matchLabels:
workload: counter
containers:
- name: counter
image: ko://github.com/agent-substrate/substrate/demos/counter
wakeupProbe:
httpGet:
path: /readyz
port: 80
volumeMounts:
- name: data
mountPath: /home/counter
resources:
limits:
- name: cpu
quantity: "1"
- name: memory
quantity: 512Mi
snapshotConfig:
onPause: SNAPSHOT_CONTENT_SCOPE_FULL
onCommit: SNAPSHOT_CONTENT_SCOPE_FULL
storageLocation: gs://${BUCKET_NAME}/ate-demo-counter/
sandboxConfig:
sandboxClass: SANDBOX_CLASS_GVISOR
configName: gvisor-default
volumes:
- name: data
durableDir: {}From the substrate docs (demos/counter/counter-template.yaml.tmpl), trimmed, not run here.
The workerSelector matches the pool's metadata labels, which is how a template claims its pool. The template's own resources size the sandbox; the pool's resources size the pod that hosts it.
That split is the practical sizing rule. You scale the worker pool on how many actors are running at the same moment, not on how many exist. A pool sized to the population pays for idle pods, which is the thing Substrate exists to remove; a pool sized to the mean with no headroom turns every burst into waiting. The repo ships an autoscaled WorkerPool demo that drives a HorizontalPodAutoscaler from the ate_workerpool_workers metric, which counts assigned workers, so the pool can follow real concurrency instead of a guess.
One actor's life
The architecture doc gives the lifecycle as a state machine. An actor is created suspended. A resume moves it to running on some worker. From running it can be suspended, which uploads a checkpoint and frees the worker, or paused, which keeps a short-term checkpoint on the node. A paused actor can be resumed, and that resume is pinned to the node that holds its snapshot, or suspended, which uploads the node-local snapshot and ends the pin.
[*] --> SUSPENDED : CreateActor
SUSPENDED --> RESUMING : ResumeActor
RESUMING --> RUNNING : restore / boot complete
RUNNING --> SUSPENDING : SuspendActor
SUSPENDING --> SUSPENDED : checkpoint complete
RUNNING --> PAUSING : PauseActor
PAUSING --> PAUSED : node-local checkpoint complete
PAUSED --> RESUMING : ResumeActor (pinned to the snapshot's node)
PAUSED --> SUSPENDING : SuspendActor (uploads the node-local snapshot)
SUSPENDED --> [*] : DeleteActorFrom docs/architecture.md, verbatim.
Two exits are easy to miss. RevertActor takes a running, paused, or crashed actor back to suspended, discarding any local pause checkpoint and keeping the last completed snapshot. And eviction has a clock: per the API guide, when an actor's worker pod is evicted, the actor gets SIGTERM and 30 minutes to be suspended, after which it is killed and marked crashed, losing everything since its last snapshot.
Drive one actor, my-counter-1, through the documented transitions. Watch where its state lives: on a worker, on a node's disk, or in object storage, and which worker it lands on next.
Worker pool
Node disk
Object storage
Transitions follow the state machine in docs/architecture.md. The docs state that a resume after a pause is pinned to the snapshot's node; they do not say whether the worker is released during the pause, so the figure marks that worker as not documented. Which free worker a suspended actor lands on is the scheduler's choice; the figure rotates through free workers to show it can move.
The resume path is where the design earns its keep. From the counter demo, you create an actor from its template and reach it through the router with one header:
# create the actor from the template in its atespace
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter
# expose the router locally
kubectl port-forward -n ate-system svc/atenet-router 8000:80
# any request with this header resumes the actor if it is suspended
curl -X POST \
-H "ate-target-actor: ate-demo-counter/my-counter-1" \
http://localhost:8000
# inspect, hibernate, and remove it
kubectl ate get actor my-counter-1 -a ate-demo-counter
kubectl ate suspend actor my-counter-1 -a ate-demo-counter
kubectl ate delete actor my-counter-1 -a ate-demo-counterFrom the substrate docs (demos/counter/README.md), not run here.
Behind that curl, atenet-router, which is Envoy with an ext_proc external processor, reads ate-target-actor, asks ate-api-server to resume the actor, and waits. The control plane claims a free worker from the actor's pool, the node agent atelet tells ateom inside the worker pod to restore the snapshot, and the router opens an authenticated tunnel to the worker's atunnel listener on port 443 and forwards the original request. The client sees one slow request instead of an error.
Note what does not happen: nothing in Substrate suspends an idle actor on its own. The architecture doc says that when the user or a higher-level system is done with an actor, "it can request the Agent Substrate to suspend the actor." Idle detection belongs to the layer above, which is one of the jobs AX takes on in part 2.
When the pool is momentarily full, the router does not fail fast. Its request parking feature holds the request and retries the resume with backoff until a worker frees up or the park budget runs out. The budget is set by --parked-request-budget and defaults to 5 seconds; after it, the client gets a 503 reading "no free workers available." A resume already in flight when the budget expires is not canceled, so a slow restore is served late rather than failed.
What a snapshot holds
Resume speed and correctness both come down to what the snapshot captured, and Substrate makes that a per-template choice with two scopes. A Full snapshot holds process memory, the root filesystem delta on top of the image, and any DurableDir volumes: everything needed to come back hot. A Data snapshot holds only the DurableDir contents and discards memory and the rest of the filesystem, which is cheaper to write and store.
Each scope is set per trigger. onPause picks what a pause captures on the node, onCommit picks what a suspend uploads, and onCommit must be a subset of onPause. Every template also gets a Golden Snapshot, a full capture taken once from a temporary boot when the template is created, which is what a new actor first resumes from. After that, each actor resumes from its own last snapshot.
A Data snapshot needs a boot source for everything it dropped. onResume.fromData picks it: ColdBoot, the default, starts the containers fresh with the volume contents restored; Golden restores the golden snapshot's warm memory and serves the actor's own data on top, and is currently micro-VM only. The counter demo uses Full for both triggers so that an in-memory counter survives a suspend, and its README spells out the alternative: with Data scope, "only the durable-volume counter would survive."
Three rules fall out for anyone writing a workload:
- If a value must survive a Data-scope suspend, write it under a
DurableDirmount, never only in memory. - Do not read an actor's identity from an environment variable. The API guide projects identity fields as files on a per-actor mount "precisely so they carry the correct values after a resume from a shared snapshot"; an environment variable baked into the golden snapshot would be frozen for every actor restored from it.
- Give every container a
wakeupProbe.ResumeActorreturns only after each probed container answers 200, and when every container declares one, the template controller skips its default wait of about 20 seconds before taking the golden snapshot.
How many durable volumes you get depends on the isolation class. A micro-VM template may declare several, since they are subdirectories of one shared virtio-fs mount; a gVisor template is limited to one until gVisor accepts more than a single durable mount.
Two isolation classes
Each WorkerPool picks a sandbox class, and each class has its own ateom image that drives checkpoint and restore inside the worker pod.
- gVisor (
ateom-gvisor, the default) runs the workload underrunscand uses gVisor's own checkpoint and restore of the sandboxed process tree. The architecture doc notes it "currently requires arunscversion with the--allow-connected-on-saveflag to work around a bug in networking resumption during checkpointing." - Micro-VM (
ateom-microvm) runs the workload in a Kata Containers guest on the Cloud Hypervisor VMM, captures a memory-only VM snapshot, and restores it on demand withuserfaultfddemand paging, so pages load as the guest touches them.
The sandbox binaries are not baked into worker images. They come from a cluster-scoped SandboxConfig, which the template names through sandboxConfig.configName, as in the gvisor-default reference above. The choice between classes is a trade you can read off the docs: gVisor is the default and is what the demos use, and micro-VMs get multiple durable volumes and the Golden resume source today. Micro-VMs also need /dev/kvm on the node; the repo's local guide covers Linux hosts with KVM and Apple Silicon Macs through Lima.
The parts, and what each one owns
| Component | Runs as | Owns |
|---|---|---|
ate-api-server | control plane service | actor lifecycle, scheduling onto workers, snapshot coordination; state in PostgreSQL |
atecontroller | Kubernetes controller | reconciles CRDs, for example a WorkerPool into a Deployment |
atelet | node DaemonSet | pulls images, drives sandbox lifecycle through ateom, streams snapshots to and from GCS or S3 |
ateom | inside each worker pod | runs, checkpoints, and restores the workload with runsc or Kata and Cloud Hypervisor; hosts atunnel |
atenet-router | Envoy with ext_proc | ingress: resolves the target actor, triggers resume, parks requests |
kubectl-ate | kubectl plugin | create, get, suspend, and delete actors and templates |
Summarized from docs/architecture.md and docs/glossary.md.
The split mirrors the premise. Kubernetes handles the slow, low-frequency work of creating and scaling pods, which it does well. Substrate handles the high-frequency work of deciding which actor occupies which pod right now, in a store built for that write rate. Outbound traffic goes through atunnel and an egress policy point that trusts nothing the actor claims; part 3 is about that contract and about what is not built yet.
A worked example: sizing a pool for coding agents
Take a team running 400 coding agents that each wake for model calls and tool runs. Suppose you measure, from your own traces, that a typical agent is doing work 5% of the time and that at the busiest minute of the day the share rises to 9%. Those numbers are illustrative; measure your own.
- Mean concurrency is 400 times 0.05, or 20 running actors. Peak concurrency is 400 times 0.09, or 36.
- Sizing to the mean with 25% headroom gives 25 workers, which covers ordinary load but leaves peak bursts parked.
- Sizing to the peak with 10% headroom gives 40 workers, against 400 always-on pods: ten times fewer.
- Each worker pod in the demo pool requests 1 GiB of memory with a 1 GiB limit, so 40 workers hold 40 GiB where 400 pods would hold 400 GiB.
What to check before trusting the plan: that each template's memory limit fits inside the worker's limit, since the pool advertises its limits as the per-actor ceiling; that the snapshot bucket is in the same region as the cluster, since every resume after a suspend streams the snapshot back from it; that every new node carries the ate.dev/substrate-version label; and that your clients tolerate a request that may wait up to the park budget before it is served.
When it goes wrong
| Symptom | Likely cause | Fix from the docs |
|---|---|---|
| Requests return 503 "no free workers available" | pool smaller than peak concurrency for longer than the park budget | raise replicas, autoscale on ate_workerpool_workers, or raise --parked-request-budget |
| New nodes never host workers | node lacks the version label | kubectl label node <node> ate.dev/substrate-version=<build version>, or put the label on the node pool |
| In-memory state is gone after resume | Data-scope commit, so memory cold-boots | use Full scope on commit, or keep the state in a DurableDir |
| Every actor reports the same identity | identity read from an environment variable frozen in the golden snapshot | read the per-actor metadata files instead |
| Actor marked crashed after a node drain | eviction took longer than the 30 minute suspend window | suspend actors before draining, and treat the last snapshot as the recovery point |
| Resume works for memory but connections break | gVisor networking resume bug | use a runsc build with --allow-connected-on-save |
If you adopt it, in this order
- Measure how much of the time your agents are actually busy, from traces, before touching the cluster. The whole case rests on that number.
- Decide what state must survive a suspend, and pick Full or Data scope per trigger to match.
- Move anything that must survive a Data-scope suspend onto a
DurableDir, and read identity from the projected files. - Pick the isolation class: gVisor unless you need several durable volumes, the Golden resume source, or VM-level isolation.
- Add a
wakeupProbeto every container so resumes block until the workload is ready. - Size the pool for measured peak concurrency, then put an autoscaler on assigned workers.
- Decide who suspends idle actors, because Substrate will not do it for you. That is where AX comes in.
Substrate is a narrow system. It does not build agents, pick models, or decide when an agent is idle. It makes one bet, that agents spend most of their lives waiting, and turns that waiting into pods you do not have to pay for. The next two parts read what is built on top of that bet and what the bet leaves open.
Keep reading