Agent Substrate 1.0 Joins the CNCF: One Flag Picks the Gateway
On November 5, Agent Substrate released 1.0 and became a CNCF Sandbox project. If you run agents on Kubernetes with it, two things changed for you. Upgrades now follow a written skew policy, and no 0.x cluster can upgrade to 1.0. The release also asks you to pick, at install time, the proxy that enforces the egress policy of every agent. I ran both proxies on a test cluster. They refuse requests in different ways, and they apply a change at very different speeds.
Substrate runs each agent as an actor that lives in a snapshot and borrows a warm worker pod only while it works. Three essays in September described that design: the warm pool, Google AX on top of it, and the egress path and its gaps. The design still holds at 1.0. Many manifests and commands in those essays do not, and the install section below lists them. One disclosure: I am an ambassador for the Agentic AI Foundation, which hosts agentgateway, one of the two proxies.
One actor, coder-1, has an https rule for api.github.com that replaces its Authorization header with a Kubernetes Secret. Move the knob to switch the dataplane, then send, rotate, and remove. The containers, the CA key holder, the 10 s policy cache, and the 300 s credential cache come from the v1.0.0 manifests and the source of both proxies. The 403 bodies are the ones I saw on my cluster. A request follows each change by 2 s, and a repeat send comes 60 s after the request before it. The timing of the animation is illustrative.
1.0 and the move to the CNCF
The first release, v0.1.0, came on September 10. Release v1.0.0 came on November 5. In those eight weeks the project published five minor releases and three release candidates, and most of them broke something. Release v0.5.0 replaced the snapshot content scopes with a fidelity ladder. Release v0.6.0 removed PauseActor and renamed the worker binary from ateom to ateworker.
The move to the CNCF ran in parallel. The repository went public on May 13. On September 8 the maintainers applied to the CNCF Sandbox. An automated pre-review flagged the project as too new, because it was 3.9 months old and the guideline asks for six. The TOC vote opened on September 22 and passed on September 28, with 8 of 11 binding votes in favor. The contribution agreement was signed on October 20, and the onboarding began.
Onboarding shows what a donation changes in practice. Google released the agent-substrate GitHub organization from its enterprise account, and the CNCF invited it on October 26. A second organization, agent-substrate-plugins, now holds cloud integrations such as the GCP Secret Manager credential provider. On October 28 the project made DCO sign-off mandatory and turned off the Google CLA. At v1.0.0 the copyright line reads "The Agent Substrate Authors."
The owner is neutral now, and the maintainers are still mostly one company. The maintainers file at v1.0.0 names 19 people: 16 from Google and 3 from Solo.io. Google's announcement of November 5 counts more than 100 contributors. The same post moved gVisor and OpenRL into the Sandbox and made Substrate generally available on GKE.
At KubeCon North America in Salt Lake City, November 9 to 12, the project explained the donation. The keynote "Agent Substrate: Open Layers for Agentic Systems" ran on November 10, and its recording is on the CNCF channel. The speakers were Jaana Dogan, Dmitry Berkovich, and Tim Hockin of Google, Eitan Yarmush of Solo.io, and Luiz Scheidegger of Anthropic. Hockin and Berkovich gave the longer session "Kubernetes Solutions for Agent-Shaped Problems" that afternoon. Christian Posta's "Pods Are Not Enough for AI Agents, Meet Agent-substrate" followed on November 11. A lightning talk on November 9 showed kagent 1.0 running one Substrate actor per conversation.
The community event Agent Substrate Day, also on November 9, was more open about gaps. Carlos Santana's talk on EKS lists no AWS credential provider yet and no automated idle suspension. Mike Moore's talk on a live visualizer reports 18 agents on two worker pods, with reserved capacity 4.5 times lower than always-on pods. He also argues that CPU and memory are the wrong autoscaling signals, because each worker holds a fixed number of actors. I know those two talks only from their abstracts.
One flag picks the proxy
Every outbound actor connection leaves through the atenet-egress pod, and every inbound request enters through atenet-router. At 1.0, one installer flag sets the proxy in both pods: --atenet-dataplane=envoy, the default, or --atenet-dataplane=agentgateway. The Substrate API does not change with the flag. Actors, policies, and kubectl ate work the same. The flag changes the containers, the code that enforces a policy, the place of the CA key, and the shape of a refusal.
On the Envoy dataplane, the egress pod runs three containers. Envoy terminates the tunnel from the actor and routes each request. Beside it, a Substrate process named ext-proc decides each request. A third container, sdsmint, mints the certificates that Envoy uses to intercept TLS. It is the only container that holds the CA private key. The router pod runs two more containers, so the Envoy dataplane is five containers in all.
On the agentgateway dataplane, each pod runs one container, agentgateway v1.7.0, pinned by digest in the v1.0.0 manifests. The Substrate policy code is built into agentgateway. No sidecar sits in the path, and the CA key lives in the same process that parses actor traffic.
One request through each dataplane
Take one actor, coder-1, with an https rule for api.github.com. The rule replaces the Authorization header with a value from a Kubernetes Secret. The actor sends a placeholder, and the gateway must put the real token in. The machine above follows this request.
On Envoy, the request leaves the worker pod through the actor's tunnel and reaches Envoy in atenet-egress. Envoy passes it to ext-proc as an external processor call. ext-proc verifies the actor certificate and looks up the policy of the actor. It keeps that policy in a cache for 10 s by default, and the atenet flag --egress-policy-cache-ttl sets the time. Then it reads the Secret and replaces the header. With the bundled Secret provider, the next request gets the new value.
On agentgateway, the same request reaches one process. It asks ate-api-server for the current policy on every request, at api.ate-system.svc:443. It checks the host against the rules and applies only the effects of the winning rule. Then it fetches the credential, and it reuses that value for 300 s, per actor and per Secret reference. A provider entry can set cacheTtl to change that time, and the Substrate installer sets none.
So the two caches sit at opposite ends of the request. Envoy caches the policy and reads the credential fresh. agentgateway reads the policy fresh and caches the credential. A removed rule is slow on Envoy, and a rotated Secret is slow on agentgateway.
The two also fail differently. The credential injection guide lists the status codes, and three rows change with the dataplane. A rule that names a provider class that the gateway does not serve fails with 500 on Envoy and 403 on agentgateway. The same split applies when no provider is configured. It also applies when an actor JWT mint fails for a reason other than an outage. A provider that times out returns 503 on both.
What I measured on a test cluster
I ran both dataplanes on one test cluster, on an x86 virtual machine with 16 vCPU and 23 GB of memory. Unless I give a count, each number below comes from one run.
At idle, I took one memory sample of each container soon after start. The Envoy dataplane used about 91 MB in five containers across the router and egress pods. agentgateway used about 38 MB in two. I switched the running cluster from Envoy to agentgateway with the same deploy command and the new flag value. The command took 32 s, and two running actors stayed RUNNING. I sent no requests during the switch, so I cannot say how many would fail. One allowed request took 0.037 s on Envoy and 0.035 s on agentgateway, which says nothing about load.
Next I removed one rule on each dataplane and polled about every 2 s. Envoy served new requests at 1 s and 3 s and refused them from 6 s. agentgateway refused the first sample, under 2 s. A reused HTTP/1.1 or HTTP/2 connection behaved like a new one: the next request on it got 403 after the same delay.
Then I changed one Secret three times on each dataplane, while the actor sent about one request per second. On agentgateway, the upstream saw the new value after 280.5, 200.9, and 100.6 s. In that time 233, 168, and 84 requests still carried the old value. On Envoy, the new value arrived after 1.0 to 1.6 s. No request failed on either dataplane.
Pick a change and how often the agent calls the host. Each lane shows the worst case after the change, where the cache entry is fresh at the moment you change it. Red is time when requests still use the old rule or the old Secret. The marks under a lane are the times I measured on my test cluster.
Cache defaults from the atenet source at v1.0.0 and the agentgateway source at v1.7.0. agentgateway reads the policy on every request, so it has no policy cache to set. Measured times come from one series of changes per dataplane.
Neither dataplane closes a connection that is already open. A response that had started before the removal ran to its end, 39 s later on Envoy and 18 s later on agentgateway. Those are the ends of my test responses, not a limit of the gateway. An open TLS passthrough connection served a second request 20 s after its rule was gone.
To stop an agent at once, remove the rule and then suspend the actor. An asynchronous suspend returned in 0.54 s on Envoy and 0.64 s on agentgateway, and the upstream saw the connection close within 2.2 s. After a resume, the old sockets were dead, so the agent must be able to reconnect.
The installer starts the agentgateway egress container with RUST_LOG=debug. For 100 requests at that level, the gateway wrote 4,380 lines. In 200 of them it printed the value of a custom header that the actor sent. It masked Authorization and printed no injected secret. At info, the same 100 requests wrote 100 lines and no header value. The installer writes debug again on each run, so set the level after every install:
# agentgateway dataplane: lower the egress log level after each ate-setup run
kubectl -n ate-system set env deploy/atenet-egress RUST_LOG=info
# stop an agent now: remove the rule, then suspend the actor
# --mode=sync returns after the snapshot is in the bucket
kubectl ate suspend actor coder-1 -a <atespace> --mode=sync
Refusals looked different too. An actor with no policy got a connection reset on Envoy and a 403 on agentgateway. A request to a host outside the policy got 403 on both. The body was egress denied on Envoy and actor egress policy denied on agentgateway. A tool that sorts refusals has to know both sets of answers.
The last number matters for every agent that calls a model through the egress gateway. Under an https rule, a new connection cost about 0.6 s on both dataplanes, at the median of 10 to 30 requests. Almost all of it was the TLS handshake with the interception endpoint. On a reused connection, the same request took 10 to 12 ms. So keep one HTTP client with a connection pool for the life of the actor.
What the dataplane does not do
kagent ships its own distribution of Substrate. When I ran kagent v1.0.0, both atenet pods ran agentgateway v1.7.0, and the values that its installer passes name no dataplane. The kagent controller writes one egress policy per actor through the Substrate API. So a kagent user gets the agentgateway behavior above, including the 300 s credential cache.
The egress policy cannot choose a model or a tool. At 1.0, a rule allows a host and a port. It picks http, https, or tlsPassthrough, and it can replace headers with a Secret or an actor JWT. agentgateway as a product does more. It routes requests to model providers, applies budgets, and federates MCP servers under one endpoint. The Substrate dataplane configures none of that. To use it, I ran a second agentgateway as a normal Deployment and allowed only that host in the actor's policy:
# the egress rule I ran: the actor reaches only the model gateway,
# and the egress gateway puts a Secret value into X-Lab
rules:
- https:
hostnames: ["modelgw.model-gw.svc.cluster.local"]
effects:
replaceHeaders:
- header: X-Lab
credentialUri: ate-secret://k8s.io/default/cred-ns/gw-client-key/token
The actor sends a placeholder in X-Lab, and the egress gateway replaces it with the token key of the Secret gw-client-key in namespace cred-ns. I used a separate header because the model gateway replaces Authorization with its own provider key. The model gateway then picked the backend from the model name in the request. On both dataplanes, the placeholder from the actor reached no backend.
An actor JWT in place of the Secret would tell the model gateway which actor calls. The sandbox signs its requests covers those tokens. I have not run that form yet. For the model and tool rules on the second gateway, read agentgateway in production.
Installing 1.0, and leaving 0.x
No 0.x release upgrades to 1.0. The upgrade runbook says to install again, and the release notes of v1.0.0-rc.3 say the same about v0.6.0. Before 1.0 the project merged its database migrations into one schema, so a 0.x database does not migrate. Your actors and snapshots stay behind. Install 1.0 on a new cluster, create your templates again, and move traffic. The release notes give the install from the published images:
git clone --depth 1 --branch v1.0.0 https://github.com/agent-substrate/substrate.git
cd substrate
go run ./cmd/ate-setup deploy ate-system \
--image-repo ghcr.io/agent-substrate/substrate --image-tag v1.0.0 \
--storage-bucket-name=<bucket> \
--credential-provider='{"name":"k8s.io"}' \
--ingress-allowed-spiffe-ids=<SPIFFE IDs of your ingress clients> \
--atenet-dataplane=agentgateway # leave this line out for Envoy
Then change what you carried over from the September essays. These forms fail or do nothing at 1.0:
- A WorkerPool uses
apiVersion: ate.dev/v1. The CRD serves only v1, so av1alpha1manifest fails to apply. - An ActorTemplate needs
resources.limits.cpuMillicoresandmemoryBytes.snapshotConfig.preferredFidelityreplaces theonPauseandonCommitscopes. PauseActorandkubectl ate pauseare gone. Use suspend, with--mode=syncwhen you need the snapshot in the bucket before the call returns.- Nothing reads the node label
ate.dev/substrate-versionnow, and the old kind install script is gone.ate-setup deployinstalls everything. - The router requires a client certificate on its allowlist. A plain port-forward with a header does not reach an actor. The command
kubectl ate proxy actordoes that work for you. - Authorization exists now. ate-api-server checks every call against an embedded OpenFGA model, and the default bootstrap owner is the shared
ate-clientServiceAccount. Set your own owners before anyone else uses the cluster.
The last item reverses a warning from the September egress essay, which said that the control plane had no authorization. It has authorization at 1.0. The default owner is still wide, and the authorization guide opens with that warning.
Upgrading within 1.x
From 1.0 on, the project promises compatibility in writing. The contribution guide says that minor releases do not remove features or break compatibility. It names four kinds of compatibility: the API, the CLI, data on disk and in the database, and rollback. It also names what the promise excludes. APIs behind preview gates can change or go away, and so can log lines and human-oriented CLI output. Machine-oriented output, such as kubectl ate get -o json, must stay compatible.
The version skew policy turns that promise into an order. It ranks the components: podcertificate-controller, atelet, the workers, ate-api-server, ate-controller, atenet, and then the clients. A component can be one release newer than any component after it. It must never be newer than a component before it. So during an upgrade, a new atelet drives old workers, and old clients talk to a new API server.
The order follows the call path. The API server reaches the workers only through atelet. A new API server can depend on new worker behavior behind an existing field, so the workers move first. The database is the one shared piece. The first new API server replica migrates the schema, and a rollback does not undo that migration. Until every replica is new, a client that sends a new field can reach an old replica, which rejects it with INVALID_ARGUMENT.
Three rules for operators follow. You can skip patch releases, so 1.0.1 goes straight to 1.0.5. You cannot skip a minor release, so 1.0 goes to 1.1 before 1.2. You can roll back to the release you came from, and no further. Take a database backup before the API server step.
The components in the order of the skew policy. Tap a version on any row to build a mixture, or run the upgrade from 1.0 to 1.1 one runbook step at a time. The check reads the two rules: a component can be one release newer than those below it, and never newer than those above it.
Order, rules, and step list from the version skew policy and the upgrade runbook at v1.0.0. Steps 7 and 8 change no row here. What running actors see comes from the runbook, plus the step 2 failures that I saw in my upgrade runs.
The runbook has a cost that the policy does not show. When the worker image changes, each actor on an old worker gets SIGTERM and then 30 minutes to be suspended. Substrate does not suspend it for you. I ran two upgrades from v1.0.0-rc.3 to v1.0.0 and one rollback, on a second VM with 24 vCPU and one kind node. An actor whose process exits on SIGTERM crashed in 4 of 4 pool rolls. An actor with a SIGTERM handler survived 3 of 3.
While atelet restarted in step 2, suspends and resumes on the node failed for 24 s in one run and 34 s in the other. So I do the suspend earlier than the runbook says. Before step 2, stop new requests and suspend every actor with --mode=sync, the command above. Keep requests stopped until step 3 has replaced the workers, because a request resumes an actor. Then the worker step finds no running actors, and the 30 minutes do not matter.
Where it breaks
An actor crashes each time a pool rolls. Its process exits on SIGTERM, so the 30 minute window never starts. Add a handler, or suspend every actor before the upgrade as above. After the roll, kubectl ate revert actor takes a crashed actor back to its last snapshot.
ate-api-server refuses to start after an upgrade to 1.0. Each segment of --snapshot-location must now match ^[A-Za-z0-9_-]+$, and no segment can be atespaces. Fix the path before you upgrade. During a later upgrade, a client gets INVALID_ARGUMENT. It sent a field from the new release to an old API server replica. Hold new fields until every replica runs the new release.
An agent still reaches a host after you removed its rule. On Envoy, wait for the 10 s policy cache to expire, or set a lower --egress-policy-cache-ttl. On both dataplanes, an open stream or passthrough connection keeps working, so suspend the actor. On agentgateway, an upstream rejects a key after rotation. The gateway sent the cached old value for up to 300 s, so keep the old key valid for five minutes after you rotate.
A model call through an https rule returns 503 on both dataplanes. The upstream uses a certificate from a private CA. The egress gateway verifies upstream certificates against the system roots of its own container, so it does not know your CA. On my cluster, a patch that added the CA to that trust store fixed it. The next installer run drops the patch, so keep it in the tool that applies your manifests.
A refusal looks different after you switch the dataplane. A tool that expects 500 for a provider misconfiguration gets 403 on agentgateway. A connection reset on Envoy becomes a 403 with a body. Match on both, and read the egress logs, which name the reason on each dataplane.
Which proxy I run
I use Envoy where actors run untrusted code under https interception rules. Envoy keeps the CA key away from the process that parses actor traffic, and a rotated Secret reaches the next request. Envoy is also the only dataplane that accepts the experimental extra egress processor. I use agentgateway where memory per cluster matters, where a team already operates agentgateway, or where kagent made the choice.
On agentgateway, I limit who can read Secrets in ate-system and who can open a shell in atenet-egress. I also set its log level to info. With either proxy, I plan rotations and rule removals around the cache that the proxy keeps. My full walkthrough of this cluster, from the install to failure drills, is Agent Substrate 101 on my course site.
Sources: the v1.0.0 release notes, the atenet policy cache at v1.0.0, and the Substrate policy in agentgateway at v1.7.0. The CNCF Sandbox application and onboarding threads give the donation dates.
Keep reading