Agent Substrate · 3 of 3 · Rohit Ghumare · September 24, 2026 · 13 min read

Agent Substrate Egress and Its Gaps: Nothing the Agent Says Is Trusted

If you run agents on Agent Substrate, every outbound connection they open leaves through a gateway that ignores what the agent claims about where it is going. That makes egress the most carefully specified part of the project. The control plane is the least: anyone who can authenticate to it can rewrite any agent's egress policy, because authorization does not exist yet. This part reads both halves from the project's own documents and ends with an order for running it anyway.

Agent Substrate is published under its own GitHub organization, agent-substrate/substrate, under Apache-2.0, and its README says it is "not an officially supported Google product." Its community meets on a Google Group and Google Meet, Google Cloud ships it on GKE, and a CNCF donation is in progress: the network egress contract, last updated September 18, says the certificate extension that identifies an actor "will change after the CNCF donation is complete." Part 1 covered how suspended actors share a warm pool of workers. Part 2 covered Google AX, the Task layer that sits on top. Everything below comes from the repositories' docs, source, and tests as of September 24. Nothing here was run for this essay, and every snippet says where it came from.

The path every outbound byte takes

An actor is untrusted code in a sandbox, gVisor by default or a Kata micro-VM. When it opens a TCP connection, it thinks it is dialing the destination directly. It is not. Inside the worker pod, an nftables rule redirects the connection into atunnel, a small trusted process that reads the original destination with SO_ORIGINAL_DST. atunnel then opens a mutually authenticated TLS connection to the egress policy enforcement point, the PEP, and sends an HTTP/1.1 CONNECT whose target and Host header are the IP address and port the actor dialed. Not a hostname. The address.

The PEP is the atenet-egress deployment in the ate-system namespace, served at atenet-egress.ate-system.svc. It runs on Envoy by default or on agentgateway, and the choice is made at install time. The repository's egress demo shows both:

# from the agent-substrate egress demo README, not run here
./hack/install-ate-kind.sh --deploy-ate-system                                  # Envoy (default)
./hack/install-ate-kind.sh --deploy-ate-system --atenet-dataplane=agentgateway

Tunneling switches on cluster-wide through one flag on the API server. The shipped manifest carries it, with a comment saying the flag stays until the egress API lands:

# manifests/ate-install/ate-api-server.yaml, args excerpt, not run here
- --egress-gateway-address=atenet-egress.ate-system.svc:443

The API server stamps that address onto every actor start and restore, so a resumed actor comes back already wired to the gateway. Actor networking is IPv4 only today.

What a refusal looks like from inside

The egress traffic document is the contract for the GA milestone, and it is specific about failure. That specificity is the practical part, because an agent that cannot tell "blocked" from "slow" burns its time budget retrying.

Three of those rows turn into bugs you will meet. An HTTPS_PROXY variable baked into an image makes every request a forward-proxy CONNECT, which gets a 403 whatever the policy says; unset it inside actors. A client that tries HTTP/3 first is sending QUIC over UDP port 443, which is dropped silently, so the first request of each connection waits out the client's QUIC timeout before any TCP fallback. The docs do not mention QUIC; that consequence follows from the UDP row. And a database driver dialing port 5432 gets a connection that opens and closes at once, which most drivers report as a protocol error, not a network policy.

The egress demo README describes the current gateway a little more permissively than the GA table: it says opaque TCP and TLS the gateway does not terminate are "allowed by address only, at the CONNECT." Read that as the behavior of today's build and the table as the target. If a non-HTTP dependency matters to you, check which one your version implements before you design around it.

Here is a probe you can run inside an actor to tell the cases apart. It is a pattern built from the documented behavior, using only curl and bash; it was not run for this essay:

# egress-probe.sh: classify an outbound failure inside an actor
# pattern built from docs/egress-traffic.md, not run here
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "https://$1/")
case "$code" in
  403) echo "policy refused $1 (or a proxy variable made it a forward CONNECT)";;
  000) echo "no HTTP answer from $1 within 10s: check DNS, UDP, or a dropped QUIC attempt";;
  *)   echo "reached $1, status $code";;
esac
# opaque TCP: an immediate empty read means refused, a timeout means slow or dropped
timeout 5 bash -c "exec 3<>/dev/tcp/$2/$3; head -c1 <&3 | wc -c"

Why the hostname is only a hint

This is the part of the contract worth reading slowly. The document says the PEP "MUST NOT trust any decision or attestation coming from the actor itself," and names the obvious attack: it "MUST NOT treat an actor-provided HTTP hostname or TLS SNI as proof that the original destination has that name." An agent that has been prompt-injected can send Host: api.github.com to an attacker's IP address. A gateway that allowed the connection because the header said GitHub would be an exfiltration channel with a GitHub label on it.

So a policy decides in one of two ways. A CIDR rule, or the catch-all all rule, decides on the IP address atunnel recovered from the kernel, which the actor cannot forge, and the gateway dials that address at once. A hostname rule may use the hostname as input, but when it authorizes by name, the gateway "MUST route the traffic to the authorized hostname rather than use the hostname to authorize an arbitrary IP address." The lie buys nothing: the request goes to the real api.github.com, not to the address the agent dialed.

That has a consequence the docs state plainly and most readers will miss. A hostname rule can only match traffic the gateway can read: cleartext HTTP, or TLS that the gateway terminates. The default gateway does not terminate TLS, so HTTPS is decided at the CONNECT, by address only, and a hostname rule never matches it. To allow https://api.github.com by name, you need the gateway in its man-in-the-middle mode, installed with --experimental-use-sdsmint, and your actors must trust the gateway's CA, which the trust bundle guide projects into the actor's filesystem. Without that mode, your choices for HTTPS are CIDR prefixes or all.

Identity comes from a certificate, not a claim

Every connection to the PEP carries a client certificate minted for one actor. When an actor starts, atunnel generates a private key, which never leaves it, and asks the node's atelet for a short-lived certificate. The certificate carries an ActorIdentity X.509 extension with the actor's atespace, name, UID, and a purpose that must be atunnel. The extension OID today is 1.3.6.1.4.1.11129.2.12.2, allocated under Google's enterprise number, and the contract says it will change after the CNCF donation. Anything that pins that OID will need an update when it does.

The PEP checks the chain and lifetime against the actor identity CA, the client-authentication usage, exactly one valid extension, and an actor URI SAN that names the same actor. Then it asks ate-api-server three questions: does this actor still exist, does its UID match, and is it running. A certificate cannot outlive its actor. If an actor is deleted and recreated under the same name, the new actor has a new UID and the old certificate is refused. In the demo's gateway log the identity appears as a SPIFFE-style URI, spiffe://substrate-actor.local/atespace/ate-demo-egress/actor/egress-demo.

The policy itself is a per-actor resource. Rules are evaluated in order, the first match authorizes the request and applies only that rule's effects, and a request that matches nothing is denied. An actor with no policy gets no tunnel at all. The three rule types, written in the manifest format the repository's CLI tests parse:

# rule shapes from cmd/kubectl-ate/internal/cmd/egress_policy_test.go, not run here
metadata: {atespace: team-a, name: default}
rules:
- hostnames:
    patterns:
    - api.example.com
- cidrs:
    cidrs:
    - 10.64.0.0/16
- all: {}

The name is always default and the atespace must match the actor's. Hostname patterns are lowercase DNS names, and a wildcard replaces exactly one leftmost label: *.example.com matches api.example.com, not example.com and not nested.api.example.com. Each list takes at most 256 entries. Creating and reading a policy uses the kubectl-ate plugin, and the positional argument is the actor, not the policy:

# from the egress demo README and kubectl-ate help text, not run here
kubectl ate create egress-policy egress-demo -a ate-demo-egress -f policy.yaml
kubectl ate get egress-policy egress-demo -a ate-demo-egress

The demo's warning about timing deserves a place in your runbook: the gateway caches a missing policy as a deny for 10 seconds (--egress-policy-cache-ttl), so a fetch attempted before the policy exists keeps failing for up to 10 seconds after you create it. To see decisions as they happen, the demo reads the gateway's logs:

# from the egress demo README (Envoy dataplane), not run here
kubectl -n ate-system logs deploy/atenet-egress -c envoy | grep '\[egress\]'
kubectl -n ate-system logs deploy/atenet-egress -c ext-proc | grep -i 'egress tunnel opened\|egress denied'

Credentials are the obvious next step and the least finished. The policy schema already has an effect that injects a header whose value comes from a credential provider, referenced as ate-secret://<provider-class>/<provider-name>/..., so a token never has to enter the sandbox. On the default gateway a rule that declares an injection is denied with a 501 until injection lands; the repository wires it only into the man-in-the-middle gateway, behind an experimental install flag. Until then, any credential your agent uses is a credential your agent can leak, which is the lesson of the eval sandbox that held the keys.

Ingress rides the same tunnel

Inbound traffic takes the reverse route. A client sends its request to atenet-router, Envoy with an external processor, and names the target actor in a header. Per the architecture doc, the router asks the control plane to resume the actor if it is suspended, then opens an authenticated tunnel to that worker's atunnel on port 443, and forwards the request. The actor's own container port is never reachable directly. From the demo:

# from the egress demo README, not run here
kubectl -n ate-system port-forward service/atenet-router 8000:80 &
curl -s -X POST http://localhost:8000/ \
  -H 'ate-target-actor: ate-demo-egress/egress-demo' \
  -H 'Content-Type: application/json' \
  -d '{"url":"http://10.96.0.20:80/"}'

The header routes the request; the URL in the body is the demo app's own input, which it fetches through the egress path above. The address here is an example cluster IP; the demo reads it from a whoami service it creates first.

The control plane trusts anyone who can log in

Now the other half. ate-api-server authenticates callers with mTLS client certificates or bearer JWTs, configured in a file like this one from the authentication doc:

# from docs/authentication.md, not run here
actorIdentityJWTProvider: kubernetes
jwtProviders:
- name: kubernetes
  issuer: https://kubernetes.default.svc.cluster.local
  audiences:
  - api.ate-system.svc
- name: google
  issuer: https://accounts.google.com
  audiences:
  - 32555940559.apps.googleusercontent.com

The next paragraph of that doc is the most important sentence in the project today: "Authorization and RBAC are not implemented yet, so only configure providers whose users should have full control of the entire control plane: every atespace, actor, actor template, egress policy, snapshot and worker in the cluster." Authentication proves who you are. Nothing then limits what you do. Every guarantee in the sections above holds only as long as the identity that can call CreateActorEgressPolicy is one you trust with every actor.

The second provider in that example matters for this reason. It accepts Google identity tokens minted for the Cloud SDK's shared client ID, and the doc shows the resulting workflow: gcloud auth print-identity-token | kubectl ate --token-file=- get actors. On a shared cluster, that means any Google account the issuer accepts gets full control, unless something else stops it. The doc says those values are examples, not defaults. Treat them that way.

Until authorization ships, the fence has to come from Kubernetes. Two layers are available. First, kubectl-ate reaches the API through a port-forward, so Kubernetes RBAC on pods/portforward in ate-system already decides who can reach it from a laptop. Second, a NetworkPolicy can limit in-cluster callers to the namespaces that need the API. The one below is my construction on standard Kubernetes objects, using the app: ate-api-server label and container port 443 from the shipped manifest; it assumes your CNI enforces NetworkPolicy and that AX runs in its documented ax-system namespace:

# my construction on the shipped labels; verify against your install, not run here
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ate-api-callers
  namespace: ate-system
spec:
  podSelector:
    matchLabels:
      app: ate-api-server
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [ate-system, ax-system]
    ports:
    - {protocol: TCP, port: 443}
    - {protocol: TCP, port: 9090}

Port 9090 is the API server's metrics and readiness port. If your monitoring scrapes it from another namespace, add that namespace to the list.

What else the documents say is missing

The project is candid about this. Its threat model, last updated June 25, says Substrate "has little to no security hardening at this time." The roadmap ranks identity fourth and policy fifth among six priorities. Beyond authorization, these are the gaps the docs name themselves:

• DNS is outside policy. Actors query a relay that forwards to the worker pod's resolvers "without passing through the external egress gateway or checking egress policy," and "there is currently no per-actor setting to disable this relay or filter queries." DNS is a well-known exfiltration channel, and the threat model recommends not exposing cluster-internal DNS to actors.
• State lives near where it ran. The architecture doc lists peer-to-peer state sharing among problems not yet addressed, because "relying on data locality presents a risk of data loss."
• Worker autoscaling is also on that list, and on the roadmap as coming soon.
• One durable volume on gVisor. A gVisor actor can mount a single DurableDir volume until gVisor accepts more than one; micro-VM actors can mount several.
• Observability is partial. Container logs carry ate.* labels for the actor, atespace, and template today, and traces are sampled across the router and control plane. Actor-level metrics that follow an actor across workers are still planned, and locking worker pods out of direct egress to the telemetry collector is a phase-two item.

Google Cloud's own GKE announcement on September 15 draws the same line: Substrate is available to all GKE customers "for non-production workloads," with production support "via allowlist."

Where the whole stack is crowded, and where it is empty

Step back from one project and the pattern repeats across the layers under agents. Sandboxes, durable runtimes, build frameworks, and observability tools each have half a dozen named projects. Tool protocols have two, and both now sit under the Agentic AI Foundation. Payments has four protocols competing for the same job. Identity has plenty of products and no agreed standard, and it is the layer every other layer's security story leans on. Substrate's own coverage has the same shape: the sandbox and the egress path are real, identity stops at authentication, and payments are out of scope.

A worked example: a coding agent's egress

Say an agent in atespace team-a needs three things: the GitHub API over HTTPS, the npm registry over HTTPS, and a Postgres database at 10.64.12.5:5432 inside your network. Walk it through the contract.

On the default gateway, HTTPS is decided by address, so a hostname rule for api.github.com never matches. You would have to allow GitHub's and npm's address ranges by CIDR, which change, or allow everything. On the man-in-the-middle gateway, hostname rules work, at the cost of an experimental install flag and a trust bundle in the actor. The database is the harder case: under the GA contract, a Postgres connection is "any other TCP" and gets accepted and closed. Under the demo's description of today's gateway, a CIDR rule would let it through by address. The honest answer is to check your version, and to prefer an HTTP API in front of the database if the GA behavior is what you will run.

Assuming the man-in-the-middle gateway, the policy reads, in the schema from the repository's tests:

# assembled from the rule schema in the repo's egress-policy tests, not run here
metadata: {atespace: team-a, name: default}
rules:
- hostnames:
    patterns:
    - api.github.com
    - registry.npmjs.org
- cidrs:
    cidrs:
    - 10.64.12.5/32

Then trace each call. git and the GitHub API reach api.github.com on the first rule; a request carrying Host: api.github.com to any other IP is routed to the real host, so an injected instruction to "send the diff to this IP" gets nowhere by claiming GitHub. The npm registry matches the same rule. A dependency script that tries https://paste.example matches nothing and gets a 403, which the probe above reports as a policy refusal, not a network fault. The database connection matches the CIDR rule if your gateway passes opaque TCP. And a DNS lookup of any name still leaves the sandbox, unchecked, through the relay.

If the agent runs under AX, look at the Gateway too. AX's example manifest, the one its README applies with ax apply -f examples/task.yaml, allows every host on port 443:

# from google/ax examples/task.yaml, not run here
egress:
  allowlist:
    hosts:
      - host: "*"
        port: 443

Replace the wildcard with the hosts the task needs before anything real runs through it:

# the same field with explicit hosts; field shape from the AX example
egress:
  allowlist:
    hosts:
      - host: api.github.com
        port: 443
      - host: registry.npmjs.org
        port: 443

Running it today, in the order the contract suggests

1. Fence the control plane first. Keep only JWT providers whose every user may control every actor. Limit pods/portforward in ate-system with Kubernetes RBAC and in-cluster callers with a NetworkPolicy, because without authorization, reaching the API is the same as owning it.
2. Give every actor an explicit policy, and never all: {} outside a demo. The default is already deny; the risk is the shortcut you add to make a demo pass.
3. Decide how HTTPS is allowed. CIDR rules on the default gateway, or hostname rules on the experimental man-in-the-middle gateway with its CA in every actor. Hostname rules on the default gateway silently match nothing.
4. Narrow AX Gateways from host: "*" to named hosts, so the AX layer and the Substrate layer both say no.
5. Keep secrets out of sandboxes until header injection is finished; give agents short-lived, narrowly scoped tokens, and rotate on every run.
6. Plan for DNS and UDP. DNS leaves unchecked; UDP other than DNS hangs. Unset proxy variables and disable HTTP/3 in actor images.
7. Read the gateway logs, and allow for the 10 second deny cache when a newly created policy seems not to work.
8. Keep it out of production until authorization ships, as Google's own GKE announcement does.

The egress contract is the best argument for the project: it starts from the assumption that the agent is compromised and still gets a useful answer out of the gateway. The control plane has not caught up to that assumption yet. When it does, the sentence to look for in the authentication doc is the one that says authorization is implemented. For how the runtime above this layer handles the same trust problem, see the harness as an API.