Agent Substrate · 2 of 3 · Rohit Ghumare · September 24, 2026

Google AX (Agent Executor): Task, Workspace, Gateway, Model

Google's AX turns running an agent on Kubernetes into four declarative objects you apply with one command: a Task that runs in a sandbox, a Workspace prepared before it starts, a Gateway that fences its network, and a Model that holds the provider config. The part that changes how you write the agent is what happens on suspend: the container is stopped, /workspace is saved, and on resume the agent comes back as a fresh process over the same files. Anything it kept only in memory is gone, so an AX agent has to keep its working state on disk.

AX lives at github.com/google/ax under Apache 2.0, with a landing page at agentexecutor.io. Google introduced it as Agent Executor in a Google Cloud post in May 2026, which also announced Agent Substrate as "a new open-source project also announced today." On September 20 a single commit rebuilt it, "moving away from a single CLI with an embedded Python harness toward a general-purpose orchestration layer," and the relaunch reached 662 points on Hacker News. The README still warns that major breaking changes are likely before a stable release, and the API group is ax.io/v1alpha1. One naming note: "AX" is also used for "agent experience," a design term coined by Mathias Biilmann. That is unrelated; this essay is about Agent Executor.

This is part 2 of a three-part series. Part 1 covers the runtime underneath, Agent Substrate: actors, warm workers, snapshots, and isolation. Part 3 covers the egress contract and what nobody has shipped yet. Every manifest and command below comes from the AX repository's docs and examples, read on September 24, 2026, and none of it was run for this essay.

Four binaries around one Redis

The rebuilt AX is three server-side binaries and a CLI, and the design doc explains the first decision in one sentence: "Storing millions of short-lived tasks as Kubernetes CRDs pushes etcd past its comfort zone (single-digit GB storage limits, write-rate bottlenecks, control plane degradation)." So AX keeps its own state in Redis instead of the Kubernetes API.

• ax is the CLI. It is deliberately shaped like kubectl: apply, get, describe, watch, delete, plus suspend, resume, and ssh. It follows your active kube context and tunnels to the control plane in the background.
• ax-server is a stateless gRPC API on port 8080 (the ax.v1alpha1.AX service). It validates a manifest, writes it to Redis as a hash, and publishes an event.
• ax-controller is a pool of reconcilers that read the event stream with XREADGROUP. They provision atespaces and actors on Agent Substrate, apply egress policy, and drive each task toward its desired state. You scale it by adding replicas.
• ax-task-runner is PID 1 inside every task container. It prepares the workspace, serves a metadata endpoint, and starts and supervises your agent command.

Installing it is two steps, and the second one assumes a lot: a Kubernetes cluster, ko, a registry the cluster can pull from, and a reachable Agent Substrate control API. That last one is why AX is not a laptop tool yet. Substrate on GKE is open to all GKE customers for non-production workloads, with production support by allowlist.

go install github.com/google/ax/cmd/ax@latest

# needs ko, a registry, and a reachable Agent Substrate control API
# (in-cluster default: api.ate-system.svc.cluster.local:443)
make deploy AX_IMAGE_REPO=<your-registry>

From the google/ax README, not run here. make deploy applies Redis, then builds and applies ax-controller and ax-server with ko, all into the ax-system namespace.

Task: the unit that suspends

The concepts doc calls a Task "the smallest unit of isolated execution," and it means small on purpose: "An agent is not one process that runs to completion." A task is one sandbox with an image, a command, resource limits, environment variables, a gateway reference, and one or more workspace bindings. An agent that fans work out creates more tasks; AX does not model the tree for you.

apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: task123
  atespace: default
spec:
  image: "ghcr.io/my-org/my-agent-image"
  command: ["python", "agent.py"]
  env:
    - name: ENVIRONMENT
      value: "production"
  resources:
    requests: { cpu: "500m", memory: "1Gi" }
    limits:   { cpu: "2",    memory: "4Gi" }
  workspaces:
    - name: default-workspace
      path: "/workspace"
      goal: "Install dependencies and run the test suite"
  gateway:
    name: default-gateway
  debug: true   # serve guest services so `ax ssh` works; off by default

From docs/manifests.md, with the resources block folded onto single lines. Not run here.

A task reports a one-word phase (Running, Suspended, Failed, Terminating) and three conditions. The one to wait on is Ready.

The lifecycle commands mirror kubectl, with two agent verbs. ax ssh only works on a task that set spec.debug: true, because the guest services behind it allow arbitrary process execution and file access inside the sandbox.

ax apply -f examples/task.yaml       # Task + Workspace + Gateway + Model in one file
ax get tasks
ax watch task task123                # stream phase and condition changes
ax ssh task123 -- ls -la /workspace  # needs spec.debug: true
ax suspend task task123              # checkpoint and pause
ax resume task task123
ax delete task task123               # blocks until the sandbox is torn down

From the google/ax README, not run here.

What survives a suspend

The runner contract is precise about this, and it is the most important paragraph in the repo for anyone writing an agent: "The /workspace volume is what survives suspend and resume. Agent Substrate snapshots it when a task is suspended and restores it into a fresh container when the task is resumed, so the runner will see the same files but a new process tree." Suspend sends SIGTERM to the command's process group, waits ten seconds, and kills whatever is left. The demo script says the same thing after it suspends: the workspace has been checkpointed and the sandbox is gone.

In Substrate's vocabulary (see part 1) that behaves like a Data-scope snapshot with the default ColdBoot resume: a durable directory comes back, process memory does not. Substrate can also take Full snapshots that keep process memory, but AX's runner documents only the workspace. So write the agent as if every suspend were a crash with ten seconds of warning. The pattern below is ours, built from that contract; it is not code from the AX repo.

import json, os, signal, sys

STATE = "/workspace/.agent/state.json"   # inside the volume that survives

def load():
    try:
        with open(STATE) as f:
            return json.load(f)
    except FileNotFoundError:
        return {"step": 0, "plan": None, "done": []}

def save(state):
    os.makedirs(os.path.dirname(STATE), exist_ok=True)
    tmp = STATE + ".tmp"
    with open(tmp, "w") as f:
        json.dump(state, f)
        f.flush(); os.fsync(f.fileno())
    os.replace(tmp, STATE)               # atomic swap, never a half-written file

state = load()                           # a resume starts here, not where it stopped

def on_term(signum, frame):
    save(state)                          # well inside the 10 second grace period
    sys.exit(0)

signal.signal(signal.SIGTERM, on_term)

A pattern built from the documented runner contract (SIGTERM, ten second grace, /workspace survives), not from the AX repository. Save after every completed step too, so a hard kill loses at most one step.

The runner also serves the task's own config back to it on port 80, so an agent can read its spec without an SDK. /readyz returns 503 until the workspace is prepared and 200 after, which is exactly what the controller polls.

# from inside a task
curl -s "$AX_METADATA_URL/metadata/v1alpha1/ax/task"
curl -s "$AX_METADATA_URL/metadata/v1alpha1/ax/workspaces"

From docs/sandbox.md, not run here.

Workspace: setup done once

A Workspace declares everything an agent needs before its first useful action: git repositories to clone, MCP servers and MCP registries it may call, and skill registries with the path to materialize skills into. Declare it once and bind it from as many tasks as you like.

apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
  name: default-workspace
  atespace: default
spec:
  git:
    - name: origin
      repo: "https://github.com/chalk/chalk.git"
      branch: "main"
  mcp:
    registries:
      - provider: google
        query: "mcp.tags:build"
    servers:
      - name: git-tools
        endpoint: "http://git-mcp.default.svc.cluster.local:8080"
  skills:
    registries:
      - provider: google
        query: "skills.tags:nodejs"
    path: "/.agents/skills"

From docs/manifests.md and examples/task.yaml, not run here.

A binding can carry a goal, a plain-language description of the environment the task needs. On first boot the runner hands that goal to Google's Antigravity agent, which installs toolchains or dependencies until the environment matches. That agent needs GEMINI_API_KEY inside the container and gets ten minutes by default; AX_BOOTSTRAP_TIMEOUT takes a Go duration to change it. The task stays not ready until every workspace, including that agent run, has finished.

Setup must happen once, and the runner contract says why: "Resume restarts the container, and re-cloning into a restored workspace would destroy the agent's state." The default runner writes a marker file under /ax for each workspace path and skips setup on later boots. Several workspaces can be bound to one task, each at its own path, and the first one is the command's working directory:

spec:
  workspaces:
    - name: my-service          # mounted at /workspace/my-service, the working directory
      goal: "Install dependencies and run the test suite"
    - name: team-tools
      path: "/workspace/tools"  # explicit mount path

From docs/manifests.md, not run here. The manifests doc links an examples/multi-workspace.yaml that does not exist in the repo as of September 24.

Gateway: the network edge, declared

A Gateway declares the listeners a task exposes and an egress allowlist of hosts and ports it may reach. Tasks get no Kubernetes Service or Ingress of their own. Every request goes through Agent Substrate's atenet-router, which reads one header, ate-target-actor: <atespace>/<task>, resumes the task first if it was suspended, and proxies the request to whichever worker it lands on.

The example gateway in the docs allows every host on port 443, with a comment to tighten it in production. Here is the same schema with an explicit list. The hosts are illustrative values, not from the repo; the field names are the documented ones.

apiVersion: ax.io/v1alpha1
kind: Gateway
metadata:
  name: locked-gateway
  atespace: default
spec:
  listeners:
    - name: http
      port: 8080
      protocol: HTTP
  egress:
    allowlist:
      hosts:
        - host: "generativelanguage.googleapis.com"
          port: 443
        - host: "github.com"
          port: 443

Field names from docs/manifests.md; host values are ours. Not run here.

Reaching a task from your machine is a port-forward plus the header. Note that a request to a suspended task wakes it, so a health checker pointed at a task keeps it resumed.

kubectl -n ate-system port-forward svc/atenet-router 8001:80
curl -H "ate-target-actor: default/task123" http://localhost:8001/readyz

From docs/networking.md, not run here.

Two cautions from the roadmap. Full continuous reconciliation of Gateway specs is listed as work to do, so an edited allowlist is not promised to reach tasks that already reference the gateway; recreate those tasks after a change. And the egress enforcement underneath is Substrate's, which part 3 reads in detail.

Model: configuration, not a model

The docs put it bluntly: "A Model is not a model." It is a named configuration: provider, model identifier, generation parameters, and a reference to the Kubernetes secret that holds the key. Declaring it once means rotating a key or pinning a new model version is one ax apply. AX's own components read it too, for example when planning a workspace from a goal.

kubectl create secret generic anthropic-api-secret \
  --from-literal=ANTHROPIC_API_KEY="sk-ant-..."

apiVersion: ax.io/v1alpha1
kind: Model
metadata:
  name: claude-model
  atespace: default
spec:
  provider: anthropic
  model: claude-opus-5
  secretKey:
    name: anthropic-api-secret
    key: ANTHROPIC_API_KEY
  parameters:
    maxTokens: 16000
    temperature: 0.9

From docs/manifests.md, not run here. The Google variant sets provider: google, model: gemini-3.8-flash, and a secret holding GEMINI_API_KEY.

Gemini is the default in the examples and Anthropic is documented as the second provider. The runner contract adds one detail worth knowing: the task container gets GEMINI_API_KEY in its environment when the atespace has a Gemini credential configured, because the goal bootstrap needs it. If your agent code should not see that key, do not use goals in that atespace.

What each primitive becomes underneath

AX adds names; Agent Substrate does the work. The controller creates one Substrate actor per task, named after the task. It provisions a dedicated actor template for each distinct image and environment, so different runners can share an atespace. The workspace lives in the durable directory Substrate snapshots, and the gateway turns into Substrate egress policy plus routes through atenet-router. Model has no counterpart below: Substrate knows nothing about language models, and the config stays in AX's Redis and a Kubernetes secret.

That leaves a seam. AX records a task's phase in Redis. Substrate records the actor's status in its own PostgreSQL store, which its glossary chose because these records "change too frequently for etcd." Nothing shares a database, and the two agree only as long as ax-controller keeps reconciling. The figure below traces each operation through the stores it writes.

When suspend pays

Suspend exists because agents mostly wait: on a model, on a tool, on a person. A sandbox held open during those waits costs a warm worker for nothing. Today you call suspend yourself, through ax suspend or the SuspendTask RPC, from whatever orchestrates your agents. Automatic suspension on idleness is on the roadmap, not in the release.

Suspending is not free either. The next resume boots a fresh container, the runner starts again, and your agent reloads its state from /workspace. AX does not publish a cold-resume time for that path. Substrate's GKE post claims "sub-500ms resume operations" for its own actors, but AX's documented behavior is a restart over restored files, so the figure takes the resume cost as your input.

Run the defaults and the policy that matters shows up at once. Holding the worker for everything keeps it for the whole 2 hours 24 minutes. Suspending only for the human review cuts held time to about 24 minutes, and adding the 12 minute CI run cuts it to about 13. Suspending on every thirty-second model call trims held time by under three minutes more and puts a resume, with its latency, in front of ten calls. A wait is worth suspending when it is clearly longer than the resume cost plus the time your agent needs to reload its state from disk.

A worked example: a fix that waits for review

Say a team runs a bug-fix agent per ticket. The ticket repo is a Workspace with a goal of "Install dependencies and run the test suite." The task uses a locked Gateway that reaches only the model provider and the git host, and the Model points at the team's key. The orchestrator applies one multi-document file per ticket and watches for Ready:

1. ax apply -f ticket-4812.yaml writes three objects to Redis. The controller creates an actor named ticket-4812, the runner clones the repo, and Antigravity installs dependencies within its ten minute budget. WorkspaceReady turns true.
2. The agent plans, edits, and calls the model eight times. It writes its plan and completed steps to /workspace/.agent/state.json after each one.
3. CI runs for twelve minutes. The orchestrator suspends the task when CI starts and resumes it from the CI webhook, because a twelve minute wait is far longer than a ten second resume.
4. The agent opens a pull request and waits for a human. The orchestrator calls ax suspend task ticket-4812. The runner gets SIGTERM, the agent saves state, and the workspace is snapshotted to object storage. The worker returns to the pool.
5. Two hours later a reviewer comments. The orchestrator calls ax resume task ticket-4812, or any request through atenet-router with the ticket's header wakes it. The agent starts from state.json, reads the review, and pushes a fix.

That is the "waits of 5 minutes or more" policy in Fig. 2: about 13 minutes of worker time held against about 2 hours 25 minutes of wall time, with two resumes. Without suspend the worker is held for all of it. Check the Ready condition's reason (TaskSuspended) and the state file after each resume in your first week; if the agent ever redoes a finished step, it is keeping state in memory.

When it goes wrong

What is not there yet

The roadmap is candid about the distance to production. On the object model: token and timeout budgets, approval policies, and a Sandbox resource for isolation profiles. On the actor side: migration to Substrate's new Actor API, splitting workspace setup into its own actor with its own least-privilege credentials, idleness detection with automatic suspend, and forking a running task into parallel branches. On the edge: gateway reconciliation, a swappable Google-managed gateway, SPIFFE workload identity for tasks and gateways, governance hooks, and OpenTelemetry traces and agent trajectories collected at the runner and gateway. Below all of it, Substrate itself authenticates callers but does not yet authorize them, which part 3 covers.

Reaction on Hacker News split along the same line, as InfoQ summarized. Platform engineers liked not paying for idle sandboxes; others pointed out that clusters, registries, and ko are real operational weight for something marketed as ergonomic.

An adoption order that follows the objects

1. Start on a non-production cluster with Substrate. GKE offers it for non-production workloads to everyone; production is allowlist-only.
2. Apply a Model and its secret first. Decide per atespace whether the Gemini credential should exist, because the task container sees it when it does.
3. Write a Gateway with named hosts. Never ship the "*" on 443 from the example.
4. Add a Workspace without a goal, then with one. Confirm clones and MCP config by reading /metadata/v1alpha1/ax/workspaces from inside the task.
5. Make the agent crash-only. State in /workspace, saved atomically after each step and on SIGTERM, before you ever call suspend.
6. Suspend from your orchestrator on long waits only. Human review and long CI runs, not individual model calls.
7. Pin versions and turn debug off. CLI version, runner image digest, and spec.debug: false everywhere real code runs.