Four Tiers for One KV Cache

A 256K-token conversation with Qwen3-235B-A22B leaves 50 GB of KV cache behind. One Ironwood host has 768 GiB of HBM across four chips. After the FP8 weights, that is room for about ten caches of that size. An agent host keeps more sessions open than that, so most caches leave HBM between turns. The cost is in where they go and how fast they come back. On November 9 Google announced two TPU systems and a library for that problem. The library, TPU Sync, treats HBM, host DRAM, storage, and the data center network as one tiered memory for KV caches. This essay does the arithmetic per tier, and shows when a tier pays and when it does not.

Google announced TPU 8tx and TPU 8i Max on November 9, 2026, in a post by Amin Vahdat. The same day, Fenghui Zhang and Weiren Yu published the TPU Sync post on the Google Cloud blog. The code is on GitHub under Apache 2.0. The hardware numbers below come from those two posts and from the April TPU 8 deep dive. The link rates come from the TPU Sync post and the TPU7x documentation, because every published TPU Sync measurement ran on seventh-generation Ironwood hosts. The model numbers come from the config.json of each model. Nothing here ran on a TPU.


Machine · One KV cache, four tiers

One prefill host and one decode host, each with four TPU v7x chips. A prompt card enters the HBM tank and leaves eight KV cubes behind. Tier to host DRAM moves them down the PCIe rod. Spill drops them to the storage shelf. Stream sends them over the data center network to the decode host. Reload brings them back into an HBM tank for the next turn. The cache bytes come from the Qwen3-235B-A22B config.json. The prefill time comes from the chip's published FP8 peak at the utilization you set. The NIC rate is the one-NIC rate that TPU Sync measured. The PCIe rate starts at half of the published 223 GB/s, the conservative reading explained below. The storage rate is an assumption. Both are knobs. The verdict compares each reload with a fresh prefill.

Two more systems, and what is published

The April deep dive gave the baseline. A TPU 8i chip has 288 GB of HBM at 8,601 GB/s, 384 MB of on-chip SRAM, and 10.1 petaflops of FP4 peak. A TPU 8t chip has 216 GB at 6,528 GB/s, 128 MB of SRAM, and 12.6 petaflops. The deep dive says the 8i SRAM can host a larger KV cache entirely on silicon. For the model in the next section, 384 MB is about 2,000 tokens of cache. Both chips sit on Arm-based Axion host CPUs.

TPU 8i Max puts two 8i chips in one package. Google says it doubles the compute, the SRAM, and the HBM bandwidth of a single 8i. It fits twice the silicon in the same rack footprint. Two times the published numbers gives 768 MB of SRAM and 17.2 TB/s of HBM bandwidth per package. Google did not publish those totals, the HBM capacity of the package, or whether the two chips act as one device. Doubled bandwidth with doubled compute keeps the bytes per flop where they were. So each chip reads its share of a KV cache at the 8i rate. The gain that Google names is density: more compute per square foot and per watt.

TPU 8tx uses the 8t silicon on 5th Gen AMD EPYC hosts, with the Boardfly network that Google built for 8i. Google says data and model weights stay protected and isolated while in use. It claims up to 2x lower latency than TPU 8t, with a near-zero performance penalty. AMD's release adds the mechanism. The host uses AMD Secure Encrypted Virtualization, and the protection extends to the attached accelerators. An accelerator's identity and security configuration are checked before it gets direct access to protected memory. AMD says this helps keep prompts, weights, and intermediate data encrypted in CPU memory and between host and accelerator. A KV cache in host DRAM fits that description of intermediate data. Neither post says whether the TPU Sync DRAM tier runs inside that boundary on 8tx, so I treat it as an open question.

Both are new packages and hosts around chips that already exist. For a serving stack, the library is the larger change. Every TPU Sync number below was measured on Ironwood, the seventh generation, in July 2026. Google says the results will vary with configuration and traffic.

The state that has to move

A transformer keeps one key and one value vector per token per layer. Qwen3-235B-A22B has 94 layers, 4 KV heads, and a head dimension of 128, from its config.json. In BF16 that is 2 × 94 × 4 × 128 × 2 bytes, or 192,512 bytes per token. At 256K tokens the cache is 50.5 GB. At 1M tokens it is 202 GB. Qwen3-Coder-480B-A35B has 62 layers and 8 KV heads, so 253,952 bytes per token, and 66.6 GB at 256K. Those two models are the ones Google tested. The bandwidth essay showed that the cache is read in full on every decode step. Here the cache moves the other way, out of the chip when the chip needs the room.

An Ironwood host has four chips with 192 GiB of HBM each, 960 GB of host RAM, and one PCIe Gen5 x16 link per chip. Weights in FP8 for the 235B model take about 235 GB, so the four chips keep roughly 590 GB for everything else. That is eleven 256K sessions, if nothing else used the memory. The runtime reserves part of HBM, so Fig. 2 counts ten. Every other open session waits outside HBM. The next sections compute what it costs to bring one back.

Reload against recompute

There are two ways to put a session's cache back into HBM. The chip can compute it again from the tokens, which is a prefill. Or the host can copy it back from wherever it was parked. Each has a cost in seconds, and the two costs grow with the context length at different rates.

A reload is bytes divided by a link rate. The TPU Sync post reports "up to 223 GB/s of aggregate D2H and H2D throughput" per Ironwood host. It gives 256 GB/s as the PCIe Gen5 ceiling. The README calls the same path about 200 GB/s bidirectional. The ceiling suggests one direction, because four Gen5 x16 links carry 256 GB/s each way. The words suggest the sum of both directions. I use the conservative reading, 111.5 GB/s each way, in the text and as the default in every figure. If the other reading is correct, every DRAM time below is half as long.

The post also measured about 192 Gbps on one 200 Gbps NIC and about 350 Gbps across two. So the 50.5 GB cache comes back from pinned host DRAM in 0.45 seconds, and crosses the network in 2.1 seconds on one NIC. Google's own measurement for the 235B model was a median KV transfer of about 230 ms, down from several seconds. The post does not give the prompt length of that test, so I cannot compare it with these figures.

A prefill is floating point operations divided by a compute rate. The linear part is 2 × active parameters per token, so 44 GFLOP per token for the 22B active parameters. The attention part grows with the square of the context. With causal masking it is 2 × layers × query width × n² in total, where the query width is 64 heads × 128, or 8,192. At 256K tokens the attention term is 1.06 × 10¹⁷ FLOP and the linear term is 1.15 × 10¹⁶. Four Ironwood chips have 18.5 petaflops of FP8 peak. At 40% utilization, which is my assumption and a knob in both figures, the prefill takes 15.9 seconds. The DRAM reload is 35 times faster. The network reload is 7.6 times faster. A storage shelf at 3 GB/s takes 16.8 seconds and loses.

The storage number shows the rule. A slow tier does not pay at every context length. Per token, a DRAM reload costs 1.73 microseconds and a prefill costs at least 5.96 microseconds, so DRAM wins at any length. A tier slower than that per-token prefill cost only wins once the quadratic attention term has grown enough. Setting the two costs equal gives the break-even context for a tier of rate B. It is n* = (bytes per token × F × utilization ÷ B − 2P) ÷ (2 × layers × query width). Here F is the peak of all the chips, and P is the active parameter count. For one NIC that is about 10K tokens. For a 3 GB/s shelf it is about 270K tokens. Below those lengths, computing the prefix again is faster than reading it from that tier. SGLang's HiCache applies the same rule at a small scale. It refuses to prefetch a storage hit under 256 tokens.

Fig. 1 · Tier break-even ruler

Each row is a tier with its read rate. The marker is the context length where a reload from that tier takes exactly as long as computing the prefix again. To the right of the marker the tier pays. Move the context cursor to read both times at one length. The NIC rates are the TPU Sync measurements on Ironwood. The DRAM rate is half of the published 223 GB/s, the conservative reading. The chip peak is from the TPU7x documentation. The utilization and the storage rate are assumptions.

Prefill utilization (assumption)40%
Storage read rate (assumption)3 GB/s
Context cursor256K

Prefill FLOPs = 2 × active parameters × n + 2 × layers × heads × head dimension × n², the second term with causal masking. Reload = n × bytes per token ÷ tier rate. One request, one link, no contention, no overlap. Both are upper bounds on how good a tier can look.

What TPU Sync does on the way down

The library is a C++ layer under JAX and PyTorch. Its post names four mechanisms, and each one removes a copy or a stall from the path above. First, it reads the device buffer behind a JAX array or a PyTorch tensor in place, through the PJRT raw buffer handle. It moves slices between HBM and page-locked host memory in one hardware DMA. The tensor keeps its TPU tile layout in HBM, in DRAM, on disk, and on the network. It lands ready for the matrix unit with no serialization and no bounce buffer.

Second, it does not lock the buffer or copy it on the device. A block transfer runs while the chip computes on the same cache. Pacing keeps a large transfer from delaying a kernel launch. Third, nothing in the data path goes through XLA, so moving blocks or resharding weights between different parallel layouts never triggers a compile. Fourth, the transfers run on native threads outside the Python interpreter, so the serving loop never waits on them.

Above that transport sit three parts. A global KV store maps prefix hashes to physical blocks and runs LRU eviction and tier demotion. A controller writes transfer plans, coordinates moves across nodes, and holds leases. A per-host cache manager tracks block state across HBM, pinned DRAM, and an experimental POSIX storage backend. That backend writes to local SSD or to Managed Lustre, which sells throughput at up to 1,000 MB/s per TiB provisioned. The DRAM tier can live in /dev/shm, so a serving process can restart and re-attach to a warm cache instead of computing every prefix again. The post calls the tiers 1, 2, and 3, and the machine above draws them in that order.

For prefill and decode on separate hosts the path has three steps. The prefill host writes the finished blocks into pinned DRAM by DMA. The controller streams them host to host over the data center network to the assigned decode host. The decode host pulls them into HBM while it keeps decoding other requests. Each step overlaps the next. The engines sit on top through their own interfaces. vLLM-TorchTPU uses it as a KV connector. SGLang-TorchTPU uses it as the disaggregation transfer backend and as the I/O backend for HiCache. The SGLang guide covers HiCache tier by tier. On GKE, llm-d 1.0 adds prefix-affinity routing, so a request lands on the replica that already holds its prefix in some tier.

The post reports these results from July lab runs on Ironwood. For the 480B model under low concurrency, TTFT fell by an average of 90% with the disaggregated path. For the 235B model it fell by 60%. On a 64K-context prefix caching workload, average TTFT fell by up to 90%, throughput rose by over 50%, and P99 inter-token latency fell by 30%. The post says host DRAM expands effective cache capacity by an order of magnitude when the weights take most of HBM. On one host, 590 GB of free HBM beside 960 GB of RAM is a gain of 2.6x. An order of magnitude needs less than about 107 GB of free HBM per host, or DRAM pooled across hosts.

Sizing the pool

The tiers only help if the working set of live sessions fits somewhere faster than a prefill. Three numbers decide that. The first is how many sessions HBM holds after the weights. The second is how many more sessions the DRAM tier holds. The third is the reload traffic those sessions put on the PCIe link, which is shared with every other transfer on the host. If reloads alone need most of the link, a bigger DRAM pool stops helping, because the sessions in it cannot come back fast enough.

Fig. 2 · Session residency ledger

Open sessions are dealt into HBM, host DRAM, and storage in that order, on Ironwood hosts with four chips each. HBM is 4 × 192 GiB per host minus the weights, with a 10% runtime reserve as an assumption. The DRAM pool is a slider, because the serving process and the OS need some of the 960 GB. Every session in DRAM or storage reloads its whole cache once per turn, and the turn gap sets how often. The verdict counts the sessions per tier and the share of the PCIe link those reloads take.

Hosts1 host
Context per session256K
Open sessions64
Pinned DRAM per host (assumption)760 GB
Seconds between turns60 s
sessions by tier
HBMhost DRAMstorage
PCIe link share from reloads

Bytes per token from each config.json in BF16. Weights at 1 or 2 bytes per parameter from the parameter count in the model name. HBM and host RAM as published. PCIe at 111.5 GB/s each way, the conservative reading of the published 223 GB/s. A session in storage also crosses PCIe on its way back, so it counts in the link share. The storage rate itself is not modeled here. Fig. 1 has it.

Read the ledger with the break-even in mind. HBM holds the sessions that decode now. DRAM holds the ones that will decode within a few seconds, and they come back faster than any prefill. Storage holds the ones that return minutes later, and it only pays for long contexts. Short sessions that fall out of DRAM are cheaper to compute again than to fetch from a slow shelf. A cache manager that fetches them anyway makes TTFT worse.

Try it on a TPU, or build the same tiers on GPUs

TPU Sync is open source today. The repository publishes wheels for JAX as tpu-sync-jax and for PyTorch as tpu-sync-torch. Only dev nightlies exist until the first stable release, so pip needs --pre. For a stable deployment, the README says to pin a tagged release or the commit in lkg.version rather than track main. It also warns that the old tpu-raiden-* package names are retired and are not controlled by the project. It needs a v7x or v6e TPU VM, and the host offloading example in examples/kv_host_offloading runs the DRAM tier on a live serving topology. The three environment variables below turn on the /dev/shm warm boot.

# on a v7x or v6e TPU VM, Python 3.12
# option 1: the nightly wheel
pip install --pre tpu-sync-jax

# option 2: build from source at the last verified commit
git clone https://github.com/google/tpu-sync && cd tpu-sync
git checkout $(cat lkg.version)
./build.sh jax

# keep the DRAM tier across a serving process restart
export RAIDEN_SHM_KEY="raiden_cache"
export RAIDEN_SHM_MODEL_UID="qwen3-235b-a22b-fp8-tp4"
export RAIDEN_SHM_SERVER_NAME="server_8000"
ls -la /dev/shm/ | grep raiden_cache

Commands and variable names from the TPU Sync README. The model uid is mine. Not run here.

On GPUs the same three tiers exist in the open engines, and the flags are documented. In vLLM the OffloadingConnector copies finished KV blocks into pinned host memory by DMA while the model runs. A second spec adds secondary tiers, staged through that CPU tier. For a CPU-only setup, the docs state the sizing rule plainly. Make cpu_bytes_to_use larger than the aggregate GPU KV cache. A smaller CPU tier only mirrors what the GPU already holds and adds no hit rate. The command below opens a 512 GB DRAM tier and a filesystem tier under it. Compare that size with the GPU KV cache size that vLLM logs at startup.

vllm serve Qwen/Qwen3-235B-A22B-FP8 --tensor-parallel-size 4 \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 512000000000,
      "block_size": 64,
      "eviction_policy": "lru",
      "secondary_tiers": [
        { "type": "fs", "root_dir": "/mnt/nvme/kv_cache",
          "n_read_threads": 32, "n_write_threads": 16 }
      ]
    }
  }'

Keys and values from the vLLM KV offloading usage guide. The model and the path are mine. Not run here.

SGLang's HiCache is the same shape with different names. The flag --enable-hierarchical-cache turns on the host tier, and --hicache-ratio 2 sizes it at twice the GPU pool. The backend flag --hicache-storage-backend fast_file adds a local NVMe tier with parallel reads and background eviction. Its docs state the same capacity rule. With write-through, every token on the GPU also has a host copy. A host pool no larger than the GPU pool leaves no room for storage hits, so they are dropped. The counter sglang:storage_prefetch_unfulfilled_tokens_total{reason="host_capacity"} rises when that happens.

python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --tp 4 \
  --page-size 64 \
  --enable-hierarchical-cache \
  --hicache-ratio 2 \
  --hicache-mem-layout page_first \
  --hicache-io-backend kernel \
  --hicache-write-policy write_through \
  --hicache-storage-backend fast_file \
  --hicache-storage-backend-extra-config \
  '{"file_storage_path":"/mnt/nvme/hicache","max_size":"3Ti","read_workers":8}'

Flags from the HiCache best practices page. Not run here.

LMCache gives vLLM a third option through "kv_connector": "LMCacheConnectorV1" with "kv_role": "kv_both", and sizes the DRAM tier with LMCACHE_LOCAL_CPU=True and LMCACHE_MAX_LOCAL_CPU_SIZE in GB. Its docs now mark the in-process mode as deprecated in favor of a separate process, so read the current page before you pick it. All three tools pin the host memory, and pinned memory counts against ulimit -l. The LMCache example clamps its buffer to half of RAM and 90% of the memlock limit for that reason.

The sizing recipe is the same on every stack. Take bytes per token from config.json. Take the free HBM after the weights, and divide to get the tokens the chip holds. Set the DRAM tier to the working set of sessions that return within a few seconds. Make it larger than the device pool, or it adds nothing. Add storage only for sessions longer than the break-even in Fig. 1. Their return rate times their size must stay under the storage rate. Then check the reload traffic against the PCIe link in Fig. 2.

What November 9 changes for Agent Compute Engine

In the October essay, the Agent Compute Engine idle signal fired when a tool call finished and the loop went back to the model. The November post lists three pauses that it suspends across: a database lookup, an external software tool, and human feedback. Human feedback fits the October signal, because no command is in flight while the harness waits. A lookup or tool that the harness runs outside the sandbox fits too. A lookup that runs as a command inside the sandbox does not, because that command is still in flight while it waits. The post does not say how the host tells a blocked command from a working one. It also does not say where the reply goes while the microVM is suspended. Until the customer preview opens, read the new sentence as a stated goal.

Where it breaks

SymptomLikely causeFix
The host tier is on, and the hit rate did not movethe host pool is no larger than the device pool, so it mirrors what HBM already holdssize cpu_bytes_to_use or --hicache-ratio above the device pool, and watch the startup log for pool tokens
Storage hits are dropped and the prompt is computed againa storage hit is staged through the host pool first, and the host pool had no free roomleave host room beyond the device pool, and watch storage_prefetch_unfulfilled_tokens_total
Startup fails with a pinned memory allocation errorthe pinned buffer exceeds ulimit -l or the RAM left after the model serverraise the memlock limit, or clamp the tier to half of RAM as the LMCache example does
TTFT got worse after a storage tier was addedthe sessions it serves are shorter than the break-even context for its ratekeep short prefixes in DRAM or recompute them, and set the storage prefetch policy to best_effort or timeout
A restarted TPU server starts cold, or host RAM is still full after it stoppedRAIDEN_SHM_MODEL_UID changed, so the segment was rejected, or the /dev/shm files were never unlinkedkeep the uid stable across a binary update, and remove /dev/shm/raiden_cache_* when a model is retired
The TPU Sync extension crashes far from the call that used itthe extension was built against one jaxlib and loaded beside anotherrebuild with RAIDEN_JAX_VERSION set to the installed version, in the environment you serve from

A decision order

  1. Compute bytes per token for your model from config.json, and the free HBM after the weights. That gives the sessions the chip holds with no help.
  2. Measure the gap between turns for your traffic. Sessions that return within seconds belong in pinned DRAM, sized above the device pool.
  3. Find the break-even context for each slower tier at your measured link rates. Put only sessions longer than that on it.
  4. Add the reload bytes per second of every parked session, and keep the total well under the PCIe link. The link also carries prefill output and decode traffic.
  5. Turn on the warm restart for the DRAM tier, and test a binary update before you rely on it.
  6. Measure TTFT per tier in production, because every number above is a one-request upper bound on how good a tier can look.

None of those steps needs a TPU. The vLLM and SGLang commands above open the same tiers on a GPU host. The link rates to use in the figures are the ones you measure on your own hosts.

rg
Rohit Ghumare

CNCF Ambassador and Google Developer Expert. I build agent infrastructure and write about the fundamentals underneath the AI stack. On November 9, 2026, I read the TPU 8tx and 8i Max announcement, the TPU Sync post, and the TPU Sync README as they appeared. Beside them I read the April TPU 8 deep dive, the TPU7x documentation, and the vLLM, SGLang, and LMCache offloading docs. Nothing here ran on a TPU.

Related: Tokens per Second Is a Memory Bandwidth Number · Local LLM hardware · More posts · X