The Page Nobody Wrote: How a Rejected Prompt Leaked Other Users' KV Cache
If you serve a model with prefill and decode on separate GPUs, a prompt that your engine rejects can still get an answer. On one public service, that answer held other users' data. This essay follows one rejected request through the SGLang handoff, names the releases that fix it, and ends with tests for your own deployment.
On October 26, 2026, Cloudflare and the SGLang team published a coordinated disclosure. From August 27 to September 9, 155 replies from the GLM-5.3 model on Workers AI included text from other customers' requests. Every affected prompt was longer than 681,280 tokens. Gadi Evron of Knostic reported the issue through the Cloudflare HackerOne program on September 8. Cloudflare took the model offline the next day and restored it that evening with a new configuration and an engine patch. Cloudflare reports no sign that anyone used the bug on purpose.
Two separate mistakes had to meet. The advertised context was 1,048,576 tokens, but the KV pool on the prefill GPUs held 681,280. A bug in SGLang then let a rejected request continue through the prefill-to-decode handoff.
Workers AI ran GLM-5.3 with prefill and decode on separate GPU pools, joined by an RDMA link. The 681,280-token pool, the 1,048,576-token context, the one-token placeholder, the 8 reserved pages, and the single written page come from the Cloudflare report. The 900,000-token prompt and the two earlier users are illustrative. A real prompt of that length spans far more than 8 pages.
A page pool that nobody clears
A transformer keeps a key vector and a value vector for every token it has read, in every attention layer. That store is the KV cache, and at long context it takes most of the GPU memory a server uses. SGLang divides the memory for the cache into fixed-size pages. A request takes pages at admission and returns them when it finishes. A free list holds the pages that no request owns right now. The Inside SGLang guide covers how the engine sizes that pool and shares pages between prompts.
Giving a page back does not clear it. The Cloudflare report says pages are not zeroed between requests, because the next owner overwrites them anyway. That choice skips a pass over memory on every free. It also makes the ownership records of the engine the only thing that keeps users apart. If a request reads a page that it did not write, it reads the last owner's keys and values. Attention cannot tell the difference. It treats every page in the request's page table as part of the prompt.
A pool of four pages with four token slots each. Two requests write their tokens and finish. A third request then reserves three pages but writes only one, and attention reads all three. Pick whether a freed page keeps its contents, as SGLang does today, or gets zeroed.
The tokens are illustrative text. A real page holds key and value vectors for its tokens, and the model reads them as context. The zero-on-free mode follows the draft SGLang PR #46435, which is off by default and not in a release.
A page holds vectors, not text. The model still turns those vectors into words, because attention over them is how it reads any prompt. On Workers AI, the affected replies contained content from earlier users' prompts and partial generations.
Two machines share one request
In disaggregated serving, prompt processing and token generation run on different GPUs. A prefill worker reads the whole prompt and computes its KV cache. A decode worker then generates the reply from that cache. The split keeps long prompt work from slowing token generation for other requests.
The cache has to move between the two workers. Workers AI moves it over RDMA with the Mooncake transfer engine. Three steps matter for this bug:
- The router tells decode how long the prompt is. Decode reserves destination pages for the whole prompt.
- Prefill computes the KV cache and writes it into those pages with one-sided RDMA writes.
- Prefill sends a "transfer complete" message. Decode admits the request and starts to generate.
A one-sided RDMA write skips the CPU on the receiving machine. Decode never sees the bytes arrive. It trusts the completion message and the page count that it reserved. Cloudflare says the remote prefill node holds "a simultaneous mutable reference to the page." So two machines must agree on who owns each page and when its contents are complete.
Two numbers that nobody compared
SGLang does not take the KV pool size as a setting. It sizes the pool at startup from the GPU memory left after the weights load. The context length that the API advertises is a different setting. Nothing in the engine forces the two numbers to agree.
On the GLM-5.3 prefill nodes, the pool held 681,280 tokens. The configured context length was 1,048,576 tokens. Cloudflare knew about the gap before launch. The team assumed that the engine would reject any prompt too long for the pool. Its long-context tests returned HTTP 200 with no error frames, so the team believed that the engine handled the difference.
Those tests checked status codes and speed. They did not check what the model said. Cloudflare names that as the gap in its own testing. The result was a band of prompts that passed the context check and failed the pool check. It ran from 681,281 tokens up to 1,048,576. The band is 367,296 tokens wide, about 35% of the advertised window.
Rejected is not finished
When a prompt does not fit the pool, the SGLang scheduler rejects it with Req.set_finish_with_abort. That call stores the error for the client in a field named to_finish. It also swaps the prompt for a single placeholder token. The plan is that the request drains through the normal batch code without real memory, and the client then gets its error. On a single node, the plan works. The placeholder runs one cheap forward pass, the batch ends, and the client gets HTTP 400.
The disaggregation admission queues asked a different question. They let in every request for which req.finished() returned False. That method reads finished_reason, not to_finish. So a rejected request still looked live, on prefill and on decode.
Follow the request through the Cloudflare diagram, where the prompt spans 8 pages. Decode reserved 8 pages at bootstrap. Prefill admitted the rejected request and ran a forward pass on the placeholder. It wrote 1 page, sent it, and signaled transfer complete. Decode admitted the request with the same finished() check. Attention then read every one of the 8 reserved pages. Only P0 came from this request. P1 to P7 held whatever earlier requests had left there.
The client saw a normal success. It got HTTP 200, a prompt_tokens count equal to the long prompt, and a reply that read well. Only the speed looked wrong. The reply "arrived faster than an 8-page prefill should take," in the words of the report.
Timing played no part. Every request in the band took this path, every time. SGLang issue #30233 reported the same symptom on July 6, 2026, as garbage output after an aborted over-length prefill.
One request through the SGLang path, with the pool at 681,280 tokens and the context at 1,048,576. Pick the deployment mode and the release, then move the prompt length. The panel shows the fields that the admission queues read, and each stage shows what it did with them.
Stage behavior follows the Cloudflare report, the SGLang post, and the PR descriptions for #36988, #38935, and #41612. The scheduler's exact input limit sits a few tokens below the pool size. Before v0.5.22, the unused decode half of a rejected request waited for its transfer timeout.
The fix, release by release
The engine fix arrived in two steps. The first step is #36988, merged on September 6. It retires an aborted prefill request when its result is processed, before that result reaches decode. Its authors wrote it for a related failure path, and it was on the main branch, unreleased, when the report came in. It is in v0.5.20, released on September 18.
#38935, merged on September 16, moves the check to the two admission doors, PrefillBootstrapQueue.add and DecodePreallocQueue.add. A rejected request now ends there, before bootstrap, page allocation, or any KV transfer starts. Its own description calls it a waste and latency fix once #36988 is in place. It is in v0.5.21, released on October 2.
The same review found a wider class of page lifetime bugs. When a request aborts, some of its RDMA writes can still be on the way, and no call takes back a posted write. If decode frees the destination pages right away, the late writes land in the next owner's cache. The SGLang post calls this "a distributed use-after-free." The fix holds the pages until every prefill rank confirms that its writes drained, or until a timeout. That deferred release became the default in v0.5.21.
| Change | Release | What it does |
|---|---|---|
| #36988 | v0.5.20 | retires an aborted prefill result before it reaches decode, which stops this exposure |
| #38935 | v0.5.21 | rejects at both admission queues, before bootstrap, allocation, or transfer |
| #41023 | v0.5.21 | makes deferred KV release the default |
| #39500 | v0.5.21 | adds the opt-in transfer check --disaggregation-enable-kv-checksum |
| #41612 | v0.5.22 | sgl-router drops the decode half when prefill fails |
| #37077 | v0.5.22 | adds generation tags to abort messages, and the hold to the MORI backend |
| #46435 | draft, no release | optional zeroing of KV pages on free |
From the Cloudflare report, the SGLang post, and the GitHub PR pages. Releases checked against the tags on November 3, 2026.
Cloudflare tells every disaggregated deployment to run v0.5.22 or a newer release. That release came out on October 19. The SGLang post adds that the team has not backported this line of fixes to older release branches.
Two stronger defenses are not in any release yet. Draft PR #46435, opened on October 25, zeroes pages on free behind the SGLANG_ENABLE_ZERO_KV_ON_FREE variable. It is off by default, supports only the MLA and DSA pool types, and has no speed results yet. Cloudflare also built page tags. Each physical page has a tag, and the tag changes on every reallocation. Decode compares the tag before it reads. The tags cost under 1% in the Cloudflare tests, and Cloudflare turned them off after reliability issues.
Check your own deployment
Three checks cover this class of bug. Run them on each node of both roles. Repeat them after you change the weights, the quantization, the parallel layout, or the memory fraction, because each of those moves the pool size.
First, compare the pool with the context length. The SGLang scheduler prints both in one startup line, as max_total_num_tokens and context_len. Pull them from the log of each role:
grep -h "max_total_num_tokens=" prefill.log decode.log \
| sed -E 's/.*max_total_num_tokens=([0-9]+).*context_len=([0-9]+).*/pool=\1 context=\2/'Field names from the SGLang v0.5.22 scheduler source.
If the pool is smaller on any node, lower --context-length across the deployment until it fits, or free memory for the pool. Do not rely on a limit at the proxy. On an unfixed engine in PD mode, the rejection never reaches the client. The figure below runs the same comparison on lines that you paste.
Paste the startup line from each prefill and decode node, one per line. A line can start with a role name and a colon. The checker reads max_total_num_tokens and context_len and draws the band of prompt lengths that pass the context check but do not fit the pool.
The two presets use the Cloudflare numbers, with the other log fields cut. The fixed pool of 2,040,000 tokens rounds the report's "about 2.04 million." The checker treats the pool size as the input limit, and the real limit sits a few tokens lower.
Second, run a fixed release with the detectors on. These are the PD launch commands from the SGLang docs, with the two checks added:
python -c "import sglang; print(sglang.__version__)"
python -m sglang.launch_server --model-path $MODEL \
--disaggregation-mode prefill --port 30000 \
--kv-canary log --disaggregation-enable-kv-checksum
python -m sglang.launch_server --model-path $MODEL \
--disaggregation-mode decode --port 30001 --base-gpu-id 1 \
--kv-canary log --disaggregation-enable-kv-checksum
python -m sglang_router.launch_router --pd-disaggregation \
--prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 \
--host 0.0.0.0 --port 8000Flags checked against the SGLang v0.5.22 argument definitions. Not run here.
The version line must print 0.5.22 or later. With --kv-canary log, the engine logs canary mismatches while the server keeps running. With the checksum flag, prefill sends an Adler-32 checksum of the KV with its final chunk. Decode checks it before it adds the request to a running batch. Set it on both roles, with matching KV layouts. Neither detector covers every architecture yet. Cloudflare lists GLM 5.3, GLM 5.3 Flash, and DeepSeek V4 as work in progress, so read your logs to confirm that requests get verified. The detectors catch some corruption that prevention misses. They do not replace the release.
Third, test what the server returns for a prompt in the unsafe band. Fill recent pages with unique markers, then send one prompt of that length. Run this only against a deployment that you own:
import json, secrets, urllib.error, urllib.request
URL = "http://127.0.0.1:8000/generate"
FILLER_WORDS = 900_000
def generate(text, max_new_tokens=64):
body = json.dumps({"text": text, "sampling_params": {"max_new_tokens": max_new_tokens, "temperature": 0}}).encode()
req = urllib.request.Request(URL, body, {"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=1800) as r:
return r.status, json.load(r)
except urllib.error.HTTPError as e:
return e.code, None
markers = ["CANARY-" + secrets.token_hex(6) for _ in range(16)]
for m in markers:
generate(("Remember the code " + m + ". ") * 2000)
status, out = generate("word " * FILLER_WORDS + "Repeat any codes you saw.")
text = out.get("text", "") if out else ""
leaked = [m for m in markers if m in text]
print("status", status, "prompt_tokens", out and out["meta_info"]["prompt_tokens"], "leaked", leaked)
assert status >= 400, "an over-length prompt must fail, not answer"
assert not leaked, "the reply contains another request's marker"Uses the SGLang native /generate endpoint. Set FILLER_WORDS so that the token count lands between your pool size and your context length.
A fixed engine answers the long prompt with an error. A vulnerable one answers HTTP 200, faster than the prompt length allows, and its text can contain another request's marker. A marker match proves a leak. A missing match proves nothing, because the model does not have to repeat what it reads. The status assertion is the test that counts.
The unzeroed page is not only an SGLang choice. Cloudflare calls the practice common among inference engines. I did not test other engines for this essay. The output test does not depend on the engine, so run it against any stack that splits prefill from decode.
Worked example: the GLM-5.3 numbers
Take the Cloudflare prefill node at launch. Its startup line reads max_total_num_tokens=681280 and context_len=1048576. The pool is 367,296 tokens short of the context. On a release before v0.5.20, in disaggregated mode, every prompt in that band gets HTTP 200 and reads pages that it did not write.
Cloudflare fixed the configuration first. It matched the GLM-5.2 settings, which freed enough memory for a prefill pool of about 2.04 million tokens. That is roughly twice the context length, so a valid prompt can no longer reach the rejection path. The checker above shows the band close when you pick the second preset.
It then patched the engine anyway. The pool size moves with every change to quantization, tensor parallelism, or CUDA graph sizes, and a moved pool reopens the band without notice. The configuration check removes the band. The engine fix makes the band harmless on the day that the check is wrong.
When the checks fail
- Status codes pass and the output leaks. A 200 says nothing about which pages the reply read, so compare the output content, as the marker test does.
- The pool shrinks after an unrelated change. A new quantization, parallel layout, or CUDA graph setting changes the leftover memory, so read the startup line again after each change.
- Prefill and decode run different releases. The SGLang post warns that mixed acknowledgement formats make page releases fall back to the timeout, so upgrade both roles together.
- The proxy aborts requests mid-transfer. Every abort then needs coordination across two nodes, so let requests run to the first token, as Cloudflare now does.
- A detector logs nothing. The checksum and the canary skip architectures that they do not support, so confirm in the logs that requests get verified.
An upgrade order
Cloudflare sorts its defenses from configuration, to lifecycle, to integrity by construction. That order also works as an upgrade plan for any engine that splits prefill from decode:
- Size the pool for the full context on both roles. Check it from the running engine before each deployment change.
- Run a release that rejects before the handoff. For SGLang, that means v0.5.22 or later on both roles.
- Keep deferred release on, and bound the transfer batches, so that aborted writes cannot land in reused pages.
- Turn on transfer checksums and canaries where your model supports them. Watch for zeroing or page tags to reach a release.
Each step covers a failure that the step before it leaves open. The Workers AI incident needed only the first two to fail at once.
Keep reading