Cloudflare Workers AI Security Postmortem: A Runbook for Cross-Tenant KV Cache Leaks

A player-facing AI feature returned a response containing another studio's prompt text. Not a crash. Not a 500. A coherent, on-topic answer delivered with HTTP 200 and a token count that matched the caller's own request.

That is the worst shape a multi-tenant bug can take: the output is plausible, so nobody files a ticket. Cloudflare hit exactly this on August 27, 2026, when 155 responses from the GLM-5.3 Workers AI model contained data from other customers' sessions. Every affected request was longer than 681,280 tokens. The fix required two changes — one configuration, one engine patch — and the detection story is more instructive than the fix.

Here is the runbook: what broke, how to detect the same failure class in your stack, how to remediate it, and how to stop it recurring.

What Actually Broke

Two independent problems overlapped. Either one alone would have been survivable.

Problem one: a capacity mismatch nobody asserted on

Cloudflare deployed GLM-5.3 with a 1,048,576-token context window. The KV cache pool on the prefill GPUs held 681,280 tokens.

That gap matters because the pool size is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after loading model weights. The advertised context length is a separate config value. Nothing in the system forces the two to agree, and nothing warns you when they diverge.

Pre-launch testing passed. HTTP status codes were 200, streaming bodies had no error frames, throughput looked normal. The tests validated transport, not content.

Problem two: a rejected request that still reported itself alive

When a prompt exceeded the pool, SGLang's scheduler rejected it. The rejection path recorded the error for the client and swapped the prompt for a one-token placeholder so the request could drain through the batch machinery without allocating memory. On a single node, that worked fine: the placeholder ran, the batch finished, the client got an error.

In disaggregated prefill/decode mode — where prefill GPUs build the KV cache and ship it over RDMA to decode GPUs that generate tokens — the sequence went like this:

1. Bootstrap. The router told decode the prompt was 8 pages long. Decode reserved 8 KV pages.
2. Prefill admission. The queue admitted any request where req.finished() returned False. A rejected request stored its rejection in to_finish, not finished_reason, so finished() still returned False. The rejected request was admitted.
3. Prefill forward pass. The prompt had already been replaced by a one-token placeholder. Prefill ran a trivial forward pass on that single token.
4. KV transfer. Prefill sent one page of KV to decode and signaled "transfer complete."
5. Decode admission. Decode's queue used the same finished() check and accepted the handoff.
6. Decode attention. Decode generated tokens while attending over all 8 reserved pages. Only page 0 had been written by this request. Pages 1 through 7 still held other tenants' prompts and partial generations.
7. Response. The client got HTTP 200, a matching prompt_tokens count, and a coherent answer assembled from stale cache pages.

Every request over the 681,280-token threshold behaved this way. The bug had been filed upstream as issue #30233 on July 6, 2026, and the affected deployment had not yet picked up the fix.

Detect It Before Your Players Do

You cannot test your way to isolation with status codes. You need three signals.

Signal 1: pool size versus context length, read from the running engine

Check this at startup and in CI, not from a config file. The pool is derived from leftover memory, so a quantization change, a tensor-parallelism change, or a CUDA graph size tweak silently moves it.

# deploy_gate.py - run before promoting any disaggregated inference config
import json, os, sys, urllib.request

CONTEXT_LEN = 1_048_576      # the model's advertised window
HEADROOM    = 1.25           # pool must exceed context, not merely match it

def server_info(base_url: str) -> dict:
    with urllib.request.urlopen(f"{base_url}/get_server_info", timeout=10) as r:
        return json.load(r)

def kv_pool_tokens(info: dict) -> int:
    # SGLang reports the pool it sized at startup; field names vary by version
    for key in ("max_total_num_tokens", "kv_pool_tokens", "max_total_tokens"):
        if key in info:
            return int(info[key])
    raise RuntimeError("KV pool size not exposed by this build - fail closed")

def main(prefill_url: str) -> None:
    pool = kv_pool_tokens(server_info(prefill_url))
    required = int(CONTEXT_LEN * HEADROOM)
    print(f"kv_pool={pool} required={required} context={CONTEXT_LEN}")
    if pool < required:
        print("FAIL: pool smaller than context window; over-length prompts reach the rejection path")
        sys.exit(1)
    print("OK: pool covers the full context window")

if __name__ == "__main__":
    main(os.environ.get("PREFILL_URL", "http://localhost:30000"))

Run it against every prefill node before the load balancer sees traffic. A non-zero exit blocks the deploy.

Signal 2: canary markers in the KV cache

Plant a unique sentinel string in a long prompt, let the request finish, then send a second long prompt that should never contain it. Check the returned text, not the status code.

import uuid, requests

def canary_check(base_url: str, token: str, long_prompt_tokens: int) -> None:
    sentinel = f"CANARY-{uuid.uuid4()}"
    filler = "lorem ipsum " * (long_prompt_tokens // 2)

    requests.post(base_url, json={
        "prompt": f"{sentinel} {filler}",
        "max_tokens": 16,
    }, headers={"Authorization": f"Bearer {token}"}, timeout=300)

    # A fresh, unrelated request must never surface the previous sentinel
    r = requests.post(base_url, json={
        "prompt": f"Summarize this: {filler}",
        "max_tokens": 16,
    }, headers={"Authorization": f"Bearer {token}"}, timeout=300)

    assert sentinel not in r.text, f"KV cache leak: {sentinel} surfaced in a fresh response"

Run this against the longest prompt you intend to accept, at exactly the boundary, and at boundary + 1 token. The boundary case is where the rejection path fires.

Signal 3: latency that is too good

A full-length prefill takes measurable time. If an 8-page prompt returns in the time a 1-page prompt should take, you are reading someone else's cache. Log prefill duration indexed by prompt length and alert on the p5 of the long-context bucket. An anomalously fast long prompt is a leak signature, not a win.

The Fix, In Order

1. Make the pool hold the full context

Cloudflare changed the GLM-5.3 configuration to match GLM-5.2, freeing enough GPU memory that the prefill pool now holds roughly 2.04 million tokens — about twice the 1,048,576-token context. A prompt that fits the context window now always fits the pool, so the rejection path is never reached for a valid request.

2. Make the rejection path safe anyway

Pool sizing alone is fragile. Any change to quantization, tensor parallelism, CUDA graph sizes, or a model version moves the pool in a way that is easy to miss.

The engine patch retires a request carrying a pending rejection at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. The decode node also verifies that the number of tokens transferred matches the number it reserved. That landed upstream as PR #38704, with follow-ups in #38935, and shipped in SGLang v0.5.20 through v0.5.22.

If you run SGLang anywhere, pin to v0.5.22 or later. This is not a "schedule it next sprint" upgrade.

3. Verify with the repro, not with vibes

Cloudflare validated both changes against the researcher's reproduction script and their own canary-marker tests before restoring the model. Two independent tests, one adversarial and one internal. That is the correct bar before you re-enable a model or a fleet.

Why "It Returned 200" Is Not a Test

The root testing failure was narrow and completely ordinary: the team validated HTTP response codes and engine performance at long context lengths, without validating model output and coherence. Benchmarking passed because benchmarks ask "did it respond?" not "did it respond correctly?"

For a hosted AI feature, the correctness check is cheap. For your own game backend, the equivalent gap is usually wider. Here is the transferable rule: any shared resource pool needs an assertion that ties the size of the pool to the size of the largest request the system will accept, and that assertion must run against the live process, not a config file.

The Same Bug Class Lives In Game Backends

Replace "KV cache pages" with "session slots" and the incident reproduces almost line for line.

You allocate a fixed pool of match slots, player state slots, or actor buffers. A request reserves 8 slots. An admission check tests state != Finished instead of state == Valid, so a rejected reservation gets admitted anyway. The slot is handed to the next lobby with only slot 0 initialized. Slots 1 through 7 still hold the previous match's player IDs, inventory, and position data.

This is the same failure mode we covered in architecting game backends to survive compromises: the leak does not need to be a sophisticated attack, it just needs an ownership model that the code does not actually enforce.

Here is the buggy pattern and the fix, in C#:

public sealed class SessionSlotPool
{
    private readonly SessionSlot[] _slots;
    private readonly Queue<int> _free = new();

    // BUG: release without invalidation leaves stale state for the next tenant
    public void ReleaseUnsafe(int[] indices)
    {
        foreach (var i in indices) _free.Enqueue(i);
    }

    // FIX: invalidate on release, stamp a generation, verify on read
    public void Release(int[] indices)
    {
        foreach (var i in indices)
        {
            _slots[i].Reset();          // zero player list, inventory, match id
            _slots[i].OwnerId = null;
            _slots[i].Generation++;     // stale handles now fail the check
            _free.Enqueue(i);
        }
    }

    public bool TryRead(int index, long expectedGeneration, out SessionSlot slot)
    {
        slot = _slots[index];
        return slot.Generation == expectedGeneration && slot.OwnerId != null;
    }
}

The generation counter is the cheap part. It converts "we hope the slot was overwritten" into a hard, testable rejection. Any read against a stale handle fails instead of silently returning another player's data.

The Isolation Checklist

1. Assert pool size against context length at startup, from the running process. Not from config. A non-zero exit blocks the deploy.
2. Treat "rejected" as a terminal state distinct from "finished." Abort at admission, before any allocation or handoff, and return the original error.
3. Verify transfer counts on both sides. If decode reserved 8 pages, it must confirm 8 pages arrived. A one-page transfer reporting "complete" is the exact bug.
4. Test the failure path with content assertions. Canary markers, sentinel strings, boundary-plus-one prompts. Check the response body, never the status code.
5. Re-run the gate on every capacity change. Quantization, tensor parallelism, batch size, and model version all move the pool. Treat each as an isolation-relevant change.

Item 5 is the one teams skip. Every capacity change is also a security change, because the pool is derived from leftover memory rather than declared. The same zero-sum thinking applies to any resource pool you size by hand — the trade-off math in the zero-waste server hibernation analysis is the same arithmetic.

What This Costs

Isolation is not free, and the trade-off is honest.

Doubling the KV pool from 681,280 to roughly 2.04 million tokens consumes GPU memory that would otherwise hold more concurrent requests. Fewer concurrent requests means higher cost per generated token. On a prefill fleet, that headroom is capacity you cannot sell.

There is a cheaper alternative: keep the small pool and reject over-length prompts with an explicit HTTP 400 at the edge, before they reach the scheduler. That preserves throughput but fails legitimate long-context requests — which, for an agentic coding model, is most of the product.

Cloudflare chose correctness at the capacity cost. For a player-facing feature, that is almost always right. The alternative — a fast, cheap, plausible response containing another player's data — is not a trade-off you can defend in a postmortem.

Where Isolation Burden Actually Sits

Every multi-tenant system makes this decision: which shared resources do you pool and reuse, and who owns the invalidation. The answer for GPU KV pages is "the inference engine, badly, until v0.5.22."

For your game's player state, the answer is usually "your database, and it is on you." If you would rather not own the pool semantics at all, account-bound state services shift the surface: horizOn provides account-bound Cloud Save with revision-aware reads and writes, where the client resolves conflicts rather than your servers guessing which slot is stale.

That does not eliminate isolation work — nothing does. It narrows it to the boundaries you actually control.

What To Do This Week

Audit every shared pool you own and ask three questions. Does the pool size provably cover the largest request the system accepts? Does a rejected request terminate before it allocates anything? Does the receiving side verify that what arrived matches what it reserved?

If any answer is "we assume so," you have the same bug waiting for a long enough request. Start with the pool whose contents would be most damaging in another player's response.

If you are ready to stop maintaining that pool for player state, try horizOn free — authentication, account-bound Cloud Save, leaderboards, remote config, and crash reports, without standing up a database and a session model from scratch. The API docs show the integration surface in about ten minutes.

---

Source: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang