What Cloudflare's Container Vulnerability Taught Us About Multi-Tenant Game Backend Security

Your containerized game backend spins up a fresh VM for every match, every lobby, every analytics batch. When that container shuts down, the data is gone — right?

Not necessarily. In September 2026, security researcher Oren Yomtov from Accomplish discovered that Cloudflare Containers had a cross-tenant data exposure vulnerability. Residual disk blocks from destroyed containers were readable by unrelated workloads on the same host. Recovered data included directory structures, database pages, and structurally complete SQLite databases.

If you run game backends on any multi-tenant container platform, this vulnerability's root cause — a Linux storage configuration called skip_block_zeroing in the dm-thin subsystem — is something you need to understand. This post breaks down what went wrong, provides a detection and remediation runbook you can apply today, and covers the architectural principles that prevent this class of vulnerability from reaching your players' data.

How Thin Provisioning Creates Cross-Tenant Data Exposure

Cloudflare Containers run workloads inside Firecracker micro-VMs on multi-tenant hardware. Each container gets a writable root disk backed by Linux device-mapper thin provisioning (dm-thin). Firecracker exposes this disk to the guest VM as /dev/vdc.

Thin provisioning works by deferring physical storage allocation. When you create a 10 GiB virtual disk, it consumes almost no physical space. Blocks are allocated on-demand from a shared pool when the guest writes to a previously unmapped region. Cloudflare's affected pools used a 64 KiB thin-block size.

Here's where the vulnerability lives. When a container is destroyed, its thin-volume mappings are deleted, and the physical blocks return to the shared pool for reuse. The affected pool was configured with:

skip_block_zeroing

With this flag, dm-thin skips zeroing newly allocated blocks before making them accessible to the next tenant. A full 64 KiB write overwrites the entire recycled block. But a partial write — say, 4 KiB — replaces only that region. The remaining 60 KiB can retain data from the block's previous owner.

This is not a theoretical edge case. The researcher recovered directory structures, database pages, and complete SQLite databases from residual blocks across 18 of 24 production placements and 20 of 22 underlying nodes spanning four continents.

Why Partial Writes Are the Key Vector

Reading an unmapped region of a new thin disk returns zeroes — dm-thin serves zeros without allocating a physical block. That's safe. The exploit works because a small write triggers allocation of a recycled 64 KiB block without zeroing it first.

The attack pattern:

1. Create a container on a Workers Paid account.
2. Open /dev/vdc (the writable root disk).
3. Identify 64 KiB-aligned regions corresponding to ext4 free space.
4. Write one aligned 4 KiB block into each target region.
5. Read back the full 64 KiB blocks.
6. Examine only the 60 KiB that the attacker did not overwrite.

Step 4 is the critical moment. The 4 KiB write causes dm-thin to allocate a physical block from the shared pool. Because zeroing is disabled, the remaining 60 KiB may contain residual data from whoever previously owned that block.

What the Researcher Actually Validated

The researchers used ext4 directory block checksums (the metadata_csum feature) to distinguish their own test filesystem blocks from foreign blocks. Across six production placements:

• 5,614 testable directory blocks examined
• 0 blocks attributed to the researchers' own filesystem
• 2,700 distinct foreign directory inodes identified through checksum analysis

The recovered file types were not junk. They included structurally meaningful SQLite databases — the kind of data that, in a game backend context, could contain player save states, session tokens, or inventory records.

Detection Playbook: Finding Residual Data Harvesting in Your Infrastructure

If you operate containerized game backends on multi-tenant infrastructure, you need two layers of detection: configuration auditing and runtime I/O anomaly analysis.

Step 1: Audit Your dm-thin Configuration

Run this script on every host that runs your containers:

#!/bin/bash
set -euo pipefail

echo "=== dm-thin Pool Configuration Audit ==="
echo ""

# Find all thin pool devices on this host
for pool in $(dmsetup status --target thin-pool 2>/dev/null | awk '{print $1}'); do
    echo "Pool: $pool"
    TABLE=$(dmsetup table "$pool")
    echo "  Full table: $TABLE"
    
    if echo "$TABLE" | grep -q "skip_block_zeroing"; then
        echo "  ⚠️  WARNING: skip_block_zeroing is ENABLED"
        echo "  Recycled blocks may retain previous tenant data."
        echo "  Action: Remove skip_block_zeroing from pool table."
    else
        echo "  ✅ Block zeroing is active (default dm-thin behavior)"
    fi
    echo ""
done

echo "=== Thin-block size check ==="
for pool in $(dmsetup status --target thin-pool 2>/dev/null | awk '{print $1}'); do
    BLOCK_SECTORS=$(dmsetup table "$pool" | grep -oP '\d+ \d+ thin-pool' | awk '{print $1}')
    BLOCK_KB=$((BLOCK_SECTORS * 512 / 1024))
    echo "  Pool $pool: block size = ${BLOCK_KB} KiB"
    if [ "$BLOCK_KB" -le 64 ]; then
        echo "    ⚠️  Small block size amplifies residual data exposure."
        echo "    Larger block sizes reduce the ratio of leftover bytes per partial write."
    fi
done

If skip_block_zeroing appears anywhere in your pool configuration, you have the same vulnerability class that Cloudflare patched. Fix it immediately — do not wait for a scheduled maintenance window.

Step 2: Monitor for the Characteristic I/O Signature

The exploit produces a detectable signature: small writes followed by disproportionately large reads. A 4 KiB write allocates a 64 KiB block; a subsequent read recovers the full 64 KiB. Container telemetry where read volume exceeds write volume by >10x is suspicious.

#!/usr/bin/env python3
"""
Detect anomalous write-then-read patterns characteristic of
cross-tenant residual data harvesting in thin-provisioned containers.

Usage: python detect_residual_harvesting.py io_events.jsonl

Input: JSONL file with one event per line:
  {"container_id": "abc123", "operation": "write", "size_bytes": 4096}
"""

import json
import sys
from collections import defaultdict

ANOMALY_THRESHOLD = 10.0  # read_bytes : write_bytes ratio

def analyze_io_events(events_file):
    with open(events_file) as f:
        events = [json.loads(line) for line in f]

    containers = defaultdict(
        lambda: {"writes": 0, "reads": 0, "write_bytes": 0, "read_bytes": 0}
    )

    for event in events:
        cid = event.get("container_id", "unknown")
        op = event.get("operation")
        size = event.get("size_bytes", 0)

        if op == "write":
            containers[cid]["writes"] += 1
            containers[cid]["write_bytes"] += size
        elif op == "read":
            containers[cid]["reads"] += 1
            containers[cid]["read_bytes"] += size

    print("=" * 60)
    print("CONTAINERS WITH ANOMALOUS READ/WRITE RATIOS")
    print("=" * 60)

    flagged = 0
    for cid, stats in sorted(containers.items()):
        if stats["write_bytes"] == 0:
            continue
        ratio = stats["read_bytes"] / stats["write_bytes"]
        if ratio > ANOMALY_THRESHOLD:
            flagged += 1
            print(f"\n🚩 Container: {cid}")
            print(f"   Writes: {stats['writes']:,} ops | "
                  f"{stats['write_bytes']:,} bytes")
            print(f"   Reads:  {stats['reads']:,} ops | "
                  f"{stats['read_bytes']:,} bytes")
            print(f"   Ratio:  {ratio:.1f}x — investigate")

    if flagged == 0:
        print("\n✅ No anomalous containers detected.")
    print(f"\nScanned {len(containers)} containers. Flagged: {flagged}")
    return flagged

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python detect_residual_harvesting.py <io_events.jsonl>")
        sys.exit(1)
    sys.exit(1 if analyze_io_events(sys.argv[1]) > 0 else 0)

In the Cloudflare incident, the researchers' proof-of-concent produced exactly this signature. Cloudflare's security team built detection signatures from the PoC, applied them to historical disk I/O telemetry, and confirmed that the only matching activity came from the researchers and Cloudflare's own engineers during authorized validation. No third-party exploitation was found.

The lesson: if you're collecting container I/O telemetry, you can audit retroactively. If you're not collecting it, you're flying blind.

Remediation Runbook

If your audit reveals skip_block_zeroing or similar exposure, here's the remediation sequence Cloudflare executed — and the order matters.

Step 1: Re-enable Block Zeroing on All Pools

Remove skip_block_zeroing from your dm-thin pool configuration. This is a runtime change on the thin-pool device, but it only protects new allocations. Blocks already mapped into running containers and cached image snapshots remain vulnerable.

# This is a simplified representation — actual dm-thin 
# pool recreation commands vary by your orchestration layer.
# The core principle: restart the pool WITHOUT skip_block_zeroing.

dmsetup reload <pool-name> --table "0 <size> thin-pool <meta_dev> <data_dev> <data_block_size> 0"
dmsetup suspend <pool-name>
dmsetup resume <pool-name>

Cloudflare merged this fix within approximately 6 hours of the initial report. The researchers independently confirmed the PoC stopped working after this change.

Step 2: Retire All Running Container Disks

Blocks already mapped into active thin devices retain their (unzeroed) content. The only way to eliminate residual data is to destroy and recreate every container disk.

Cloudflare drained hosts during off-peak hours and restarted VMs on each host. This is a rolling operation — for a game backend, schedule it during your lowest-traffic window. Expect a brief interruption per host. If your matchmaking layer supports graceful handoff, players on draining hosts get migrated rather than disconnected.

Step 3: Clear Cached Image Snapshots

This step is easy to miss. Container orchestration layers typically cache OCI image layers as dm-thin snapshots for fast container startup. These cached snapshots were created before the fix and can contain residual data in unused regions (including ext4 free space). A new container that inherits a cached layer can read residual bytes from /dev/vdc without ever allocating a new block.

Cloudflare removed all pre-mitigation cached snapshots across the affected fleet, completing cleanup 15 days after the initial report. The cached layers were recreated using zeroed allocations.

Recreation order matters. If you clear caches before enabling zeroing, you've just pushed unzeroed blocks right back into fresh snapshots.

Step 4: Validate That the Fix Holds

After remediation, run the detection script against new I/O telemetry for at least 48 hours. Compare the write/read ratios to your pre-patch baseline. The characteristic high-ratio pattern should disappear entirely.

You should also verify the dm-thin table on each host post-reboot:

# Confirm no host still has skip_block_zeroing
dmsetup table 2>/dev/null | grep -q "skip_block_zeroing" && \
    echo "CRITICAL: Host $(hostname) still has skip_block_zeroing" || \
    echo "OK: $(hostname) — block zeroing active"

Architectural Principles to Prevent This Class of Vulnerability

The Cloudflare incident is one instance of a broader class of multi-tenant storage isolation failures. Here are the principals that protect your game backend — whether you manage your own containers or rely on a platform.

Principle 1: Never Disable Block Zeroing in Multi-Tenant Pools

The skip_block_zeroing flag exists for performance — zeroing 64 KiB blocks on every allocation adds I/O overhead. On single-tenant hardware where only your code runs, the tradeoff might be acceptable. On shared infrastructure where containers are recycled across tenants, it's a security defect.

Rule: Any dm-thin pool that allocates blocks to more than one tenant identity must have block zeroing enabled. Enforce this at the infrastructure-as-code layer so no host can be provisioned without it.

Principle 2: Treat Container Disks as Ephemeral and Dangerous

Your game backend should never assume a container's disk is clean at startup. Even with block zeroing enabled, there are edge cases with caching layers, snapshot inheritance, and kernel bugs.

Write your container entrypoint to format or wipe the writable volume on boot:

#!/bin/bash
# Ensure a clean writable disk on container start
if [ -b /dev/vdc ]; then
    # Overwrite with zeros (slow but thorough)
    # For a game backend, the writable disk is typically small (1-4 GiB)
    dd if=/dev/zero of=/dev/vdc bs=1M count=4096 status=progress
    
    # Create a fresh filesystem
    mkfs.ext4 -F /dev/vdc
    
    mount /dev/vdc /workspace
fi

This adds a few seconds to container startup. For a game match that runs 5–45 minutes, that's acceptable. For sub-second cold-start requirements, use the faster approach: blkdiscard (which triggers TRIM and may zero blocks depending on the storage backend) combined with mkfs.ext4 -F.

Principle 3: Monitor I/O Ratios as a Security Signal

Most container monitoring focuses on CPU, memory, and network. Add disk I/O to your security telemetry. The ratio of bytes read to bytes written is a strong signal for residual-data harvesting attacks — legitimate game workloads write and read in balanced patterns. A container that writes 4 KiB chunks and then reads back 60+ KiB per region is anomalous.

If your backend already logs container lifecycle events, you can correlate I/O anomalies with tenant identity and creation timestamps to narrow the blast radius of a potential incident. For game developers who want built-in crash reports and user logs without managing this telemetry infrastructure themselves, platforms like horizOn provide these capabilities out of the box — meaning you spend less time building plumbing and more time on game logic.

Principle 4: Implement Defense in Depth for Player Data

Even if your container disk leaks, the damage is limited if the data on it is encrypted or meaningless without a key. Store player save states, session tokens, and inventory data encrypted at rest. The encryption key should live in a secrets manager, not on the container disk.

This is the same principle we covered in our analysis of the Star Citizen data breach and how to architect game backends to survive compromises — defense in depth means assuming any single layer will eventually fail.

Best Practices Checklist

1. Audit every dm-thin pool configuration before deployment. Add a pre-flight check to your container orchestration pipeline that fails if skip_block_zeroing is present. This is a five-line CI check that prevents an entire class of vulnerabilities.

2. Collect container disk I/O telemetry with tenant attribution. You cannot detect exploitation you don't observe. Log write/read volumes per container, correlated with tenant ID and host ID. Retain at least 30 days for retroactive investigation.

3. Zero or wipe container writable volumes at boot. Don't rely on the storage layer alone. A fresh mkfs.ext4 on startup adds 2–5 GiB of write overhead but guarantees no residual data survives container recycling.

4. Drain and recycle containers during security patches, not just after. Enabling block zeroing only protects new allocations. Existing mappings in running containers and cached image snapshots must be explicitly cleared. Always follow the fix with a fleet-wide recycle.

5. Encrypt sensitive player data at rest with external key management. If a residual block leaks, ciphertext without the key is useless. Player save states managed through horizOn's account-bound Cloud Save use revision-aware writes and client-side conflict resolution — but the underlying disk isolation is still your responsibility if you self-host containers.

Timeline: How Cloudflare Responded

For reference, here's how fast a well-resourced team can move on a critical infrastructure vulnerability:

Time (UTC)	Action
Sep 4, 15:26	Researcher reports through HackerOne
Sep 4, 18:45	Security incident opened, production setup confirmed
Sep 4, 21:27	Runtime fix merged with reuse test
Sep 4, 23:15	Rollout begins
Sep 7, 06:13	Rollout complete, cleanup begins
Sep 14, 10:50	Researcher confirms PoC no longer works
Sep 19, 15:03	All pre-mitigation cached snapshots cleared

That's under 3 hours from report to confirmed root cause, and under 8 hours to a merged fix. The full fleet cleanup took 15 days — normal for rolling operations across a global infrastructure.

The speed matters. Every hour between a vulnerability report and a fix is an hour where exploitation is possible. If you operate your own container infra, build the runbook before you need it.

The Bottom Line

The Cloudflare container vulnerability was not a zero-day in the cryptographic sense. It was a known storage configuration tradeoff — performance over security — that was inappropriate for multi-tenant workloads. The fix was a single configuration change plus a fleet recycle.

If you run game backends on shared container infrastructure, audit your dm-thin pools today. Run the detection script against your I/O telemetry. And if you'd rather not manage container disk isolation, fleet recycling, and I/O telemetry pipelines yourself, horizOn handles authentication, crash reports, cloud save, leaderboards, and the other backend services your game needs — so you can focus on shipping your game, not debugging storage-layer security in production at 3 AM.

---

Source: How Cloudflare addressed a cross-tenant data exposure vulnerability in Containers