MySQL Connection Pooling Caching Gone Wrong: Inside a 0.001% Bug That Broke Production

Your backend returns empty result sets for queries that always return data. The error rate is 0.001% — roughly one in every 100,000 requests. Your telemetry shows nothing. Traces show nothing. And the only clue you have is that it never happens when caching is disabled.

That is exactly the situation Cloudflare's Hyperdrive team found themselves in while taking MySQL support in their connection pooler to general availability. The bug they found is a masterclass in how MySQL connection pooling caching can silently break when two layers disagree about the format of the bytes moving over the wire. If you run a pooled, cached database layer for your game backend — and you should — this failure mode applies to you.

This post breaks down the bug, the protocol details that made it possible, how to detect and remediate it, and what it means for your own game backend architecture.

Why Connection Pooling Exists at All

Establishing a TCP + TLS + MySQL handshake to a database takes a minimum of five network round trips. If your game server opens a fresh connection per request, you are paying that cost on every leaderboard read, every save-game fetch, every inventory lookup.

Pooling fixes this by holding warm connections open and lending them out. MySQL is friendlier to this than Postgres — Postgres spawns an entire OS process per connection, while MySQL connections are comparatively cheap — but a popular game with thousands of concurrent players can still exhaust connection limits quickly.

Pooling introduces its own problem, though: session state leakage. If player A's query sets a session variable and player B gets the same connection, you have a correctness bug. Poolers solve this with connection cleanup between checkouts. MySQL provides a lightweight command for exactly this: COM_RESET_CONNECTION.

And that is where the first subtle bug in this story lives.

Bug #1: Encoding Drift After Connection Resets

The Hyperdrive team negotiated utf8 in the startup handshake. That worked — until players started using emojis in chat and character names, and the data came back mangled.

The problem: utf8 in MySQL is a deprecated alias for utf8mb3, which supports only 1–3 bytes per Unicode code point. Full UTF-8 (including emoji, CJK extensions, and most non-Latin scripts your global player base will use) requires up to 4 bytes per code point. The fix is negotiating utf8mb4 instead.

But here is the trap that catches nearly everyone:

-- Startup handshake: negotiate utf8mb4. All good.
-- Then, between checkouts, the pooler runs:
COM_RESET_CONNECTION;
-- Surprise: the server does NOT reset to the negotiated charset.
-- It resets to the SERVER DEFAULT, e.g. latin1.

COM_RESET_CONNECTION does not restore the connection to the character set you negotiated at startup. It resets to the server's default charset — which on many MySQL instances is still latin1. After the first reset, every pooled connection silently degrades to a 1990s-era encoding.

The remediation: re-apply your session configuration after every reset, not just at startup:

COM_RESET_CONNECTION;
SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci;
SET SESSION time_zone = '+00:00';

Rule of thumb: treat the connection as fully unconfigured after every reset, and re-assert everything you care about. Anything you set once at connect time will eventually leak away.

Bug #2: The 0.001% Empty-Response Bug

This is the one that took days to find. A high-volume customer reported a small but consistent rate of empty query responses where non-empty responses were guaranteed. The error occurred about 0.001% of the time — too rare to reliably capture an end-to-end trace, too frequent to ignore.

The team combed telemetry, increased trace rates, searched for network anomalies. Nothing. The one clue that cracked it: the bug only occurred on instances with caching enabled.

The Architecture That Made It Possible

Hyperdrive uses a split topology:

• Client node — runs near the Worker, handles local caching, forwards queries onward.
• Endpoint node — runs in the data center nearest the origin database, holds the actual connection pool.

Both nodes use the CDN cache local to their data center. When they run in the same data center, they share one cache. That sharing is a performance win — but it carries a hidden precondition: both nodes must cache the exact same bytes for the same query response.

The Protocol Mismatch

MySQL's client/server protocol has a Connection Phase where client and server negotiate capabilities — optional features like CLIENT_DEPRECATE_EOF, which changes how result sets are terminated:

• Without CLIENT_DEPRECATE_EOF: EOF packet after column definitions, then another EOF at the end of the result set.
• With it: the metadata EOF is omitted and the final EOF becomes an OK packet.

Hyperdrive supports the capability and negotiates it with the origin MySQL server. But the client library on the other side — say, mysql2 for Node.js, which does not currently support CLIENT_DEPRECATE_EOF — may not. So Hyperdrive must translate: rewrite result sets to insert EOF packets, convert OK packets back to EOFs, so the bytes match what the Worker's driver expects.

The consequence: the bytes Hyperdrive writes back to the client are in a different format than the bytes it receives from the origin server.

The Faulty Assumption

When Client caching was implemented for Postgres, the team baked in an assumption: you can cache exactly the bytes you wrote back to the client. For Postgres's wire protocol, that was true. For MySQL, it is not.

Now assemble the failure:

1. The Endpoint services query Q, caches response bytes in Hyperdrive capability format into the shared cache.
2. The Client (same data center) gets a request for Q, finds the cached entry, and serves it to the Worker without translating capabilities — because the bytes look like they were already written for the Worker.
3. The Worker's driver, which never negotiated CLIENT_DEPRECATE_EOF, misparses the OK-terminated result set — producing empty results, hangs, or protocol desyncs.

The bug fires only when Client and Endpoint colocate and one reads the other's cache entry. That is why it was 0.001% and invisible to ordinary tracing. It is the wire-protocol equivalent of a desync bug — two components holding different assumptions about shared state. If you have ever debugged state divergence between client prediction and server authority, the shape of this failure will feel familiar; it is the same class of problem described in fixing the Unreal Engine RPC replication issue that breaks your states, just at the database wire layer.

The fix, conceptually: cache in canonical format (what the origin sent), and translate to the requesting client's capabilities at read time. Never cache post-translation bytes when the translation depends on per-client negotiation.

Runbook: Detecting Cache-Corruption in Your Own Stack

If you run a pooler or cache layer in front of MySQL (or anything with a stateful wire protocol), here is how to apply this postmortem.

Detection

• Correlate errors with topology. The bug appeared only on instances where Client and Endpoint shared a data center and cache. Log which node served each response and whether it came from cache. If error rate correlates with cache hits, suspect format mismatch between writer and reader.
• Hunt rare-but-consistent errors, not spikes. A 0.001% error rate never triggers alerts. Set up a query in your error tracking for repeated occurrences of the same signature regardless of volume. Consistency at low rate is a stronger signal of a logic bug than a traffic-correlated spike.
• Test with capability asymmetry deliberately. Connect one test client with CLIENT_DEPRECATE_EOF and one without, enable caching, and hammer identical queries. Most protocol-translation bugs surface within minutes under this asymmetry.

Remediation

• Canonicalize cache entries. Store the origin's raw response format; translate on read. This is the correct fix for any pooler whose clients have heterogeneous capabilities.
• Re-assert session state after resets. As shown above, COM_RESET_CONNECTION restores server defaults, not your negotiated settings.
• Version your cache keys. If you change any protocol behavior (capabilities, encoding, response framing), include a format version in the cache key so old entries cannot be served by new code mid-deploy.

# Cache key hygiene, in pseudocode
def cache_key(query_normalized: str, schema_version: int, format_version: int) -> str:
    # format_version bumps whenever wire-format handling changes,
    # preventing stale-format entries from poisoning new deploys
    return sha256(f"{schema_version}:{format_version}:{query_normalized}")

Prevention

• Write wire-protocol conformance tests against two client libraries, not one. The CLIENT_DEPRECATE_EOF gap was only visible because a popular Node.js driver didn't support it.
• Assume nothing about "the same bytes" across layers. Any component that rewrites messages must not assume its output is byte-identical to its input.
• Load-test with cache enabled from day one. The Hyperdrive team validated reliability for months before the cache-specific bug surfaced — because the intersection of features (pooling × caching × capability translation) is where rare bugs live.

Five Best Practices for Pooling and Caching in Game Backends

1. Always negotiate utf8mb4, never utf8. Player names, chat, and save data will contain 4-byte characters. Then re-assert SET NAMES utf8mb4 after every COM_RESET_CONNECTION, because the reset restores server defaults, not your handshake.
2. Cache canonical data, not client-formatted data. If any layer transforms a response before sending it, the transformed bytes must never enter a shared cache. Store the origin format; translate per-reader.
3. Add format versions to cache keys. One-line change, saves you from stale-protocol entries during rolling deploys.
4. Alert on error consistency, not just error rate. A 0.001% recurring signature with no traffic correlation is a logic bug. A traffic-correlated spike is a capacity problem. Route them differently.
5. Test clients with mismatched capabilities. Your pooler's correctness depends on handling every client library your team (or your SDK users) might connect with.

The Backend Architecture Lesson

The deepest lesson here is not about MySQL at all. It is about layered systems with per-peer negotiation. The moment you insert a middlebox — a pooler, a proxy, a cache — that speaks protocol on both sides, you inherit the obligation to translate faithfully and to cache only formats that are universally valid. Postgres worked; MySQL broke; nothing about the caching logic was wrong except one assumption inherited from a different protocol.

For a game studio, this is a strong argument against hand-rolling your own pooler-plus-cache layer. The engineering effort to get the happy path working is maybe a week; the engineering effort to get the rare paths right — capability negotiation, encoding drift, cache co-location — is what took a dedicated infrastructure team days of investigation to hunt down a single 0.001% bug. Cloudflare's own admission is worth repeating: the difference between beta and GA was "just 20 bug fixes in a trenchcoat."

If your game backend needs reliable persistence without you maintaining a pooler, a cache invalidation scheme, and a protocol-translation layer, that is where a Backend-as-a-Service earns its keep. horizOn handles account-bound Cloud Save with revision-aware reads and writes — your client resolves conflicts, and you skip the entire pooled-connection-plus-cache correctness minefield for player data. Pair it with leaderboards and remote configuration, and the query-latency problems discussed in this post mostly stop being your problem.

Next step: audit your own pooler config today. Check three things — the negotiated charset, whether session state is re-asserted after connection resets, and whether your cache stores canonical or translated bytes. If you can't answer all three in ten minutes, you have work to do — and if you'd rather not do it at all, give horizOn a look and see if its Cloud Save and leaderboards cover what your database layer currently does by hand.

---

Source: Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug