Bots Survived the Yandex Outage — Your Game Backend Architecture Should Too

Your monitoring dashboard shows traffic dropped 40% overnight. You breathe a sigh of relief — the DDoS must have stopped. But Cloudflare's analysis of the October 2023 Yandex outage proved this assumption catastrophically wrong: malicious bots didn't die when Yandex data centers went dark. They migrated to OVH, Hetzner, and DigitalOcean within minutes. Your aggregate metrics lied to you while attackers kept running.

If your game backend runs on a single cloud provider with IP-based bot detection, you're one infrastructure event away from the same blind spot. This runbook breaks down what the Yandex outage reveals about bot behavior during infrastructure failures, and how to architect game backends that survive them.

What Breaks: When Your Infrastructure Provider Goes Dark

When a major infrastructure provider experiences an outage, four critical systems fail simultaneously — and they fail in ways that look like improvements to naive monitoring.

Your authentication layer goes offline. If your player authentication depends on a single provider's identity service, players can't log in. During the Yandex outage, services tightly coupled to Yandex infrastructure saw complete service interruption. Players got error screens; bots that don't authenticate kept hitting endpoints elsewhere.

Your bot detection loses its reference frame. Most game backends track malicious IPs by ASN or hosting provider. When those IPs go dark, your detection systems mark them as "resolved" and stop looking. Meanwhile, the same bot fleet re-appears on new IPs at a different provider.

Your rate limiting becomes useless. IP-based rate limits are tied to specific address ranges. When bots migrate, they get fresh IPs with clean reputation scores. Your rate limiter treats them as new legitimate traffic.

Your traffic aggregates become misinformation. Cloudflare's Yandex analysis showed AS13238 (Yandex Search) traffic consisted almost entirely of legitimate crawlers — the drop there was real. But AS200350 (Yandex.Cloud) saw malicious vulnerability scanning and probing decline while high-volume automation simply relocated. Aggregate traffic charts showing a 99% drop told you the wrong story.

The core failure mode: your monitoring thinks things got better when the threat just moved.

How to Detect: Spotting Bot Migration in Real-Time

Detection requires monitoring at multiple layers, not just aggregate traffic volume. Each layer catches a different aspect of the migration pattern.

Layer 1: Traffic Source Correlation

Instead of tracking total requests per second, monitor the distribution of traffic sources across autonomous system numbers (ASNs). When one provider's traffic drops, check if other providers' traffic increases proportionally. A correlated drop-and-rise pattern across ASNs is the signature of bot migration.

Here's a Python detector that identifies this pattern:

import time
from collections import defaultdict
from dataclasses import dataclass
from typing import Dict, List, Optional

@dataclass
class TrafficSnapshot:
    timestamp: float
    source_asn: str
    request_count: int
    unique_ips: int
    error_rate: float

class BotMigrationDetector:
    """Detects correlated traffic drops and rises across ASNs
    to identify bot farm migration during infrastructure failures."""

    def __init__(self, migration_threshold: float = 0.3):
        self.history: Dict[str, List[TrafficSnapshot]] = defaultdict(list)
        self.migration_threshold = migration_threshold
        self.window_seconds = 300  # 5-minute rolling window

    def record_snapshot(self, snapshot: TrafficSnapshot):
        self.history[snapshot.source_asn].append(snapshot)
        self._prune_old_data()

    def _prune_old_data(self):
        cutoff = time.time() - self.window_seconds
        for asn in self.history:
            self.history[asn] = [
                s for s in self.history[asn]
                if s.timestamp > cutoff
            ]

    def detect_migration(self) -> Optional[Dict]:
        """Returns migration report if suspicious pattern detected."""
        declining_asns = []
        rising_asns = []

        for asn, snapshots in self.history.items():
            if len(snapshots) < 2:
                continue

            mid = len(snapshots) // 2
            first_half = snapshots[:mid]
            second_half = snapshots[mid:]

            avg_first = sum(s.request_count for s in first_half) / len(first_half)
            avg_second = sum(s.request_count for s in second_half) / len(second_half)

            if avg_first > 0:
                change_ratio = (avg_second - avg_first) / avg_first

                if change_ratio < -self.migration_threshold:
                    declining_asns.append({
                        'asn': asn,
                        'drop_pct': abs(change_ratio) * 100,
                        'baseline': avg_first
                    })
                elif change_ratio > self.migration_threshold:
                    rising_asns.append({
                        'asn': asn,
                        'rise_pct': change_ratio * 100,
                        'current': avg_second
                    })

        if declining_asns and rising_asns:
            return {
                'type': 'bot_migration',
                'declining': declining_asns,
                'rising': rising_asns,
                'confidence': self._calculate_confidence(declining_asns, rising_asns)
            }
        return None

    def _calculate_confidence(self, declining, rising) -> float:
        """Higher confidence when drop and rise magnitudes match."""
        total_drop = sum(d['drop_pct'] for d in declining)
        total_rise = sum(r['rise_pct'] for r in rising)

        if total_drop == 0:
            return 0.0

        ratio = min(total_rise, total_drop) / max(total_rise, total_drop)
        return round(ratio * 100, 2)


# Usage: detect Yandex-style migration
detector = BotMigrationDetector(migration_threshold=0.25)

# Yandex traffic collapsing
detector.record_snapshot(TrafficSnapshot(
    timestamp=time.time() - 240,
    source_asn="AS13238", request_count=15000,
    unique_ips=8500, error_rate=0.02
))
detector.record_snapshot(TrafficSnapshot(
    timestamp=time.time() - 120,
    source_asn="AS13238", request_count=1200,  # 92% drop
    unique_ips=900, error_rate=0.45
))

# OVH traffic surging (bots migrated here)
detector.record_snapshot(TrafficSnapshot(
    timestamp=time.time() - 240,
    source_asn="AS16276", request_count=5000,
    unique_ips=3200, error_rate=0.01
))
detector.record_snapshot(TrafficSnapshot(
    timestamp=time.time() - 120,
    source_asn="AS16276", request_count=18500,  # 270% increase
    unique_ips=2800, error_rate=0.08
))

migration = detector.detect_migration()
if migration:
    print(f"Bot migration detected: {migration['confidence']}% confidence")
    for d in migration['declining']:
        print(f"  {d['asn']}: -{d['drop_pct']:.0f}%")
    for r in migration['rising']:
        print(f"  {r['asn']}: +{r['rise_pct']:.0f}%")

This detector watches for correlated traffic drops and rises across multiple ASNs. When AS13238 drops 92% and AS16276 simultaneously rises 270%, it flags probable bot migration rather than legitimate traffic changes. The confidence score increases when drop and rise magnitudes match — a hallmark of coordinated migration.

Layer 2: Behavioral Fingerprinting

Don't just track where traffic comes from — track how it behaves. Bots migrating between providers retain identical behavioral signatures: the same request intervals, header patterns, session durations, and error-retry behavior. IP reputation resets when they move; behavior does not.

Build a behavioral fingerprint for each session and hash it:

import hashlib
import statistics

def fingerprint_session(requests: list[dict]) -> str:
    """Generate a behavioral fingerprint from a session's request pattern."""
    intervals = []
    for i in range(1, len(requests)):
        delta = requests[i]['timestamp'] - requests[i-1]['timestamp']
        intervals.append(round(delta, 3))

    if len(intervals) < 2:
        return "insufficient_data"

    # Key behavioral features
    interval_mean = round(statistics.mean(intervals), 3)
    interval_stdev = round(statistics.stdev(intervals), 3)
    total_duration = round(requests[-1]['timestamp'] - requests[0]['timestamp'], 3)
    error_count = sum(1 for r in requests if r.get('status_code', 200) >= 400)
    retry_count = sum(1 for r in requests if r.get('is_retry', False))
    unique_paths = len(set(r['path'] for r in requests))
    user_agent_count = len(set(r.get('user_agent', '') for r in requests))

    fingerprint_data = (
        f"{interval_mean}|{interval_stdev}|{total_duration}|"
        f"{error_count}|{retry_count}|{unique_paths}|{user_agent_count}"
    )

    return hashlib.sha256(fingerprint_data.encode()).hexdigest()[:16]


# Example: same bot session detected at two different providers
session_provider_a = [
    {'timestamp': 1000.0, 'path': '/api/login', 'status_code': 200},
    {'timestamp': 1000.5, 'path': '/api/profile', 'status_code': 200},
    {'timestamp': 1001.0, 'path': '/api/match/join', 'status_code': 429},
    {'timestamp': 1001.5, 'path': '/api/match/join', 'status_code': 200, 'is_retry': True},
    {'timestamp': 1002.0, 'path': '/api/score', 'status_code': 200},
]

session_provider_b = [
    {'timestamp': 2000.0, 'path': '/api/login', 'status_code': 200},
    {'timestamp': 2000.5, 'path': '/api/profile', 'status_code': 200},
    {'timestamp': 2001.0, 'path': '/api/match/join', 'status_code': 429},
    {'timestamp': 2001.5, 'path': '/api/match/join', 'status_code': 200, 'is_retry': True},
    {'timestamp': 2002.0, 'path': '/api/score', 'status_code': 200},
]

fp_a = fingerprint_session(session_provider_a)
fp_b = fingerprint_session(session_provider_b)

print(f"Provider A fingerprint: {fp_a}")
print(f"Provider B fingerprint: {fp_b}")
print(f"Same bot: {fp_a == fp_b}")

When you see the same behavioral fingerprint appearing on new IPs at a different provider shortly after the original provider went down, that's confirmed migration — not coincidence.

Layer 3: Authentication State Monitoring

Track the ratio of authenticated vs. unauthenticated requests over time. During infrastructure failures, legitimate players struggle to authenticate (shifting the ratio toward unauthenticated), while bots that never authenticate continue hitting your endpoints unchanged.

This layer is particularly valuable because it separates real players from automated traffic regardless of source provider. If you're using horizOn's account-bound sessions with multiple authentication methods (email, Google, Apple, anonymous), you can monitor authentication success rates per method. When email auth fails but Google auth succeeds, you've identified a partial infrastructure failure — not a bot migration. When all auth succeeds but unauthenticated traffic spikes at a new provider, that's migration.

How to Remediate: Failover Strategies That Work

Detection without remediation is just watching your game die in real-time. Here are three failover mechanisms you can implement this week.

DNS-Based Failover

Configure your DNS with short TTLs and health checks. When your primary region fails, DNS automatically routes traffic to secondary regions. This doesn't require any application changes.

Recommended settings:

• TTL: 60-120 seconds (shorter means faster failover, but more DNS load)
• Health check interval: 10-15 seconds
• Health check endpoint: /health returning 200 with < 500ms response time
• Failover trigger: 3 consecutive health check failures

The tradeoff: short TTLs increase DNS query volume by 10-20x during normal operation. For a game with 50,000 daily active players, this adds roughly $2-5/month in DNS costs — negligible compared to the downtime it prevents.

Application-Level Circuit Breakers

Implement circuit breakers in your game client that detect when a service is failing and route to alternatives automatically:

using System;
using System.Net.Http;
using System.Threading.Tasks;

public class ServiceCircuitBreaker
{
    private readonly string _primaryEndpoint;
    private readonly string[] _fallbackEndpoints;
    private readonly int _failureThreshold;
    private int _failureCount;
    private DateTime _lastFailure;
    private CircuitState _state = CircuitState.Closed;
    private int _currentFallbackIndex;
    private readonly object _lock = new();

    public enum CircuitState { Closed, Open, HalfOpen }

    public ServiceCircuitBreaker(
        string primary, string[] fallbacks, int failureThreshold = 5)
    {
        _primaryEndpoint = primary;
        _fallbackEndpoints = fallbacks;
        _failureThreshold = failureThreshold;
    }

    public async Task<HttpResponseMessage> SendRequest(HttpRequestMessage request)
    {
        lock (_lock)
        {
            if (_state == CircuitState.Open)
            {
                if (DateTime.UtcNow - _lastFailure > TimeSpan.FromSeconds(30))
                    _state = CircuitState.HalfOpen;
                else
                    return SendToFallback(request).GetAwaiter().GetResult();
            }
        }

        try
        {
            using var client = new HttpClient();
            client.Timeout = TimeSpan.FromSeconds(5);
            var response = await client.SendAsync(request);

            if (response.IsSuccessStatusCode)
            {
                lock (_lock) { _failureCount = 0; _state = CircuitState.Closed; }
                return response;
            }

            RecordFailure();
            return await SendToFallback(request);
        }
        catch (TaskCanceledException)
        {
            RecordFailure();
            return await SendToFallback(request);
        }
    }

    private void RecordFailure()
    {
        lock (_lock)
        {
            _failureCount++;
            _lastFailure = DateTime.UtcNow;
            if (_failureCount >= _failureThreshold)
                _state = CircuitState.Open;
        }
    }

    private async Task<HttpResponseMessage> SendToFallback(HttpRequestMessage request)
    {
        for (int i = 0; i < _fallbackEndpoints.Length; i++)
        {
            int idx = (_currentFallbackIndex + i) % _fallbackEndpoints.Length;
            try
            {
                var fallbackRequest = new HttpRequestMessage(
                    request.Method,
                    _fallbackEndpoints[idx] + request.RequestUri.PathAndQuery
                );
                using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(5) };
                var response = await client.SendAsync(fallbackRequest);
                if (response.IsSuccessStatusCode)
                {
                    _currentFallbackIndex = idx;
                    return response;
                }
            }
            catch { continue; }
        }

        throw new HttpRequestException("All endpoints failed");
    }
}

// Usage in your game client
var breaker = new ServiceCircuitBreaker(
    primary: "https://api-primary.game.example.com",
    fallbacks: new[] {
        "https://api-secondary.game.example.com",
        "https://api-tertiary.game.example.com"
    },
    failureThreshold: 3
);

var request = new HttpRequestMessage(HttpMethod.Get, "/api/leaderboard/global");
var response = await breaker.SendRequest(request);

This pattern prevents cascading failures when one provider goes down. As we covered in our analysis of architecting server fallbacks, automatic failover is the difference between a 30-second hiccup and a 4-hour outage.

Distributed Rate Limiting

When traffic fails over to a new region, your rate limiting needs to work there too. IP-based rate limits that are local to each server become useless during failover because the same bot gets a clean slate at every endpoint.

Implement distributed rate limiting using a shared store accessible from all regions:

import redis
import time

class DistributedRateLimiter:
    """Sliding window rate limiter backed by Redis.
    Works identically across all deployment regions."""

    def __init__(self, redis_url: str, max_requests: int = 100,
                 window_seconds: int = 60):
        self.redis = redis.from_url(redis_url)
        self.max_requests = max_requests
        self.window = window_seconds

    def is_allowed(self, identifier: str) -> tuple[bool, int]:
        """Returns (allowed, remaining_requests)."""
        key = f"ratelimit:{identifier}"
        now = time.time()
        window_start = now - self.window

        pipe = self.redis.pipeline()
        pipe.zremrangebyscore(key, 0, window_start)  # Remove expired entries
        pipe.zadd(key, {f"{now}:{id(identifier)}": now})  # Add current request
        pipe.zcard(key)  # Count requests in window
        pipe.expire(key, self.window)
        results = pipe.execute()

        request_count = results[2]
        remaining = max(0, self.max_requests - request_count)

        if request_count > self.max_requests:
            # Remove the request we just added — it's over limit
            pipe.zremrangebyscore(key, now, now)
            pipe.execute()
            return False, 0

        return True, remaining


# Usage: same limiter works from any region
limiter = DistributedRateLimiter(
    redis_url="redis://shared-redis.internal:6379",
    max_requests=60,   # 1 request per second average
    window_seconds=60
)

# Identify by player ID, not IP — survives migration
allowed, remaining = limiter.is_allowed("player:8f3a2b1c")

Notice the identifier uses player ID, not IP address. Bots migrating to new IPs still carry the same player fingerprint. This approach catches them even after provider-level migration.

How to Prevent: Architecture Patterns for Resilience

Remediation is reactive. Prevention means building architecture that assumes infrastructure failure will happen — because it will.

Pattern 1: Multi-Provider Deployment for Critical Services

Deploy your authentication, matchmaking, and bot detection across at least two cloud providers. This doesn't mean running your entire stack twice — it means the services that must survive an outage are provider-independent.

Cost reality: Multi-provider deployment for critical services typically adds 30-50% to your infrastructure bill. For a game backend serving 10,000 concurrent players, that's roughly $800-1,500/month additional. The cost of a 2-hour outage for the same game — accounting for lost IAP revenue, player churn, and support tickets — averages $5,000-12,000.

Minimum viable multi-provider:

• Authentication: primary + secondary provider
• Player data / save state: replicated across providers
• Matchmaking: at least one fallback region
• Leaderboards and analytics: can tolerate brief outages (queue and replay)

Pattern 2: Stateless, Provider-Agnostic Bot Detection

Design your bot detection to work identically regardless of where traffic originates. The Yandex outage proved that IP-based detection is fundamentally fragile — bots get new IPs the moment they migrate.

Build detection around behavioral features that survive infrastructure changes:

from dataclasses import dataclass
from typing import List

@dataclass
class SessionFeatures:
    request_count: int
    duration_seconds: float
    interval_variance: float      # Low variance = bot-like regularity
    error_retry_rate: float       # Bots retry aggressively; humans quit
    unique_endpoints: int         # Bots hit predictable endpoint patterns
    header_consistency: float     # Bots reuse identical headers

class BehavioralClassifier:
    """Bot classifier based on behavioral features, not IP addresses.
    Survives provider migration because behavior doesn't change."""

    # Weights calibrated from production bot traffic analysis
    WEIGHTS = {
        'interval_variance': 0.30,   # Most discriminating feature
        'error_retry_rate': 0.25,
        'header_consistency': 0.20,
        'endpoint_diversity': 0.15,
        'session_duration': 0.10,
    }

    def classify(self, features: SessionFeatures) -> float:
        """Returns bot probability from 0.0 (human) to 1.0 (bot)."""
        score = 0.0

        # Low interval variance → bot (humans are irregular)
        if features.interval_variance < 0.1:
            score += self.WEIGHTS['interval_variance']
        elif features.interval_variance < 0.3:
            score += self.WEIGHTS['interval_variance'] * 0.5

        # High retry rate → bot (humans give up after 1-2 failures)
        if features.error_retry_rate > 0.8:
            score += self.WEIGHTS['error_retry_rate']
        elif features.error_retry_rate > 0.5:
            score += self.WEIGHTS['error_retry_rate'] * 0.5

        # Perfect header consistency → bot (humans have browser variations)
        if features.header_consistency > 0.95:
            score += self.WEIGHTS['header_consistency']

        # Very few unique endpoints → bot (humans explore)
        if features.unique_endpoints <= 3 and features.request_count > 20:
            score += self.WEIGHTS['endpoint_diversity']

        # Suspiciously precise session duration (e.g., exactly 300s)
        if abs(features.duration_seconds % 60) < 2:
            score += self.WEIGHTS['session_duration']

        return min(score, 1.0)


# Usage
classifier = BehavioralClassifier()

bot_session = SessionFeatures(
    request_count=150, duration_seconds=300.0,
    interval_variance=0.02, error_retry_rate=0.95,
    unique_endpoints=2, header_consistency=0.99
)

human_session = SessionFeatures(
    request_count=45, duration_seconds=1847.3,
    interval_variance=0.67, error_retry_rate=0.15,
    unique_endpoints=12, header_consistency=0.73
)

print(f"Bot score: {classifier.classify(bot_session):.2f}")   # ~0.85
print(f"Human score: {classifier.classify(human_session):.2f}")  # ~0.10

This classifier doesn't care which ASN the request comes from. When a bot migrates from Yandex.Cloud to OVH, its behavioral fingerprint stays the same — and the classifier still catches it.

Pattern 3: Decouple Authentication from Infrastructure

Your authentication system should survive any single infrastructure failure. Use multiple authentication providers so players have fallback options for accessing their accounts.

Implement authentication with graceful degradation:

class MultiProviderAuth:
    """Authentication with automatic provider fallback."""

    def __init__(self, providers: list):
        # Order by reliability — primary first
        self.providers = providers

    async def authenticate(self, credentials: dict) -> dict:
        last_error = None

        for provider in self.providers:
            try:
                result = await provider.authenticate(credentials)
                return {
                    'success': True,
                    'provider': provider.name,
                    'player_id': result['player_id'],
                    'session_token': result['session_token']
                }
            except Exception as e:
                last_error = e
                continue

        return {
            'success': False,
            'error': 'All authentication providers failed',
            'last_error': str(last_error)
        }

This pattern matters because during the Yandex outage, services tightly coupled to Yandex identity infrastructure couldn't authenticate users at all. Players got locked out; bots that don't authenticate kept running. A multi-provider auth strategy means your players keep playing even when your primary auth provider is down.

Pattern 4: Observability That Spans Providers

Your monitoring needs to see across all providers simultaneously. Split monitoring into separate concerns:

Traffic distribution monitoring: Track request volume per ASN, per region, per provider. Alert when the distribution shifts by more than 20% in a 5-minute window.

Authentication health monitoring: Track auth success rates per method and per provider. Alert when any provider's success rate drops below 95%.

Bot detection monitoring: Track the ratio of flagged-to-total sessions per provider. Alert when the ratio drops below baseline at one provider while rising at another — that's migration.

Cross-provider correlation: This is the critical piece. Individual provider metrics can look healthy while the overall system is under attack. Build dashboards that show all providers side-by-side so you can spot migration patterns at a glance.

Best Practices: 5 Steps to Bot-Resilient Game Backend Architecture

1. Monitor migration, not just volume. Set up cross-provider traffic correlation that detects when traffic drops in one place and rises in another. Aggregate metrics alone would have fooled you during the Yandex outage — AS200350 traffic dropped, but malicious activity continued at OVH and Hetzner.

2. Implement circuit breakers on every external dependency. Your game client, leaderboard service, and matchmaking logic all need automatic failover. The circuit breaker pattern prevents cascading failures when one provider goes down. Test them monthly by killing endpoints in staging.

3. Use behavioral detection, not IP detection. Behavioral analysis (request intervals, retry rates, header patterns) works regardless of where bots are hosted. IP-based detection fails the moment bots migrate to a new provider with clean IP reputation. The behavioral classifier above catches bots that survive 99% traffic drops.

4. Design authentication for graceful degradation. Use horizOn's account-bound sessions with multiple authentication methods (email, Google, Apple, anonymous). When one auth provider is unavailable, players can use another method to maintain access to their game state. This prevents the worst-case scenario: players locked out while bots keep running.

5. Test your failover quarterly. Simulate provider outages in staging. Verify that DNS failover triggers within your TTL, circuit breakers open correctly, and rate limiting works in fallback regions. Document the time from failure detection to full recovery. Untested failover is not failover — it's hope.

Conclusion

The Yandex outage proved that malicious actors have better infrastructure resilience than most game backends. Bots didn't die when Yandex data centers went dark — they migrated to other providers within minutes and kept running. Your aggregate traffic dashboard showed improvement. Reality got worse.

Start by deploying the behavioral bot detector from this article in your staging environment. Run it for one week to establish baselines, then simulate a provider outage by blocking traffic from one ASN and observing whether the migration detector fires. The patterns you discover will reveal exactly where your current architecture is vulnerable.

For game studios building authentication and player data systems that survive infrastructure failures, horizOn provides account-bound sessions with revision-aware cloud save — services designed so your players keep their progress even when the unexpected happens. Try it free or explore the API documentation to see how account-bound sessions integrate with your existing backend.

---

Source: Yandex outage: Data shows that instead of going dark, malicious bots went on the move