Voxel Streams and World Snapshots: Engineering a High-Performance User Generated Content Backend Architecture

Your players just spent 40 hours building a floating cathedral in your voxel RPG, and now they want to share it with 10,000 strangers—is your backend ready to foot the 4TB egress bill? Most developers treat User-Generated Content (UGC) as a simple file-upload problem, but when you are dealing with voxel-based worlds like Enshrouded, the technical reality is far more complex. You aren't just hosting a file; you are architecting a distributed world-state synchronization system that must remain performant, cost-effective, and secure.

The Carcinisation of Indie Games: The Shift from Game to Platform

The recent 'Forging the Path' update in Enshrouded introduces Adventure Sharing, a feature that allows players to package their world states and share them with the community. This move highlights a growing trend in the industry often referred to as the 'Roblox event horizon.' Games are no longer static experiences; they are becoming platforms for creation. This 'carcinisation' of games means that regardless of your genre, if you want long-term player retention, you will eventually have to deal with the technical debt of a user generated content backend architecture.

For a game like Enshrouded, which relies heavily on voxels, the challenge is double-fold. Voxels allow for infinite creativity and total destructibility, but they generate massive amounts of data. A single highly-detailed base can easily exceed 50MB of raw voxel data. Multiply that by 100,000 players, and your storage costs alone will kill your studio before the game even hits Version 1.0.

The Voxel Data Payload Problem

To understand how to architect a backend for this, we first need to look at what is actually being shared. In a voxel engine, the world is typically divided into chunks (e.g., 16x16x16 or 32x32x32). Each chunk contains voxel IDs, light data, and often metadata for entities like chests or crafting stations.

When a player 'shares an adventure,' the game must perform a 'World Snapshot.' This isn't just a copy-paste of the save folder. It requires:

1. Pruning: Removing transient data (like dropped items or active monster AI states) that doesn't need to be in the shared version.
2. Serializing: Converting the in-memory octree or grid structures into a flat buffer.
3. Compression: Applying algorithms like LZ4 or Zstandard to reduce the payload.

If you don't handle this correctly, you'll run into the same issues described in The Unreal Engine Multiplayer Sync Bug Ruining Your World States And How To Fix It, where mismatched world states between the client and the shared snapshot lead to corrupted structures or 'ghost' voxels.

Delta Compression: Only Ship What Changed

A common mistake in UGC architecture is uploading the entire world state every time a player makes a small update to their shared 'Adventure.' Instead, you should implement Delta Compression. By comparing the current state of the voxel chunks against the 'Base World' (the procedural seed), you only need to store the modifications (deltas).

If the base world says chunk (10, 5, 2) is a solid mountain, and the player dug a tunnel through it, your backend only needs to record the 'Air' voxels that replaced the 'Stone' voxels. This can reduce a 10MB chunk to a few kilobytes.

Engineering the Backend Pipeline

Once the client has generated a compressed snapshot or delta, the backend takes over. A robust user generated content backend architecture consists of three main layers: The Ingestion Layer, The Storage Layer, and The Discovery Layer.

1. The Ingestion Layer (Validation and Virus Scanning)

Never trust the client. A malicious user could upload a 'snapshot' that is actually a 2GB file filled with zeroes (a Zip Bomb) or a payload designed to exploit your voxel deserializer. Your ingestion server must:

• Validate the file header and size before accepting the full stream.
• Run the payload through a sandbox deserializer to ensure it doesn't crash the server.
• Generate a unique Content Hash (like SHA-256) to prevent duplicate uploads.

2. The Storage Layer (Blobs vs. Metadata)

Separate your binary data (the voxel blobs) from your searchable data (player name, adventure tags, rating).

• Blobs: Use S3-compatible object storage. For high-traffic games, use a Content Delivery Network (CDN) to cache these blobs at the edge.
• Metadata: Use a relational database like PostgreSQL for indexing. This allows players to search for 'High Fantasy' or 'Level 10-20' adventures with sub-millisecond latency.

3. The Discovery Layer (Real-Time Updates)

When a new adventure is published, you want other players to see it immediately. Rather than having clients poll the API every 30 seconds, use WebSockets to push new content notifications. For a deep dive on setting this up, check out our guide on how to Ditch Http Polling An Unreal Engine Websockets Tutorial For Real Time Backends.

Implementation: A C# UGC Manager Example

Below is a simplified example of how you might structure a UGC upload manager in a Unity-based game. This code handles the compression and the multi-part upload process to ensure large voxel files don't time out on slow connections.

using System;
using System.IO;
using System.IO.Compression;
using System.Threading.Tasks;
using UnityEngine;
using UnityEngine.Networking;

public class AdventureUploader : MonoBehaviour
{
    private const string API_URL = "https://api.yourgame.com/v1/ugc/upload";

    public async Task ShareAdventure(string adventureId, byte[] rawVoxelData, AdventureMetadata metadata)
    {
        // 1. Compress the data locally to save bandwidth
        byte[] compressedData = await CompressData(rawVoxelData);
        Debug.Log($"Compressed world state from {rawVoxelData.Length / 1024}KB to {compressedData.Length / 1024}KB");

        // 2. Create the Multi-part form
        WWWForm form = new WWWForm();
        form.AddField("adventureId", adventureId);
        form.AddField("title", metadata.Title);
        form.AddBinaryData("worldBlob", compressedData, "world.vox", "application/octet-stream");

        // 3. Send to the backend
        using (UnityWebRequest www = UnityWebRequest.Post(API_URL, form))
        {
            var operation = www.SendWebRequest();
            while (!operation.isDone) await Task.Yield();

            if (www.result != UnityWebRequest.Result.Success)
            {
                Debug.LogError($"UGC Upload Failed: {www.error}");
            }
            else
            {
                Debug.Log("Adventure shared successfully!");
            }
        }
    }

    private async Task<byte[]> CompressData(byte[] data)
    {
        using (var outputStream = new MemoryStream())
        {
            using (var gZipStream = new GZipStream(outputStream, CompressionMode.Compress))
            {
                await gZipStream.WriteAsync(data, 0, data.Length);
            }
            return outputStream.ToArray();
        }
    }
}

[Serializable]
public class AdventureMetadata
{
    public string Title;
    public string Description;
    public string CreatorId;
}

The Cost of Doing It Yourself

Architecting this pipeline manually is a significant undertaking. You need to manage:

• Load Balancers: To handle spikes when a famous streamer shares an adventure.
• Database Sharding: When your metadata table hits millions of rows.
• CDN Invalidations: Ensuring that when a player updates their adventure, the old version isn't served from the cache.

Building this yourself usually takes 2-3 months of dedicated senior engineering time. This is where horizOn provides massive value. Instead of building the 'plumbing' for binary blob storage and metadata indexing, horizOn provides a pre-architected UGC module. You simply define your metadata schema, and horizOn handles the global distribution, file validation, and scaling automatically. This lets you focus on making the 'Adventure' gameplay fun rather than worrying about S3 bucket policies.

5 Best Practices for UGC Backend Architectures

1. Implement Content-Addressable Storage (CAS): Use the hash of the voxel data as the filename. If two players share identical bases, you only store the physical file once, saving massive amounts of storage.
2. Use Asynchronous Ingestion: Don't make the player wait for the backend to process the file. Return a '202 Accepted' immediately and use a background worker to handle validation and thumbnail generation.
3. Tiered Storage Strategy: Keep the top 100 most popular adventures in a 'Hot' cache (Redis/CDN) and move older, unplayed adventures to 'Cold' storage (S3 Glacier) to keep costs low.
4. Version the Schema: As you update your game, your voxel format will change. Your backend must store a format_version integer with every upload so old adventures can be migrated or deprecated gracefully.
5. Rate Limit Everything: UGC is the easiest way to DDoS a game server. Implement strict rate limits on how often a single IP or PlayerID can upload or search for content.

Conclusion: The Future of Shared Worlds

As Enshrouded demonstrates, the technical bar for indie games is rising. Players expect to be able to share their creations seamlessly, and 'Adventure Sharing' is fast becoming a standard feature rather than a luxury. By focusing on a robust user generated content backend architecture early in development, you avoid the painful refactors that come when your community outgrows your infrastructure.

Ready to scale your multiplayer backend without the headache of managing raw servers? Try horizOn for free or check out the API docs to see how we handle high-volume UGC and world-state persistence.

---

Source: Enshrouded approaches the Roblox event horizon with the addition of Adventure Sharing in its latest update