ARCANUM
← Blog
architecture

Shared-nothing: how Hanamori's servers scale without locks

Every thread owns its data. Ring buffers are the only way work crosses between them. How Hanamori's meadow and realm servers are built that way.

12 Oct 2026 · 6 min read

Hanamori is the multiplayer action game we run on Rune. A meadow server hosts the shared overworld. Behind each portal, realm servers run instanced dungeons. Both must serve sockets, a 20 Hz simulation, password hashing and a database connection at once.

The usual answer is one shared world guarded by locks, which stalls when the server is busiest. Ours: no thread shares mutable state with another. Each owns its data, and a message in a ring buffer is the only way anything crosses between threads.

One owner per piece of state

ThreadOwnsDoes
NetworkSockets, sessionsuWebSockets loop: decodes client frames, drains each player's output. A pump thread triggers the flush at 30 Hz
SimThe world and the map data20 Hz tick: reads every player's input, simulates, writes updates back
AccountJoin and login stateChecks names and credentials. Argon2id runs here, so it stalls neither Rune nor Sim
RuneThe Rune client, subscriptions, producersReplays state on boot, writes records, runs transactions, tails the log

A realm server has the same shape, with several workers in place of one Sim. Each worker owns a pool of dungeon instances and the rings of the clients it accepted, and ticks them at 20 Hz. An instance id picks its worker, so an instance always lives on one thread.

Shared data (item tables, terrain variants) is loaded at startup, passed as const and never written again. The map data is not safe to read from two threads, so it stays on Sim.

The payoff: game logic is single-threaded. A pickup, a portal jump or a hit runs start to finish on one thread. No data races, no lock ordering, no half-applied state. Concurrency lives in one place, the rings.

Rings are the only bridge

NetworkuWebSockets loop Accountjoin · Argon2id Sim20 Hz tick Runetopics · transactions join · close character writes records admit positions · txns records per-client in per-client out ring (many producers, one consumer)
Threads in the meadow server and the rings between them

Every arrow is a fixed-size ring in the Agrona layout, the design behind Aeron. One implementation, many producers and one consumer, used two ways:

  • Inboxes. Each thread has one inbox ring (Account, Sim, Rune, and a command inbox for Network). Any thread that needs it writes there.
  • Per-client rings. Each player gets an in-ring and an out-ring when they join. Network writes decoded frames into the in-ring. Each tick Sim drains every in-ring, simulates, and writes updates into the out-ring, which Network drains to the socket. A busy player can fill their own ring, never someone else's.

Ring ownership moves with the connection: Network creates it, Account holds it while deciding, Sim owns it once the player is admitted. It is freed only after Network reports the connection closed. The realm follows the same protocol and frees rings on an explicit FREE_RINGS message, always the last word about a connection.

Inside a ring

One preallocated block, a power-of-two size, then a trailer of 12 cache lines holding tail, a cached head, head, a correlation counter and a consumer heartbeat, each on its own line. Records sit back to back with an 8-byte header (length, message type), padded to 8 bytes. Head and tail are 64-bit positions that only grow; the slot is position & (capacity - 1). The largest message is one eighth of the capacity.

A write is claim, copy, commit. This is the real path from Network to Sim:

// network thread: one client frame → this player's in-ring
const auto off = ctx.inRing->tryClaim(templateId, frame.size());
if (off < 0) {  // the client floods faster than Sim drains: drop
    metrics_.ringbufferClaimFailures.inc();
    return;
}
ctx.inRing->buffer().putBytes(off, frame.data(), static_cast<std::int32_t>(frame.size()));
ctx.inRing->commit(off);        // publish: Sim can now see it

// Sim thread, once per tick: drain every admitted player's in-ring
work += it->second.in->read(
    [&](std::int32_t type, auto& buf, std::int32_t idx, std::int32_t len) {
        onClientFrame(id, type, std::span(buf.buffer() + idx, len));
    }, MAX_INPUT_MSGS_PER_CONN);
  1. Claim. Producers race for the tail with one compare-and-swap; the loser retries. A full ring returns an error immediately.
  2. Mark in progress. The record's length is first stored as a negative number. The consumer stops at any non-positive length, so it never reads a half-written message.
  3. Commit. The producer stores the positive length with release ordering. The payload becomes visible with it. No fences, no locks.
  4. Consume. The consumer reads in order, zeroes the consumed bytes, and advances the head with one ordered store.

A record that would straddle the end gets a padding record and restarts at the beginning, so every message stays contiguous and handlers decode it in place.

Why it's fast

  • Nobody waits. No mutex on these paths. The worst case for a producer is a failed claim that returns at once.
  • Cache lines stay put. Only producers write the tail, only the consumer writes the head, each on its own line. Producers keep a cached head and re-read the real one only when the ring looks full.
  • Batching. A consumer drains a ring in one pass, up to a per-read limit, over contiguous memory.
  • No allocation on the hot path. Messages are encoded straight into the ring and decoded where they lie. The stock ring read allocates through a std::function, so the realm reads its dozens of rings per pass with an allocation-free read of its own.
  • Steady ticks. Sim and the realm workers tick at 20 Hz, the network flush at 30 Hz, on a frame-sync idle strategy that sleeps most of the interval and then yields until the frame is due. Account uses a backoff strategy and costs almost nothing when idle.

Backpressure is explicit

A full ring is a fact the producer handles. Each decides per message type:

  • Network never sleeps or retries in the event loop. A full in-ring drops that input frame (input is lossy by design) and counts it.
  • Sim's out-ring drops a self-state update, which the next tick replaces. A message the client cannot do without kicks the connection once; it reconnects and gets a full snapshot.
  • Admits, connection closes and Rune transactions are never dropped. They wait in an in-order backlog behind the full ring.

Trade-offs

  • A thread is the unit of scale. One Sim thread simulates the whole meadow. A realm server spreads instances over a configured number of workers, and capacity grows by adding realm servers, not by splitting a world across cores.
  • Ring sizes are configured. Capacity is fixed per ring, so overflow is a real case with its own handling.
  • Ownership must be explicit. Exactly one owner at a time, and a free only after the last user is confirmed gone.
  • Data is copied. Every crossing is a copy. For small, frequent messages that is far cheaper than the synchronisation it replaces.

Between processes the bridge is Rune, not rings: see Hanamori on Rune.

← All posts