Hanamori on Rune, part 3: the realm handoff
One hero from the shared field into a dungeon and back, across two servers that never connect: ownership records, heartbeats, tickets, rewards and every failure branch. With the real code.
In part 1 we met topics, views and the transaction queue. In part 2 we followed the state that changes constantly. This part follows one hero from the meadow (Hanamori's open-world server), through a portal, into a dungeon run by a different server, and back with the loot. Two servers that never talk have to agree, at every moment, on who currently owns the hero, and they have to keep agreeing when packets vanish. The network, as usual, had other plans.
1. Who owns the hero? Ownership records
Ownership is character_location (compact, key = characterId). Each record has an owner (0 meadow, 1 realm), a state (0 inMeadow, 1 pending,
2 inRealm), the serverId and instanceId, an ownerEpoch that counts handoffs, the meadow position to return to (returnX/Z/Yaw), a deadline, and three counters, xpRecs, goldRecs and
invRecs, which come back in section 7.
Both servers fold this topic: they read the records in log order and keep a state machine per character (hloc::Fold). A record is applied only if hloc::valid allows the move. Everything else is void and ignored.
Because the log has a single order, two servers reading it reach the same answer without ever speaking.
| Move | Written by | Rule |
|---|---|---|
| inMeadow@N to pending@N+1 | meadow | owner = Realm, non-zero serverId and instanceId |
| pending to inRealm | realm | same epoch, server and instance |
| pending or inRealm to inMeadow@N+1 | realm (leave) or meadow (reclaim) | owner = Meadow, same serverId |
| deadline reclaim | meadow | valid only over pending; void once inRealm has folded |
| identical record | either | a valid no-op (a "reassertion") |
When a record folds as void, its writer reasserts its own state (reassertAfterVoid). Reason: this is a compacted topic, and a void record could otherwise be the latest per key and eventually be
the only thing left in the log. Reasserting makes sure compaction keeps a valid record.
2. Heartbeats and realm liveness
Each realm writes a beat to realm_servers once a second (kBeatMs = 1000) on its fast producer, key = serverId. The beat carries publicUrl, maxInstances, liveInstances,
freeSeats (max(0, maxInstances * 7 - sum(maxPlayers))), draining, startedAt and beatAt. The meadow's RealmDirectory keeps the latest beat per server and applies two rules:
- Dead after 3 s.
DEAD_AFTER_MS= 3000 since the last beat.newlyDeadreports each death exactly once, until that server beats again. - Who gets the next dungeon.
pick(need, now, exclude)returns a live, non-draining server withliveInstances < maxInstancesandfreeSeats >= need, preferring the most free seats and breaking ties by the lowest id.
The topic is transient. After a broker restart it is empty, and every realm repopulates it within a second, which is faster than any human will complain.
3. Asking for a dungeon
The request is a record on realm_requests with kind 1 (create) or 2 (cancel), a targetServerId, portalId, instanceId, seed, rank, theme, maxPlayers,
partySize, expiresAt and a random requestId. The realm subscribes with a broker-side filter on targetServerId, so the broker never sends it other servers' requests.
The answer goes to realm_events (key instanceId) with a kind: 1 created, 2 rejected, 3 playerJoined, 4 playerLeft, 5 ended, 6 cleared.
Who shares a dungeon is decided by PortalBindings in the meadow. A solo hero gets a dedicated binding (maxPlayers 1). A party shares one joinable binding per (portal, party), with a cap per rank
(DEFAULT_RANK_CAPS = 3, 4, 5, 7 and MAX_PARTY = 7). Each seat is Empty, Waiting, Ticketed or Inside.
On the realm, onRequest ignores expired requests. onCreate checks the sizes (maxPlayers 1 to 7, partySize not above maxPlayers) and rejects with BAD_REQUEST, DRAINING or FULL; otherwise
it mints an instance id with makeInstanceId and sends the create to a worker thread.
// InstanceId.h: [serverId:16][incarnation:16][counter:32] // the incarnation field separates ids minted by different runs of the same server uint64_t makeInstanceId(uint16_t serverId, uint16_t incarnation, uint32_t counter);
Duplicates are expected, because the network likes to repeat itself. A repeated request is answered from the realm's memory (kRequestMemoryMs = 60 s). A cancel makes the worker end the instance, and an unknown or already-ending
instance is answered with ended again. On the meadow side, localReq maps the wire request id back to the seat that asked. An id it does not know is history being replayed after a reset, so it is ignored, and in particular it is never cancelled.
If the realm does not answer within CREATE_TIMEOUT_MS (5 s) the binding retries on a different server, up to MAX_TRIES = 3 times. After that, the player gets REALM_UNAVAILABLE with NO_SERVER. An idle live binding is probed every 5 s with a cancel,
so a dungeon nobody uses does not hold seats forever.
4. The ticket
Handoff::issue builds one transaction with three records, in this order:
character_location:pendingat epoch+1, owner Realm,deadline= now + 20 s (PENDING_DEADLINE_MS), andreturnX/Z/Yawset to the portal's spot.realm_tickets: the hash (SHA-256 hex) of 32 random bytes fromRAND_bytes,expiresAt= now + 15 s (TICKET_TTL_MS), plus a snapshot of the hero: name, stats, gold,xpTotaland up to 60 inventory slots packed into 29-byteTicketSlots (packTicketInventory).meadow_positions: one last position record, at the portal.
The raw secret lives only in memory (tickets_). The log has only its hash, and the meadow wipes the raw bytes with OPENSSL_cleanse after sending them.
// Handoff (shape): the redirect has two preconditions void tryRedirect(Seat& s) { if (!s.written) return; // the ticket transaction has committed if (!s.ownPendingFolded) return; // our own pending record came back through the fold host_.toAway(s); // remove entity, keep socket as "away", write final position send(RealmRedirect{ s.instanceId, s.portalId, s.url /* from the beat */, s.rawTicket }); cleanse(s.rawTicket); // OPENSSL_cleanse }
Why both conditions? "Committed" says the data is in the log. "Folded" says the meadow itself has seen its own record as valid, so it will not send a hero into a dungeon on a claim the log has already
disowned. toAway removes the entity from the world but keeps the WebSocket open as an "away" socket, so the player can still get a reply. If the ticket transaction fails and the record never folded, the player gets NO_SERVER and the seat is freed.
On the browser side, netEnterPortal sends EnterPortal and resends every 150 ms for up to 1500 ms (ENTRY_RESEND_MS) until the server answers, because the server silently ignores an enter request from a hero who is out of range. The answers are
PortalWait (the queue status), RealmRedirect, PortalFull or RealmUnavailable. A waiting seat is released if the hero strays past 6 m (WAIT_RANGE), and the client mirrors that rule.
// hanamori client: parts/19d-net.js (shape) case 'RealmRedirect': const ws = new WebSocket(msg.url); ws.onopen = () => ws.send(encode('JoinRealm', { instanceId: msg.instanceId, ticket: msg.ticket, bakeId, protocolVersion: 3 })); // RealmLayout means we are in; RealmJoinFail / RealmEnded mean we are not.
5. Joining the dungeon
The browser opens a WebSocket to the realm and presents the raw ticket. The realm hashes it with SHA-256 and runs checkTicket. The checks run in this order: version, present, hash (constant-time compare), used, instance, expiry (now >= expiresAt).
Failures map to a JoinFail code: BadTicket 1, Expired 2, Used 3, WrongInstance 4, Full 5, BadVersion 6, Ended 7; every JOIN_FAIL is followed by FREE_RINGS to release the connection's buffers.
If the ticket passes and the fold shows a pending record that is the realm's own, the realm writes one critical transaction: character_location inRealm (same epoch, serverIncarnation = its startedAt_) plus a playerJoined event.
Only when its own inRealm record has come back through the fold as valid (onOwnInRealm) does it send ADMIT to the worker. If the fold says void, because the meadow reclaimed first, the join fails with Expired.
Some special cases follow from that rule:
- Reconnect: if the hero is already
inRealmand owned by this realm, it is admitted with no write. - Newer ticket: a newer ticket for the same hero supersedes an older one (
onTicket); a replayed older ticket is ignored; a used ticket stays used (spent_). - Slow write: a write that has not folded within
kJoinWaitMs(5 s) is abandoned and the join fails Expired. If it folds later anyway, the realm sends the hero straight back. - Lost race: a meadow reclaim that folds after the realm's
inRealmwins. The realm evicts the hero and writes no leave record, because the meadow now owns the hero.
6. Rewards inside the dungeon
onReward on the realm writes one ledger entry per XP gain (reason 1 = kill, source = instance id), one per gold pickup (reason 0) and one inventory record per changed slot (quantity 0 = cleared). Each carries the
realm's writerId and the epoch. These go through GroupTxnQueue, which:
- packs whole items first-in-first-out, up to
MAX_RECORDS= 256 records per transaction; - retries an in-doubt commit with backoff from 100 ms to 2000 ms, and treats 60 s of doubt as fatal;
- on a definitive refusal, bisects: each item is resent on its own, in order, and only the item that fails is dropped (
dropped_); - treats a failure of a critical item (a join, a leave or an ended) as fatal, since the process cannot sensibly continue with an unknown owner;
- aborts and resends the same transmission, with backoff, when the refusal is a transient local one (back-pressure).
The leave record waits for the rewards in flight (inflight) and carries only the counts of the ones that were committed. A reward that failed definitively is not counted, so the meadow never waits for it.
7. Leaving and coming back
The leave (leave) is one critical transaction: character_location inMeadow at epoch+1, owner Meadow, the same serverId carried over, plus the xpRecs, goldRecs and invRecs counts, plus a playerLeft event, plus an ended
event if this was the last hero. The reason is one of Victory 1, Retreat 2, Defeat 3, Disconnect 4 or Ended 5.
On the meadow, onLocation sees InMeadow after a non-InMeadow state and starts a barrier (startBarrier). Rewards and the location record live on different topics, so nothing guarantees the leave record is folded after the rewards it counts.
The barrier waits until the number of reward records folded from this realm (onRewardRecord, matching server and epoch, writerId != 0) matches the counts in the leave record. The cap is BARRIER_CAP_MS = 5 s.
What if the counts are zero? That is what a fence, a restart release or a leave without rewards produces. The meadow cannot tell "no rewards" from "rewards not here yet" by counting, so it writes its own reward markers (writeRewardMarkers: a nonce record on xp, gold and inventory) and resumes only when all three markers come back through the fold
(onRewardMarker). By then, everything written earlier is behind them in the log. A deadline reclaim resumes at once. Finally host_.resume respawns the hero at the return spot and sends RESUME.
// Handoff::startBarrier (shape) void startBarrier(Seat& s, const Location& leave) { if (leave.deadlineReclaim) return resume(s); // nothing to wait for if (leave.xpRecs + leave.goldRecs + leave.invRecs == 0) return writeRewardMarkers(s); // wait for 3 markers to fold s.want = {leave.xpRecs, leave.goldRecs, leave.invRecs}; // wait for counts_ to match, 5 s cap }
8. When things go wrong
Everything above has a failure branch, and each branch is a rule the code actually applies. Here they are in one place.
| What breaks | What the code does |
|---|---|
| The ticket transaction fails | If the pending record never folded, the player gets NO_SERVER and the seat is freed. |
| The realm never admits the hero | reclaimPastDeadline writes a reclaim when the 20 s deadline passes. A playerJoined event is only a hint; it holds the reclaim for one more 20 s window. |
| The hero reconnects mid-handoff | reissue rewrites the realm_tickets row alone (same epoch, ticketOnly) when the client asks again (onAdmit, onRequestTicket). |
| The realm socket drops | The client sends RequestRealmTicket every 4 s (REJOIN_RETRY_MS) for 60 s (REJOIN_WINDOW_MS), then gives up; the server ends the seat and sends RESUME. |
| A realm stops beating | After 3 s newlyDead triggers startFence (below), then a reclaim of every non-InMeadow character on that server, and its bindings are dropped. |
| A realm restarts | Two defenses. A beat with a different startedAt means a new incarnation, and the meadow releases what the old one held (releaseOldIncarnation); a beat with an older startedAt is replay and ignored. And at boot the realm folds character_location behind its own marker and releases every hero held by an earlier incarnation of itself. |
| The meadow restarts | onBootDone rebuilds the bindings from the fold and watches the pending deadlines. For QUIET_MS (3 s) it will not declare any realm dead or reclaim anyone, because beats it has not yet seen are not deaths. The same quiet window follows onTopicsReset. |
| A realm is paused (stop-the-world, VM freeze) | It notices that its own beat has not come back for kSelfFenceMs (2 s) and freezes: workers get FREEZE, waiting joins fail Expired, new creates are rejected DRAINING, and writes are held in order. When a beat stamped after the pause returns, it resumes and shifts its join and drain timers forward by the paused time. |
| A realm was fenced | Its transactional producer gets fenced, the process exits with code 3. |
Fencing a dead realm
The realm writes with the transactional id realm-<serverId>. Rune says that a newer producer with the same id fences the old one: its next operation fails. So fenceRealm_ on the meadow's Rune thread creates a producer with that id and runs
initTransactionsAndWait. After that, nothing the old realm still has in flight can commit. Only then does the meadow reclaim (onFenced(ok)), with deadline=false. A FenceGate adds care: the session must be connected and have just answered a ping, and be up for kFenceSettleMs (3 s), so an unhealthy session does not start fencing. A failed fence is retried after 1 s (FENCE_RETRY_MS), never inside the quiet window.
9. The whole flow in one paragraph
The meadow picks a live realm from heartbeats, asks it for a dungeon through a filtered request, writes a pending ownership record, a hashed ticket and a final position in one transaction, and only after it sees its own pending record fold does it redirect the hero. The realm checks the ticket,
writes inRealm and waits for it to fold before admitting. Rewards go out as ledger and inventory entries; the leave record carries their counts; the meadow counts them back before it resumes the hero. If anyone disappears, deadlines, fences and the fold decide who owned the hero, and the log has exactly one answer.
Nothing in this required the two servers to be able to open a socket to each other, which was the original idea.