The network layer
Three threads, two transports, and one rule: the SDK measures and exposes, the game decides. This page follows a byte from the world thread to a slow client, then lists every option.
The shape of it
A server::Game is the rules of your world. Everything between the game and the wire (accepting connections, writing bytes, measuring links, refusing floods) is the network layer, and a server::App assembles it for you. You usually touch it in three places: the server::Options you give the App, the GameOptions of your Game (radius, budget, focus), and the rules that read p.link().
The sections go in the order a message travels. Threads and the App (1, 2); the options (3); the two transports (4); what each client is sent every tick (5); who sees what (6); how many bytes they may be sent (7); how the server measures a link and what the game does with it (8); backpressure (9); the limits on what a client may send (10); several games in one process (11); the numbers (12). The reference is the last section.
Names: server:: is aether::server. Headers are under include/aether/server/, include/aether/ecs/Interest.h and include/aether/spatial/Grid.h. For the rules of the game itself see Server; for what the client does with all this see Client.
1. The three threads
When you call app.listen() the SDK starts three threads and the thread that called it waits for them. Each has one job, and they share nothing but a few ring buffers.
| Thread | Name | Runs |
|---|---|---|
| Server layer (network I/O) | not named by an option | The transport's event loop: accepts connections, reads messages, enforces the rate limiter, writes the bytes the world queued, measures the path. uWebSockets for WebSocket, GNS for GameSocket. |
| NetworkProcessor | Options::networkThreadName ("aether-network") | A steady pump: Options::flushRate (30) times a second it asks the server layer to run flush(), which copies each client's output ring to its connection. |
| World | Options::worldThreadName ("aether-world") | Your Game: Options::tickRate (20) ticks a second. Reads input, runs the rules, commits, and writes every client's tick message. |
The thread names are what the OS shows; the OS keeps 15 characters. Notice how the three are separated:
- Two rings per client. An input ring (
inputBufferBytes, 32 KiB) is written by the network thread and read by the world thread. An output ring (outputBufferBytes, 128 KiB) is written by the world thread and read by the network thread, which writes it to the connection. Each ring has one writer and one reader. The largest message a ring takes is one eighth of its size (16 KiB for the output ring by default). - Only the world thread writes output. The one exception is
Interest::sendwithworkers > 1, where a worker thread sends whole clients while the world thread waits insidesend(). Anything else that wants to talk to a client mustpost()to the world thread. - The world buffer. Connect and disconnect notices travel from the network thread to the world thread through one many-to-one ring (
Options::worldBufferBytes, 1 MiB). post(fn)is the one call that works from any thread.Game::postandHost::postqueuefnunder a mutex; it runs on the world thread at the start of the next tick, before the inputs are applied.
One tick, in order
- The world reads the world buffer:
onConnect(which admits the player, callsonJoinand attaches the client to the interest) and disconnects. - For every player it reads at most
Options::rate.perTick(16) messages of that client; the rest wait in the ring. Each goes to the interest (subscribe, ignore, don't know), then the inputs (queued only), then the message handlers. - The schedule runs: input phase (posted functions, then the queued inputs are applied), simulate (
update(dt)and your systems), commit (every view takes a snapshot), send (Interest::send()writes each client's tick message), then queuedsendandbroadcastmessages go out, after the tick message with the state they are about. - The world pings every client that is due (
pingIntervalMs, 1000). - Later, on the network thread's next flush (30 Hz by default, not aligned to ticks), each client's output ring is copied to its connection until the connection is full (section 9).
Each connection also gets, every linkReportIntervalMs (250), a link report written by the network thread into its input ring; that is how the world learns about capacity, loss and backlog (section 8).
Next: starting and stopping all this.
2. App: listening, stopping, signals
server::App takes your world (a Game, a Host, or any WorldProcessor) and the Options, and runs it. It is not copyable.
#include <cstdio> #include <cstdlib> #include "meadow/World.h" int main(int argc, char** argv) { ecs::FileStore store("meadow.db"); World world(store); server::App app(world, {.tickRate = 20}); const std::uint16_t port = argc > 1 ? std::atoi(argv[1]) : 9000; return app.listen(port, [](std::uint16_t p) { std::printf("meadow on %u\n", p); }) ? 0 : 1; }
Notice:
listen()blocks untilstop()is called or the process gets SIGINT or SIGTERM. It is[[nodiscard]].- The optional second argument is
onListening(port), called once, on the thread that calledlisten(), with the bound port. Port 0 picks a free port, and this callback is how you learn which. - It returns false if the port could not be bound, or if the world's
onStartthrew (the world never ticks, the server stops at once, and the exception text goes toOptions::onError,stderrby default). It returns true after a clean stop. - A port that cannot be bound fails before anything else is started.
The two listen forms
| Call | Transport |
|---|---|
app.listen(9000) | WebSocket (uWebSockets, plain, no TLS). The port is a TCP port. |
app.listen(server::GameSocket{27015}) | GameNetworkingSockets over UDP. #include <aether/server/GameSocketServer.h> and link aether_gns. The port is a UDP port; 0 picks a free one. |
#include <aether/server/GameSocketServer.h> server::App app(world, {.tickRate = 20}); return app.listen(server::GameSocket{27015}) ? 0 : 1;
Everything else (onListening, the return value, stop(), the options) is the same for both. server::Transport is the interface behind it: make(handler, options) builds a Server, and you can implement it for a transport of your own.
Stopping
app.stop() closes every connection, delivers the disconnects, calls the game's onClose, and listen() returns. It is safe from any thread and from a signal handler. A stop() that arrives before listen() has started everything is applied as soon as it has.
With Options::handleSignals (true) the App installs handlers for SIGINT and SIGTERM that stop the server, and restores the previous handlers before listen() returns. Turn it off if your program handles signals itself.
The stop order matters and is fixed: the server layer closes every connection and waits for the disconnects to reach the world buffer, so the world is closed last.
To choose the threads or the pipeline yourself, assemble the pieces as src/server/App.cpp does: a WorldManager, makeSessions(manager, options), a WebSocketServer or GameSocketServer, a NetworkProcessor and your WorldProcessor (start, or attach plus your own tick() loop). The reference lists each.
Next: the options you give the App.
3. Options: a guided tour
server::Options is one struct for every piece; each piece reads only the fields named for it. Every field has a default that is a sensible server, so server::App app(world) works. Invalid values are caught where the piece is built: a value that can never work (a negative time, a zero that would spin, an empty path) throws std::invalid_argument whose text is aether::server::Options::<field>: <why>. Values that merely make no sense at the edges are clamped, and the table says so: the tick and flush rates to at least 1, the buffer sizes up to the smallest that works.
The tour, by what you want to change.
How fast the world runs
server::App app(world, {.tickRate = 30}); // 30 world ticks a second
tickRate (default 20) is the world's ticks a second, clamped to at least 1. Each tick is one commit and one send per client, so it is also how often clients can hear from you. Game::dt() is 1 / tickRate and Game::time() is tick / tickRate seconds. A world that is mostly idle (a lobby, a turn-based game) can set worldIdle = server::Idle::FrameSyncBackoff: it ticks at tickRate while the tick did work and halves its rate after each tick that did none, down to 1 a second, and is back to full as soon as a tick does work (the first message wakes it within a second). The default FrameSync is a steady rate whether or not there was work.
networkIdle is the same choice for the flush pump, and its default is FrameSync on purpose: the pump very often reports "no work", so FrameSyncBackoff would throttle it toward one flush a second and batch updates into bursts. Use Backoff only for a transport that is mostly silent where latency up to a second is acceptable.
How many clients, how big a message
server::App app(world, {.maxClients = 200, .maxMessageBytes = 2048});
maxClients (1024): further connections are refused (WebSocket answers 503 Service Unavailable with the body "full"; GNS closes the connection with the reason "refused"); 0 refuses all. maxMessageBytes (4096, at least 1): the largest message a client may send; a larger one closes that client. The input ring is raised to at least 8 times this.
Where it listens, and for how long a silent client lives
host (empty = all interfaces) is the bind address: an IPv4 or IPv6 literal, or for WebSocket a host name; a host that cannot be bound makes listen return false. idleTimeoutSeconds (32, 0..65535, 0 = never; over GNS about 24 days) closes a silent connection. The client answers the server's pings, so a healthy client is never silent. stopPollMs (50, at least 1) is how often the WebSocket loop checks for a stop request (the GNS loop reacts within its own pollIntervalMs).
Measuring the link
pingIntervalMs (1000; 0 = no pings, and rttMs then stays 0), rttCapMs (250), rttMaxSampleMs (2000), rttSmoothing (4) and linkReportIntervalMs (250; 0 = never, and the link's rates stay 0) shape what p.link() reads. Section 8 explains each.
Buffers and the write pipeline
inputBufferBytes (32 KiB), outputBufferBytes (128 KiB), worldBufferBytes (1 MiB) are rounded up to a power of two of at least 1 KiB (the world buffer: at least 32 KiB and enough for a connect notice). backpressureBytes (64 KiB) is "stop writing to a connection with this much unsent". maxMessagesPerFlush (0 = no limit) bounds the messages written to one client per flush. stalledSeconds (300; 0 = never) is the dead-peer safety net. Section 9 explains how they fit.
The limits on what a client sends
rate is a RateLimit (section 10): perSecond 30, burst 60, bytesPerSecond 16384, burstBytes 32768, violationWindowMs 2000, perTick 16. It is on by default.
Errors in the threads
onError is called when the world's or the network agent's work throws (the agent then carries on). It runs on the thread that threw: the world thread, or the network thread. Empty means print to stderr.
Per transport
server::App app(world, {.webSocket = {.path = "/game", .healthPath = "/healthz", .maxLifetimeMinutes = 120}});
webSocket (WebSocketOptions) and gameSocket (GameSocketOptions) hold what only one transport reads; their fields are in section 4 and the reference. The complete table of every field, with units and ranges, is in the reference.
Next: the two transports.
4. The two transports
Aether has two flavours of the server layer. Both carry the same messages, in the same order, through the same pipeline: the first byte of every message is its type, then the body, little endian.
- WebSocket (
WebSocketServer,app.listen(port)): uWebSockets over TCP, binary messages. The URL's query string becomes the connection's query. No TLS in the SDK: terminate TLS in front of the server (a proxy or a tunnel); the client refuseswss://. - GameNetworkingSockets (
GameSocketServer,app.listen(server::GameSocket{port})): Valve's open-source library over UDP. Messages are reliable and ordered, and the connection is always encrypted. Every message the server sends over GNS is sent reliable; unreliable sends are not built.
| WebSocket | GNS | |
|---|---|---|
| Library, link | uWebSockets; part of aether | GameNetworkingSockets; link aether_gns (clients: aether_client_gns) |
| Under the hood | TCP | UDP, reliable ordered messages, encrypted |
| Connect parameters | The URL's query: ws://host:9000/?name=Ayla; maxQueryBytes 2048, longer gets 414 URI Too Long. The path must match webSocket.path. | No URL. The first message is a hello carrying the query text (below). |
address() | The remote address text. | ip:port as GNS prints it (an IPv4 peer appears as [::ffff:a.b.c.d]:port). |
| One wire message is | One binary WebSocket message. A text frame closes the connection. | One GNS message. |
bytesIn / bytesOut | Wire bytes including WebSocket framing. | Message payload bytes (GNS framing and UDP headers are not visible). |
| Congestion control | The kernel's TCP. | The SDK's pacing (section 9), option gameSocket.congestionControl. |
| Servers per process | Several are fine. | One looping at a time. The second loop() returns false at once. |
| Closes | The close code and reason travel in the close frame. A peer that is not reading (more than backpressureBytes unsent) is closed at once and the reason is lost. | Codes 3000..4999 travel as GNS's end reason (code minus 2000) with the reason text as its debug string (at most 127 bytes). The client puts them back. |
What does not depend on the transport: the rate limiter, the write pipeline and its backpressure, stalledSeconds, the close codes 4000, 4001, 4290 and 4291, every Link field, and the contract check (the same Game admit code).
The WebSocket server
It accepts upgrades on webSocket.path (a uWebSockets pattern, "/*" by default so any path works; "/game" to restrict). A plain GET to webSocket.healthPath ("/health") answers ok, for a load balancer; an empty healthPath means no health endpoint (404). With maxClients reached, the upgrade is answered 503 Service Unavailable; with a query longer than maxQueryBytes, 414. The other WebSocket fields map to uWebSockets settings and are in the reference table.
The GNS server and the hello
A WebSocket client says who it is in the URL. GNS has no URL, so a client that has something to say before it joins (a ?name=Ayla, the contract of its .aether file, maxBytesPerSecond) sends it as its first message: [kHelloType = 9][the text of a query string], for example name=Ayla&contract=0123456789abcdef&schema=meadow.aether. The server layer then announces the connection with that text as its query, so p.query("name") and the contract check work exactly as on WebSocket. The client library does this for a gns:// URL.
The server has to be told to wait for it: gameSocket.helloWaitMs.
helloWaitMs | Behaviour |
|---|---|
| 0 (the default of the option) | Connections are announced at once with an empty query, and a hello is refused: the client is closed with the reason "this server does not read a hello: set Options::gameSocket.helloWaitMs". |
| 1 to 60000 | A connection is announced when its first message arrives (the hello, or any other message, in which case the query is empty), or after this many milliseconds from its accept, whichever is first. A client that sends no hello waits this long to join. A hello that arrives after the wait is refused ("a hello must be the first message"). A hello longer than 2048 bytes closes the client ("the hello is longer than 2048 bytes"). |
listen(GameSocket) changes the default. App::listen(server::GameSocket) needs the hello (the App reads the query), so when helloWaitMs is 0 it builds the server with 10000 (a copy of the options; the App's own are not changed). Set helloWaitMs yourself to wait longer or shorter. Keep it well above one resend (about a second): a hello that arrives after the wait is refused. A server with a .contract needs the hello, since without it every client is refused ("the client sent none").The GNS options
Besides helloWaitMs and congestionControl (section 9): pollIntervalMs (1; 0 = no sleep, a busy loop on a full core), maxMessagesPerPoll (64), deferQueueBytes (64 KiB task ring), stopDrainMs (200), sendBufferBytes (0 = GNS's 512 KiB), sendRateMin / sendRateMax, nagleTimeUs (-1 = GNS default), noNagle, mtuPacketSize, timeoutInitialMs, timeoutConnectedMs, flushAfterCork (true), the four fakePacket* fields for tests (they are global in GNS: they affect every GNS connection in the process while the server loops, and are reset when loop() returns), debugOutput and debugLevel (4), and a raw config list of ConfigValue{key, value} applied last. All are in the reference table.
Next: what a client is actually sent.
5. What a client is sent
A client connected to a Game receives a small, fixed vocabulary of message types. Types 0 and 1 are reserved, your own messages start at 16.
| Type | Direction | Meaning |
|---|---|---|
| 1 | both | Ping: the server sends {int64 t}; the client answers with the echo {int64 t, uint16 interpMs}. |
| 2 | server to client | The tick message. |
| 3, 4, 5 | client to server | Subscribe [view u16][entity u64]; ignore [view u16][entity u64][ms u32]; don't know [view u16][entity u64]. |
| 6 | server to client | The layout of a view. |
| 7 | server to client | The welcome [entity u64][tick rate u16][input rate u16]. |
| 8 | client to server | An input batch. |
| 9 | client to server | The hello (GNS only). |
| 10 | server to client | The table of message and move ids: [last u8] then entries [kind u8][id u8][name length u8][name]. |
| 11 | server to client | A warning [text]: the client's onWarning. |
| 16 to 255 | both | Game messages, ids from the table. |
When a client attaches, it is sent, in order: the welcome, the id table, then (in the first send) the layout of every view it can be sent, and the first tick message. A client does not know a view's fields until its layout arrives, and no entry of that view is sent before it (a layout the output ring refuses is tried again on the next send, and until it has gone out that view is held back for that client). The welcome is sent again when the player is assigned a new entity.
The tick message
Once per send (once per tick for a Game), each client gets a type-2 message: [stamp i64][last input sequence u32], then sections [view u16][entries u16] followed by that many entries. The stamp is the registry's last commit, the game's tick number. The input sequence is the last input the server applied for that client (the client's ack, see Client: corrections).
| Entry | Layout | Meaning |
|---|---|---|
| Full (0) | [kind u8][entity u64][row] | The whole row (Type::blockLength bytes). |
| Delta (1) | [kind u8][entity u64][mask][changed fields] | The mask has one bit per delta field in select order, (n+7)/8 bytes; then only the changed delta fields, each at its width. |
| Unsubscribed (2) | [kind u8][entity u64] | The row is gone for this viewer. |
| Snap (3) | like a Full | A Full of an entity the viewer already has, marked as a discontinuity (a portal, a respawn). A viewer without the entity gets a plain Full. |
| Leave (4) | [kind u8][entity u64][cell u32] | Only inside the crowd path's cell pieces: the entity is no longer in that cell. |
- Entries are written in the order Unsubscribed, Deltas, Fulls, and a view may have more than one section. A tick that does not fit one message (the output ring's largest message, 16 KiB with the defaults) is split at entry boundaries, and every part repeats the header.
- If the ring refuses a message, its entries and those of later messages keep their old state and are tried again on the next send. Fulls start at an offset that moves on every send, so a ring that is always short does not starve the same entities.
- Every entry carries the full 8-byte entity id.
Delta, history and every(k)
For an entity the viewer is subscribed to, the server compares the view's delta fields against the row the viewer was last sent, taken from the view's snapshots (history). Only the fields that differ go; if none differs, no bytes. If that row has aged out of the history (or there is none), all the delta fields go in the Delta, not a Full. A changed row is encoded once per tick and copied. Fields that are not delta fields travel only in a Full.
The view's history(n) is how many commits older than the last are kept; the default, 1, is enough if every viewer is sent every tick. A view with every(k) (it takes part in a viewer's send only when (send number + viewer number) % k == 0), or a viewer whose budget leaves entries out for k sends, needs history(k) or more, or every Delta carries all the delta fields. every(k) is not the crowd path's crowdEvery: every sits a whole view out, crowdEvery makes cells take turns. See Views.
every(k). A viewer's own player row in a view that sits out under every(k) is still sent (the owner-row rule, always on in a Game), and nothing else of the view is looked at. Send owner-only views every tick: every(k) can delay the visible effect of an action up to k-1 ticks.The idle header and keep-alives
A client whose send has no entries, no layout and the same input sequence as the last header it was sent gets no message at all, until idleHeaderTicks sends in a row wrote none; then the header goes as a keep-alive. A changed input sequence is always sent at once. In a Game, GameOptions::idleHeaderTicks is 0, meaning the game's tick rate (about one keep-alive a second); 1 means a message every tick. The low-level Interest defaults to 1.
The consequence on the client: the stamp skips ticks and session.tick().count counts messages, so use session.serverTime() for time.
Next: who is in the message.
6. Interest management: who sees what
Each viewer has its own view of the world, and the server works it out again on every send. Per viewer and view there are exactly two lists: subscribed (entity to the commit it was last sent at, and how many sends it has been out of sight) and ignored (entity to the time the ignore ends). The neighbours are found again on every send; no other state is kept per viewer and entity on the exact path.
void onStart() override { view<Walker>().spatial("x", "y"); // rows within the viewer's radius of its own player (default radius 64) view<Purse>().ownerOnly(); // each client gets only the row of its OWN player entity // view<Score>().global(); // every client gets every row, wherever it is // view<Board>(); // sent to nobody unless shown: p.show(e) }
What each send does for a viewer
- The grids are brought up to date (one grid per spatial view; at the start of
send(), not in the commit). A row whose position is not finite as a float stays out of the grid and is counted inInterest::unplaced(). - The viewer's radius is read, once per viewer per send.
- The neighbours are the entities of each spatial view within the radius of the viewer's own position, plus the entities given to
show(). The viewer's position is that of its player entity's row in the spatial view with the lowest id that has one; a viewer with no entity has no grid neighbours, only the shown and focused ones. - For each neighbour: ignored, nothing. Subscribed, a Delta if something differs. Neither, a Full, and on an
Autoview the server subscribes the entity at once, when the ring took the Full. - A subscribed entity that is not a neighbour this send (it left the radius, lost its row, or was hidden) is kept for
unsubscribeDelaysends and receives nothing meanwhile; if it is a neighbour again in time, nothing happened and its Deltas carry on. After the delay the client is sent Unsubscribed and the entry is removed. A destroyed entity is Unsubscribed on the next send regardless.
The radius
The radius is not per view. It is GameOptions::radius: a number, or a rule of the Player asked once per player per send. The SDK clamps nothing: what it answers is used. The default is 64 world units. A NaN or negative answer is a bug in the game: a debug build stops with a message naming the client (stopOnBadRadius), a release build uses 0 (the viewer sees only what is shown to it by hand) and counts it in Game::stats().badRadius. With no radius set, a debug build prints one notice to stderr the first time a view has rows beyond 64.
Game({.radius = 400}); Game({.radius = [](const Player& p) { return p.link().congested() ? 150.f : 400.f; }});
The rule is asked every send, so it can flap; keep hysteresis in p.user<T>() if you do not want the edge of the radius to churn.
Subscriptions, cap, ignore
Setting (InterestPolicy) | Default | Meaning |
|---|---|---|
unsubscribeDelay | 3 | Sends a subscribed entity (for the crowd path, a held cell, at most 255) may be out of sight before the client is sent Unsubscribed (the cell is dropped). 0: on the first send. |
cap | 2000 | Subscriptions per viewer over all views. At the cap a neighbour on neither list gets no Full until a subscription is removed. A crowd cell counts by the cell; a cell that does not fit is not taken, a held cell is kept; the cell with the viewer's own row is taken whatever the cap says. |
maxIgnoreMs | 600000 | A longer client ignore is cut to this. |
maxIgnored | 4096 | Ignore entries per viewer over all views; an ignore past it is dropped. |
cellSize | 32 | World units; finite and above 0. See the grid, below. |
These live in the low-level policy: Game({.interest = {.cap = 500, .unsubscribeDelay = 5}}).
Auto and Explicit views
A view is Subscribe::Auto by default (Game::view<Row>() makes it subscribable). A neighbour on neither list gets a Full and, on an Auto view, is subscribed at once. On a Subscribe::Explicit view the entity gets a Full every send until the client sends subscribe or ignore: the Full is only an offer, and no Unsubscribed is sent for an entity that was never subscribed.
The client's three messages: subscribe is dropped if the entity is dead, has no row, is ignored or is not a neighbour now, and at the cap it is answered with Unsubscribed in the next tick message; on an Auto view it is accepted and changes nothing. ignore means nothing about the entity for ms (cut to maxIgnoreMs), also after it leaves and returns; it is dropped for a dead entity, for ms 0, and when the viewer already has maxIgnored entries; ignoring an ignored entity replaces the end time. don't know (a Delta for an entity the client has no row of) removes the subscription, sends nothing, and makes the entity a "neither" neighbour again, so it gets a Full next send. A body of the wrong length, an unknown view or a view that is not subscribable counts in Interest::dropped(client).
Showing, hiding and focusing by hand
p.show(e)/p.hide(e)makeea neighbour of this viewer in every subscribable view wherever it is (except owner-only views, which only ever show a viewer its own player), until hidden. InonJointhey are applied when the client attaches.p.focus(e)/p.unfocus(e):eis sent to this viewer entity by entity, in every send, wherever it is, and under a budget before the crowd (inside thefocusshare). It is this viewer's business alone: for everybody elseestays on the crowd path. Independent of show and hide. A target that dies is dropped from the focus. Under a budget, this is how a game says "this one matters".- A player's own entity is shown to it in every view (owner-only stays private from others). Assign
p.entity = spawn(...)to respawn; the Game rebinds before the next send, sends a new welcome, and the client'ssession.me()follows.ecs::kNoEntityis a spectator or a dead player.
The grid
Each spatial view has a spatial::Grid: a uniform grid on a plane, unbounded (cells exist only where entities are, negative coordinates included). The default cellSize is 32 world units, "about the usual radius": a query then looks at about 9 cells. A match is by the true distance to the entity centre (dx*dx + dy*dy <= (range + radius)^2), not by cell. Insert, move and remove take handles; a move is cheap when the entity stays in its cell; nothing is allocated in steady state. It is not thread safe. The grid is also a public class you can use on its own (reference).
The crowd path: cell blocks
The default for every view that is spatial, Subscribe::Auto, every(1) and has a radius is not "entity by entity" but by grid cell. The entries of a cell are encoded once per send and the same bytes go to every viewer that holds the cell, so the cost follows entities plus players times cells, not players times neighbours. Such a view keeps no subscription per row for these viewers; what is kept is one 8-byte record per held cell.
- What a viewer holds: every cell that touches its radius circle, "a little more than the circle, never less". The extra is at most one cell's diagonal. A game that needs "nothing beyond the circle" (a far player must not see it) sets
cellSizesmall against the radius. - What stays on the exact path: the viewer's own row, focused entities, entities shown to or ignored by the viewer, entities with no finite position, entities more than 4095 cells from the first one seen (a cell id has 13 bits of x and y), and views without a radius. The exact path's rows are the only ones with Unsubscribed entries and the cap counted per entity.
- The wire: sections whose view has the high bit set hold pieces
[cell u32][op u8][entries u16]. Op 0 applies the entries; 1 (reset) removes every row the client holds of the cell, then applies; 2 (drop) removes them, the cell is no longer held; 3 (hold, under a budget only). A message that is more than one section group ends all but the last with[view 0xFFFF][entries 0]("the send goes on"), and the client makes its removals at the end of the last. The game sees none of this: it has rows. - Contents of a block: a row that entered the cell is a Full; one that was there and changed is a Delta (or a Snap); one that left is a Leave. A viewer's first send of a cell, or any send after it missed one, is a reset block. A cell that did not change since the viewer's last send costs one record compare and no bytes.
- Four consequences to design around: the radius is accurate to a cell; a per-entity
weightorders only the entries sent by entity; an entity shown to some player leaves the crowd path for everyone while it is shown; and under a budget the order differs from per-entity urgency. - Limits: a viewer that lags more than 255 sends on a cell loses the Leaves for it (a hole until the other cell is sent); a cap still takes new cells in key order, not nearest first;
every(k)does not apply to cells (usecrowdEvery).
Sending in parallel
InterestPolicy::workers (Game({.interest = {.workers = 8}})) is the number of threads that send viewers in parallel: 1 (default) is the world thread alone; N above 1 adds N-1 threads that live as long as the Interest, each sending whole viewers, and what every client is sent is the same bytes whatever N is. 0 means one per hardware thread (at most 16; "usually more than is worth it: the send is memory bound, try 4 to 8"). Fewer than 32 viewers are always sent by the world thread. radius and budget are still asked on the world thread; weight, cellWeight and cut are called from the workers, several at once: they must only read game data that does not change during the send and keep no unsynchronised state.
Next: how many bytes.
7. Budgets: how many bytes a player gets
A game that sets nothing sends everything: all changed rows in the radius, every tick. That is right for a small world and ruinous for a crowd or a weak link. A budget caps the bytes of rows a player is sent, and lets the game say what matters when everything does not fit.
The unit: bytes a second
Game({.radius = 400, .budget = 80'000}); // 80 KB/s of rows per player Game({.budget = [](const Player& p) -> std::size_t { return p.link().headroom(); }});
GameOptions::budget is a number or a rule of the player: bytes per second of rows for one player. The game sends once a tick, and each send gets budget / tickRate bytes: the SDK carries the remainder per player, so 80,003 bytes a second at 20 Hz is 4,000 four times and then 4,001, and over a second exactly 80,003. SIZE_MAX means no limit. Every Link member is also per second, so the natural rule is the link itself.
InterestPolicy::budget (a function of the Client) is bytes per send, not per second. GameOptions::budget converts for you.What counts, and what does not
The bytes of Delta and Full entries (and cell blocks) count. On top, not counted: message headers, section headers, Unsubscribed entries, cell drops and holds, and the player's own row, which is sent by itself in every send so that a predicting client has its own state with the input ack of the same tick whatever the budget (its cell competes like any other, so the row can arrive twice; the client keeps a row the server sent in an ordinary section).
- The budget is an average, not a cap on one send. A block larger than the whole budget goes now and then: what is left of each send in which it was the most urgent is saved as credit (at most its size) and added to the next budget. So a
headroom()budget is exceeded now and then; leave margin if the link is tight. - Budget 0: only headers and Unsubscribed. A budget smaller than a view's Full (9 bytes plus the row) never sends that view's Fulls (exact path).
- With no budget (or
SIZE_MAX),weightis never called and everything in the radius goes every send.
Urgency: what goes first
When everything does not fit, entries are ranked by urgency and taken highest first while they fit; one that does not fit is skipped and smaller ones are still tried.
urgency (exact path) = view priority x weight(viewer, other, distance) x (sends waited + 1)
urgency (a cell) = view priority x cellWeight(viewer, cell) x (sends waited + 1)
cellWeight defaults to 1 / (1 + distance in cells); a cell the client has none of: x 1024
- Priority is the view's
priority(n), default 1; 0 is refused at build ("priority must be at least 1"). - Weight (
GameOptions::weight,(const Player&, Entity other, float distance)) defaults to1 / (1 + distance); NaN and negative count as 0; weight 0 is sent only when budget is left over. The distance is between the two in that view, and 0 if either has no position there. - Waited: for a Delta, the sends in a row in which the entity had a Delta that did not go out whole; for a Full, the sends in a row in which this viewer had a Full left out. So a left-out entry is sent later, not never.
- Conflation, for free: a left-out update is not queued. Its record keeps the commit it was last sent as of, so the next time the entity goes it carries everything that changed since: state, not a queue.
- When everything fits, no game weight is set and the ring has room, urgency is not computed at all and entries go in the order they were walked (views by priority).
weight does not rank the crowd. Under a budget the rows around the player are sent a cell at a time, so weight only orders the entries sent entity by entity: the player's own row, focused and shown entities, and views without a radius. To make one entity matter, p.focus(entity).Tiers: cutting a Delta instead of dropping it
Fields can have a tier: delta(&Hero::emote, Tier::Low) or delta("hp", Tier::High) on the view builder (High, Normal the default, Low). A Delta that does not fit whole is cut to its High and Normal fields, then to High only, if that fits and has a changed field. A subscribed entry holds one sent commit per tier, so the left-out fields keep their own baseline and travel in a later Delta. The wire is unchanged: a cut Delta just has fewer bits set in its mask. A Full is never cut; one that does not fit waits and its urgency grows. Tiers matter only under a budget.
The game's say over the crowd
These are all optional; with none set the bytes are those of the basic crowd path.
explicit World(ecs::Store& store) : Game({ .radius = 400, .budget = [](const Player& p) -> std::size_t { return p.link().headroom(); }, .focus = 0.3f, .cellWeight = [](const Player&, const ecs::Cell& c) { return c.mine || c.distance < 1 ? 3.f : 1.f / (1.f + c.distance); }, .nearReserve = {.after = 60, .share = 0.1f}, .crowdEvery = [](const Player& p) { return p.link().congested() ? 4u : 1u; }, .cut = [](const Player&, const ecs::Cell& c) { return c.distance < 1 ? ecs::Cut::Never : ecs::Cut::LowAndNormal; }, .store = &store, .contract = meadow::contract}) {}
| Option | Default | What it does |
|---|---|---|
focus (std::optional<float>) | 0.5 | The share (0 to 1) of the budget that entries sent entity by entity (own row, focused and shown entities, views with no radius) take first. They take max(focus x budget, budget - what the crowd wants); the crowd takes the rest plus its credit. 0: they take only what the crowd leaves (an entity the player focuses is still in its cell's block within the radius, so focusing never makes it less visible); 1: they may take everything (for a game whose few exact entries must never wait). Values outside 0..1 count as the nearest end. A share of bytes, not a rule about entities. |
cellWeight | nearer first, 1 / (1 + distance) | Replaces only that factor in a cell's urgency. ecs::Cell says x, y (the grid cell, covering [x*cellSize, (x+1)*cellSize)), distance (cells; 0 if the player is in it or touching it), entities, bytes (what its block costs this send), waited (sends since this player last had the cell up to date; default 1), isNew (the player holds none of it), mine (it holds the player's own row). 0 means "only with room left over"; NaN and negative count as 0. Asked once per cell the budget has to choose among, only in a send whose budget cannot take every cell, and not at all while unset. |
nearReserve (ecs::NearReserve) | {after = 30, within = 1.5f, share = 0.25f} | A viewer that walks keeps getting cells it has none of, and those come before refreshing a cell it holds, so without a rule the cells right around it could wait forever. A cell within within cells that has waited after sends is paid out of share of the budget kept for such cells. {.share = 0} switches it off. A cell the game gave weight 0 never counts as near. |
crowdEvery (number 1..255 or rule) | unset: every send | The crowd's cells are sent to a player at most every this many sends. A cell takes its turn when (send number + a per-cell offset) is a multiple of it, so half of the cells go in each send at 2; it is then sent as the changes since its last turn (a catch-up block), and a cell waits at most twice that. The player's own row goes every send; a cell the player has none of yet is sent at once. Not every(k), which sits a whole view out. |
cut (ecs::Cut) | Cut::Low | When a block of changes does not fit, how far it may be cut by tier: Low may leave the Low fields out, LowAndNormal the Normal ones too (only High is guaranteed), Never cuts nothing (the whole block goes when it fits, else it waits). Fulls, Snaps and Leaves are never cut. A cut block is encoded once per cell and commit; the viewer's record is marked behind, and the fields left out come with the cell's next uncut block. Nothing is lost. |
idleHeaderTicks | 0 | Section 5. |
A cell that is out of step is sent, for a policy with crowdEvery or a view with several tiers under a budget, as the changes since the commit the viewer has it as of (a catch-up block: Leaves, Fulls of entrants, Deltas, Snaps) when that is smaller than the whole cell (always with tiers); other views send such a cell whole. Under a budget a cell whose whole block would be larger than about 2 KiB is split into 2, 4 .. 64 slices (the low 6 bits of the cell id): an entity belongs to the slice its entity slot hashes to and stays in it for life; a cell splits at once when it grows and merges back only when its slices would be four times smaller than the limit. Without a budget nothing is split, except a cell whose block would be more than a quarter of the smallest ring, which is split like a budgeted one.
Two readouts tell you what all this did to a player: p.crowdDelay() is how long the longest-waiting cell around it had waited in its last send (the sends since it was last up to date, in milliseconds from the tick rate; 0 before the first send or with no crowd), and p.focusDelay() is the same for the entities it focuses. They are measurements, not forecasts.
Next: the link, which is what a budget is usually a function of.
8. The link: what the server measures
Budgets and radii are most useful as functions of how the player's connection is doing. The server measures each connection and exposes it as a server::Link, read with p.link(). It decides nothing from it by itself: the game writes the rules.
explicit World(ecs::Store& store) : Game({ .radius = [](const Player& p) { return p.link().congested() ? 150.f : 400.f; }, .budget = [](const Player& p) -> std::size_t { if (p.link().limit) return p.link().limit; // what the client asked for return p.link().headroom(); // what the link can take }, .crowdEvery = [](const Player& p) { return p.link().congested() ? 4u : 1u; }, .store = &store, .contract = meadow::contract}) {} void update(float) override { for (Player& p : players()) if (p.link().queueMs > 5000) p.kick("connection too slow"); }
Notice:
p.link()returns aLinkby value (a copy). Itsleftand the ring part ofbacklogare filled in when you ask.- A rule that reads
backlogis a feedback loop with a tick of delay. Keep hysteresis inp.user<T>()if a rule flaps. p.kick(reason)ends the connection after what is already in its output buffer is written; the client'sonClosegets 4001 and the reason (at most 123 bytes).
Every field of Link
| Field | Meaning | On WebSocket | On GNS |
|---|---|---|---|
rttMs (u16) | Smoothed ping round trip, each sample capped at rttCapMs. 0 = not measured yet. | The server's ping (type 1) and the client's echo: the same on both transports (below). | |
jitterMs (u16) | Smoothed deviation of the round trip. 0 = not measured yet. | The same ping samples. | |
loss (float 0..1) | The fraction of what the server sent that was lost. 0 = unknown or none seen. | TCP segments retransmitted over segments sent, from TCP_INFO, smoothed loss += (ratio - loss) / 4 per report. A symptom: TCP retransmits, so it is not a lost message. | 1 minus the connection quality the remote reports (a negative quality, too little data, reads as 0). A slow figure: a 5-second average that GNS updates every 20 to 30 seconds. |
capacity (u32, bytes/s) | What the path is estimated to carry. 0 = unknown (also for the first report of a connection). | The kernel's delivery rate (tcpi_delivery_rate, a lower bound while the connection does not fill the path), made into an estimate by a CapacityEstimator (below). | With congestionControl (default): 1.25 times the path rate the transport measures and paces at (2 times during start-up), taken as it is. With it off: GNS's fixed send rate. |
sending (u32, bytes/s) | Bytes a second the server wrote to the connection. | Counted by the server layer, wire bytes including WebSocket framing. | Message payload bytes. |
receiving (u32, bytes/s) | Bytes a second the client sent. | As above. | As above. |
backlog (u32) | Bytes accepted but not yet on the wire. | uWebSockets' buffer plus the kernel's unsent bytes (tcpi_notsent_bytes). | GNS's pending bytes, plus, with congestion control, what is in flight above the limit. |
queueMs (u32) | How long a byte written now waits before it leaves. | backlog * 1000 / capacity (or / sending if capacity is 0), capped at 60000; 60000 when there is a backlog and nothing is draining; 0 for no backlog. | With congestion control: derived the same way from the backlog. Off: GNS's own queue time, capped at 60000. |
left (u32) | Delta and Full entries the last send left out or cut (the budget, or a full ring). | Only in Player::link() (Interest::deferred). | |
interpMs (u16) | The client's interpolation delay, from its last echo. 0 = not reported. | The ping echo. | |
limit (u32, bytes/s) | What the client asked for (SessionOptions::maxBytesPerSecond, in the connect query). 0 = no cap. | Never enforced by the SDK. It is a number for your rules. | |
Two helpers, exactly
bool congested() const { return queueMs > 100 || left > 0; } std::uint32_t headroom() const { return capacity > 4 * backlog ? capacity - 4 * backlog : 0; }
congested() is "what to shrink the radius by". headroom() is what the link can take in bytes a second, like the budget: the capacity minus the queue it still has to pay, because the backlog is owed within the next quarter of a second, so a link that has a queue gets less until the queue is paid. It is 0 when the capacity is unknown. It does not subtract sending: that already contains what the budget sent, and subtracting it made the rule alternate between the whole link and nothing.
How the numbers are measured
- Round trip. The world pings each client every
pingIntervalMs(1000). The echo{t, interpMs}is timed on the network thread on arrival. Only the echo of the last ping sent counts (forged or late ones are ignored). A round trip below 0 or aboverttMaxSampleMs(2000) is ignored. A sample is clamped to[1, rttCapMs](the lower bound so loopback does not read as "not measured"), thenrtt += (sample - rtt) / rttSmoothingin integer arithmetic (rttSmoothing4; 1 = no smoothing), so differences under about 4 ms do not move it. The jitter isjitter += (|sample - rtt before| - jitter) / rttSmoothing, rounded; the first sample has none.pingIntervalMs = 0turns pings off, andrttMsthen stays 0. - Transport numbers.
capacity,loss,backlog,sending,receiving,queueMsarrive as a link report written by the network thread into the client's input ring everylinkReportIntervalMs(250; 0 = never, and the rates stay 0). The first report goes one flush pass after the connect; a report is skipped if the input ring is full.Client::link()is updated while the client's input is read, so it changes at most once per tick. Player::link()reads the backlog live. It is the socket's part (user space and kernel, published by the network thread every flush pass) plus the bytes still in the world's output ring that the network thread has not taken. A rule that saw only the socket would send a slow link more and more behind a queue it cannot see (a 128 KiB ring is 8 seconds at 128 kbit/s). The ring part is added toqueueMsat the capacity (orsendingif capacity is 0); if the rate is unknown nothing is added, because a fresh connection's welcome used to read 60000 and a kick rule kicked every player on arrival.- WebSocket capacity. One raw delivery-rate sample spikes (a measured 400 MB/s on a 16 KB/s link), and a budget made of a spike sends the whole crowd at once. So
CapacityEstimatoruses the acknowledged bytes: while data waited at both ends of the interval, what was acknowledged in the interval is the sample; an interval that did not fill the path does not lower the estimate and only probes upward (times 1.5 a second) while the application uses it (it delivered at least half of it, nothing is queued, loss is flat). It falls at once, has a floor of 4000 bytes a second, and starts at 64 KiB a second.
rttMs never reads above rttCapMs. With the default 250, a rule such as p.link().rttMs > 300 never fires. Raise rttCapMs if a rule needs a higher range.loss and capacity are not what they look like on WebSocket. loss over WebSocket is TCP retransmissions, a symptom, not message loss; over GNS it is slow. capacity over WebSocket is a lower bound while the app is not filling the path.Close codes 4001 and 4291: kick and stall
Two ways the server ends a connection because of its link. p.kick(reason) is the game's rule (code 4001). Options::stalledSeconds (300, 0 = never) is the safety net: a connection whose output has been blocked that long is closed with code 4291 and the reason "the connection did not take what the server sent for 300 s"; Client::closedBy() is then OutputStalled, and Game::stats().closedStalled counts it. "Blocked" means the flush pass wrote nothing although the output ring is not empty and connection.full(backpressureBytes); any progress resets it ("a slow reader is not a dead one"). It is checked at the flush rate. It is not a slow-client policy: that is yours, through p.link(). Over WebSocket, a peer with more than backpressureBytes unsent is closed at once, so the reason may not reach it.
Next: the machinery that keeps the queue short.
9. Backpressure: why a slow client does not slow the server
A client on a weak link takes bytes slower than the world makes them. Something has to give, and it must never be a thread.
The write pipeline
On each flush pass, for each connection, the network thread reads messages from the client's output ring and, for every message, checks connection.full(backpressureBytes). If the connection is full it stops; the rest stays in the ring. Otherwise it sends. Writes are corked, so a transport can batch them. backpressureBytes defaults to 64 KiB: "stop writing to a connection with this much unsent".
When the ring is full, the world's Client::send returns SendResult::Full, and the interest keeps those entries' old state and retries on the next send. That is why rows are state and not a queue: a newer state replaces an older one that never went. Queued game messages (send and broadcast) that find a full ring wait for the next flush in order, up to one ring's worth of bytes per client, and only beyond that are dropped and counted (Players::dropped()). A message too big for the ring is dropped and counted. Nothing ever blocks a thread.
The game's part is the budget: a budget made from headroom() keeps the backlog short, because headroom() falls as the backlog grows. Up to backpressureBytes can wait below the pipeline, which on a slow link is seconds of data on both transports. The SDK's own default policy is only stalledSeconds.
WebSocket: the kernel does the control
TCP congestion control keeps in flight only what the path carries; the rest stays in the socket's send buffer, where the SDK reads it as backlog. The server counts uWebSockets' own queue plus the kernel's unsent bytes (TCP_INFO), because the kernel takes over a megabyte on loopback, or its whole send buffer on a real path, before uWebSockets sees any backpressure: counting uWebSockets alone let a slow link hold seconds of data while reporting 0. The getsockopt takes the socket lock, so it is not read per message: the last kernel figure plus the bytes written since is an upper bound, and the kernel is asked only when that bound crosses backpressureBytes, at each link report, and at most every 2 ms for a connection that had a queue.
GNS: pace it, and limit what is in flight
GNS has no congestion control of its own. Its send rate is set once at connect (clamped between 128 KiB/s and 1 MiB/s) and does not react to loss or delay; lost reliable data is resent at the same rate; the only refusal is at 512 KiB queued. Over a slow link, GNS sent into the bottleneck and the bottleneck dropped, the backlog stayed low, capacity read GNS's configured rate, and a headroom() budget flooded the client.
With gameSocket.congestionControl = true (the default) the transport does two things for each connection with data pending or unacknowledged, every 50 ms:
- Pacing. It sets GNS's per-connection send rate to its estimate of the path. GNS's limiter releases bytes at that speed and holds the rest in its own queue, where the SDK sees it as
backlog. The estimator follows BBR version 1's rules, adapted to a polled status: a path rate from the highest delivery sample of the last 10 rounds (a round is the longer of 500 ms and the lowest ping plus 50 ms), start-up at 2 times, drain, a steady cycle of 1.25, 0.75 and six 1.0 gains, and an empty-path check to re-measure the lowest round trip. A connection starts at 131072 bytes a second (GNS's own default) with a limit of 10 packets in flight until its first sample; the lowest estimate is 4000 bytes a second. A loss rule sets a ceiling when the share of resent bytes in one of GNS's 5-second statistics intervals reaches 2%. - An in-flight limit.
limit = max(4 packets, 2 * pathRate * lowestRtt + pathRate * 100 ms), never abovemax(4 packets, 1.5 * pathRate)(a packet is 1248 stream bytes). Whileunacked > limitthe rate is brought down to 0.75 of the lower of the path rate and the latest sample, and the connection reports full.
What the pipeline sees: buffered() = pending + (unacked above the limit), and full(bp) = buffered() > bp || unacked > limit. A message of any size is admitted whenever the connection is not full, so one larger than the limit cannot wait forever. The game just sees capacity = 1.25 x pathRate, so a headroom() game offers slightly more than the path carries, which keeps GNS, not the game, the limiter; the cost is about 60 ms of standing backlog on a busy connection that has room.
congestionControl = false is GNS's own rate: no backpressure, and capacity reads 131072 on every link. sendRateMin and sendRateMax (0 = none) bound the paced rate with it on, and are GNS's own SendRateMin and SendRateMax with it off.Closing with data waiting
end() lingers until what the connection accepted is delivered; close() drops it. At server stop, gameSocket.stopDrainMs (200) is how long to wait for what GNS accepted to be sent and acknowledged before closing; 200 ms is about 3 KB at 128 kbit/s. A server that must deliver a last message to slow clients raises it. Client to server traffic is not paced: clients send inputs, a few hundred bytes a second.
What is not built
Unreliable sends are not built, so there is no state-over-unreliable mode and no head-of-line avoidance on loss. The measured limits of the pacing are listed in section 12.
Next: the other direction, what a client may send.
10. The rate limiter
On the network thread, before anything else and on both transports, every connection passes an inbound limiter. It is on by default.
server::App app(world, {.rate = {.perSecond = 30, .burst = 60}}); // these are the defaults
Field of RateLimit | Default | Meaning |
|---|---|---|
perSecond | 30 | Messages a second sustained. 0 turns off the whole limiter: the message limit, the byte limit and the violation rules. |
burst | 60 | The bucket's size. 0 = perSecond. |
bytesPerSecond | 16384 | Bytes a second of the messages that are counted. 0 = no byte limit. |
burstBytes | 32768 | The byte bucket's size. 0 = bytesPerSecond. |
violationWindowMs | 2000 | The 2nd and 3rd violation count only inside this window of the 1st. |
perTick | 16 | The world reads at most this many messages of one client per tick; the rest waits in its input ring. 0 = all. (This is why an input ring can fill.) |
How it works. Each connection has a token bucket for messages and a second for bytes, refilled continuously (fractions are carried). A message needs one message token and, if the byte limit is on, tokens for its size including the type byte (a message larger than the bucket needs it full). The byte bucket catches thirty 4 KiB messages a second, which the message count would allow.
- Not counted: the ping echo (type 1) and the held-input message (type 8). Inputs have their own budget,
server::InputPolicy. Plain game messages are counted; a batch of up to 32 inputs is one message. - A violation is a message that finds an empty bucket, or one that passes the bucket but finds the world's input ring full (counted in
ringFull). - 1st violation: the message is dropped and the client is told with a type-11 warning whose text is built from the limits, for the defaults "too many messages: the limit is 30 messages a second (burst 60), 16384 bytes a second; what is over it is dropped". The client's
onWarninggets it. - 2nd inside the window: dropped silently.
- 3rd inside the window (2000 ms from the 1st): the connection is closed with code 4290 and the reason "closed for sending too much: the limit is 30 messages a second (burst 60), 16384 bytes a second";
Client::closedBy()isTooManyMessages. - A message that passes ends the run (the count resets), and a first violation after the window has run out starts a new run.
A limit per message type
on<Chat>([this](Player& p, const Chat& c) { broadcast(Said{String<16>(p.query("name", "anon")), c.text}); }) .limit(2, std::chrono::seconds(1));
on<Chat>(fn).limit(count, per) (or messages().limit<Chat>(count, per)) allows at most count messages of that type per per from one client, a bucket of count. It throws std::invalid_argument unless both are positive. Over the limit the message is dropped and counted (Dropped::perType), with the same warn, drop, close sequence: the warning reads "too many Chat messages: the limit is 2 per 1000 ms; what is over it is dropped", and the close, code 4290, "closed for sending too many Chat messages".
Counters and other limits
p.dropped() (and Client::dropped()) is Dropped{rate, ringFull, perType} with total(); Game::stats() is {dropped, closedForRate, closedStalled, badRadius} over all players, those who left included. A body of the wrong size is dropped and counted in Messages::dropped(); a throwing message handler is counted in failed() and reported (stderr, or onError); the tick carries on. maxMessageBytes (4096) closes a client that sends more; maxClients (1024) refuses connections; WebSocket maxQueryBytes (2048) answers 414.
Next: more than one game in a process.
11. Host: several games in one process
An App serves one WorldProcessor. server::Host is such a world that holds several Games and sends each client to one of them, by the URL it connected with.
World walker(store); Lobby lobby; // another server::Game server::Host host; host.add("meadow", walker).add("lobby", lobby, {.tickRate = 5}); // the first added is the default server::App app(host, {.tickRate = 20}); // ws://host:9000/?game=lobby
Notice:
- The client chooses with
?game=namein its query. Nogame=means the first game added. A name that does not exist disconnects the client. - Tick rates. A game ticks at the App's rate, or at that rate divided by a whole number (App at 20: 20, 10, 5, 4, 2, 1). Any other rate is an error at start that names the valid ones: "server::Host: game 'lobby' wants 7 ticks a second, but the server runs at 20. A game's rate must be 20 divided by a whole number (20, 10, 5, 4, 2, 1), or set the server's tickRate to a multiple of 7." Games of the same rate run on different ticks so their work does not pile onto one tick.
- A game's clients get its rate in the welcome, and its stamps count its own ticks.
- One thread. All the games run on the App's one world thread, one after the other. For more isolation run one
Appper game on its own port. - Add every game before
listen(); adding later throwsstd::logic_error("the server has started, and a game added now would never be opened. Add every game before App::listen() (in your Host's constructor)"). host.post(fn)runsfnon the world thread at the start of the next App tick, before any game's tick; any thread.
Next: the numbers.
12. The numbers
Numbers only mean something with their conditions. Every figure below is quoted with the conditions the source gives. None is a promise about your machine, your game or your network.
Arithmetic that motivated the crowd path (not measurements)
Everyone seeing everyone with 5,000 players is 25 million pairs; at 10 Hz and about 10 bytes per mover that is about 500 KB/s per client and 250 million pair evaluations a second. For a 100-player PvP fight: about 24 KB/s per client at 20 Hz and 10,000 pairs a tick. Before the cell blocks, the send was over 98% of the tick in a crowd, and 5,000 players kept 25 million per-pair records (0.6 GB).
The crowd path under a budget
Conditions: optimised build, 8 send threads, 100 real clients with the rest stand-ins, 10 Hz, a 200 m square, cell size 32, on a shared machine (so about plus or minus 3% on a send time). "Send" is the world thread's time in Interest::send(), mean and p99 in milliseconds over a 15 s window after a warm-up longer than the first fill (28 s at 5,000 players). Bytes are per client per second (budget times 10 Hz, plus the own row and headers). "Newcomer" is the time until a client that joins mid-run holds every row of the square. Tool: crowd_bench. The budget is bytes per send here.
| Players | Budget per send | Send ms, mean / p99 | Bytes per client per second | Newcomer sees the full square |
|---|---|---|---|---|
| 1,000 | 1,000 B | 0.63 / 0.77 | 10,316 | 3.6 s |
| 1,000 | 4,000 B | 0.74 / 1.00 | 40,474 | 0.9 s, 0.8 s |
| 1,000 | 16,000 B | 0.99 / 1.40 | 161,025 | 0.2 s |
| 1,000 | none | 0.94 / 1.11 | 227,807 | 0.1 s |
| 2,000 | 1,000 B | 1.25 / 1.53 | 10,382 | 7.3 s |
| 2,000 | 4,000 B | 1.36 / 1.69 | 40,143 | 1.71 s, 1.69 s |
| 2,000 | 16,000 B | 1.87 / 2.39 | 160,841 | 0.4 s |
| 2,000 | none | 3.59 / 4.62 | 457,277 | 0.09 s |
| 5,000 | 1,000 B | 5.22 / 6.16 | 11,201 | 19.4 s |
| 5,000 | 4,000 B | 5.72 / 6.63 | 40,223 | 4.2 s, 4.3 s |
| 5,000 | 16,000 B | 6.70 / 8.50 | 161,257 | 1.0 s |
| 5,000 | none | 17.47 / 20.24 | 1,150,075 | 0.12 s |
Reading it: with a budget the send time is nearly flat in the player count compared with no budget (5,000 players: 5.72 ms at 4,000 B against 17.47 ms unbudgeted), and the bytes per client follow the budget. The price is the time to fill a newcomer's view, which is roughly the square's size over the budget. The source records that the pass mark "send time falls with the budget" was met at 2,000 and 5,000 players, and at 1,000 players only for 1,000 B, 4,000 B and 16,000 B (16,000 B against none is a tie, 0.94 to 0.99 against 0.94 to 0.97 ms), so it is not strictly met there. In the same run, slice sizes were measured at 2,000 players with a 4,000 B budget and 8 threads: 512 B slices, 179 records per viewer per send, 5.9 ms; 1,024 B, 95 records, 3.4 to 3.7 ms; 2,048 B, 54 records, 2.4 to 2.6 ms; 4,096 B, 49 records, 2.3 ms, which is why slices are split above about 2 KiB.
Pacing GNS: WebSocket against GNS
Conditions (phase 0): a UDP relay link, one client in a dense crowd, two runs per row. At a 16,000 B/s link: WebSocket's capacity read 19,900 and it delivered 16,000 B/s with no drops; GNS without congestion control read 131,072, the relay dropped 1,430 datagrams, and the age of the view was at the measure's ceiling; GNS pinned at 16,000 B/s matched WebSocket (no drops). On an uncapped link, GNS's default never exceeded 131,072 B/s, against WebSocket's 687,000.
Conditions (kernel-shaped link, tests/netns): one connection, 20 s, a dense crowd, one run per row, drops counted from the kernel queue. GNS with congestion control received 15,522 B/s at a 16,000 B/s link with 50 ms delay (8 drops, down from 19), 58,683 B/s at 64,000/50 ms (8 drops, was 30), 226,321 at 250,000/50 ms (0 drops), and 14,776, 57,886 and 226,940 at the same rates with 200 ms (21, 20 and 46 drops, down from 41, 94 and 286). All the drops are in the first seconds: GNS sends at 131,072 B/s for about 0.7 s before the first sample. With 1% and 5% random loss at 64,000 B/s it received 58,052 and 54,372 B/s.
Weak spots the source lists: a busy connection on a path with room pays about 60 ms (measured in a 100-player fight over the relay: own row p50 105 ms and p99 173 ms with congestion control; 47 and 90 ms with it off; WebSocket 30 and 55 ms); twenty connections through one bottleneck are not solved (in the model the median loss was 33%; for two connections the median was 2.6% and the worst 8.7%); shallow queues lose about 2% in a probing round; on a slow link the lowest ping reads high; and a collapse of two orders of magnitude leaves most of a minute of data in flight. The status read for 4,000 connections cost 5.5 ms of the loop thread.
Reference
Every public item of the network layer. "Throws" means an exception; defaults are those of the headers. Times are milliseconds unless the name says otherwise, rates are bytes a second unless stated.
server::Options
struct Options { /* the fields below */ void validate() const; }; // throws std::invalid_argument naming the field
validate() checks every field at once and is called by every piece, so a bad Options is reported by whichever piece is built first. The text is aether::server::Options::<field>: <reason>. The sizes of the buffers are rounded up to a power of two.
| Field | Type | Default | Valid, and what a bad value does | Meaning |
|---|---|---|---|---|
tickRate | int | 20 | clamped to at least 1 | World ticks a second. |
worldBufferBytes | size_t | 1 MiB | raised to fit a connect notice, at least 32 KiB; rounded up to a power of two | The world buffer, any thread to the world. |
pingIntervalMs | int | 1000 | negative throws | How often the world pings each client; 0 = no pings (rttMs stays 0). |
rttCapMs | int | 250 | below 1 throws | A round-trip sample above this counts as this. |
rttMaxSampleMs | int | 2000 | below 1 throws | An echo whose round trip is longer than this is ignored. |
rttSmoothing | int | 4 | below 1 throws | rtt += (sample - rtt) / rttSmoothing; 1 = no smoothing. |
worldIdle | Idle | FrameSync | The world thread's pacing (see Idle). | |
worldThreadName | std::string | "aether-world" | The name of the world thread (the OS keeps 15 characters). | |
onError | std::function<void(const std::exception&)> | empty: stderr | Called when the world's or the network agent's work throws (the agent then carries on), on the thread that threw. | |
flushRate | int | 30 | clamped to at least 1 | Flush requests a second. |
networkIdle | Idle | FrameSync | The flush pump's pacing. Keep FrameSync (see section 3). | |
networkThreadName | std::string | "aether-network" | The name of the network thread. | |
host | std::string | "" (all interfaces) | a host that cannot be bound makes listen return false | Bind address: an IPv4 or IPv6 literal, or a name for WebSocket. |
stopPollMs | int | 50 | below 1 throws | How often the WebSocket loop checks for stop(). GNS reacts within pollIntervalMs. |
maxClients | size_t | 1024 | Further connections are refused (WebSocket: 503; GNS: closed "refused"); 0 refuses all. | |
maxMessageBytes | size_t | 4096 | below 1 throws | The largest message a client may send; a larger one closes it. |
rate | RateLimit | see below | a negative member throws | What a client may send. On by default. |
stalledSeconds | int | 300 | negative throws | Close (4291) a connection whose output has been blocked this long; 0 = never. |
idleTimeoutSeconds | int | 32 | outside 0..65535 throws | Close a silent connection; 0 = never (GNS: about 24 days). |
inputBufferBytes | size_t | 32 KiB | rounded up to a power of two of at least 1 KiB; raised to 8 x maxMessageBytes if smaller | Per client, client to world. The largest message is 1/8 of it. |
outputBufferBytes | size_t | 128 KiB | rounded up to a power of two of at least 1 KiB | Per client, world to client. The largest message is 1/8 of it (16 KiB). |
backpressureBytes | size_t | 64 KiB | Stop writing to a connection with this much unsent. | |
linkReportIntervalMs | int | 250 | negative throws | How often the world gets each client's byte rates and backlog; 0 = never (the link's rates stay 0). |
maxMessagesPerFlush | int | 0 | negative throws | Most messages written to one client per flush; 0 = no limit. |
handleSignals | bool | true | SIGINT and SIGTERM stop the App. | |
webSocket | WebSocketOptions | see below | WebSocket only. | |
gameSocket | GameSocketOptions | see below | GNS only. |
Fixed on purpose, not settings: the reserved message types (ping is 1), the layout of the ping echo, the link report and the connect notice, a ring's message limit of capacity/8, the WebSocket frame arithmetic used for byte counts, and the batch of 1024 messages a flush reads from a ring per call.
Example: server::App app(world, {.tickRate = 30, .maxClients = 200, .rate = {.perSecond = 60, .burst = 120}, .stalledSeconds = 60}); (fields in declaration order).
enum class Idle { FrameSync, FrameSyncBackoff }: a steady rate, or a rate that halves after each tick that did no work, down to 1 a second, and returns to full as soon as a tick does work.
server::RateLimit
| Field | Default | Meaning, and 0 |
|---|---|---|
int perSecond | 30 | Messages a second sustained. 0 = no limit at all (the byte limit and the violation rules go with it). |
int burst | 60 | The bucket's size. 0 = perSecond. |
int bytesPerSecond | 16 * 1024 | Bytes a second of counted messages. 0 = no byte limit. |
int burstBytes | 32 * 1024 | The byte bucket's size. 0 = bytesPerSecond. |
int violationWindowMs | 2000 | The 2nd and 3rd violation count only inside this window of the 1st. |
int perTick | 16 | Messages of one client the world reads per tick. 0 = all. |
server::WebSocketOptions
| Field | Default | Valid | Meaning |
|---|---|---|---|
std::string path | "/*" | starts with / | The URL pattern that upgrades to a WebSocket (a uWebSockets pattern: "/*", "/game"). |
std::string healthPath | "/health" | empty or starts with / | GET here answers ok; empty = no health endpoint (404). |
size_t maxQueryBytes | 2048 | 1..60000 | Longest connect query accepted (longer: 414). Also sizes the world buffer's minimum. |
bool keepAlivePings | true | uWebSockets sendPingsAutomatically: protocol-level WebSocket pings, not aether's type-1 ping. | |
bool compression | false | permessage-deflate with a shared compressor; byte counters then count uncompressed sizes. | |
size_t maxBackpressureBytes | 0 | uWebSockets maxBackpressure: unsent bytes per socket uWebSockets tolerates; 0 = no uWebSockets limit (the pipeline applies backpressureBytes). | |
bool closeOnBackpressureLimit | false | needs maxBackpressureBytes > 0, else throws | Close a socket that passes maxBackpressureBytes. |
bool resetIdleTimeoutOnSend | false | uWebSockets resetIdleTimeoutOnSend: sending counts as activity for the idle timeout. | |
int maxLifetimeMinutes | 0 | 0..65535 | Force-close a socket after this many minutes; 0 = never. |
bool thinStream | true | Linux TCP_THIN_LINEAR_TIMEOUTS on every accepted socket: with fewer than four packets in flight a lost packet is retried at a steady interval rather than a doubling one. Retransmit timing only, never what is sent. No effect on other systems. |
server::GameSocketOptions and ConfigValue
| Field | Default | Valid | Meaning |
|---|---|---|---|
int pollIntervalMs | 1 | not negative | Sleep between loop iterations; 0 = no sleep (a busy loop, a full core). |
int maxMessagesPerPoll | 64 | at least 1 | Messages taken from the poll group per receive call (the loop calls it until empty). |
size_t deferQueueBytes | 64 KiB | Capacity of the defer() task ring, rounded up to a power of two (at least 4 KiB); a full ring makes defer() fail. | |
int stopDrainMs | 200 | not negative | At stop, wait at most this long for what GNS accepted to be sent and acknowledged; 0 = do not wait. |
int sendBufferBytes | 0 | not negative | GNS SendBufferSize per connection; 0 = GNS's 512 KiB. |
bool congestionControl | true | Pace GNS at the measured path rate (section 9). Off: GNS's own fixed rate, no backpressure, capacity reads GNS's rate. | |
int sendRateMin, int sendRateMax | 0, 0 | not negative | Bytes a second; 0 = none. With congestion control: the lowest and highest rate GNS is paced at. Without: GNS's own SendRateMin and SendRateMax (0 = GNS default). |
int nagleTimeUs | -1 | GNS NagleTime in microseconds; 0 = no Nagle delay; negative = GNS default. | |
int mtuPacketSize | 0 | not negative | Largest UDP payload sent, bytes; 0 = GNS default. |
int timeoutInitialMs | 0 | not negative | How long a connect may take; 0 = GNS default. |
int timeoutConnectedMs | 0 | not negative | Silence before a connection is dropped; 0 = derived from idleTimeoutSeconds. |
bool noNagle | false | Send every message with the NoNagle flag (at once instead of batched; defeats cork). | |
bool flushAfterCork | true | Connection::cork flushes the connection after its writes, so they leave together now. | |
int helloWaitMs | 0 (10000 under listen(GameSocket)) | 0..60000 | Section 4. |
float fakePacketLossSendPct, fakePacketLossRecvPct | 0, 0 | 0..100 | Percent of outgoing / incoming packets dropped, for tests. Global in GNS: affects every GNS connection in the process while the server loops; reset when loop() returns. |
int fakePacketLagSendMs, fakePacketLagRecvMs | 0, 0 | not negative | Delay added to outgoing / incoming packets; same global rule. |
std::function<void(int level, std::string_view message)> debugOutput | empty (GNS prints nothing) | GNS's debug hook, process-wide: installed while the server loops, removed when loop() returns. Called on GNS's own threads, possibly concurrently: it must be thread safe. Replaces any hook the program installed. | |
int debugLevel | 4 | 1..8 | 1 bug, 2 error, 3 important, 4 warning, 5 msg, 6 verbose, 7 debug, 8 everything. |
std::vector<ConfigValue> config | empty | Raw GNS config values, applied last (they win over the fields above). A key GNS defines only globally is applied process-wide and reset when loop() returns. |
struct ConfigValue { int key = 0; std::variant<std::int32_t, std::int64_t, float, std::string> value; }: key is an ESteamNetworkingConfigValue number from GNS's steamnetworkingtypes.h, and the value's type must match what GNS documents for that key.
server::App, GameSocket, Transport, Server
| Signature | Behaviour |
|---|---|
explicit App(WorldProcessor& world, Options options = {}) | Not copyable. Reads every field of options and passes them to the pieces. listen validates them (throws std::invalid_argument). |
[[nodiscard]] bool listen(std::uint16_t port, std::function<void(std::uint16_t port)> onListening = {}) | WebSocket. Blocks until stopped. onListening is called once with the bound port (0 picks a free one). False if the port could not be bound or onStart threw; true after a clean stop. |
[[nodiscard]] bool listen(const Transport& transport, std::function<void(std::uint16_t)> onListening = {}) | The same over another transport. transport is only used while listen starts. |
void stop() | Closes every connection, delivers the disconnects, calls onClose; listen returns. Safe from any thread and from a signal handler. |
struct GameSocket final : Transport { std::uint16_t port; explicit GameSocket(std::uint16_t p); } | The GNS transport (GameSocketServer.h, link aether_gns). Port 0 picks a free UDP port. make() builds the server; a 0 helloWaitMs becomes 10000 in a copy. |
struct Transport { virtual std::unique_ptr<Server> make(ConnectionHandler&, const Options&) const = 0; } | Which flavour of Server an App listens with. |
class Server: virtual bool loop(), bool loopOn(std::thread&), virtual void stop(), virtual bool defer(std::function<void()>), virtual std::uint16_t port() const | loop() binds and runs on the calling thread until stop(); false if it could not bind; before returning it closes every connection and waits until ConnectionHandler::pending() is false. loopOn starts it on a new thread and returns once it is listening (false if it could not bind: join the thread). stop() is safe from any thread and from a signal handler. defer runs a task on the server's thread, soon; false if not looping (the task is dropped). port() is the bound port while listening, else 0. |
WebSocketServer(std::uint16_t port, ConnectionHandler&, Options = {}), GameSocketServer(...) | Not copyable; nothing is bound until loop(); the constructor throws std::invalid_argument for an invalid Options. GameSocketServer::loop() returns false at once if another is looping in the process. Port 0 picks a free port (GNS refuses 0 itself, so the server finds one with a plain socket and binds it, retrying if another process takes it in between). |
detail::tcpInfoReads() | How many getsockopt(TCP_INFO) calls the WebSocket server has made in this process (for tests). |
NetworkProcessor, WorldProcessor, WorldManager, Sessions
| Item | Behaviour |
|---|---|
NetworkProcessor(Server&, ConnectionHandler&, Options = {}) | The flush pump. Reads flushRate, networkIdle, networkThreadName, onError. void start() runs doWork() on a new thread at flushRate; void close() stops and joins (harmless if not started); int doWork() makes one flush request (at most one pending) and returns 1 if the last flush did any work, else 0. The destructor closes. |
WorldProcessor | Derive and implement the protected methods. void start(WorldManager&, const Options& = {}, std::function<void()> onStartFailed = {}) runs the world on its own thread (returns once onStart finished or threw; throws std::logic_error if already started or attached; if onStart throws, the world never ticks, onClose is not called, onStartFailed is called once). void attach(WorldManager&, const Options& = {}) calls onStart on the calling thread, then you call int tick() yourself. void close() stops, reads the world buffer one last time, calls onClose exactly once, frees the clients; do not call it from the world thread. Protected: tickNumber() (1 during the first tick, always larger than the last: what registry.commit takes), tickRate(), rate() (the RateLimit), nowMs() (read once per tick, steady clock, never goes backwards), and the virtuals onStart(WorldManager&), onConnect(Client&), onDisconnect(Client&), onMessage(typeId, span), doWork() (returns the work done, 0 for none), onClose(). Reads tickRate, worldIdle, pingIntervalMs, rttCapMs, rttMaxSampleMs, rttSmoothing, worldThreadName, onError. World buffer type ids from kFirstReservedWorldType (1 << 30) up are the pieces' own; yours are 1 to that minus 1. Close a world before destroying the object you derived. |
WorldManager(const Options& = {}) | Owns the world buffer, a many-to-one ring any thread may write (worldBuffer()); the world reads it at the start of every tick, and records of your own type ids reach onMessage. Reads worldBufferBytes and webSocket.maxQueryBytes; validates the options. |
Sessions makeSessions(WorldManager&, const Options& = {}) | The default session pipeline for a server layer: per connection it allocates the rings, tells the world about the connect and disconnect, checks the wire envelope, moves client messages into the input ring and the world's into the connection, and reports the link. |
server::Host and HostGameOptions
| Signature | Behaviour |
|---|---|
Host& add(std::string name, Game& game, HostGameOptions options = {}) | Adds a game (the first one is the default). Returns *this. Throws std::logic_error after the server has started. The games are not WorldProcessors here: Host drives them through open, admit, drop, advance. |
struct HostGameOptions { int tickRate = 0; } | Ticks a second; 0 = the App's. Must be the App's rate divided by a whole number. |
static int everyFor(int base, int rate, std::string_view game = {}) | How many App ticks make one tick of a game that wants rate on an App at base. Throws std::invalid_argument naming the rates that would work if rate is not base divided by a whole number. |
Game* game(std::string_view name) | The game of that name, or null. |
void post(std::function<void()> fn) | Runs fn on the world thread at the start of the next App tick, before any game's tick. Any thread. |
server::Link, Dropped, Client
Link fields: see section 8 for the meaning and the measurement of each. struct Link { uint16 rttMs = 0, jitterMs = 0; float loss = 0; uint32 capacity = 0, sending = 0, receiving = 0, backlog = 0, queueMs = 0, left = 0; uint16 interpMs = 0; uint32 limit = 0; bool congested() const; uint32 headroom() const; }. Dropped { uint64 rate = 0, ringFull = 0, perType = 0; uint64 total() const; }: messages dropped since the client connected. RttSettings { capMs = 250, maxSampleMs = 2000, smoothing = 4 } is what the world hands to each Client from the options.
| Client member | Behaviour |
|---|---|
ClientId id() const, std::string_view query() const, std::string_view address() const | A process-unique id (never reused); the connect query ("token=abc&map=3"); the remote address text. |
void disconnect(), void disconnect(std::uint16_t code, std::string_view reason) | Close after the messages already in the output buffer are written; onDisconnect still follows. World thread; harmless to repeat. The second form tells the client why: code 3000..4999 is the game's (4000 is "rejected"), reason at most 123 bytes (cut if longer); only the first call's reason counts. |
ClosedBy closedBy() const | None (the game's own disconnect), TooManyMessages, OutputStalled. |
const Link& link() const | Updated while the client's input is read. Player::link() adds left and the live backlog. |
Dropped dropped() const, size_t outputBytes() const, std::uint32_t socketBytes() const, float outputFill() const | Drop counters; bytes in the output ring not yet taken; bytes the connection holds not on the wire as of the last flush pass (live); the output ring's fill, 0..1. |
OneToOneRingBuffer& input(), output() | The raw rings. An output record is the whole wire message, type byte first, with the type as its ring type id. If you read the input ring yourself, pass every record to handleSystem first. |
[[nodiscard]] SendResult send(std::uint8_t type, std::span<const std::uint8_t> body), send(type, size, write), template <class M> send(const M&) | Copy a message into the output buffer (types 2..255; with Interest, yours start at 16). Zero copy form: write is not called unless the result is Ok; if it throws nothing is sent. Types 0 and 1 throw std::invalid_argument. Ok, Full (no room), TooLarge (never fits; more than 1/8 of the ring). The struct form needs the client attached to an Interest (ids() throws std::logic_error otherwise). |
int receive(OnMessage&&, int limit = INT_MAX), bool handleSystem(...) | Calls onMessage(type, body) for each message the client sent, oldest first, at most limit; returns how many. Ping echoes and link reports are consumed here, updating link(). |
Player: network members
| Member | Behaviour |
|---|---|
Link link() const | By value. left filled in; backlog is the socket's live figure plus the output ring; queueMs includes the ring part. |
std::chrono::milliseconds crowdDelay() const, focusDelay() const | The longest wait of a crowd cell / of a focused entity in the last send. 0 before the first send, with no crowd, or none focused. |
void kick(std::string_view reason) | client.disconnect(4001, reason). |
Dropped dropped() const | Messages of this player dropped instead of handled. |
void show(ecs::Entity), hide(ecs::Entity) | show throws std::logic_error if the Players has no interest; hide does nothing then. Applied at attach if called in onJoin. |
void focus(ecs::Entity), unfocus(ecs::Entity) | Same: focus throws, unfocus does nothing. |
std::string query(std::string_view key, std::string fallback = {}) const, T& user<T>() | A ?key=value of the connect query (the hello over GNS); a per-player struct of your own, made on first use (one type per game: another type throws std::logic_error). |
Game::stats() returns Stats { Dropped dropped; std::uint64_t closedForRate, closedStalled, badRadius; } since the Game started, over every player (those who left too). Queued messages dropped are Players::dropped(); message handler failures are Messages::failed() and Messages::dropped().
GameOptions: the network fields
In declaration order (a designated initialiser must keep it): interest (InterestPolicy), inputs (Server), radius, budget, weight, focus, cellWeight, nearReserve, crowdEvery, cut, idleHeaderTicks, then store, contract, loadTimeout.
| Field | Type | Default | Section |
|---|---|---|---|
radius | Rule<float, const Player&> | unset: 64 | 6 |
budget | Rule<std::size_t, const Player&>, bytes a second | unset: no limit | 7 |
weight | std::function<float(const Player&, ecs::Entity other, float distance)> | unset: 1 / (1 + distance) | 7 |
focus | std::optional<float> | nullopt: 0.5 (or .interest = {.focus = ..}) | 7 |
cellWeight | std::function<float(const Player&, const ecs::Cell&)> | unset | 7 |
nearReserve | ecs::NearReserve | {30, 1.5f, 0.25f} | 7 |
crowdEvery | Rule<std::uint32_t, const Player&> | unset: every send | 7 |
cut | std::function<ecs::Cut(const Player&, const ecs::Cell&)> | unset: Cut::Low | 7 |
idleHeaderTicks | std::uint32_t | 0 (the tick rate) | 5 |
server::Rule<T, Args...> is "a number or a function": Rule(T value), Rule(F fn) for a callable returning something convertible to T; bool set(), T value(), bool isFunction(), T operator()(Args...). The SDK never decides the number: what the function returns is used. Unset: T{}; check set().
ecs::InterestPolicy, NearReserve, Cell, Cut
| Field | Default | Meaning |
|---|---|---|
float cellSize | 32 | Grid cell size, world units; finite and above 0. |
Rule<float, const Client&> radius | unset: 64 | Radius per viewer; not clamped. |
bool stopOnBadRadius | true unless NDEBUG | Stop on a NaN or negative radius, else use 0 and count. |
std::uint32_t cap | 2000 | Subscriptions per viewer (cap 0 is refused at construction). |
std::uint32_t unsubscribeDelay | 3 | Sends out of sight before Unsubscribed; 0 = first send. |
std::uint32_t maxIgnoreMs, maxIgnored | 600000, 4096 | Longest ignore; ignore entries per viewer. |
std::function<size_t(const Client&)> budget | unset: no limit | Bytes per send; 0 = only headers and Unsubscribed. |
weight, weightFor | unset | (Entity viewer, Entity other, float distance); or taking the Client (used instead of weight if set). Called only when a budget is set; threads as in section 6. |
float focus | 0.5 | Exact-entries share of the budget. |
cellWeight, cut | unset | Taking (const Client&, const Cell&). |
NearReserve nearReserve | {30, 1.5f, 0.25f} | Section 7. |
Rule<std::uint32_t, const Client&> crowdEvery | unset: every send | 1..255. |
std::uint32_t workers | 1 | Section 6. |
tickType, subscribeType, ignoreType, dontKnowType | 2, 3, 4, 5 | Each must be above 1; the four (five with sendLayouts, adding layoutType) must differ. |
bool sendLayouts, layoutType | true, 6 | Send each viewer the layout of every view once. Off: the game sends them. |
bool sendWelcome, welcomeType | false, 7 | The welcome. Off by default in the low-level Interest; a server::Game always has it on. |
bool ownerRowEveryTick | false | The viewer's own row goes in every send, also for a view sitting out under every(k). Always on in a Game. |
std::uint32_t idleHeaderTicks | 1 (a Game sets the tick rate) | At least 1. |
NearReserve { std::uint32_t after = 30; float within = 1.5f; float share = 0.25f; }. Cell { std::int32_t x = 0, y = 0; float distance = 0; std::uint32_t entities = 0, bytes = 0, waited = 1; bool isNew = false, mine = false; }. enum class Cut : std::uint8_t { Never, Low, LowAndNormal }. enum class Entry : std::uint8_t { Full = 0, Delta = 1, Unsubscribed = 2, Snap = 3, Leave = 4 }; kFirstGameType = 16.
ecs::Interest: public members
The low-level piece Game is built on. World thread only; one per Registry; it must not outlive the registry, and every client must be detached before it is destroyed. Interest(Registry&, InterestPolicy = {}) throws std::invalid_argument for a policy that breaks a rule (cell size, cap 0, message types) and std::logic_error if the registry already has one.
| Member | Behaviour |
|---|---|
policy(), registry() | The policy and the registry. |
setTickRate(int), setInputRate(int), setIdleHeaderTicks(std::uint32_t), tickRate(), inputRate() | The rates the welcome announces (set before the first attach; inputRate() is never 0: the tick rate if unset). |
attach(Client&, Entity player), setPlayer(Client&, Entity), detach(Client&), attached(const Client&) | attach throws std::logic_error if already attached or if the registry has no subscribable view. setPlayer (respawn) re-sends the welcome; throws if not attached. detach sends nothing; a no-op if not attached. |
show(client, view, e), hide(...), show(client, e), hide(client, e), focus(client, e), unfocus(client, e) | Section 6. The view form throws std::logic_error if the view is not subscribable; all throw if the client is not attached. |
snap(Entity) | This commit's change of e is a snap, in every view: each viewer that has e is sent a Snap entry in the first send that includes it (also under every(k) and a budget). Call it in the tick that changes the row, before the commit. |
handle(client, type, body[, nowMs]) | True if type is the subscribe, ignore or don't-know type (handled or dropped). Throws std::logic_error if the last commit had no time (without nowMs). |
setInputSequence(client, std::uint32_t) | The ack that travels in the tick header. |
std::size_t send(), send(std::int64_t nowMs) | Sends every viewer its tick message for the last commit; returns the bytes written (bodies without the type byte). Before the first commit it does nothing. |
ids(), enableWelcome(), observe(...), unobserve(...) | The id table; welcome and owner-row on; hooks for attach/detach/player change. |
radius(client), subscribed(client[, view[, e]]), ignored(client[, view, e]), dropped(client), deferred(client), budget(client), crowdDelay(client), focusDelay(client), layoutsPending(client), entriesLast(client) | Per viewer, throwing std::logic_error if not attached: the radius of the last send; subscription counts; ignore counts; dropped client messages; Delta and Full entries the last send left out or cut; the last send's budget (SIZE_MAX: no limit); the longest waits; views whose layout has not gone out; entries written last send. |
viewers(), badRadius(), unplaced(), lastSendStats() | Viewer count; radius answers that were NaN or negative; rows not in a grid because a position is not finite. lastSendStats() is benchmark instrumentation. |
spatial::Grid<Id>
#include <aether/spatial/Grid.h> aether::spatial::Grid<int> grid(32.f); auto h = grid.insert(7, 10.f, 20.f); // id, x, y [, height [, radius]] grid.move(h, 12.f, 20.f); std::vector<int> near; grid.queryCircle(0.f, 0.f, 50.f, near); // appends ids within 50 of (0, 0) grid.remove(h);
| Member | Behaviour |
|---|---|
explicit Grid(float cellSize, float cellHeight = 0.f, std::size_t expectedEntities = 1024) | Throws std::invalid_argument unless cellSize is finite and above 0 and cellHeight finite and at least 0. cellHeight > 0 adds vertical cells (use for real vertical spread; in a mostly flat world the height filter alone is cheaper, 9 cells instead of 27). |
| Handle insert(Id id, float x, float y, float height = 0.f, float radius = 0.f) | x and y are the two ground axes of your world. Throws std::invalid_argument if a coordinate or the radius is not finite or the radius is negative; std::length_error for too many entities. Handles are reused after remove. |
void move(Handle, float x, float y, float height = 0.f) | Cheap when the entity stays in its cell. Throws std::invalid_argument for a non-finite coordinate. An unknown handle is ignored. |
void remove(Handle) | An unknown handle (kNoHandle, already removed) is ignored. |
size(), cellSize(), cellHeight() | Entities in the grid; the sizes. |
visitCircle(x, y, range, visit, HeightBand = {}), visitBox(x, y, halfX, halfY, visit, band = {}) | Calls visit(const Id&) once for every match: the distance to the entity's centre at most range + radius (box: its circle overlaps the rectangle). No order. A non-finite centre or a negative or NaN range matches nothing. Do not insert, move or remove inside visit. With vertical cells, give every query a HeightBand. |
queryCircle(x, y, range, std::vector<Id>& out, band = {}), queryBox(...) | Appends the matching ids to out (not cleared). |
struct HeightBand { float min = -inf, max = inf; } | Matches entities whose [height - radius, height + radius] overlaps [min, max]. |
Not thread safe: use one grid from one thread. No allocation by move, remove or a query in steady state. Cell coordinates are clamped to plus or minus 2^30.
The transport's contract: server::Connection, PathInfo, ConnectionHandler
For writing a transport of your own (Transport, Server). PathInfo { static constexpr uint32 kUnknown = 0xFFFFFFFF; uint32 capacity = 0; float loss = 0; uint32 queueMs = kUnknown; bool appLimited = false; bool measured = false; uint64 bytesAcked = 0; } is what a transport says about the path to a peer, read about every linkReportIntervalMs; a field it cannot give is 0 (or kUnknown for queueMs, which is then worked out from the backlog). measured means capacity is the transport's own estimate and the client uses it as is; otherwise the client runs a CapacityEstimator on it.
Connection virtual | Meaning |
|---|---|
query(), address() | The connect parameters; the remote address. |
bool send(std::span<const std::uint8_t>), cork(const std::function<void()>&) | Queue one whole message (false if closed); run the writes now so a transport that can batch puts them on the wire together. |
buffered(), full(backpressureBytes), queued(), path() | Bytes accepted not yet sent; whether the write pass should stop (default buffered() > bp); what was last measured of the queue without a guess (default buffered()); the PathInfo (default nothing). |
end(), end(code, reason), close() | Close after what was sent has gone out (the second form with a close code and reason, if the transport has them); close now. |
bytesIn, bytesOut, session | Wire bytes counted by the server layer (whoever reports them resets them); the pipeline's own state. |
ConnectionHandler (what a server layer calls; the session pipeline implements it): onConnect, onMessage(Connection&, span) (one whole binary message; may close the connection), onWritable, onDisconnect, flush() (writes what the world queued and does other periodic work; the NetworkProcessor asks for it at its rate, and the server layer calls it while stopping), pending() (true while there is work only flush() can finish).
Message types and close codes
| Constant | Value |
|---|---|
server::kPingType | 1 |
aether::kWelcomeType, kInputType, kHelloType, kIdsType, kWarningType | 7, 8, 9, 10, 11 |
aether::kFirstGameType | 16 |
aether::kCloseTooManyRequests, kCloseStalled | 4290, 4291 |
Game-chosen: p.kick(); refusal | 4001; 4000 |
See also: Client for what the other end does with all of this, Views for spatial, every, history, priority and delta(Tier), Server for Game and its inputs, and Testing for the test server that runs this stack in a single test.