Rune API
Connect, log in, define a typed topic, produce, subscribe, run a transaction, pick a delivery guarantee, have the broker check a write, query a view, hold a lease, grant access, handle errors.
Connect
One session to any node; leader redirects and reconnects are handled for you.
from your game loop
Authentication
On by default. Each broker session logs in with SCRAM-SHA-512; the password never crosses the wire. A client without credentials is answered AuthenticationFailed.
- 44 AuthenticationFailedWrong or missing credentials. Ends the session. A rate-limited login is retried after the broker's hint.
- 45 NotAuthorizedNo grant for that request. Fails only that request and is never re-sent.
- 46 SessionRevokedThe user was dropped, disabled or terminated. Terminal: no reconnect.
- 47 PasswordChangeRequiredA one-time password may only be changed.
A client with credentials refuses a broker that skips the login, unless allowUnauthenticatedBroker is set. Revoking READ ends a live subscription with SubscriptionFailReason::NotAuthorized, a group consumer with GroupFailReason::NotAuthorized.
Topics
Defined in Rune QL from the rune console. A typed topic has at most one key field and validates every write; a record goes to the partition its key hashes to.
replication, durability, retention
transient topic: replicated in memory, never written to disk. For high-rate, short-lived state.
autoincrement: the broker assigns the key of every record written with key 0
The key is a uint64, int64 or uint32. It needs a new, persisted, compacted topic without retention limits, and it turns on key.index and modulo routing: a record goes to partition id mod partitions. The ids of partition p of P are n * P + p: dense per partition, never reused, with gaps after a failover, a rejected record or an aborted transaction. Past autoincrement.max a create is refused with 58 AutoIncrementExhausted and nothing is stored. The key field, its type and the partition count never change. Writing one: Checked writes.
import: write rows with the ids they already have
A record with an explicit key is an import: an Insert stored as sent, after the counter is moved above it. It needs the IMPORT grant, which nothing implies, and a key at most autoincrement.max that belongs to its partition. A plain upsert is refused. Live creates that follow get fresh ids.
generate C++ types from the registry
or define a topic from code
Produce
Acks::None · Leader · All, plus the View flag (Checked writes). trySend is the non-blocking form.
many records: sends within lingerUs are batched automatically
many records as one atomic append to one partition
On a topic with constraints the batch is also judged as one: a single rejected record rejects the whole batch (an atomic batch, kAttrAtomic). Sent with b.send(cq, token), it completes once with the code, the constraint id and the batch's base offset, and none of its records become visible.
zero-copy: write the record straight into the send buffer
zero-copy batch
typed topic: generated flyweight, routed by the key
Checked writes
The broker judges every write against the rules of the topic's views and key index. A rejected record is stored but never visible: no view folds it, no consumer receives it. Add Acks::View and the writer is told the verdict. Needs a broker with checked writes; an older one completes the send OpStatus::BrokerTooOld and nothing is judged.
typed writes: insert, update, upsert, remove, with a verdict
insert | 56 KeyExists when the key is stored. A key-0 create on an autoincrement topic is an insert |
update · removeExisting | 57 KeyNotFound when the key is absent |
upsert · remove | never refused by the key index. A delete is never refused by a constraint either |
Insert and update need a topic with a key index (with (key.index = on), implied by autoincrement) and an idempotent producer with View, or a transactional one. Plain upserts and deletes need neither. A local refusal throws std::invalid_argument from the …AndWait forms and returns SendResult::WriteNotAllowed from trySend.
what a rejection carries
r.code() | 55 ConstraintViolation from a unique or check view, 56, 57 from the key index; 0 when the write was accepted |
r.constraintId() | viewObjectId << 16 | ordinal: ordinal 0 is the view's unique, k its k-th constraint. 0 is the key index |
r.offset() | the rejected record's own offset |
r.key() | the id the broker assigned to a key-0 create; 0 for every other write and for an import |
client.constraintName(id) | {view, constraint}, or std::nullopt for 0. A dropped or replaced view still resolves. A miss waits for one catalog refresh (at most 5 s): call it only from the thread that owns the Client |
racing writers: the loser reads the winner's row from the unique view
batches report every record
A claimBatch or raw Tier-2 batch of two or more records stores the accepted records and reports the rest. An atomic p.batch(...) is all or nothing and completes BrokerError once, with the constraint id and the base offset. A batch of one record keeps the single-record form. A View batch holds at most 4096 records.
autoincrement create
A key-0 create needs an idempotent producer with Acks::All, plus View where a constraint can reject it. A producer never mixes key-0 creates with other records in one batch. A lost ack is retried and answers the original ids; a retry the broker can no longer match completes StoredIdUnknown: it was stored, its id is unknown. After an insertAndWait times out, look the row up through a unique view before inserting again. Explicit ids are imports: they answer key() 0 and need the IMPORT grant.
A retry inside the broker's window replays the original verdicts. Outside it the write completes OpStatus::VerdictUnknown: stored, verdict unknown.
Subscribe
Push delivery, in order per partition. key and value are zero-copy spans, valid inside the handler.
Consumer group
Register before connectAndWait(). Offsets auto-commit; gc.commit() commits on demand.
Transaction
Any producer with a transactionalId. Every produce path works inside it: send, batch, tryClaim, claimBatch, typed claim. ReadCommitted readers see the trade only after commit.
constrained sources: a no vote is a result, not an exception
At commit the leader of every partition the transaction wrote judges all its records together against every unique, check and key-index rule, and one no aborts the whole transaction. A commit that returns ok means the rows are final on every partition; View on a transactional producer is ignored. A retried commit after a coordinator failover gets the same no. Every other failure still throws. From a yes vote to the commit or abort marker the rows it wrote are locked: another writer's record on one of them waits for the marker. Deletes are never rejected. Within a partition a reader sees a transaction entirely or not at all.
- 62 TxnTooLargeMore than
RUNE_TXN_JUDGED_RECORDS_MAX(default 4096) records on one partition. A send that would pass the limit is refused with the same code. - 63 TxnConflictAnother writer wrote one of its keys after this transaction's write and before its commit.
Delivery guarantees
Chosen per producer and per consumer group.
at most once: no ack, no retry; commit before handling
at least once: retried until acked, deduplicated by the broker; commit after handling
exactly once: consume, produce and commit the input offsets in one transaction
Materialized views
Defined in Rune QL, kept current on every write, read by key. Persisted by default: checkpointed and recovered on restart, no rebuild. Add transient to keep one in memory only. No separate database.
one row per character, keyed by account and character
per-user stats over all of a user's characters
rules the broker enforces: one character per name (any case), at most 12 characters per user
A write that breaks a rule is refused with 55 ConstraintViolation and never becomes visible. A transaction that breaks one fails at commit with the same code and the constraint id, and nothing in it is applied.
unique projection view | no two source records may produce one view key. key lower(name) makes names unique in any case. A unique aggregate is refused (61) |
constraint … check (…) | a predicate over the view's own row (= <> < <= > >= against a constant, joined by and or not). A record is refused when its row fails it and the record changes a column the predicate reads. Projection or aggregate views |
| key index | the Insert and Update intents, 56 and 57 (Checked writes) |
A record that touches several constrained views must pass all of them, or changes none. Records in a batch are judged one at a time, in order, each seeing the accepted ones before it. Deletes are never rejected, so model a ledger as appended entries. A group already over its limit can shrink and never grow. NULL in a check passes. A unique view built over existing duplicates lets the smallest source key own the name; rune stats counts the other claimants as uniqueConflicts, and they can only shrink. Creating a constraint on a large topic pauses delivery to its consumers until the view has caught up; writes are never refused for it. create or replace keeps judging with the old definition until the new one has caught up. A constraint needs a persisted source (else 39), and at most 32 verdict views per topic (the key index counts as one), a view object id below 65536 and a plan record of at most 65,536 bytes (else 60).
wide and derived keys: up to 128 bytes of columns, lower() of a string
A key component is an integer, bool or string(N) column; lower(name) as nameKey keys a string(N) by its lowercased value (Unicode simple lowercasing, the version pinned in the plan) and takes floor(3N / 2) bytes. A projection must stay identifiable: include the source key in the view key or declare it unique, else 59 ViewKeyExpressionInvalid. A view keyed by anything but the source key, and every aggregate view, needs a one-partition source (36). An aggregate group folds the latest record per source key, so to count events per user key the source by a per-event id (an autoincrement loginId), not by the user.
transient view: in memory only, rebuilt from its topics on restart. A view may read a keyed transient topic; a persisted one keeps its checkpointed rows across a restart.
query: a point read by the full key, or a scan by its leading columns
where full key | one row |
where userId = … | the rows under that prefix; limit caps them |
| any other condition | refused before anything is sent |
from code: build the key, read a row or a prefix
read your own write, and page through a prefix
ViewKeyBuilder encodes the key exactly as the broker folds it: setInt, setUInt, setBool, setString; key() needs every component, prefix() every one but the last, and lower() is applied for you. A page holds at most 256 rows and 61,440 row bytes; limit lowers the row cap. Pages come in table order, not key order. The reads need a broker with checked writes, else BrokerTooOld and nothing is sent. Status codes: kOk, kAbsentKey, kNotReady and kBusy (retry), kNotAuthorized, kCursorInvalidated.
A position update rewrites only that character's row. A user rename updates every row of that user. Deleting a character retracts its row and its contribution to user_stats.
inspect: definition, then the counters from each partition's leader
show and rune stats need DESCRIBE on the object. rune stats (topic | view) NAME [--partition N] [--json] prints every partition's counters in id order (a counter the partition lacks is not printed), one line per constraint, and for a unique view uniqueConflicts. From code: client.getStatsAndWait(rpc::StatsScope::TopicPartition, topicId, partition, 5s) returns a StatsResult with get(StatId::verdictLag) and constraints; kBrokerTooOld means the seed broker predates it and nothing was sent.
Lease topics
A lease is a key held for a time to live and renewed by keepalives. When the holder stops, the key expires on its own. A lease topic is a typed, keyed topic whose rows carry four system columns the broker adds (lease_version, lease_ttl_ms, lease_holder, lease_remaining_ms). Every change to a lease is fenced by its version.
lease.ttl | the ttl a grant gets when it names none. 30 000 ms |
lease.min_ttl · lease.max_ttl | a shorter ttl is raised to the floor (at least 1000 ms); a longer one is refused (66). Up to 9 000 000 000 000 ms. A transient lease topic sets lease.max_ttl explicitly |
lease.max_keys | live leases per partition, 1 to 16 777 216, creation-only. Past it a grant is refused with 65 |
lease.expiry_batch · lease.checkpoint | expiry records per batch (1 to 4096); checkpoints (on by default) that record the remaining time of long leases. Also lease.checkpoint_interval and lease.checkpoint_rate |
The CLI parses show leases, show lease and revoke lease but does not run them yet; read and revoke leases from code.
grant, keep alive, revoke
keep leases alive from the game loop
A session sends keepalives at a third of the ttl. A lease that was not renewed in time, or that the broker no longer has, is lost: its bound producers fail first (DeliveryFailed), then onLost fires once and a fenced revoke goes out. A lease that is only at risk fails the unacknowledged and new writes of its producers the same way (DeliveryFailed, LeaseAtRisk) and fires onAtRisk(ref, Entered); after the next answered keepalive (Cleared) their writes go out again. Grants, updates, revokes and keepalives need WRITE on the topic; timeToLive and reads need READ. A persisted lease topic survives a restart; a transient one forgets every lease when all replicas of a partition restart, its watchers get 43 PartitionDataReset and holders grant again. Needs a broker with lease support (an older one: openLeaseTopicAndWait throws BrokerTooOld); a session below client version 18 is answered 39.
Users, roles, grants
Rune QL statements, run as an admin with rune ql -f. Explicit allow, default deny. Every change is an event in the replayable __security log.
first admin: once per cluster, proves possession of the cluster key
users.rql, run with rune ql --user admin --password-file admin.pw -f users.rql --broker node1:40200
| operations | READ WRITE CREATE ALTER DROP DESCRIBE ADMIN IMPORT |
| resources | CLUSTER TOPIC VIEW GROUP TRANSACTIONAL_ID |
| match | an exact name, or PREFIX 'p' |
| passwords | 12 characters or more; at most two active; read from a file or stdin, never -e |
from code
Revoking READ ends the affected subscriptions with NotAuthorized; revoking WRITE fails the next produce. Dropping or disabling a user, or TERMINATE SESSIONS, answers every live session SessionRevoked. A permission denied on an object you cannot see looks the same as a missing object.
Errors
*AndWait throws std::runtime_error (timed out, session lost, broker error code 45 (NotAuthorized): ...). A broker error on a DDL or schema call is returned in the result: errorCode and detail. The token form completes with an OpStatus and the code.
| 2 NotLeader | retried after a metadata refresh; 3 Redirect is followed |
| 6 Internal | a DDL answered 6 is in doubt: it may still apply. Retry with the same request id |
| 7 UnknownTopic · 8 UnknownPartition | |
| 10 MinIsrViolation | acks=all with too few in-sync replicas: retried after a backoff |
| 11–16 | consumer group: StaleGeneration, UnknownMember, RebalanceInProgress, NotCoordinator, IllegalGeneration, GroupDead |
| 17–19 | transactions: InvalidProducerEpoch, ProducerFenced, InvalidTxnState |
| 20 OutOfOrderSequence · 21 DuplicateSequence | idempotent produce; a known duplicate counts as success |
| 22 MessageTooLarge | a batch above the broker's limit; a record above 16 KiB is rejected at append |
| 23 InvalidSchema · 24 SchemaValidationFailed | bad descriptor or partition count; a record failed validation |
| 25 TopicExists · 40 InvalidTopicName | name taken (case-insensitively); bad name |
| 26 StorageError | a log failed an fsync; the broker halts |
| 27 ConcurrentTransactions · 28 UnknownProducerId | retried; re-sequenced |
| 29 HandshakeFailed | TLS could not be negotiated, or RUNE_REQUIRE_TLS refused a plaintext client |
| 30 SchemaIncompatible · 31 SchemaConflict | a change the compatibility mode refuses; a stale expectedVersion |
| 32–38 | views: ViewExists, ViewSourceNotFound, ViewFieldUnresolved, ViewKeyInvalid, ViewNotCoPartitioned, ViewSourceTransient, ViewNotAlterable |
| 39 InvalidRequest | a malformed request or an out-of-range DDL setting; never retried |
| 41 ViewSourceIsView | a view cannot read another view |
| 42 TransientLogFull | a transient partition's ring is full of unreplicated records: retried after 100 ms |
| 43 PartitionDataReset | the transient data a subscription resumed from is gone; it restarts at the log start |
| 44–47 | authentication: AuthenticationFailed, NotAuthorized, SessionRevoked, PasswordChangeRequired |
| 48 AuthUnavailable | the broker's security state is not current: back off and reconnect, rotating brokers |
| 49 RecordEgressUnavailable · 50 UnstableOffsetCommit | 49: retried a few times, then fails the subscription or group consumer. 50: retried after a backoff while a transaction holds the offset, never surfaced |
| 51–54 | namespaces: NamespaceSuspended, NamespaceLimitExceeded, Throttled (carries a retry-after), ViewPaused |
| 55–57 | a write broke a view rule: ConstraintViolation, KeyExists, KeyNotFound. Never visible; the constraint id names the rule and the offset the record. In WriteResult and BatchVerdicts; in a transaction, in TxnCommitResult |
| 58 AutoIncrementExhausted | the counter reached its maximum; nothing is stored |
| 59–61 | view definition errors: ViewKeyExpressionInvalid, ViewConstraintInvalid, ViewUniqueInvalid |
| 62 TxnTooLarge · 63 TxnConflict | a transaction voted no at commit; it is already aborted: begin the next one |
| 64 LeaseNotFound | no live lease for the key, or at another version |
| 65 LeaseLimitExceeded · 66 LeaseTtlTooLarge | lease.max_keys or the namespace lease limit reached; a ttl above lease.max_ttl |
| codes 55–66 | sent only to sessions at the matching client version; an older client is answered 39 InvalidRequest |
OpStatus::BrokerTooOld · kBrokerTooOld | the seed broker predates checked writes, stats or leases: nothing was sent, and nothing was judged. Upgrade the broker |
OpStatus::VerdictUnknown · StoredIdUnknown | a retry fell outside the broker's window: the write was stored, its verdict or its autoincrement id is unknown. Look it up through a view |
OpStatus::RecordsRejected | a batch of two or more records had rejections; BatchVerdicts lists them |