zzstoatzz.io
in-memory redis
redis
in-memory
testing
docs: architecture overview
Captures the port story end-to-end:
- what's translated (store / scripting / commands inline / persistence)
- what's NOT translated (PyO3, Tokio, RESP wire server, MessagePack)
- complete command surface grouped by upstream parity
- data model (single-mutex Store, the ordered-structure tradeoffs)
- EVAL implementation (fresh Lua 5.1 VM per script, redis.call as C fn)
- pubsub callback contract (sync delivery, do not re-enter the store)
- differences from upstream-rust in a side-by-side table
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Opus 4.7 (1M context)
date
(May 24, 2026, 12:58 AM -0500)
commit
d8f1f1bf
d8f1f1bf3dc7fa759f2b1f9ca5b029a09a93677b
parent
7405ff17
7405ff17a73e3b3c648786a92ba8b701971a9fa0
+158
+158
docs/architecture.md
+158
docs/architecture.md
···
1
+
# burner-redis-zig architecture
2
+
3
+
zig 0.16 port of [`PrefectHQ/burner-redis`](https://github.com/PrefectHQ/burner-redis) — an embedded, in-process Redis-compatible engine. The upstream is a Rust crate with PyO3 Python bindings; this port is the Rust *core* translated to Zig, with the Python/PyO3/Tokio layer removed (Zig consumers call the engine directly).
4
+
5
+
## what's translated
6
+
7
+
| upstream | zig port | notes |
8
+
|-------------------------|-----------------------------------|------------------------------------------|
9
+
| `src/store.rs` | `src/store.zig` | Store + all Redis data types + commands |
10
+
| `src/scripting.rs` | `src/scripting.zig` | Lua 5.1 engine via [ziglua](https://github.com/natecraddock/ziglua) |
11
+
| `src/commands/*.rs` | inlined into `scripting.zig` dispatchInner | upstream's per-command files were arg-parsers; we have them in the dispatcher |
12
+
| `src/persistence.rs` | `src/persistence.zig` | custom binary format (BRDB), not MessagePack — we don't need wire compat with the Rust upstream |
13
+
| `src/lib.rs` (PyO3) | not ported | zig consumers call store methods directly |
14
+
15
+
## what's not translated
16
+
17
+
- the Python/PyO3 binding layer
18
+
- the Tokio runtime + async glue (we use std.Io + cancellation primitives instead)
19
+
- replication, AOF, RDB compatibility (we have our own snapshot format)
20
+
- the actual RESP wire server (consumers call methods directly; no socket)
21
+
22
+
The result: a ~5000-line zig module that runs the entire Redis-compatible engine in-process, with the same EVAL scripts as real Redis.
23
+
24
+
## command surface
25
+
26
+
All commands grouped by upstream parity:
27
+
28
+
| group | commands |
29
+
|-----------------|----------|
30
+
| keys | DEL, EXISTS, KEYS (glob), TTL, EXPIRE |
31
+
| strings | SET (with NX/XX/EX/PX), GET, MGET, INCRBY, DECRBY |
32
+
| hashes | HSET, HGET, HGETALL, HDEL, HEXISTS |
33
+
| sets | SADD, SMEMBERS, SISMEMBER, SREM, SCARD |
34
+
| sorted sets | ZADD (with NX/XX/GT/LT/CH), ZREM, ZRANGE, ZRANGEBYSCORE, ZREMRANGEBYSCORE, ZRANGESTORE, ZCARD, ZSCORE, ZCOUNT |
35
+
| lists | LPUSH, RPUSH, LLEN, LINDEX, LRANGE, LPOP, RPOP, LREM, LSET, LTRIM, LINSERT, LMOVE, RPOPLPUSH |
36
+
| blocking lists | BLPOP-poll, BRPOP-poll (wait loop lives at caller — matches upstream's `*_poll`) |
37
+
| streams | XADD, XLEN, XDEL, XREAD, XRANGE, XTRIM (MAXLEN), XGROUP CREATE/DESTROY, XREADGROUP, XACK, XCLAIM, XAUTOCLAIM, XINFO {GROUPS, CONSUMERS, STREAM}, XPENDING {summary, range} |
38
+
| scripting | EVAL (no SCRIPT LOAD / EVALSHA yet — docket and existing prefect-server callers always EVAL directly) |
39
+
| pubsub | SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, PUBLISH, PUBSUB CHANNELS / NUMSUB / NUMPAT |
40
+
| persistence | save (path), load (path) — atomic temp+rename, expired keys filtered, scripts persisted |
41
+
42
+
## data model
43
+
44
+
A single `Store` struct owns the keyspace, an Io.Mutex, and an embedded `PubSubRegistry`. Every command method locks the mutex for the duration — matches Redis's single-threaded execution model. Sharded locking can come later if benchmarks justify it (they probably don't, for docket-sized loads).
45
+
46
+
```zig
47
+
Store {
48
+
data: HashMap<Bytes, ValueEntry>,
49
+
scripts: HashMap<sha1, source>,
50
+
pubsub: PubSubRegistry,
51
+
mutex: Io.Mutex,
52
+
}
53
+
54
+
ValueEntry { data: ValueData, expires_at_ns: ?i128 }
55
+
ValueData = String(Bytes)
56
+
| Hash(HashMap<Bytes, Bytes>)
57
+
| Set(HashSet<Bytes>)
58
+
| SortedSet { by_score: sorted ArrayList(ScoredMember),
59
+
by_member: HashMap<Bytes, f64> }
60
+
| Stream { entries, last_id, groups }
61
+
| List(DoublyLinkedList)
62
+
```
63
+
64
+
`Bytes` is `[]const u8`. Everything stored inside the Store is owned by the store's allocator (caller's bytes are duped on insert). The deinit path walks the keyspace and frees recursively.
65
+
66
+
### ordered structures
67
+
68
+
zig's stdlib doesn't have a BTreeMap, so:
69
+
70
+
- sorted set: `by_score` is a sorted `ArrayList(ScoredMember)`. Insert/remove are O(log n) lookup + O(n) shift. Fine for docket scale; consider a skip-list later for high cardinality.
71
+
- stream entries: `ArrayList(StreamEntry)`. Stream IDs are monotonic — XADD always appends to the tail. No general ordered-insert needed.
72
+
73
+
## EVAL
74
+
75
+
`LuaEngine.eval(script, keys, args)` spins up a fresh Lua 5.1 VM, pushes KEYS / ARGV / `redis.call` / `redis.pcall`, then `pcall`s the script. The `redis.call` is a C function (via ziglua) that:
76
+
77
+
1. reads its argv off the Lua stack
78
+
2. uppercases the command name
79
+
3. dispatches into a giant switch in `scripting.zig::dispatchInner`
80
+
4. calls the corresponding `Store` method
81
+
5. pushes the result back onto the Lua stack in the right RESP shape
82
+
83
+
Fresh VM per script = no state leaks. The Store's mutex is held by `Store.{method}` during each `redis.call`. Net effect: the whole script runs as a single critical section against the store, matching Redis's atomicity guarantee for EVAL.
84
+
85
+
See [databases/redis/eval-lua](https://tangled.org/zzstoatzz.io/notes/blob/main/databases/redis/eval-lua.md) for the broader pattern.
86
+
87
+
## persistence
88
+
89
+
`save(store, path)` builds the whole snapshot in memory, writes `<path>.tmp`, fsyncs, atomic-renames to `<path>`. Expired keys are filtered. The scripts cache (SCRIPT LOAD registry) is persisted alongside the keyspace.
90
+
91
+
Format is a small custom binary layout described at the top of `persistence.zig`:
92
+
93
+
```
94
+
header: "BRDB" | u32 version
95
+
data: u64 n_entries
96
+
for each: u64 key_len | key | u8 has_ttl [| i128 expires_at_ns] | u8 variant tag | (per-variant payload)
97
+
scripts: u64 n_scripts
98
+
for each: u64 sha_len | sha | u64 body_len | body
99
+
```
100
+
101
+
No MessagePack lib needed — we don't need wire compatibility with the Rust upstream.
102
+
103
+
## pubsub
104
+
105
+
`PubSubRegistry` is a synchronous callback registry. Subscribers register a callback (`*const fn (id, msg, ctx) void`); PUBLISH walks the by-channel and by-pattern indexes and invokes each matching callback under the store mutex.
106
+
107
+
Contract: the callback runs with the store mutex held. **Do not call back into the store from inside it** (deadlock). The callback should push into an Io.Queue / signal a condvar / dupe the payload — anything that defers actual work.
108
+
109
+
No tokio broadcast / async runtime. Real Redis serializes deliveries the same way (single-threaded).
110
+
111
+
## cancellation
112
+
113
+
Most methods don't take an Io.Cancelable — they hold the mutex for short critical sections and complete synchronously. The one exception is the memory backend's `consumeStream` (which docket calls): when no entries are available it sleeps via `std.Io.sleep`, which IS a cancellation point. That's how docket workers can be cancelled cleanly via `Future.cancel`.
114
+
115
+
## thread safety
116
+
117
+
The Store is single-mutex. Multiple callers can hold a `*Store` and call methods concurrently — they serialize. No internal sharding (yet).
118
+
119
+
PubSub is part of the same mutex. The publisher iterating the subscriber list runs under the same lock as keyspace mutations.
120
+
121
+
## differences from upstream
122
+
123
+
| dimension | upstream (rust) | this port (zig) |
124
+
|--------------------------|--------------------------|------------------------------|
125
+
| Python bindings | PyO3 | none — zig callers |
126
+
| async runtime | tokio | std.Io + Io.Threaded |
127
+
| persistence format | MessagePack (rmp-serde) | custom binary (BRDB) |
128
+
| Lua impl | mlua (Lua 5.4) | ziglua (Lua 5.1 explicit) |
129
+
| concurrency on lists | tokio Notify | callback API (no notify yet) |
130
+
| sorted-set internals | BTreeMap + HashMap | sorted ArrayList + HashMap |
131
+
132
+
The Lua 5.1 vs 5.4 difference matters: real Redis uses Lua 5.1, so port-targeting 5.1 means EVAL scripts behave like real Redis (no `goto`, integer-as-double, `unpack` global vs `table.unpack`).
133
+
134
+
## why all of this
135
+
136
+
[docket](https://tangled.org/zzstoatzz.io/docket) wanted to ship as a self-contained background-task system. The natural substrate is Redis (streams, consumer groups, EVAL). But you don't always want to run Redis — for tests, for small self-hosted prefect deployments, you'd rather just embed the engine.
137
+
138
+
burner-redis-zig is that embedded engine. The trick is: it runs the *same Lua scripts* docket EVALs against real Redis. So switching from redis-mode to memory-mode is a URL change, not a code change. That's what makes prefect-server's dual-backend story honest.
139
+
140
+
## status
141
+
142
+
what works:
143
+
- entire upstream-parity command surface
144
+
- 55/55 tests pass (including pubsub, persistence, all the stream observability commands)
145
+
- docket runs its perpetual loop against this engine indistinguishably from real Redis
146
+
- prefect-server's loop services run end-to-end on memory mode (5/5 SCHEDULED → 5/5 Late)
147
+
148
+
what's deferred:
149
+
- SCRIPT LOAD / EVALSHA (docket doesn't use them; consumers EVAL directly)
150
+
- BLPOP / BRPOP with internal blocking (callers use the `_poll` variants + their own wait loop)
151
+
- pub/sub with internal queueing (callbacks are synchronous; consumers wrap their own queue if they need it)
152
+
153
+
## references
154
+
155
+
- upstream rust source: [PrefectHQ/burner-redis](https://github.com/PrefectHQ/burner-redis)
156
+
- docket (the primary consumer): [tangled.org/zzstoatzz.io/docket](https://tangled.org/zzstoatzz.io/docket)
157
+
- ziglua: [natecraddock/ziglua](https://github.com/natecraddock/ziglua)
158
+
- pattern notes: [zzstoatzz.io/notes/databases/redis](https://tangled.org/zzstoatzz.io/notes)