zzstoatzz.io
Commits
Picked up by the new pre-commit hook at ~/.config/git/hooks/pre-commit
which runs `zig fmt --check` on staged .zig files. Pure whitespace /
trailing-comma drift, no behavior changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the port story end-to-end:
- what's translated (store / scripting / commands inline / persistence)
- what's NOT translated (PyO3, Tokio, RESP wire server, MessagePack)
- complete command surface grouped by upstream parity
- data model (single-mutex Store, the ordered-structure tradeoffs)
- EVAL implementation (fresh Lua 5.1 VM per script, redis.call as C fn)
- pubsub callback contract (sync delivery, do not re-enter the store)
- differences from upstream-rust in a side-by-side table
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the last piece of the upstream burner-redis surface that wasn't
already in this port. We deliberately don't carry MessagePack — the
zig engine doesn't need wire compatibility with the rust crate
(different processes, different paths) — so the format is a small
length-prefixed custom binary layout described at the top of
persistence.zig.
Behavioral parity verified against the upstream src/persistence.rs:
- crash-safe write: serialize to memory → write to {path}.tmp →
fsync → atomic rename onto {path}
- expired keys are filtered out at save time (we count first, then
emit only live entries)
- SCRIPT LOAD registry is persisted alongside `data`
- load on missing file returns Ok(false) (not an error)
- load clears in-memory keyspace + script cache before restoring
Store.save() / Store.load() lock the store mutex for the duration,
mirroring the rest of the API. persistence.zig itself is io-agnostic
beyond the file ops; tests roundtrip a string + hash + set + zset +
stream + consumer-group PEL + list and verify the consumer's
pending count survives.
Pub/sub state isn't persisted — it's tied to live subscribers, not
to keyspace data, matching the upstream's behaviour.
55/55 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round of catch-up work toward "if we call it burner-redis it should
support Redis." Everything still under the docket-driven feature flag
set ("commands the embedded engine knows about"), but no longer just
the docket-critical subset.
List rotation:
- lmove (.left/.right both ends, same-key rotation preserved)
- rpoplpush (legacy convenience for lmove(.right, .left))
- linsert (before/after a pivot)
Lists, non-blocking pop helpers (the wait loop lives at the caller
layer — same shape as the upstream blpop_poll / brpop_poll):
- blpopPoll, brpopPoll
- PoppedKV result type + freePoppedKV
Sorted sets:
- zrangestore (score-range copy into a fresh dst key)
Streams:
- xclaim (explicit ownership transfer with idle reset / retrycount /
force / justid options — complements XAUTOCLAIM's scan-by-idle)
- xinfoGroups / xinfoConsumers / xinfoStream observability triple,
each returning a typed result struct with free helpers
- xpendingSummary / xpendingRange — aggregated and per-entry forms
PubSub (new top-level: PubSubRegistry):
- subscriberRegister / subscriberRemove + Store wrapper methods
- subscribe / unsubscribe / psubscribe / punsubscribe
- publish — sync, delivers under the store mutex, contract is
"do not call back into the store from inside the callback"
- pubsubChannels (glob filter), pubsubNumSub (parallel array),
pubsubNumPat
- PubSubMessage tagged with .message / .pmessage variants for
exact-channel vs pattern delivery
Coverage: 53/53 tests pass. New tests cover each addition's happy
path plus a same-key LMOVE rotation, a missing-pivot LINSERT, and a
pattern-vs-direct PubSub delivery contrast.
Out of scope for this commit (next): MessagePack persistence and the
scripting.zig dispatchers for the newly added store-level methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Needed by docket's worker to reclaim entries from consumers that died
mid-task — otherwise PEL entries stay claimed forever and a docket
worker restart can't pick up the stuck work.
Mirrors upstream src/store.rs::xautoclaim:
- Walks every consumer's PEL in the named group.
- Filters by min_idle_ms and `start` cursor.
- Sorts by stream ID for deterministic pagination.
- Transfers each qualifying entry to the claiming consumer, even
when the underlying stream entry was trimmed (reports it in
deleted_ids so the caller can XACK and drain the PEL).
- Returns (next_start, claimed entries, deleted_ids); next_start ==
zero means we processed everything in range.
Test covers the basic transfer path (c1 reads, c2 autoclaims with
min_idle=0 → all moved). 41/41 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the next layer of Redis-compatible commands plus their EVAL
dispatchers, building on the docket-critical foundation. Surface now
covers everything pydocket / docket-equivalent flows are likely to
exercise plus the supporting string/list/set CRUD.
Strings (matching upstream src/store.rs):
set (with NX/XX, EX/PX TTL), get, mget, keys (glob), ttl, expire,
incrby, decrby (and INCR/DECR alias forms in the EVAL dispatcher).
Sets: sadd, smembers, sismember, srem, scard.
Lists: lpush, rpush, llen, lindex, lrange, lpop, rpop (single +
count forms), lrem (head/tail/all directions), lset, ltrim. Stored as
a std.DoublyLinkedList of intrusive ListNode; head is index 0.
Sorted set extras (on top of earlier ZADD/ZREM/ZCARD/ZSCORE/
ZRANGEBYSCORE/ZREMRANGEBYSCORE): zrange (inclusive index range,
negative supported), zcount.
Stream extras (on top of earlier XADD/XLEN/XDEL/XGROUP CREATE/
XREADGROUP/XACK): xread, xrange, xtrimMaxlen, streamLastId,
xgroupDestroy. Plus a top-level sweepExpired(budget) hook for
background TTL maintenance.
Helpers (free functions, since methods can't reference each other
through anytype):
globMatch — Redis-style *, ?, [..], escape with \
normalizeRange — negative indices + clamping
listLen / listAt — linked-list walk helpers
freeReadResult / freeReadResults — ownership cleanup
scripting.zig now dispatches all these via redis.call, with EVAL
tests exercising SET/GET and list RPUSH/LRANGE through the Lua
bridge. 40/40 tests pass.
Still TODO (filed under #27): LMOVE/RPOPLPUSH/LINSERT/BLPOP_POLL/
BRPOP_POLL, XAUTOCLAIM/XCLAIM/XINFO_*/XPENDING_*, ZRANGESTORE,
PubSub, persistence (MessagePack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Required by docket's promote_due.lua to clear the queue ZSET after
moving due tasks into the stream. Without this command the script
EVALs hit "unknown command" mid-flight and the post-XADD cleanup is
skipped — the queue grows unbounded while docket still functions
(because XADD already landed) but the script returns an error each
tick.
Implementation: linear pass over by_score (it's sorted), bail when
score > max, remove from both by_score and by_member, free the owned
member bytes. Empties drop the key.
Also surfaces the offending command name on the unknown-command path
so we don't have to instrument the dep again next time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds all the Store methods + Lua bridge needed to run docket's three
embedded scripts (schedule.lua, promote_due.lua, ack.lua) end-to-end
against the in-process engine:
Store methods translated 1:1 from upstream src/store.rs:
- key mgmt: delete, exists
- hash: hset (flat field/value), hget, hgetall (out-allocated pairs),
hdel (empties remove the key), hexists
- zset: zadd (with NX/XX/GT/LT/CH flags), zrem (empties remove key),
zcard, zscore, zrangebyscore (with WITHSCORES)
- stream: xadd (auto-id or explicit, validates monotonic), xlen, xdel,
xgroupCreate (with MKSTREAM), xreadgroupOne (>/PEL forms), xack
Internal helpers (nowNs, removeKey, expireIfDue, liveEntry,
getOrCreate, zsetInsert, zsetReplaceScore, zset lower/upper-bound
binary search) keep the per-command bodies close to the upstream
shape. Sorted-set indexes use a sorted ArrayList in place of
BTreeMap (zig stdlib has none; docket cardinalities are small).
scripting.zig stands up an LuaEngine that:
- spins up a fresh Lua 5.1 VM per eval (matches upstream isolation),
- opens base + string + table + math,
- exposes KEYS/ARGV globals, redis.call + redis.pcall (via
pushClosure + lightuserdata upvalue threading a DispatchCtx),
- dispatches on upper-cased command name to the matching Store
method, with per-command arg-parsing helpers (doZadd, doXadd,
doXgroup, doXreadgroup),
- converts Lua return values back into a RedisValue tagged union.
Tests added covering each command, plus an EVAL roundtrip that runs
XGROUP CREATE + XADD + XREADGROUP through the Lua bridge — the
docket-critical happy path. 23/23 tests pass.
Out of scope for this commit (filed under #27 follow-up): strings,
sets, lists, full streams (XREAD/XRANGE/XTRIM/XAUTOCLAIM/XCLAIM/
XINFO/XPENDING), full zsets (ZRANGE/ZRANGESTORE/ZREMRANGEBYSCORE/
ZCOUNT), PubSub, persistence, sweep_expired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports the upstream src/store.rs type hierarchy:
- StreamId with format/parse/cmp (mirrors commands/streams.rs StreamId)
- SortedSet with dual-index (by_score ArrayList + by_member HashMap);
ArrayList stands in for Rust's BTreeMap until we need real ordered
iteration semantics under heavy load.
- Stream + ConsumerGroup + Consumer + PendingEntry with append-only
entries (XADD only ever tails) and store-owned bytes for fields.
- ValueData tagged union mirroring the Rust enum (String/Hash/Set/
SortedSet/Stream/List).
- ValueEntry with TTL as nanoseconds since unix epoch (upstream uses
Instant; we use realtime since the engine has no separate session).
- Store with Io.Mutex (lock state lives on the struct, not a global),
data + scripts maps, and shutdown atomic.
Ownership rule: all bytes inside the Store are duped from caller
input on insert; deinit frees recursively through the variants. Tests
cover StreamId roundtrip + ordering, ScoredMember sort, Store
empty-init/deinit, and a hand-inserted string entry to exercise free.
3 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Initial scaffolding for the zig port of PrefectHQ/burner-redis. The
upstream is a Rust crate with PyO3 Python bindings; this port is the
Rust core translated to Zig, with the Python/PyO3/Tokio layer dropped
(Zig consumers call the engine directly).
- build.zig wires ziglua@8f271c8 in Lua 5.1 mode (matches Redis's Lua
version, so EVAL scripts ported from real Redis behave identically).
- src/burner.zig re-exports Store + scripting; src/store.zig has
StoreError + Store stubs; src/scripting.zig has a ziglua smoke test
that runs `return 'hello from lua'` end-to-end.
- LICENSE retains the original Prefect Technologies MIT copyright and
adds the Zig port's copyright line.
Smoke test: `zig build test` passes — Lua 5.1 boots, runs a script,
returns a string. Translation of store.rs / scripting.rs / commands/
incoming.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the port story end-to-end:
- what's translated (store / scripting / commands inline / persistence)
- what's NOT translated (PyO3, Tokio, RESP wire server, MessagePack)
- complete command surface grouped by upstream parity
- data model (single-mutex Store, the ordered-structure tradeoffs)
- EVAL implementation (fresh Lua 5.1 VM per script, redis.call as C fn)
- pubsub callback contract (sync delivery, do not re-enter the store)
- differences from upstream-rust in a side-by-side table
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the last piece of the upstream burner-redis surface that wasn't
already in this port. We deliberately don't carry MessagePack — the
zig engine doesn't need wire compatibility with the rust crate
(different processes, different paths) — so the format is a small
length-prefixed custom binary layout described at the top of
persistence.zig.
Behavioral parity verified against the upstream src/persistence.rs:
- crash-safe write: serialize to memory → write to {path}.tmp →
fsync → atomic rename onto {path}
- expired keys are filtered out at save time (we count first, then
emit only live entries)
- SCRIPT LOAD registry is persisted alongside `data`
- load on missing file returns Ok(false) (not an error)
- load clears in-memory keyspace + script cache before restoring
Store.save() / Store.load() lock the store mutex for the duration,
mirroring the rest of the API. persistence.zig itself is io-agnostic
beyond the file ops; tests roundtrip a string + hash + set + zset +
stream + consumer-group PEL + list and verify the consumer's
pending count survives.
Pub/sub state isn't persisted — it's tied to live subscribers, not
to keyspace data, matching the upstream's behaviour.
55/55 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round of catch-up work toward "if we call it burner-redis it should
support Redis." Everything still under the docket-driven feature flag
set ("commands the embedded engine knows about"), but no longer just
the docket-critical subset.
List rotation:
- lmove (.left/.right both ends, same-key rotation preserved)
- rpoplpush (legacy convenience for lmove(.right, .left))
- linsert (before/after a pivot)
Lists, non-blocking pop helpers (the wait loop lives at the caller
layer — same shape as the upstream blpop_poll / brpop_poll):
- blpopPoll, brpopPoll
- PoppedKV result type + freePoppedKV
Sorted sets:
- zrangestore (score-range copy into a fresh dst key)
Streams:
- xclaim (explicit ownership transfer with idle reset / retrycount /
force / justid options — complements XAUTOCLAIM's scan-by-idle)
- xinfoGroups / xinfoConsumers / xinfoStream observability triple,
each returning a typed result struct with free helpers
- xpendingSummary / xpendingRange — aggregated and per-entry forms
PubSub (new top-level: PubSubRegistry):
- subscriberRegister / subscriberRemove + Store wrapper methods
- subscribe / unsubscribe / psubscribe / punsubscribe
- publish — sync, delivers under the store mutex, contract is
"do not call back into the store from inside the callback"
- pubsubChannels (glob filter), pubsubNumSub (parallel array),
pubsubNumPat
- PubSubMessage tagged with .message / .pmessage variants for
exact-channel vs pattern delivery
Coverage: 53/53 tests pass. New tests cover each addition's happy
path plus a same-key LMOVE rotation, a missing-pivot LINSERT, and a
pattern-vs-direct PubSub delivery contrast.
Out of scope for this commit (next): MessagePack persistence and the
scripting.zig dispatchers for the newly added store-level methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Needed by docket's worker to reclaim entries from consumers that died
mid-task — otherwise PEL entries stay claimed forever and a docket
worker restart can't pick up the stuck work.
Mirrors upstream src/store.rs::xautoclaim:
- Walks every consumer's PEL in the named group.
- Filters by min_idle_ms and `start` cursor.
- Sorts by stream ID for deterministic pagination.
- Transfers each qualifying entry to the claiming consumer, even
when the underlying stream entry was trimmed (reports it in
deleted_ids so the caller can XACK and drain the PEL).
- Returns (next_start, claimed entries, deleted_ids); next_start ==
zero means we processed everything in range.
Test covers the basic transfer path (c1 reads, c2 autoclaims with
min_idle=0 → all moved). 41/41 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the next layer of Redis-compatible commands plus their EVAL
dispatchers, building on the docket-critical foundation. Surface now
covers everything pydocket / docket-equivalent flows are likely to
exercise plus the supporting string/list/set CRUD.
Strings (matching upstream src/store.rs):
set (with NX/XX, EX/PX TTL), get, mget, keys (glob), ttl, expire,
incrby, decrby (and INCR/DECR alias forms in the EVAL dispatcher).
Sets: sadd, smembers, sismember, srem, scard.
Lists: lpush, rpush, llen, lindex, lrange, lpop, rpop (single +
count forms), lrem (head/tail/all directions), lset, ltrim. Stored as
a std.DoublyLinkedList of intrusive ListNode; head is index 0.
Sorted set extras (on top of earlier ZADD/ZREM/ZCARD/ZSCORE/
ZRANGEBYSCORE/ZREMRANGEBYSCORE): zrange (inclusive index range,
negative supported), zcount.
Stream extras (on top of earlier XADD/XLEN/XDEL/XGROUP CREATE/
XREADGROUP/XACK): xread, xrange, xtrimMaxlen, streamLastId,
xgroupDestroy. Plus a top-level sweepExpired(budget) hook for
background TTL maintenance.
Helpers (free functions, since methods can't reference each other
through anytype):
globMatch — Redis-style *, ?, [..], escape with \
normalizeRange — negative indices + clamping
listLen / listAt — linked-list walk helpers
freeReadResult / freeReadResults — ownership cleanup
scripting.zig now dispatches all these via redis.call, with EVAL
tests exercising SET/GET and list RPUSH/LRANGE through the Lua
bridge. 40/40 tests pass.
Still TODO (filed under #27): LMOVE/RPOPLPUSH/LINSERT/BLPOP_POLL/
BRPOP_POLL, XAUTOCLAIM/XCLAIM/XINFO_*/XPENDING_*, ZRANGESTORE,
PubSub, persistence (MessagePack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Required by docket's promote_due.lua to clear the queue ZSET after
moving due tasks into the stream. Without this command the script
EVALs hit "unknown command" mid-flight and the post-XADD cleanup is
skipped — the queue grows unbounded while docket still functions
(because XADD already landed) but the script returns an error each
tick.
Implementation: linear pass over by_score (it's sorted), bail when
score > max, remove from both by_score and by_member, free the owned
member bytes. Empties drop the key.
Also surfaces the offending command name on the unknown-command path
so we don't have to instrument the dep again next time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds all the Store methods + Lua bridge needed to run docket's three
embedded scripts (schedule.lua, promote_due.lua, ack.lua) end-to-end
against the in-process engine:
Store methods translated 1:1 from upstream src/store.rs:
- key mgmt: delete, exists
- hash: hset (flat field/value), hget, hgetall (out-allocated pairs),
hdel (empties remove the key), hexists
- zset: zadd (with NX/XX/GT/LT/CH flags), zrem (empties remove key),
zcard, zscore, zrangebyscore (with WITHSCORES)
- stream: xadd (auto-id or explicit, validates monotonic), xlen, xdel,
xgroupCreate (with MKSTREAM), xreadgroupOne (>/PEL forms), xack
Internal helpers (nowNs, removeKey, expireIfDue, liveEntry,
getOrCreate, zsetInsert, zsetReplaceScore, zset lower/upper-bound
binary search) keep the per-command bodies close to the upstream
shape. Sorted-set indexes use a sorted ArrayList in place of
BTreeMap (zig stdlib has none; docket cardinalities are small).
scripting.zig stands up an LuaEngine that:
- spins up a fresh Lua 5.1 VM per eval (matches upstream isolation),
- opens base + string + table + math,
- exposes KEYS/ARGV globals, redis.call + redis.pcall (via
pushClosure + lightuserdata upvalue threading a DispatchCtx),
- dispatches on upper-cased command name to the matching Store
method, with per-command arg-parsing helpers (doZadd, doXadd,
doXgroup, doXreadgroup),
- converts Lua return values back into a RedisValue tagged union.
Tests added covering each command, plus an EVAL roundtrip that runs
XGROUP CREATE + XADD + XREADGROUP through the Lua bridge — the
docket-critical happy path. 23/23 tests pass.
Out of scope for this commit (filed under #27 follow-up): strings,
sets, lists, full streams (XREAD/XRANGE/XTRIM/XAUTOCLAIM/XCLAIM/
XINFO/XPENDING), full zsets (ZRANGE/ZRANGESTORE/ZREMRANGEBYSCORE/
ZCOUNT), PubSub, persistence, sweep_expired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports the upstream src/store.rs type hierarchy:
- StreamId with format/parse/cmp (mirrors commands/streams.rs StreamId)
- SortedSet with dual-index (by_score ArrayList + by_member HashMap);
ArrayList stands in for Rust's BTreeMap until we need real ordered
iteration semantics under heavy load.
- Stream + ConsumerGroup + Consumer + PendingEntry with append-only
entries (XADD only ever tails) and store-owned bytes for fields.
- ValueData tagged union mirroring the Rust enum (String/Hash/Set/
SortedSet/Stream/List).
- ValueEntry with TTL as nanoseconds since unix epoch (upstream uses
Instant; we use realtime since the engine has no separate session).
- Store with Io.Mutex (lock state lives on the struct, not a global),
data + scripts maps, and shutdown atomic.
Ownership rule: all bytes inside the Store are duped from caller
input on insert; deinit frees recursively through the variants. Tests
cover StreamId roundtrip + ordering, ScoredMember sort, Store
empty-init/deinit, and a hand-inserted string entry to exercise free.
3 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Initial scaffolding for the zig port of PrefectHQ/burner-redis. The
upstream is a Rust crate with PyO3 Python bindings; this port is the
Rust core translated to Zig, with the Python/PyO3/Tokio layer dropped
(Zig consumers call the engine directly).
- build.zig wires ziglua@8f271c8 in Lua 5.1 mode (matches Redis's Lua
version, so EVAL scripts ported from real Redis behave identically).
- src/burner.zig re-exports Store + scripting; src/store.zig has
StoreError + Store stubs; src/scripting.zig has a ziglua smoke test
that runs `return 'hello from lua'` end-to-end.
- LICENSE retains the original Prefect Technologies MIT copyright and
adds the Zig port's copyright line.
Smoke test: `zig build test` passes — Lua 5.1 boots, runs a script,
returns a string. Translation of store.rs / scripting.rs / commands/
incoming.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>