zzstoatzz.io
This repository has no description
weir: rename from prefect-pds + readme with architecture contrast
a weir shapes flow, measures it, and never owns the water. the name
also stops claiming pds (this is an appview + write gateway hosting a
native repo) and stops squatting the prefect prefix on the repo's
public face — io.prefect.v0.* stays as the lexicon namespace, which is
a separate, deliberately-versioned decision.
binary, package name, and env vars follow (WEIR_PORT, WEIR_DATA_DIR,
WEIR_CACHE, WEIR_REDIS_HOST/PORT, WEIR_COMMIT_EVERY). new readme leads
with mermaid diagrams contrasting prefect's self-hosted shape (LB ->
api xN -> shared postgres+redis) against weir's canonical/derived
split, and the ownership asymmetry table that is the actual thesis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Fable 5
date
(Jun 12, 2026, 1:12 PM -0500)
commit
33fb9c80
33fb9c80167378fa1fce2961fb4742d0b3459a82
parent
82925b4b
82925b4b24340f170a5f9d063484001ef0468b11
+150
-81
+128
-59
README.md
+128
-59
README.md
···
1
-
# prefect-pds
1
+
# weir
2
2
3
-
a minimal prefect server whose database is an atproto repo.
3
+
a [weir](https://en.wikipedia.org/wiki/Weir) is a low dam that shapes a
4
+
river's flow, measures it as it passes, and never owns the water.
4
5
5
-
there is no postgres. there is no sqlite. the canonical store is a
6
-
[merkle search tree](https://atproto.com/specs/repository) of orchestration
7
-
records, sealed by signed commits, persisted as a single CAR file. the
8
-
workspace is a DID. your orchestration history is cryptographically
9
-
verifiable, portable, and exportable with one HTTP call.
6
+
this is a prefect-compatible orchestration server whose database is an
7
+
[atproto repo](https://atproto.com/specs/repository): every flow, run, and
8
+
state transition is a record in a merkle search tree, sealed under signed
9
+
commits, persisted as a single CAR file. your orchestration history is
10
+
cryptographically verifiable, portable, and exportable with one HTTP call —
11
+
the server shapes and measures your flows; the water stays yours.
12
+
13
+
there is no postgres. there is no sqlite.
14
+
15
+
## the architecture, in contrast
16
+
17
+
[prefect's self-hosted architecture](https://docs.prefect.io/v3/advanced/self-hosted)
18
+
is the classic shape — a load balancer over N API servers and N background
19
+
services, all reading and writing one shared data layer:
10
20
21
+
```mermaid
22
+
flowchart TD
23
+
clients["users / UI / workers"] --> lb["load balancer"]
24
+
lb --> api["api servers ×N"]
25
+
svcs["background services ×N"] --> pg
26
+
api --> pg[("postgresql<br/>state · config · history")]
27
+
api --> rd[("redis<br/>events · coordination")]
28
+
svcs --> rd
11
29
```
12
-
prefect client ──▶ /api/* ──▶ records ──▶ MST ──▶ signed commit ──▶ repo.car
13
-
│
14
-
└──▶ appview (burner-redis) ◀── /api reads
30
+
31
+
the database is the system of record, the event stream is internal plumbing,
32
+
and your history lives inside a postgres only the operator can interpret.
33
+
34
+
weir keeps the same surface — the prefect SDK and UI speak to it unchanged —
35
+
and replaces the data layer with two pieces that have sharply different jobs:
36
+
37
+
```mermaid
38
+
flowchart LR
39
+
sdk["prefect sdk / ui / curl<br/><i>unchanged</i>"]
40
+
41
+
subgraph weir["weir — one binary, ~1 MB"]
42
+
api["prefect-compatible REST"]
43
+
gw["write gateway<br/>records → signed commits"]
44
+
av["appview<br/>filters · counts · point reads"]
45
+
end
46
+
47
+
subgraph truth["canonical — yours"]
48
+
car[("repo.car<br/>signed MST of<br/>io.prefect.v0.* records")]
49
+
end
50
+
51
+
subgraph derived["derived — disposable"]
52
+
redis[("redis<br/>(or embedded burner-redis)")]
53
+
end
54
+
55
+
sdk --> api
56
+
api --> gw --> car
57
+
api --> av --> redis
58
+
car -. "replay on boot<br/>(rebuilds the appview)" .-> redis
59
+
60
+
car --> export["getRepo → CAR<br/>verify offline · query with any engine"]
15
61
```
16
62
17
-
## the idea
63
+
the asymmetry is the whole idea:
18
64
19
-
prefect cloud's data-architecture direction is "object storage + derived
20
-
indexes + event streams" with per-workspace storage election. atproto is that
21
-
exact shape, already specified and interoperable:
65
+
| | canonical (repo) | derived (appview) |
66
+
|---|---|---|
67
+
| holds | what happened — entities + the event log | how to answer queries fast |
68
+
| shape | append-only records, TID-keyed, signed | membership sets, row caches |
69
+
| loss | unacceptable | irrelevant — replayed from the repo on boot |
70
+
| owner | **you** — export it, verify it, leave with it | the operator — delete it freely |
22
71
23
-
| orchestration concern | atproto primitive |
24
-
|---|---|
25
-
| workspace identity | DID |
26
-
| canonical state | records in an MST, signed commits |
27
-
| customer-owned storage | the repo is a file; host it anywhere |
28
-
| audit | offline verification against the signing key |
29
-
| event stream | `com.atproto.sync.subscribeRepos` (future) |
30
-
| derived reads | appview — here, embedded burner-redis |
72
+
state transitions are not rows that get updated; they are
73
+
`io.prefect.v0.event` records appended to the log (mirroring prefect's own
74
+
Event schema), and a run's "current state" is just appview material derived
75
+
from them. delete redis, restart weir, and the entire read surface rebuilds
76
+
from the signed log — that replay runs on every boot.
31
77
32
-
state transitions are append-only records keyed by TID, so the chronological
33
-
state history of a run is literally a collection scan. the appview is
34
-
disposable by construction: rebuild it by replaying the repo.
78
+
because the canonical store speaks atproto, the export story is free:
79
+
`com.atproto.sync.getRepo` hands anyone the workspace as a CAR they can
80
+
verify offline against the workspace DID and query with any engine. same
81
+
argument iceberg makes for BYOS, plus cryptographic provenance.
35
82
36
83
## try it
37
84
38
85
```bash
39
-
zig build && ./zig-out/bin/prefect-pds # port 4200, data in ./data
86
+
zig build -Doptimize=ReleaseFast
87
+
./zig-out/bin/weir # port 4200, data in ./data
40
88
41
89
# in another shell: run a real prefect flow against it
42
90
just smoke
43
91
44
-
# then look at what it wrote
45
-
just records io.prefect.v0.event # the event log as at:// records
92
+
# look at what it wrote
93
+
just records io.prefect.v0.event # the event log, as at:// records
46
94
just export-car # the whole database, signed
47
95
```
48
96
49
-
on boot the server mints (or reloads) a p256 keypair — the workspace DID is
50
-
`did:key:...`, no external services involved. kill the server, start it
51
-
again: same identity, same records, replayed from the CAR.
97
+
on boot weir mints (or reloads) a p256 keypair — the workspace DID is
98
+
`did:key:...`, zero external services. for the production shape, point the
99
+
appview at a real redis:
100
+
101
+
```bash
102
+
docker run -d --name weir-redis -p 6390:6379 redis:7-alpine
103
+
WEIR_CACHE=redis WEIR_REDIS_PORT=6390 ./zig-out/bin/weir
104
+
```
105
+
106
+
| env | default | |
107
+
|---|---|---|
108
+
| `WEIR_PORT` | 4200 | |
109
+
| `WEIR_DATA_DIR` | `data` | signing key + repo CAR |
110
+
| `WEIR_CACHE` | `burner` | `burner` (embedded) or `redis` |
111
+
| `WEIR_REDIS_HOST` / `WEIR_REDIS_PORT` | 127.0.0.1 / 6379 | |
112
+
| `WEIR_COMMIT_EVERY` | 1 | commit cadence; the durability/throughput lever |
113
+
114
+
## numbers
115
+
116
+
same seeded workload, same machine, sequential reads
117
+
([details + caveats](docs/bench-results.md)):
118
+
119
+
| pattern | weir | python+sqlite | python+postgres |
120
+
|---|---|---|---|
121
+
| point lookup | **0.2 ms** | 1.7 ms | 2.6 ms |
122
+
| active runs by state | **1.0 ms** | 4.4 ms | 4.7 ms |
123
+
| count by state | **0.2 ms** | 1.3 ms | 2.1 ms |
52
124
53
-
## what works
125
+
under 16 concurrent clients to 8,000 runs (~30k records), point reads hold
126
+
~1 ms flat and batched commits sustain 840 writes/s
127
+
([scale results](docs/scale-results.md) — including the three real bugs the
128
+
harness found, two of them upstream).
54
129
55
-
- real `prefect>=3.0` python client runs flows to completion
56
-
- flows, flow runs, task runs as entity records; state transitions as
57
-
`io.prefect.v0.event` records (see `docs/lexicons.md` and `lexicons/`)
58
-
- signed v3 commits with monotonic TID revs; CAR persisted atomically on
59
-
every commit; verified round-trip in tests via `zat.verifyCommitCar`
60
-
- `com.atproto.sync.getRepo`, `com.atproto.repo.describeRepo`,
61
-
`com.atproto.repo.listRecords`
62
-
- appview on embedded [burner-redis](https://tangled.org/zzstoatzz.io/burner-redis),
63
-
swappable behind a union (real redis / docket-backed slot in beside it)
130
+
## status
64
131
65
-
## honest limitations (v0)
132
+
an experiment with working proofs, not a product. real `prefect>=3.0`
133
+
clients run flows to completion; filters/counts/point reads serve the SDK's
134
+
read paths; the same `io.prefect.v0.*` records have been written to a local
135
+
did:key repo, a stock bsky.social account (where the public **jetstream
136
+
carried the events** — free event bus), and a private permissioned space on
137
+
[zds](https://tangled.org/zat.dev/zds) with oplog + 401s for strangers.
66
138
67
-
- **no orchestration rules** — every proposed state is ACCEPTed
68
-
- **events websocket is stubbed** — prefect 3.x local execution reports task
69
-
runs via events, so task-run records only appear when the client posts
70
-
them through the REST path
71
-
- **floats** — atproto's data model has no float type; floats are stringified
72
-
inside record bodies (API responses are unaffected — they're served from
73
-
the appview)
74
-
- **full-CAR rewrite per commit** — fine at demo scale, obviously not the
75
-
end state (block-level append is the fix)
76
-
- secrets should never enter the repo; this server stores none
139
+
deliberately absent so far: orchestration rules (every transition is
140
+
ACCEPTed), auth (localhost demo), the events websocket, blob storage,
141
+
time-window filters. the lexicon rationale lives in
142
+
[docs/lexicons.md](docs/lexicons.md), the read-path design in
143
+
[docs/access-patterns.md](docs/access-patterns.md).
77
144
78
-
## relation to siblings
145
+
## siblings
79
146
80
-
- [zat](https://tangled.org/zat.dev/zat) — provides every atproto primitive
81
-
used here (MST, DAG-CBOR, CAR, signing, TIDs)
147
+
- [zat](https://tangled.org/zat.dev/zat) — every atproto primitive here
148
+
(MST, DAG-CBOR, CAR, signing, TIDs)
149
+
- [zds](https://tangled.org/zat.dev/zds) — the PDS; permissioned spaces are
150
+
the private-workspace substrate
151
+
- [burner-redis](https://tangled.org/zzstoatzz.io/burner-redis) — the
152
+
embedded appview default
82
153
- [prefect-server](https://tangled.org/zzstoatzz.io/prefect-server) — the
83
-
full ~95%-coverage zig prefect server; its `atproto-storage` branch
84
-
explores grafting this storage layer onto the real thing
85
-
- [burner-redis](https://tangled.org/zzstoatzz.io/burner-redis) — the
86
-
embedded appview backend
154
+
full zig prefect server; its `atproto-storage` branch explores grafting
155
+
this storage layer onto the real thing
+1
-1
build.zig
+1
-1
build.zig
+2
-2
build.zig.zon
+2
-2
build.zig.zon
···
1
1
.{
2
-
.name = .prefect_pds,
2
+
.name = .weir,
3
3
.version = "0.0.1",
4
-
.fingerprint = 0x33d77a550e0a83b7, // Changing this has security and trust implications.
4
+
.fingerprint = 0x93848e0c8724d628, // Changing this has security and trust implications.
5
5
.minimum_zig_version = "0.16.0",
6
6
.dependencies = .{
7
7
.zat = .{
+3
-3
docs/bench-results.md
+3
-3
docs/bench-results.md
···
2
2
3
3
`scripts/bench-reads`: identical seeded workload (5 flows, 300 flow runs,
4
4
~870 writes via the REST API, seed 1337), then 30 iterations per read
5
-
pattern, sequential, same machine (macOS arm64 M3 Max). prefect-pds is a
5
+
pattern, sequential, same machine (macOS arm64 M3 Max). weir is a
6
6
ReleaseFast-equivalent debug build? **no — debug build**; python 3.7.4 via
7
7
`uv run`. pattern names mirror `profile_queries.py` in
8
8
`prefectlabs/object-store-orchestration`.
9
9
10
-
| pattern | prefect-pds p50/p95 | python+sqlite p50/p95 | python+postgres p50/p95 |
10
+
| pattern | weir p50/p95 | python+sqlite p50/p95 | python+postgres p50/p95 |
11
11
|---|---|---|---|
12
12
| seed write rate | 184–202/s | 160/s | 163/s |
13
13
| point_lookup | **0.2 / 0.3** | 1.7 / 2.3 | 2.6 / 3.3 |
···
57
57
58
58
- single machine, sequential requests, demo scale (300 runs), one run per
59
59
target. variance not characterized.
60
-
- prefect-pds is a Debug build — its numbers would only improve.
60
+
- weir is a Debug build — its numbers would only improve.
61
61
- python servers measured through uvicorn + pydantic validation; that's
62
62
the product surface, not the database. this benchmarks *architectures
63
63
end-to-end*, not storage engines.
+1
-1
docs/lexicons.md
+1
-1
docs/lexicons.md
···
151
151
- `io.prefect.*` squats prefect's domain. fine for an internal experiment by
152
152
prefect people; a published protocol needs the org to bless the namespace.
153
153
154
-
## what this changes in prefect-pds
154
+
## what this changes in weir
155
155
156
156
1. drop `io.prefect.flowRunState` / `io.prefect.taskRunState` collections
157
157
2. set_state writes an `io.prefect.event` record (name
+1
-1
docs/scale-results.md
+1
-1
docs/scale-results.md
···
60
60
fresh buffer and retires the old one until the next command, which is
61
61
precisely the documented slice lifetime.
62
62
63
-
also fixed locally: prefect-pds retained every full CAR ever written in the
63
+
also fixed locally: weir retained every full CAR ever written in the
64
64
repo arena (one per commit — quadratic). commit-transient memory now lives
65
65
in a per-commit scratch arena and `latest_car` is freed on replacement.
66
66
+1
-1
scripts/bench-scale
+1
-1
scripts/bench-scale
···
78
78
car_mb = car.stat().st_size / 1048576 if car.exists() else 0
79
79
rss_mb = 0.0
80
80
try:
81
-
pid = subprocess.run(["pgrep", "-f", "bin/prefect-pds"],
81
+
pid = subprocess.run(["pgrep", "-f", "bin/weir"],
82
82
capture_output=True, text=True).stdout.split()[0]
83
83
rss_kb = subprocess.run(["ps", "-o", "rss=", "-p", pid],
84
84
capture_output=True, text=True).stdout.strip()
+3
-3
scripts/test-flow
+3
-3
scripts/test-flow
···
3
3
# requires-python = ">=3.12"
4
4
# dependencies = ["prefect>=3.0"]
5
5
# ///
6
-
"""smoke test: drive a real prefect client against prefect-pds.
6
+
"""smoke test: drive a real prefect client against weir.
7
7
8
8
usage:
9
9
PREFECT_API_URL=http://localhost:4200/api ./scripts/test-flow
···
14
14
15
15
from prefect import flow, task
16
16
17
-
RESULT_DIR = Path(tempfile.mkdtemp(prefix="prefect-pds-test-"))
17
+
RESULT_DIR = Path(tempfile.mkdtemp(prefix="weir-test-"))
18
18
19
19
20
20
@task
···
41
41
result = basic_flow(3, 4)
42
42
assert result == 14, f"expected 14, got {result}"
43
43
print(f"basic_flow(3, 4) = {result}")
44
-
print("=== flow ran against prefect-pds ===")
44
+
print("=== flow ran against weir ===")
45
45
46
46
47
47
if __name__ == "__main__":
+1
-1
src/api.zig
+1
-1
src/api.zig
···
103
103
fn csrfToken(req: *httpz.Request, res: *httpz.Response) !void {
104
104
const client = (try req.query()).get("client") orelse "unknown";
105
105
try res.json(.{
106
-
.token = "prefect-pds-csrf-token",
106
+
.token = "weir-csrf-token",
107
107
.client = client,
108
108
.expiration = "2099-01-01T00:00:00Z",
109
109
}, .{});
+9
-9
src/main.zig
+9
-9
src/main.zig
···
1
-
//! prefect-pds: a minimal prefect server whose database is an atproto repo.
1
+
//! weir: a minimal prefect server whose database is an atproto repo.
2
2
//!
3
3
//! boot: load (or mint) the workspace signing key, replay the repo CAR from
4
4
//! disk, hydrate the appview, serve. env:
5
-
//! PREFECT_PDS_PORT default 4200
6
-
//! PREFECT_PDS_DATA_DIR default ./data
5
+
//! WEIR_PORT default 4200
6
+
//! WEIR_DATA_DIR default ./data
7
7
8
8
const std = @import("std");
9
9
const httpz = @import("httpz");
···
21
21
defer threaded.deinit();
22
22
const io = threaded.io();
23
23
24
-
const port: u16 = if (std.c.getenv("PREFECT_PDS_PORT")) |p|
24
+
const port: u16 = if (std.c.getenv("WEIR_PORT")) |p|
25
25
std.fmt.parseInt(u16, std.mem.span(p), 10) catch 4200
26
26
else
27
27
4200;
28
-
const data_dir: []const u8 = if (std.c.getenv("PREFECT_PDS_DATA_DIR")) |d|
28
+
const data_dir: []const u8 = if (std.c.getenv("WEIR_DATA_DIR")) |d|
29
29
std.mem.span(d)
30
30
else
31
31
"data";
···
34
34
defer repo.deinit(allocator);
35
35
36
36
var cache: cache_mod.Cache = blk: {
37
-
const backend = if (std.c.getenv("PREFECT_PDS_CACHE")) |c| std.mem.span(c) else "burner";
37
+
const backend = if (std.c.getenv("WEIR_CACHE")) |c| std.mem.span(c) else "burner";
38
38
if (std.mem.eql(u8, backend, "redis")) {
39
-
const host = if (std.c.getenv("PREFECT_PDS_REDIS_HOST")) |h| std.mem.span(h) else "127.0.0.1";
40
-
const redis_port: u16 = if (std.c.getenv("PREFECT_PDS_REDIS_PORT")) |p|
39
+
const host = if (std.c.getenv("WEIR_REDIS_HOST")) |h| std.mem.span(h) else "127.0.0.1";
40
+
const redis_port: u16 = if (std.c.getenv("WEIR_REDIS_PORT")) |p|
41
41
std.fmt.parseInt(u16, std.mem.span(p), 10) catch 6379
42
42
else
43
43
6379;
···
49
49
};
50
50
defer cache.deinit();
51
51
52
-
const commit_every: usize = if (std.c.getenv("PREFECT_PDS_COMMIT_EVERY")) |c|
52
+
const commit_every: usize = if (std.c.getenv("WEIR_COMMIT_EVERY")) |c|
53
53
std.fmt.parseInt(usize, std.mem.span(c), 10) catch 1
54
54
else
55
55
1;