weir#

a weir is a low dam that shapes a river's flow, measures it as it passes, and never owns the water.

this is a prefect-compatible orchestration server whose database is an atproto repo: every flow, run, and state transition is a record in a merkle search tree, sealed under signed commits, persisted as a single CAR file. your orchestration history is cryptographically verifiable, portable, and exportable with one HTTP call — the server shapes and measures your flows; the water stays yours.

there is no postgres. there is no sqlite.

the architecture, in contrast#

prefect's self-hosted architecture is the classic shape — a load balancer over N API servers and N background services, all reading and writing one shared data layer:

flowchart TD
    clients["users / UI / workers"] --> lb["load balancer"]
    lb --> api["api servers ×N"]
    svcs["background services ×N"] --> pg
    api --> pg[("postgresql<br/>state · config · history")]
    api --> rd[("redis<br/>events · coordination")]
    svcs --> rd

the database is the system of record, the event stream is internal plumbing, and your history lives inside a postgres only the operator can interpret.

weir keeps the same surface — the prefect SDK and UI speak to it unchanged — and replaces the data layer with two pieces that have sharply different jobs:

flowchart LR
    sdk["prefect sdk / ui / curl<br/><i>unchanged</i>"]

    subgraph weir["weir — one binary, ~1 MB"]
        api["prefect-compatible REST"]
        gw["write gateway<br/>records → signed commits"]
        av["appview<br/>filters · counts · point reads"]
    end

    subgraph truth["canonical — yours"]
        car[("repo.car<br/>signed MST of<br/>io.prefect.v0.* records")]
    end

    subgraph derived["derived — disposable"]
        redis[("redis<br/>(or embedded burner-redis)")]
    end

    sdk --> api
    api --> gw --> car
    api --> av --> redis
    car -. "replay on boot<br/>(rebuilds the appview)" .-> redis

    car --> export["getRepo → CAR<br/>verify offline · query with any engine"]

the asymmetry is the whole idea:

state transitions are not rows that get updated; they are io.prefect.v0.event records appended to the log (mirroring prefect's own Event schema), and a run's "current state" is just appview material derived from them. delete redis, restart weir, and the entire read surface rebuilds from the signed log — that replay runs on every boot.

because the canonical store speaks atproto, the export story is free: com.atproto.sync.getRepo hands anyone the workspace as a CAR they can verify offline against the workspace DID and query with any engine. same argument iceberg makes for BYOS, plus cryptographic provenance.

try it#

zig build -Doptimize=ReleaseFast
./zig-out/bin/weir                       # port 4200, data in ./data

# in another shell: run a real prefect flow against it
just smoke

# look at what it wrote
just records io.prefect.v0.event         # the event log, as at:// records
just export-car                          # the whole database, signed

on boot weir mints (or reloads) a p256 keypair — the workspace DID is did:key:..., zero external services. for the production shape, point the appview at a real redis:

docker run -d --name weir-redis -p 6390:6379 redis:7-alpine
WEIR_CACHE=redis WEIR_REDIS_PORT=6390 ./zig-out/bin/weir

numbers#

same seeded workload, same machine, sequential reads (details + caveats):

under 16 concurrent clients to 8,000 runs (~30k records), point reads hold ~1 ms flat and batched commits sustain 840 writes/s (scale results — including the three real bugs the harness found, two of them upstream).

status#

an experiment with working proofs, not a product. real prefect>=3.0 clients run flows to completion; filters/counts/point reads serve the SDK's read paths; the same io.prefect.v0.* records have been written to a local did:key repo, a stock bsky.social account (where the public jetstream carried the events — free event bus), and a private permissioned space on zds with oplog + 401s for strangers.

deliberately absent so far: orchestration rules (every transition is ACCEPTed), auth (localhost demo), the events websocket, blob storage, time-window filters. the lexicon rationale lives in docs/lexicons.md, the read-path design in docs/access-patterns.md.

siblings#

• zat — every atproto primitive here (MST, DAG-CBOR, CAR, signing, TIDs)
• zds — the PDS; permissioned spaces are the private-workspace substrate
• burner-redis — the embedded appview default
• prefect-server — the full zig prefect server; its atproto-storage branch explores grafting this storage layer onto the real thing