weir/src/ at main · zzstoatzz.io/weir · Tangled

zzstoatzz.io / weir

0

This repository has no description

0

weir / src

6 files

at main

zzstoatzz event feed: served directly from the canonical log 6w ago

event feed: served directly from the canonical log POST /events/filter and /events/count-by/{countable} are real now — and they read straight from the repo's io.prefect.v0.event collection: TID-keyed records decode back into prefect's ReceivedEvent shape, filtered by occurred range / event prefix / resource id, newest-first, with EventPage and EventCount envelopes. the histogram buckets by hour and carries each bucket's min/max occurred as start/end. this is the page where the architecture stops hiding: the feed the ui renders IS the signed event log, no derived table in between. (at scale it wants an appview index — collection scan per request is honest about being v0.) replaces the empty-envelope fallback that the graph fix introduced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

scale: real-redis appview, commit cadence, and the 8000-run gauntlet production shape lands: the Cache union gains a real-redis-over-TCP backend (PREFECT_PDS_CACHE=redis, container via orbstack), selected at boot; burner stays the zero-infra default. commit cadence becomes a knob (PREFECT_PDS_COMMIT_EVERY) — the same lever as the cloud prototype's delta-flush cadence. scripts/bench-scale drives 16 concurrent client lifecycles in cumulative tiers and found three real bugs before producing numbers: quadratic CAR retention here (fixed: per-commit scratch arena + child-allocator-owned latest_car), quadratic MST re-serialization in zat (fixed upstream on zat#mst-block-cache), and realloc-dangled response slices in the redis client under large SMEMBERS (fixed upstream on redis#retired-read-buffers). results (docs/scale-results.md): 8000 runs / ~30k records, no crashes. point reads flat at ~1ms across 32x scale; per-write commits decay O(records) as predicted; commit_every=20 flattens throughput at 840 w/s with p95 *improving* at depth. remaining walls are slopes with named fixes, not cliffs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

prefect-pds: a minimal prefect server whose database is an atproto repo no postgres, no sqlite: orchestration writes become records in an MST, sealed by signed v3 commits, persisted as one CAR file. the workspace is a did:key minted on first boot. reads are served from a disposable appview on embedded burner-redis (swappable behind a union). verified end to end: a real prefect>=3.0 python client runs basic_flow(3, 4) = 14 to Completed against it, and the resulting state history is browsable as at:// records via com.atproto.repo.listRecords and exportable as a signed CAR via com.atproto.sync.getRepo. restart reloads the repo from disk with the same identity. deps: zat v0.3.6 (MST/CBOR/CAR/signing/TIDs), httpz (zat-websocket-pin branch so the websocket package is shared with zat), burner-redis 0.0.4. v0 gaps documented in the README: no orchestration rules, no appview hydration on boot, events websocket stubbed, full-CAR rewrite per commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

the convergence demo: prefect ui served by weir + waow.tech workspace weir now serves the prefect react ui (ui-v2 bundle, copied from the python package via just fetch-ui): static SPA serving with index.html fallback, /ui-settings, real /paginate for flows + flow/task runs (the list pages use paginate, not filter), a graph-v2 stub for the run detail page, and generic empty-state fallbacks for the ui's long tail of filter/count/history/paginate queries. scripts/space-mirror is the first background service in the architecture's own idiom: it consumes weir's canonical log (listRecords polling — the firehose consumer's placeholder) and pushes new records into a permissioned io.prefect.v0.workspace space on zds as waow.tech. restart-safe via diffing against the space's existing rkeys. verified end to end: basic_flow runs against weir; the ui's paginate shows the run COMPLETED; the mirror lands all 6 records in ats://did:plc:siv7.../io.prefect.v0.workspace/weir-demo where the Pending->Running->Completed events read back with auth and strangers get 401. just demo wires the whole thing up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

events websocket: task runs arrive; compose + justfile cleanup prefect 3.x creates task runs locally and reports them only as events over ws /api/events/in — so weir now speaks that socket: upgrade with the 'prefect' subprotocol echoed when offered, the auth handshake ({type:auth} -> {type:auth_success}; every token succeeds, auth is an acknowledged gap), and ingest of task-run events: upsert into the appview + verbatim io.prefect.v0.event records + entity record on first sighting. flow-run events are skipped (already canonical via REST). the missing handshake reply was why flows hung at teardown — the client blocks on recv after sending auth. found + fixed a corruption regression in the zat encoding cache along the way: nodeCid cleared dirty without refreshing the cached encoding, so every CAR written after the second commit failed reload with BadBlockHash. fixed upstream (zat#mst-block-cache 3960e70) with a regression test there and a multi-commit reload test here. infra hygiene per review: compose.yml owns the appview redis and the justfile is reorganized around it (up/down/serve/demo/demo-flow/mirror /bench-*). scripts/demo-flow is a task-rich etl: 8 mapped transforms, a retry survivor (run_count=3), a tolerated failure. verified: etl_demo() = 204, 11 task runs in the appview (10 completed, 1 failed), 38 events in the canonical log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

prefect-pds: a minimal prefect server whose database is an atproto repo no postgres, no sqlite: orchestration writes become records in an MST, sealed by signed v3 commits, persisted as one CAR file. the workspace is a did:key minted on first boot. reads are served from a disposable appview on embedded burner-redis (swappable behind a union). verified end to end: a real prefect>=3.0 python client runs basic_flow(3, 4) = 14 to Completed against it, and the resulting state history is browsable as at:// records via com.atproto.repo.listRecords and exportable as a signed CAR via com.atproto.sync.getRepo. restart reloads the repo from disk with the same identity. deps: zat v0.3.6 (MST/CBOR/CAR/signing/TIDs), httpz (zat-websocket-pin branch so the websocket package is shared with zat), burner-redis 0.0.4. v0 gaps documented in the README: no orchestration rules, no appview hydration on boot, events websocket stubbed, full-CAR rewrite per commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago