zzstoatzz.io / weir
This repository has no description
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

weir / docs / access-patterns.md
5.5 kB

access patterns & aggregations#

how reads work over an atproto-repo-canonical store, calibrated against the cloud prototype (prefectlabs/object-store-orchestration, guidry's proofs/ directory) so the two designs can be compared honestly.

what the cloud prototype proved (and what we borrow)#

the prototype's GKE simulation (p95/whale workspace, ~440 writes/s, 30 query patterns) sorted every read into three latency bands:

band what answers it examples (their p50)
sub-30 ms redis directly, or pod-local index cache active_by_state 14ms, deployment reads 5ms
100–500 ms iceberg hour-partition scans through a byte-range cache completed_1h 112ms → completed_24h 279ms
500–900 ms delta-merge dominated (hot rows not yet flushed) task_runs_24h 869ms, by_tag_24h 918ms

three of their findings transfer directly:

  1. the live-state index is a membership structure, not a table. their active_by_state is a redis ZSET; ours is a burner-redis set per (kind, state). same shape, same O(members) cost, same answer to "what is running right now."
  2. the latency knob is unmerged-delta size, not file count. their scan_redis_deltas finding (5-min flush → 830ms; 30-s flush → ~460 deltas and falling). our analog is commit cadence: everything in the appview is always "flushed" (synchronous dual-write), so we sit at the degenerate happy case — at the cost of write latency, which is the trade to benchmark.
  3. many small objects is death. their disproven parquet-indexes proof: 500 individual delta files = 52–95× degradation. our analog decision: one CAR, batched commits — never a file per record. (the full-CAR-rewrite-per-commit we do today is the opposite extreme and fails at the other end; block-level append is the meeting point.)

their explicitly open item: "we haven't run the same queries against the production Postgres to compare head-to-head." that's the gap our benchmark fills at small scale (see scripts/bench-reads).

the read surface, mapped#

pattern (their name) their path our path
run_point_lookup redis hash / bucket JSON appview row get (flow_run:<id>)
active_by_state redis ZSET slice s:flow_run:state:RUNNING set → rows
terminal_completed_* iceberg partition scan + delta merge s:flow_run:state:COMPLETED set → rows, ISO-string sort
paginated_* OFFSET over iceberg scan sort + slice (same O(n), honest about it)
by_flow, by_deployment IndexTable predicate pushdown s:flow_run:flow:<id> membership set
deployment_list IndexTable (parquet base + redis delta) s:flow:all → rows, name sort
count_by_state ZSET cardinality scard — O(1)
time-window scans (completed_1h) hour-partition pruning TID-range scans (designed, not built): event rkeys are TIDs, and the MST is ordered — a key-range walk from TID(t0) to TID(t1) is partition pruning by construction

sorting note: ISO-8601 timestamps sort correctly as strings, so the appview never parses dates on the read path.

aggregations: three tiers#

tier 1 — maintained aggregates (answers in O(1)). counts by state, by queue, by deployment: set cardinalities updated on every transition. this is the cloud prototype's redis counter layer and it covers the dashboard's top row. cheap to add: per-day counters (c:flow_run:completed:2026-06-12) bumped on terminal events.

tier 2 — bounded scans (answers in O(candidates)). everything /filter shaped: membership-set intersection → row loads → sort → slice. fine to ~10⁵ rows per workspace in-process. this is where we live today and where the benchmark measures us.

tier 3 — analytical queries (answers in O(data), offline-friendly). the cloud answer is iceberg + datafusion, load-bearing for BYOS: "point any engine at your bucket." our answer is the same argument with a different open format: the repo CAR is the export. getRepo → decode records → parquet → duckdb/datafusion/polars. the pipeline is trivially scriptable (just export-car && duckdb), needs zero server support, and inherits the verifiability story — you can prove the parquet faithfully derives from the signed repo. group-bys over months of history don't belong on the orchestration hot path in either design.

the event log makes tier 3 unusually clean here: io.prefect.v0.event is already the fact table (occurred, event name, resource id, payload), TID rkeys give natural time partitioning, and entity records are the dimension tables. it's a star schema by accident.

what we deliberately don't do#

  • no SQL pushdown into the appview — filter shapes are compiled by hand (the cloud prototype compiles prefect filter schemas into datafusion plans; see their filter-schema-compilation.md. if our filter surface grows past a handful of shapes, that's the design to copy.)
  • no time-range filters yet (needs TID-range walks or a created-time zset)
  • no cross-entity joins on the hot path — flow_runs joined to deployments resolves through membership sets or moves to tier 3

benchmark#

scripts/bench-reads seeds an identical workload through the REST API of each target, then measures the shared read patterns. honest caveats baked in: same process, same machine, sequential requests, demo scale — this measures architecture overhead, not cloud scale. results in docs/bench-results.md.