weir vs the cloud prototype: compared learnings#

prefectlabs/object-store-orchestration (guidry) is the proof-driven prototype behind the cloud data-architecture initiative: 22 proven proofs, 3 disproven ones, a GKE whale-workspace simulator, and a rust datafusion/iceberg read layer. weir is the atproto-canonical version of the same instinct. this doc lines up what each has learned, because the overlap is not coincidence — it's the same architecture meeting different trust boundaries.

convergences we found independently#

commit-per-write doesn't scale on snapshot stores — buffer and batch. his disproven all-you-need-is-iceberg: committing every write storms the catalog (200+ commits/s → CAS losers dominate, ~720k manifests/hour, query planning walks a million manifest entries, tiny files defeat the cache). weir hit the identical wall from the other side: signing + rewriting the CAR per write decays from 855 → 118 writes/s as the repo grows. both architectures landed on the same lever — his delta buffer with flush cadence, our commit_every — and both keep unflushed writes queryable (his read-time delta merge; our appview dual-write). same physics, discovered twice.

hand-rolling the catalog is a trap — adopt a snapshot model. his disproven all-you-need-is-parquet-and-redis: the bespoke file-list HASH catalog suffered CAS storms under power-law load, cache-hostile in-place overwrites, and no schema evolution; iceberg's snapshot pointer fixed all three. weir never hit this because the atproto repo IS a snapshot model — iceberg for records, plus signatures: metadata.json pointer CAS ≈ the commit swap; manifest list ≈ the MST; immutable content-addressed data files ≈ blocks. we inherited on day one the thing he had to migrate to.

many small objects is death. his disproven state-partitioned-indexes (52–95× degradation at 500 delta files) and the iceberg tiny-files problem ↔ our one-CAR/batched-commits decision. both sides now treat "one object per record" as a known anti-pattern.

live state is not archival state. his two-tier active-run design (redis holds live attributes; the durable record holds identity + config; handoff at terminal) is exactly weir's entity-records-without-state + appview split — we cited his plane doc when we made that call, but the proofs show how load-bearing it is: it's what keeps the hot path off the snapshot store entirely.

schema evolution without migrations. iceberg's column-ID evolution (rename/drop safely, old snapshots stay consistent) and lexicon evolution (additive-only, breaking = mint v1) are different mechanisms for the same vow: never run a blocking migration again. iceberg evolves in place by ID; lexicons freeze and version. ours is stricter and simpler; his preserves renamability.

his lessons weir has not yet absorbed#

1. pushdown. his filter compilation turns prefect filters into parameterized datafusion SQL with LIMIT/OFFSET pushed down — ~40× over the N+1 approach, only the requested page materializes. weir's filter engine loads every candidate row, sorts, then slices (visible in the scale results: completed_50 grows 16 → 477ms). the fix is known (time-ordered zset, slice before loading) and his numbers say it's worth doing before any real workspace.
2. cold-start economics. his read path evolved three times: download-everything → native byte-range reads → cluster-shared redis byte-range cache with write-time priming (cold 23.5s → warm-everywhere tens-of-ms). weir's equivalent sin: hydration replays the entire repo on every boot even when the redis appview is already warm — we should skip replay when the appview is populated, and prime it at write time (which we already do; we just throw it away on restart).
3. stats-based pruning beats filename contracts. his hourly-file pruning worked (linear ~7ms/file) but iceberg's per-file min/max stats superseded it. weir's analog is better than the filename version and unbuilt: TID rkeys make the event collection a time-ordered keyspace, so an MST range walk IS partition pruning. listEvents should learn since/until key-range walks instead of full collection scans.
4. compaction wants snapshot atomicity. his pre-iceberg compaction needed deferred deletes, two-pass rewrites, version gating; iceberg collapsed it to "write new files, swap pointer, expire old snapshots." weir's block-level CAR append + periodic repack inherits the same trivial shape from content addressing — old blocks stay valid, the swap is one rename. design accordingly when the CAR rewrite finally gets replaced.
5. query-engine pooling for multi-tenancy. datafusion over duckdb because SessionContexts are ~1MB and stateless (LRU pool of 128) while duckdb forces instance-per-workspace or shared-instance-no-isolation. irrelevant at weir's single-workspace scale; decisive the day weir hosts many. the trait stack he praises (replaceable I/O layers) is the same property we exploit in zat.

what weir has that the prototype cannot express#

• verifiability. signed commits, offline verification against the workspace DID. his planes have no provenance story at all; ours is the default.
• portability as protocol, not format. his BYOS argument is "iceberg in your bucket — point any engine at it." ours is that plus identity: getRepo hands you the workspace, the DID makes it resolvable, and the format is specified by a protocol with multiple implementations.
• the seam is public. his event stream feeds internal consumers; search rewiring onto it is an initiative project. weir's firehose/oplog is the architecture's front door — the event feed UI reads the canonical log directly, and any third party could too.
• one binary. his prototype spans python + rust + redis + GCS + a catalog; weir is a ~1MB zig binary plus optional redis. different scales, but the operational floor difference is real.

shared open questions#

• concurrent throughput at production fan-in (his latency numbers are single-query; ours stop at 16 clients).
• whale-scale terminal queries (~23.5M rows) — neither side has measured.
• the postgres head-to-head he lists as open — weir's bench-reads covers it only at n=300.

verdict#

the prototype's proofs read like a de-risking document for weir's roadmap: the order of work he discovered the hard way (batch commits → pushdown → shared cache → stats pruning → snapshot compaction) is the order weir should climb, and two of those rungs we already stand on because atproto shipped them. what weir adds is the layer his stack has no vocabulary for: the store is signed, portable, and public by construction. his architecture asks "can object storage replace postgres?" — proven. weir asks "if the storage layer is a protocol, who else gets to read it?" — and that question is the product.