zzstoatzz.io
notification manager for bsky
noti.waow.tech
Don't bulk-pull follows: collection cardinality dictates access pattern
The follow=500(capped) in the eval was a tell — bulk-scraping an unbounded set is
the request-time-scraper anti-pattern this architecture exists to kill, and a silent
cap is just truncation (removing it to pull all 5k would be worse). The fix isn't the
number; it's the access pattern.
Principle (now in the doc): bounded high-signal collections (blocks, lists) are
materialized; high-cardinality/low-signal ones (follows) are maintained incrementally
from the firehose changelog (each event O(1)) and answered as scoped membership checks
("does U follow THIS actor?" via getRelationships for the few actors of interest) —
never bulk-pulled. featuresFromRepoRecords stays per-record-pure (correct for the
changelog path); the anti-pattern was only in the eval's bulk listRecords over follows,
now removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Opus 4.8 (1M context)
date
(May 31, 2026, 1:45 AM -0500)
commit
7c414592
7c414592be3624af67fd7701ab82c6f5c5d35470
parent
1b255227
1b255227f302be313c29d385011aabbfe8839762
+28
-3
+19
docs/2026-05-31-firehose-ingestion.md
+19
docs/2026-05-31-firehose-ingestion.md
···
149
149
hydration stays a **separate identity adapter** (public `getProfiles`, cached as aliases/features),
150
150
not part of the firehose path.
151
151
152
+
## collection cardinality dictates access pattern (don't bulk-pull large sets)
153
+
154
+
Not all collections are equal, and treating them uniformly (paginate everything,
155
+
cap at N) reintroduces the request-time-scraper anti-pattern through the back door —
156
+
a silent cap is just truncation, and removing the cap to pull all 5,000 follows is
157
+
worse. The rule:
158
+
159
+
| collection | cardinality / signal | access pattern |
160
+
| --- | --- | --- |
161
+
| `block`, `listblock` | small, high-signal | **materialize fully** (cheap; backfill + live) |
162
+
| `list` / `listitem` | bounded, high-signal | materialize fully |
163
+
| `follow` | high-cardinality, low *individual* signal | **never bulk-pull.** maintained incrementally from the changelog (each create/delete is O(1)); answered as a **scoped membership check** — "does U follow *this* actor?" via `getRelationships(actor=U, others=[the few we care about])`, bounded by actors-of-interest, not graph size |
164
+
165
+
So the question is never "how do we get the 500 follows" — it's "we never fetch the
166
+
follow set; we receive follow events incrementally, and we ask about specific actors
167
+
on demand." Backfill of any genuinely large collection must **stream** (the 128 MB
168
+
limit), not buffer. `featuresFromRepoRecords` is per-record-pure, so it's correct for
169
+
the changelog path as-is; the anti-pattern lives in any caller that bulk-scans.
170
+
152
171
## staged build plan (revised)
153
172
1. **Shadow coordinator** — connect to jetstream for tracked DIDs + graph collections; persist cursor;
154
173
reconnect/heartbeat. Log live events only, no writes (learn real volume/shape; outbound-WS cost).
+9
-3
scripts/eval.ts
+9
-3
scripts/eval.ts
···
13
13
14
14
const APPVIEW = 'https://api.bsky.app'
15
15
const PLC = 'https://plc.directory'
16
-
const GRAPH_COLLECTIONS = ['app.bsky.graph.block', 'app.bsky.graph.follow', 'app.bsky.graph.listitem']
17
-
const MAX_PER_COLLECTION = 500
16
+
// Only bounded, high-signal collections are bulk-read here. Follows are
17
+
// high-cardinality / low individual signal: in production they're maintained
18
+
// incrementally from the firehose changelog (one O(1) event each) and answered as a
19
+
// scoped check ("does U follow THIS actor?", via getRelationships for the few actors
20
+
// we care about) — never bulk-pulled. Bulk-scraping a follow graph is the
21
+
// request-time-scraper anti-pattern this architecture exists to kill.
22
+
const GRAPH_COLLECTIONS = ['app.bsky.graph.block', 'app.bsky.graph.listitem']
23
+
const MAX_PER_COLLECTION = 1000
18
24
19
25
type Profile = {did: string; handle?: string; displayName?: string; avatar?: string}
20
26
···
95
101
console.log(` pds: ${pds || '(unresolved)'}`)
96
102
console.log(` avatar: ${profile.avatar ? 'yes' : 'no'}`)
97
103
console.log(` records: ${GRAPH_COLLECTIONS.map(c => `${c.split('.').pop()}=${counts[c] ?? 0}${counts[c] >= MAX_PER_COLLECTION ? '(capped)' : ''}`).join(' ')}`)
98
-
console.log(` derived pds features: ${features.length} (blocked=${features.filter(f => f.key === 'viewer.blocked').length}, follows=${features.filter(f => f.key === 'viewer.follows').length}, list.member=${features.filter(f => f.key === 'list.member').length})`)
104
+
console.log(` derived pds features: ${features.length} (blocked=${features.filter(f => f.key === 'viewer.blocked').length}, list.member=${features.filter(f => f.key === 'list.member').length})`)
99
105
console.log(` labels: ${labels.length ? labels.map(l => `${l.key.slice('label.'.length)} [${l.sourceId.slice(0, 14)}…]`).join(', ') : 'none'}`)
100
106
101
107
// ── structural assertions ──