pub-search/backend/src/server/ at main · zzstoatzz.io/pub-search · Tangled

zzstoatzz.io / pub-search

0

search for standard sites pub-search.waow.tech

search zig blog atproto

0

pub-search / backend / src / server

11 files

at main

zzstoatzz feat(api): GET /document — full extracted text by AT-URI 2d ago

feat(api): GET /document — full extracted text by AT-URI search responses carry snippets only, which forces agents to re-fetch records from the author's PDS and re-flatten block structures themselves (and older leaflet shapes defeat naive extractors). the indexer already stores extracted full text; serve it. up to 25 comma-separated uris per request, read entirely from the local replica, same visibility policy as search (banned/labeled/bridgy/dead-url excluded → listed under missing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2 days ago

feat(backend): /curators — leaderboard of recommenders New endpoint flips the axis of /recommended: ranks DIDs by how many recommends they've given out, with windowed counts (drives rank) and all-time totals (displayed). Same caching shape — WindowedJsonCache keyed by Window, background-refreshed every 45s. JSON shape: { did, recommendCount, totalRecommends, uniqueDocs, lastRecommendAt }. Frontend resolves DIDs → handles client-side via bsky's getProfiles, matching how /search results render @handles. server/curators.zig reuses recommended.Window (same URL semantics) and recommended.sliceJson (structure-agnostic JSON array slicer). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2 months ago

refactor: rename tap → ingester everywhere in the live path The standalone tap was replaced by our own ingester on 2026-06-09, but the code kept the tap vocabulary — confusing in ops (alerts told you to restart a dead machine). Renames the consumer module (tap.zig → ingester.zig), spans (tap.index_record → ingest.index_record, tap.dropped → ingest.dropped, …), log strings ("tap error" → "ingester error"), env (TAP_HOST/PORT/RELAY_URL → INGESTER_*), and types/functions. fly.toml env updated in lockstep. Alert queries that key on the span/log names are updated separately, after deploy, so the rename doesn't false-fire the stall alert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

3 weeks ago

feat(api): GET /document — full extracted text by AT-URI search responses carry snippets only, which forces agents to re-fetch records from the author's PDS and re-flatten block structures themselves (and older leaflet shapes defeat naive extractors). the indexer already stores extracted full text; serve it. up to 25 comma-separated uris per request, read entirely from the local replica, same visibility policy as search (banned/labeled/bridgy/dead-url excluded → listed under missing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2 days ago

perf(subscriptions): materialize the publication join key (sargable indexed equijoin) the publication join matched on the (did, rkey) PARSED out of publication_uri per row, which defeated every index. a publisher with 621 publications full-scanned the whole subscriptions table per publication (>60s observed). store the parsed (publication_did, publication_rkey) once at write/build time and index it, so the join is an indexed equijoin. EXPLAIN goes SCAN s → SEARCH s USING idx_subscriptions_pub_did_rkey on the publisher side; /wrapped, /subscribed, /subscribers all use it. - pubkey.joinOnStored: sargable ON predicate against the stored columns - LocalDb.createSchema: idempotent ALTER + boot backfill + index so an adopted snapshot is correct immediately (no multi-step rollout) - sync.buildSnapshot: populate at build time so snapshots ship correct - migration 019: same columns + backfill + index on turso - indexer.insertSubscription: parse + bind on live ingest - rewrite all 10 joinOn call sites to joinOnStored Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

1 month ago

recommended.zig

feat(wrapped): recent vs most-recommended tabs for a curator's picks The curator section was labeled "recently recommended" but ordered by global recommend_count — so a curator's genuinely recent recommend of a niche/older doc never surfaced (it was a popularity list wearing a recency label). Split into two honestly-labeled tabs: - recent: orders by when THIS curator recommended each doc (new sort=recent; curator variant uses the curator's own recommend timestamp, author/unfiltered use the doc's latest recommend). Default tab. - most recommended: the existing popularity order (sort=top). backend: add Sort.recent + Recent{,ByAuthor,ByCurator}Query (ORDER BY only — identical projection, comptime column-drift assertion still holds); recent is uncached (handler live-fetches on null snapshot). Also trim the freshness footer to "↻ updated ~N ago" (full explanation moved to a title tooltip) — it was needlessly verbose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

3 weeks ago

recommenders.zig

feat(recommended): reveal WHO recommended a post (avatar popover) The recommend count was a dead end — it shows how many, never who. Adds a /recommenders?document=<uri> endpoint that opens up the COUNT(DISTINCT did) aggregate (deduped, recency-ordered, local-replica-first) and a plyr.fm- inspired popover of overlapping avatar circles on the score: hover-reveal on desktop, tap-reveal on mobile, springy hover-lift, shimmer skeletons, profile links resolved via typeahead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

1 month ago

fix search pagination across all surfaces

6 days ago

refactor: rename tap → ingester everywhere in the live path The standalone tap was replaced by our own ingester on 2026-06-09, but the code kept the tap vocabulary — confusing in ops (alerts told you to restart a dead machine). Renames the consumer module (tap.zig → ingester.zig), spans (tap.index_record → ingest.index_record, tap.dropped → ingest.dropped, …), log strings ("tap error" → "ingester error"), env (TAP_HOST/PORT/RELAY_URL → INGESTER_*), and types/functions. fly.toml env updated in lockstep. Alert queries that key on the span/log names are updated separately, after deploy, so the rename doesn't false-fire the stall alert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

3 weeks ago

subscribers.zig

perf(subscriptions): materialize the publication join key (sargable indexed equijoin) the publication join matched on the (did, rkey) PARSED out of publication_uri per row, which defeated every index. a publisher with 621 publications full-scanned the whole subscriptions table per publication (>60s observed). store the parsed (publication_did, publication_rkey) once at write/build time and index it, so the join is an indexed equijoin. EXPLAIN goes SCAN s → SEARCH s USING idx_subscriptions_pub_did_rkey on the publisher side; /wrapped, /subscribed, /subscribers all use it. - pubkey.joinOnStored: sargable ON predicate against the stored columns - LocalDb.createSchema: idempotent ALTER + boot backfill + index so an adopted snapshot is correct immediately (no multi-step rollout) - sync.buildSnapshot: populate at build time so snapshots ship correct - migration 019: same columns + backfill + index on turso - indexer.insertSubscription: parse + bind on live ingest - rewrite all 10 joinOn call sites to joinOnStored Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

1 month ago

perf(subscriptions): materialize the publication join key (sargable indexed equijoin) the publication join matched on the (did, rkey) PARSED out of publication_uri per row, which defeated every index. a publisher with 621 publications full-scanned the whole subscriptions table per publication (>60s observed). store the parsed (publication_did, publication_rkey) once at write/build time and index it, so the join is an indexed equijoin. EXPLAIN goes SCAN s → SEARCH s USING idx_subscriptions_pub_did_rkey on the publisher side; /wrapped, /subscribed, /subscribers all use it. - pubkey.joinOnStored: sargable ON predicate against the stored columns - LocalDb.createSchema: idempotent ALTER + boot backfill + index so an adopted snapshot is correct immediately (no multi-step rollout) - sync.buildSnapshot: populate at build time so snapshots ship correct - migration 019: same columns + backfill + index on turso - indexer.insertSubscription: parse + bind on live ingest - rewrite all 10 joinOn call sites to joinOnStored Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

1 month ago