+1 -5

CLAUDE.md

View file Reviewed

··· 3 3 ## deployment 4 4 - **backend**: push to `main` touching `backend/**` → auto-deploys via GitHub Actions 5 5 - **frontend**: manual deploy from `site/` directory (`cd site && wrangler pages deploy . --project-name leaflet-search`) 6 6 - - **tap**: manual deploy from `tap/` directory (`fly deploy --app leaflet-search-tap`) — STOPPED since 2026-06-09 (replaced by ingester; kept for rollback) 7 6 - **ingester**: manual deploy via `ingester/scripts/deploy.sh` (stages the repo-root `banned-dids.txt` into the build context first, then `fly deploy --app leaflet-search-ingester`). Don't call `fly deploy` directly — the build embeds `../banned-dids.txt` which isn't in `ingester/`'s context without staging. 8 7 - `--app` does NOT protect against deploying from the wrong directory — it only renames the target; the config (ports, env, mounts) still comes from that directory's `fly.toml`. Always `cd` into the app dir first. (2026-06-10: root-dir deploy with `--app leaflet-search-ingester` was stopped only by a volume-name mismatch.) 9 8 ··· 14 13 15 14 ## architecture 16 15 - **backend** (Zig): HTTP API, FTS5 search, vector similarity; same binary runs as the snapshot builder under `BUILDER_MODE=1` 17 17 - - **ingester** (Zig): our own firehose consumer — verifies every commit (signature + MST diff via zat), drops bridgy/non-canonical repos, re-emits over a tap-compatible `/channel` 16 16 + - **ingester** (Zig): our own firehose consumer — verifies every commit (signature + MST diff via zat), drops bridgy/non-canonical repos, re-emits over a `/channel` websocket the backend consumes (`backend/src/ingest/ingester.zig`) 18 17 - **site**: static frontend on Cloudflare Pages 19 18 - **db**: Turso (source of truth) + local SQLite read replica (FTS queries; FROZEN by construction — in-place sync deleted 2026-06-26 — refreshed only by snapshot adoption, see docs/scaling-plan.md) 20 19 - **R2**: `leaflet-search-index` bucket for builder snapshots (`INDEX_R2_*` secrets on the backend app) ··· 35 34 - channels: `staging` (default) → `staging/builds/…` + `latest.staging.json`; `prod` requires `BUILDER_ALLOW_PROD=1` and writes `builds/…` + `latest.json` (pointer uploaded LAST) 36 35 - gates before publish: doc-count tolerance vs turso, FTS sentinel, quick_check; banned DIDs + bridgy rows excluded at build time (`policy.zig`) 37 36 - completion signal: `builder: published <id> to <channel> channel` in logs/logfire 38 38 - 39 39 - ## tap operations (HISTORICAL — tap is STOPPED, replaced by ingester 2026-06-09) 40 40 - - see `docs/tap.md` for the protocol reference; `tap/` kept only for rollback 41 37 42 38 ## zig dependencies 43 39 - update a dependency hash: `zig fetch --save <url>` (fetches and updates build.zig.zon automatically)

+1 -2

docs/README.md

View file Reviewed

··· 22 22 static frontend (cloudflare pages) 23 23 ``` 24 24 25 25 - content flows in one direction: the firehose broadcasts every AT Protocol event in real-time, the [ingester](../ingester/) filters for publishing-related records and verifies each commit cryptographically (it replaced bluesky's [tap](tap.md) on 2026-06-09 — non-canonical repos like bridgy fed mirrors are dropped at the door), and the backend indexes them into turso. the serving replica is rebuilt offline and adopted atomically rather than synced in place — bulk data movement never touches the serving box. a [reconciler](reconciliation.md) periodically verifies documents still exist at their source, catching missed deletions. 25 25 + content flows in one direction: the firehose broadcasts every AT Protocol event in real-time, the [ingester](../ingester/) filters for publishing-related records and verifies each commit cryptographically (it replaced bluesky's indigo tap on 2026-06-09 — non-canonical repos like bridgy fed mirrors are dropped at the door), and the backend indexes them into turso. the serving replica is rebuilt offline and adopted atomically rather than synced in place — bulk data movement never touches the serving box. a [reconciler](reconciliation.md) periodically verifies documents still exist at their source, catching missed deletions. 26 26 27 27 ## how searching works 28 28 ··· 82 82 - [snapshot-pipeline.md](snapshot-pipeline.md) — how the keyword index ships (builder → manifest → R2 → verified adoption), what scales, and how to do production data surgery 83 83 - [scaling-plan.md](scaling-plan.md) — the plan of record: snapshot builder → R2 → verified swap → live overlay (largely executed; see status header) 84 84 - [retro-2026-06-10-cutover-cascade.md](retro-2026-06-10-cutover-cascade.md) — the outage night that produced the invariants behind that plan 85 85 - - [tap.md](tap.md) — HISTORICAL: the indigo tap sidecar the ingester replaced (protocol reference) 86 85 - [reconciliation.md](reconciliation.md) — stale document detection and cleanup 87 86 - [turso-hrana.md](turso-hrana.md) — Turso's HTTP protocol for database queries 88 87 - [migrations.md](migrations.md) — schema migration system (zug + adapter + bootstrap)

+2 -2

docs/reconciliation.md

View file Reviewed

··· 37 37 38 38 **historical drift (the main problem):** documents deleted while the tap was down are detected and cleaned up. this is the only mechanism that catches these — tap resync can't. 39 39 40 40 - **forward-looking vector leak:** the tap.zig delete handler now also calls `tpuf.delete()`, so future firehose deletes clean both turso and turbopuffer. 40 40 + **forward-looking vector leak:** the ingester.zig delete handler now also calls `tpuf.delete()`, so future firehose deletes clean both turso and turbopuffer. 41 41 42 42 ## files 43 43 ··· 46 46 | `backend/src/ingest/reconciler.zig` | background worker (~250 lines) | 47 47 | `backend/src/main.zig` | wires up `ingest.reconciler.start(allocator, io)` after `tpuf.init()` | 48 48 | `backend/src/db/migrations.zig` | `verified_at` column (in migration `001_initial_schema` — see [migrations.md](migrations.md)) | 49 49 - | `backend/src/ingest/tap.zig` | `tpuf.delete()` after `indexer.deleteDocument()` | 49 49 + | `backend/src/ingest/ingester.zig` | `tpuf.delete()` after `indexer.deleteDocument()` | 50 50 51 51 ## configuration 52 52

+1 -1

ingester/fly.toml

View file Reviewed

··· 1 1 - # leaflet-search-ingester — our own verified firehose ingester (replaces tap). 1 1 + # leaflet-search-ingester — our own verified firehose ingester. 2 2 # Worker app: no inbound http_service; it consumes the firehose outbound and 3 3 # (currently) emits comparison logs to logfire. Cursor persists to the volume 4 4 # so it resumes across restarts.

+1 -2

ingester/src/cbor_json.zig

View file Reviewed

··· 1 1 //! Serialize a decoded DAG-CBOR value (zat.cbor.Value) to atproto's dag-json 2 2 - //! form — the same JSON shape indigo `tap` emits, so the backend's extractor 3 3 - //! parses our records identically. Conventions: 2 2 + //! form — the JSON shape the backend's extractor expects. Conventions: 4 3 //! CID -> {"$link": "<base32 cidv1>"} (consumed as e.g. coverImage.ref.$link) 5 4 //! bytes -> {"$bytes": "<base64>"} 6 5 //! Everything else maps to the obvious JSON type.

+3 -3

ingester/src/channel.zig

View file Reviewed

··· 1 1 - //! Tap-compatible `/channel` websocket server. 1 1 + //! Firehose `/channel` websocket server. 2 2 //! 3 3 - //! Emits the exact frame shape the backend's tap.zig consumer expects: 3 3 + //! Emits the exact frame shape the backend's ingester consumer expects: 4 4 //! {"id":<seq>,"type":"record","record":{"action":..,"did":..,"collection":..,"rkey":..[,"record":<value>]}} 5 5 //! and accepts (ignores) the backend's {"type":"ack","id":..} replies — frames 6 6 //! replayed from the ring are delivered at-least-once and the backend's ··· 8 8 //! 9 9 //! While no client is connected (backend deploy/restart), frames land in a 10 10 //! bounded in-memory ring instead of being dropped; the ring drains in order 11 11 - //! to the next client that connects. This stands in for tap's durable outbox — 11 11 + //! to the next client that connects. This stands in for a durable outbox — 12 12 //! at our matched-event rate (~tens/min) the ring covers hours of backend 13 13 //! downtime, and anything beyond that is `/admin/backfill` territory. 14 14 //!

+7 -12

ingester/src/main.zig

View file Reviewed

··· 1 1 //! leaflet-search firehose ingester. 2 2 //! 3 3 - //! Standalone service that replaces the indigo `tap` sidecar. It consumes the 3 3 + //! Standalone service: our own verified firehose consumer. It consumes the 4 4 //! real firehose (com.atproto.sync.subscribeRepos), filters to leaflet-search's 5 5 //! collections, verifies each matched commit (signature + MST, see 6 6 - //! verifier.zig) in process, and re-emits verified records over a 7 7 - //! tap-compatible `/channel` websocket so the backend's existing consumer can 8 8 - //! point at us unchanged. 9 9 - //! 10 10 - //! Current mode: COMPARISON. Nothing points at /channel yet; every matched 11 11 - //! record logs `ingester.captured` (with a verified flag) so we can diff 12 12 - //! coverage against what tap delivers to Turso before cutting the backend 13 13 - //! over (see project_own_firehose_ingester memory). 6 6 + //! verifier.zig) in process, and re-emits verified records over the `/channel` 7 7 + //! websocket the backend consumes (live path since the 2026-06-09 cutover). 8 8 + //! Every matched record also logs `ingester.captured` (with a verified flag) 9 9 + //! so coverage stays auditable in logfire. 14 10 15 11 const std = @import("std"); 16 12 const Io = std.Io; ··· 19 15 const ch = @import("channel.zig"); 20 16 const vf = @import("verifier.zig"); 21 17 22 22 - // Collections we index — mirrors the backend's TAP_COLLECTION_FILTERS / tap.zig. 18 18 + // Collections we index — mirrors the backend's ingester collection filters. 23 19 const COLLECTIONS = [_][]const u8{ 24 20 "pub.leaflet.document", 25 21 "pub.leaflet.publication", ··· 132 128 if (tracked_ops > 0) { 133 129 // verify the commit before emitting any of its records. 134 130 // rejected/bridged commits are still logged as captured so 135 135 - // coverage comparison against tap stays honest, but they 136 136 - // never reach /channel. 131 131 + // coverage stays auditable, but they never reach /channel. 137 132 const verdict = self.verifier.verifyCommit(commit); 138 133 for (commit.ops) |op| { 139 134 if (!isTracked(op.collection)) continue;

+1 -1

ingester/src/verifier.zig

View file Reviewed

··· 19 19 raw: [33]u8, // compressed public key (secp256k1 or p256) 20 20 len: u8, 21 21 /// PDS hosted on brid.gy. policy since 7e1f071: bridgy fed content is 22 22 - /// never indexed — tap (indigo) rejected their non-canonical commits as 22 22 + /// never indexed — non-canonical commits are rejected as 23 23 /// a side effect of full MST verification; our sig_only verdict admitted 24 24 /// them and let ~20k scraper docs/day flood the corpus (2026-06-10). 25 25 bridged: bool,

+2 -2

scripts/backfill-publications

View file Reviewed

··· 56 56 57 57 SLINGSHOT = "https://slingshot.microcosm.blue" 58 58 59 59 - # corpus admission policy — mirror of backend/src/policy.zig + tap.zig 59 59 + # corpus admission policy — mirror of backend/src/policy.zig + ingest/ingester.zig 60 60 # isBridgyPds. Every write path (ingester, indexer, snapshot builder) drops 61 61 # these; a backfill must enforce the same gates or it re-admits exactly what we 62 62 # deliberately keep out. Registry of who/why/evidence: docs/exclusions.md. ··· 77 77 78 78 79 79 # both publication lexicons. leaflet stores the host in `base_path`; standard 80 80 - # stores a full `url` we strip to its host (matches backend tap.zig). 80 80 + # stores a full `url` we strip to its host (matches backend ingester.zig). 81 81 COLLECTIONS = ["pub.leaflet.publication", "site.standard.publication"] 82 82 83 83