pub-search/backend/src/ at main · zzstoatzz.io/pub-search · Tangled

zzstoatzz.io / pub-search

0

search for standard sites pub-search.waow.tech

search zig blog atproto

0

pub-search / backend / src

5 folders 11 files

at main

zzstoatzz fix(labeler): queryLabels honors the full lexicon — `*`, prefixes, sources, paging 8hrs ago

fix(ci): raise SQL validation branch quota

5 days ago

fix(indexer): don't re-stamp indexed_at on unchanged re-puts platforms mass re-put whole archives over the firehose (4,951 docs re-indexed 2026-07-21 alone, in per-author bursts). every re-put bumped indexed_at, which (a) churned the below-watermark set the snapshot builder's count gate assumes immutable — built vs expected drifted ~450 per build, 8x the ±0.1% tolerance, so the builder crash-looped and its hourly full-table scans degraded search latency — and (b) burned a turso FTS delete+insert pair per unchanged doc. indexed_at now means "content last changed": the upsert preserves it when content_hash matches, and the FTS rewrite (title+content, exactly what the hash covers) is skipped. metadata-only updates still apply and reach the replica via full-row snapshot copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

5 days ago

fix(labeler): queryLabels honors the full lexicon — `*`, prefixes, sources, paging uriPatterns was read as a single literal string matched with `uri = ?`, so the spec's wildcard (`uriPatterns=*`) looked for a subject named "*" and every generic client got `{"labels":[]}`. Reported by @hipstersmoothie.com. Now parses uriPatterns as an array (repeated params, comma-joined values, or both), supports `*` and `prefix*` patterns, and adds `sources`, `limit` (1-250, default 50) and `cursor` with a response cursor on full pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

8 hours ago

metrics: exclude the /activity liveness poll from the traffic chart The stats-page traffic chart summed every endpoint's hourly count, including /activity — which dashboard.js polls once a second per open tab. A single stats page left open therefore pins the chart at ~3,500 req/hr (a flat 1/s), 99% of it that self-poll, while real request volume sits at ~20-40/hr. That's a UI implementation detail masquerading as product traffic. getTrafficSeries() now skips endpoints where countsAsTraffic() is false (currently just /activity). Per-endpoint latency stats are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

3 weeks ago

feat(api): GET /document — full extracted text by AT-URI search responses carry snippets only, which forces agents to re-fetch records from the author's PDS and re-flatten block structures themselves (and older leaflet shapes defeat naive extractors). the indexer already stores extracted full text; serve it. up to 25 comma-separated uris per request, read entirely from the local replica, same visibility policy as search (banned/labeled/bridgy/dead-url excluded → listed under missing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2 days ago

fix(labeler): queryLabels honors the full lexicon — `*`, prefixes, sources, paging uriPatterns was read as a single literal string matched with `uri = ?`, so the spec's wildcard (`uriPatterns=*`) looked for a subject named "*" and every generic client got `{"labels":[]}`. Reported by @hipstersmoothie.com. Now parses uriPatterns as an array (repeated params, comma-joined values, or both), supports `*` and `prefix*` patterns, and adds `sources`, `limit` (1-250, default 50) and `cursor` with a response cursor on full pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

8 hours ago

chore(sync): delete in-place Turso→replica sync (footgun removal) The serving box's replica is refreshed only by verified snapshot adoption at boot (SYNC_DISABLE=1 has held in prod since the 2026-06-10 cutover). The in-place incrementalSync/fullSync path was still compiled and wired into a background thread + watchdog, neutralized only by an env flag — a footgun: drop the flag and background data movement touches the serving box again (the 06-10 outage). Delete it so the serving artifact is immutable by construction. - sync.zig: drop fullSync + incrementalSync (and the dead BATCH_SIZE); keep buildSnapshot + insert helpers (the offline builder, used by builder.zig) - db.zig: startSync now just marks the replica ready; remove syncLoop, syncWatchdog, the heartbeat atomics, and the second Turso client - off-box freshness (scripts/offline-replica-catchup) is a standalone python reimplementation — unaffected Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

1 month ago

refactor: rename tap → ingester everywhere in the live path The standalone tap was replaced by our own ingester on 2026-06-09, but the code kept the tap vocabulary — confusing in ops (alerts told you to restart a dead machine). Renames the consumer module (tap.zig → ingester.zig), spans (tap.index_record → ingest.index_record, tap.dropped → ingest.dropped, …), log strings ("tap error" → "ingester error"), env (TAP_HOST/PORT/RELAY_URL → INGESTER_*), and types/functions. fly.toml env updated in lockstep. Alert queries that key on the span/log names are updated separately, after deploy, so the rename doesn't false-fire the stall alert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

3 weeks ago

feat(labeler): rename label machine-generated → bulk-generated 'machine-generated' collides with what people mean by AI-written in 2026 — and the label explicitly does NOT mean that (original AI writing is welcome; the line is composed vs generated). 'bulk-generated' names what we actually exclude: documents generated in bulk from a dataset. Rename playbook (same as bulk-mirror → machine-generated at v6): LABEL_BULK_GENERATED constant, SCORING_VERSION 9→10 (bootstrap negates prior labels — under BOTH values, since negation matches on val — and re-judges), service declaration + /labels page + docs updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

3 weeks ago

feat(api): GET /document — full extracted text by AT-URI search responses carry snippets only, which forces agents to re-fetch records from the author's PDS and re-flatten block structures themselves (and older leaflet shapes defeat naive extractors). the indexer already stores extracted full text; serve it. up to 25 comma-separated uris per request, read entirely from the local replica, same visibility policy as search (banned/labeled/bridgy/dead-url excluded → listed under missing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2 days ago

refactor: rename mod.zig to sibling-file pattern (idiomatic zig) Zig convention uses foo.zig next to foo/ directory, not foo/mod.zig (which is a Rust pattern). Renamed all four module entry points and updated imports throughout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

4 months ago

feat(policy): kept-dids.txt — labeled but deliberately kept (festivus) thefestivusproject.com is genuinely bulk-generated (one doc per Seinfeld episode — the model's verdict is correct) and also delightful. Negating the label would make the public stream lie; carving 'unless it's funny' into the judge would put taste in the one place it must never live. The architecture already has the seam: the label states what a thing IS, the consumer decides what to DO — and consumer policy is allowed taste. kept-dids.txt (repo root, embedded like banned-dids.txt) lists accounts pub-search keeps despite a true label. policy.isKept gates the future hard-drop; /labels shows a 'kept' badge with the reasoning in its tooltip. kept ∩ banned = ∅ enforced by test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

3 weeks ago

feat(backend): serve top-author cascade from the replica — schema v2 (recommends in snapshot) /recommended-by-top-authors walked ~30k turso rows per request (the pool ranking re-joins all recommends x documents every call) and was perpetually cold because it is rarely hit: 826ms recorded, 2s+ measured cold, ~130ms warm. The parameter grid (pool 1..500 x 5 windows) is why it never fit the cache-slot model. Now the builder copies recommends into the snapshot (3,046 rows, policy-filtered, watermark-pinned, rec_count in the manifest with an exact promote gate) and the cascade runs against the local replica: identical output verified against the live turso baseline, ~1ms for ANY pool x window combo, zero turso rows read, never cold. Falls back to turso until the first v2 snapshot is adopted. SCHEMA_VERSION 1 -> 2 — the promote gate stalls freshness on v1 builds until the builder machine is recreated (by design). Scaling-plan invariant #2 applied to its last holdout: no request does corpus-proportional remote work. Also unlocks pointing the recommended/ curators cache refreshes at the replica later (retiring the standing 45s turso refresh load). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

feat(backend): builder mode — offline snapshot build + verified R2 publish BUILDER_MODE=1 runs the backend binary as a snapshot builder instead of a server: builds a fresh replica from turso into a scratch path, gates on doc-count tolerance + FTS sentinel + quick_check, compacts (VACUUM, single file), writes a manifest (sha256, byte size, source watermark, counts), and uploads artifact + manifest + latest pointer (LAST) to R2 via rclone. Never touches /data, never starts the server — scaling-plan invariant #1. - policy.zig: BANNED_DIDS shared by indexer (insert gate) and builder (build-time SQL filter) — turso still holds historical patent rows until paced cleanup; a snapshot must never resurrect them. Bridgy-flagged rows excluded the same way. - sync.buildSnapshot: keyset pagination on the uri PK (not OFFSET — linear turso row reads), 150ms pacing between pages, reuses the existing local insert path so the schema has one source of truth. - r2.zig: rclone transport adapted from typeahead (config-file remote, INDEX_R2_* env, same ops vocabulary). Channels: staging by default, prod requires BUILDER_ALLOW_PROD=1. - LocalDb.openAt: open at an explicit path (builder scratch). - main.zig test block: explicit imports for tap/policy tests — the isBridgyPds regression test was not being collected without this. Verified locally against prod turso (SKIP_UPLOAD=1): 25,390 docs / 5,730 pubs / 89,025 tags in ~6min paced, prod canary green throughout (/popular 151ms mid-build), artifact has 0 banned/bridgy rows, FTS answers, quick_check ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 month ago

feat(api): GET /document — full extracted text by AT-URI search responses carry snippets only, which forces agents to re-fetch records from the author's PDS and re-flatten block structures themselves (and older leaflet shapes defeat naive extractors). the indexer already stores extracted full text; serve it. up to 25 comma-separated uris per request, read entirely from the local replica, same visibility policy as search (banned/labeled/bridgy/dead-url excluded → listed under missing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2 days ago

refactor: remove compat.zig — use native std.Io throughout Thread io: std.Io through all modules instead of global compat bridge. Replaces compat.Mutex/Condition with Io.Mutex/Io.Condition, compat.sleep* with io.sleep(), compat.timestamp/microTimestamp with Io.Timestamp, and inlines std.c.getenv at each call site. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

3 months ago