pub search#

by @zzstoatzz.io

search ATProto publishing platforms (leaflet, pckt, offprint, greengale, and others using standard.site).

live: pub-search.waow.tech

formerly "leaflet-search" - generalized to support multiple publishing platforms

how it works#

1. ingester (ingester/, our own firehose consumer) subscribes to an ATProto relay, filters to publishing collections (pub.leaflet.*, site.standard.*, com.whtwnd.*), cryptographically verifies each commit (signature + MST diff), and re-emits verified records over a websocket channel
2. backend consumes the channel, extracts content per platform, and writes to Turso (source of truth); serves search with keyword (SQLite FTS5 on a local replica), semantic (turbopuffer ANN), and hybrid modes
3. builder (the same backend binary in BUILDER_MODE) rebuilds the keyword replica offline from Turso every hour and publishes it to R2 with a sha256 manifest — serving adopts verified snapshots instead of syncing in place, an architecture borrowed from typeahead (see docs/snapshot-pipeline.md)
4. site static frontend on Cloudflare Pages
5. mcp server for AI agents (Claude Code, etc.) — see docs/agent-surfaces.md

MCP server#

search is also exposed as an MCP server for AI agents like Claude Code:

claude mcp add-json pub-search '{"type": "http", "url": "https://pub-search-by-zzstoatzz.fastmcp.app/mcp"}'

see pub-search-mcp/server/README.md for local setup and usage details.

api#

GET /search?q=<query>&mode=keyword|semantic|hybrid&platform=<platform>&tag=<tag>&since=<date>&author=<did|handle>&format=v2
GET /similar?uri=<at-uri>&format=v2
GET /tags
GET /popular
GET /stats
GET /health

search returns three entity types: article (document in a publication), looseleaf (standalone document), publication (newsletter itself). each result includes a platform field (leaflet, pckt, offprint, greengale, whitewind, or other). use format=v2 for a wrapped response with total, hasMore, and results fields.

modes: keyword (default) uses FTS5 with BM25 + recency scoring. semantic uses voyage embeddings + turbopuffer ANN. hybrid merges both via reciprocal rank fusion.

ranking: keyword results use hybrid BM25 + recency scoring. text relevance is primary, but recent documents get a boost (~1 point per 30 days). the since parameter filters to documents created after the given ISO date (e.g., since=2025-01-01).

/similar uses Voyage AI embeddings with turbopuffer ANN search.

configuration#

the backend is fully configurable via environment variables:

the backend indexes multiple ATProto platforms — currently pub.leaflet.*, site.standard.*, and com.whtwnd.blog.entry collections (the last for whitewind). platform is stored per-document and returned in search results.

atlas#

a 2D semantic map of the entire document index: pub-search.waow.tech/atlas

documents are projected from 1024-dim voyage embeddings to 2D via PCA → UMAP, then clustered with HDBSCAN at two granularities. each point is colored by platform. zoom in to see finer cluster labels and individual document titles.

built with scripts/build-atlas (batch job, ~20s) → site/atlas.json → canvas renderer. see docs/atlas.md for details.

stack#

• Fly.io hosts the Zig search API, content indexing, and our own verified firehose ingester (zat for AT Protocol crypto)
• Turso cloud SQLite (source of truth) + local read replica (FTS queries), refreshed via sha256-verified snapshots in R2
• turbopuffer ANN vector search
• Voyage AI embeddings (voyage-4-lite, 1024 dims)
• Cloudflare Pages static frontend

embeddings#

documents are embedded using Voyage AI's voyage-4-lite model (1024 dimensions). the backend automatically generates embeddings for new documents via a background worker — no manual backfill needed. similarity search uses turbopuffer's ANN index for fast nearest-neighbor queries.