zzstoatzz.io
search for standard sites
pub-search.waow.tech
search
zig
blog
atproto
5.2 kB
91 lines
1# pub-search
2
3> pub-search indexes long-form writing across AT Protocol publishing platforms
4> (leaflet, pckt, offprint, greengale, whitewind, and self-hosted
5> standard.site sites) as one curated corpus. it exposes that corpus over an
6> HTTP API, an MCP server, and an AT Protocol labeler. bulk/scraper sources are
7> banned at ingest, so results are human writing. keyword (FTS5/BM25),
8> semantic (voyage-4-lite + turbopuffer), and hybrid (RRF) search modes.
9
10## Project contracts
11
12- [Site](https://pub-search.waow.tech) — the human surface (search + atlas)
13- [Adopting pub-search for agents](https://github.com/zzstoatzz/pub-search/blob/main/docs/agent-surfaces.md) — start here; which surface to pick and why
14- [HTTP API reference](https://github.com/zzstoatzz/pub-search/blob/main/docs/api.md)
15- [Search syntax](https://github.com/zzstoatzz/pub-search/blob/main/docs/search-syntax.md) — quotes, OR, prefix behavior
16- [Content extraction](https://github.com/zzstoatzz/pub-search/blob/main/docs/content-extraction.md) — why full text is per-platform work
17- [Source repository](https://github.com/zzstoatzz/pub-search)
18
19## Surfaces you can consume
20
21### MCP server — default for tool-calling agents
22
23- hosted: `claude mcp add-json pub-search '{"type": "http", "url": "https://pub-search-by-zzstoatzz.fastmcp.app/mcp"}'`
24- local stdio: `claude mcp add pub-search -- uvx --from 'git+https://github.com/zzstoatzz/pub-search#subdirectory=pub-search-mcp/server' pub-search`
25- a curated layer, not a 1:1 of the HTTP API. tools: `search`,
26 `get_document` (fetches + flattens the live record from the author's PDS),
27 `find_similar`, `discover_focal_post`,
28 `describe_cluster`, `recommended_by_top_authors`, `get_tags`, `get_stats`,
29 `get_popular`. prompts: `usage_guide`, `search_tips`. resource:
30 `pub-search://stats`.
31
32### HTTP API — for code-mode agents, scripts, pipelines
33
34- base URL: `https://leaflet-search-backend.fly.dev` (no auth, GET only)
35- `curl -s 'https://leaflet-search-backend.fly.dev/search?q=relay&mode=hybrid&format=v2&limit=5' | jq`
36- search returns snippets; `/document?uri=<at-uri>[,...]` returns the FULL
37 extracted article text (up to 25 uris per request) — use it to read what
38 search found instead of re-fetching records from each author's PDS.
39- endpoints: `/search`, `/document`, `/similar`, `/tags`, `/popular`,
40 `/stats`, `/activity`, `/api/dashboard`, `/subscribed`, `/subscribers`,
41 `/wrapped`, `/api/labeler`, `/health`. full parameters + response shapes
42 in api.md.
43
44### AT Protocol labeler — a subscribable trust signal
45
46- `labeler.pub-search.waow.tech` serves standard XRPC
47 `com.atproto.label.queryLabels` (HTTP) and `com.atproto.label.subscribeLabels`
48 (WebSocket). an autonomous classifier emits the `bulk-generated` label on
49 accounts that generate documents from a dataset rather than composing them;
50 those accounts are excluded from search. subscribe if you want the same
51 signal in your own pipeline. see [/labels](https://pub-search.waow.tech/labels).
52
53### the corpus itself lives on AT Protocol
54
55- every result is an `at://` record on its author's PDS. search returns enough
56 metadata (`uri`, `did`, `rkey`, `basePath`, `path`, `platform`) to build a
57 web URL or fetch the record directly (`com.atproto.repo.getRecord`, or
58 `uvx pdsx`). for reading, prefer the API's `/document` (indexed text,
59 batched) or the MCP `get_document` (live record) over rolling your own
60 extraction.
61
62## Agent guidance
63
64- read [agent-surfaces.md](https://github.com/zzstoatzz/pub-search/blob/main/docs/agent-surfaces.md)
65 first — it has a decision table (MCP vs HTTP) and the sharp edges.
66- MCP if the consumer is a tool-calling agent and you want it productive with
67 no code of yours. HTTP if your agent writes code anyway or needs pagination /
68 raw response shapes the tools don't expose.
69- hit the `fly.dev` backend directly for API calls. `pub-search.waow.tech` has
70 no API proxy — unknown paths return the SPA's HTML with a 200, so trusting
71 status codes there will "succeed" against a page of HTML. inspect bodies.
72- at least one of `q`, `tag`, or `author` is required; an empty query is
73 rejected by design.
74- filters vary by mode: `tag`/`since` apply to keyword only; `platform`/`author`
75 work everywhere; hybrid applies each filter to the half that supports it.
76- `author` accepts a handle or a DID (handles resolve server-side).
77 `search("", author="someone.com")` browses an author's everything.
78- freshness is bounded, not real-time — the keyword replica refreshes by
79 snapshot swap (minutes-to-hours of lag). `get_stats` / `/api/dashboard` show
80 current counts if staleness matters.
81
82## Source docs
83
84- `README.md` documents the overall architecture (ingester → backend → builder
85 → labeler → site → mcp).
86- `docs/agent-surfaces.md` is the canonical guide for consuming pub-search.
87- `docs/api.md` is the full HTTP endpoint reference, including URL-building
88 patterns per platform.
89- `docs/search-syntax.md` documents query syntax; `docs/search-architecture.md`
90 documents how the three modes dispatch.
91- `pub-search-mcp/server/` is the installable MCP server.