zzstoatzz.io
search for standard sites
pub-search.waow.tech
search
zig
blog
atproto
docs: bring internal docs in line with the as-built labeler + tap purge
- README: TAP_* env vars → INGESTER_*, labeler added to 'how it works'
- exclusions.md: the 'planned end-state' labeler note replaced with the
as-built reality (autonomous, negation-corrected, no review queue)
- spam-detection-plan.md: status plan → BUILT, with the deltas from plan
(label renamed, composed-vs-generated line, majority-of-3 gate)
- hydrant.md: marked HISTORICAL (tap is gone; our ingester covers it)
- api.md: /api/labeler + /admin/label documented
- docs/README: labeler in further-reading; stats footer links /labels
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Fable 5
date
(Jul 1, 2026, 11:29 PM -0500)
commit
a0761d23
a0761d23c3133161646e9650a63c027202c8b0b1
parent
c35fc776
c35fc776d532f92f3bf1445e486c5212a6cae2de
+48
-15
+5
-4
README.md
+5
-4
README.md
···
13
13
1. **ingester** (`ingester/`, our own firehose consumer) subscribes to an ATProto relay, filters to publishing collections (`pub.leaflet.*`, `site.standard.*`, `com.whtwnd.*`), cryptographically verifies each commit (signature + MST diff), and re-emits verified records over a websocket channel
14
14
2. **backend** consumes the channel, extracts content per platform, and writes to [Turso](https://turso.tech) (source of truth); serves search with keyword (SQLite FTS5 on a local replica), semantic ([turbopuffer](https://turbopuffer.com) ANN), and hybrid modes
15
15
3. **builder** (the same backend binary in `BUILDER_MODE`) rebuilds the keyword replica offline from Turso every hour and publishes it to R2 with a sha256 manifest — serving adopts verified snapshots instead of syncing in place, an architecture borrowed from [typeahead](https://tangled.sh/@zzstoatzz.io/typeahead) (see [docs/snapshot-pipeline.md](docs/snapshot-pipeline.md))
16
-
4. **site** static frontend on Cloudflare Pages
17
-
5. **mcp** server for AI agents (Claude Code, etc.) — see [docs/agent-surfaces.md](docs/agent-surfaces.md)
16
+
4. **labeler** — pub-search is also an AT Protocol [labeler](https://atproto.com/specs/label): an autonomous classifier watches the firehose and labels accounts that generate documents from a dataset (one per record) rather than composing them; labeled accounts are excluded from search. see [pub-search.waow.tech/labels](https://pub-search.waow.tech/labels)
17
+
5. **site** static frontend on Cloudflare Pages
18
+
6. **mcp** server for AI agents (Claude Code, etc.) — see [docs/agent-surfaces.md](docs/agent-surfaces.md)
18
19
19
20
## MCP server
20
21
···
53
54
|----------|---------|-------------|
54
55
| `APP_NAME` | `leaflet-search` | name shown in startup logs |
55
56
| `DASHBOARD_URL` | `https://pub-search.waow.tech/dashboard.html` | redirect target for `/dashboard` |
56
-
| `TAP_HOST` | `leaflet-search-ingester.internal` | ingester websocket host (tap-compatible `/channel` protocol) |
57
-
| `TAP_PORT` | `2480` | ingester websocket port |
57
+
| `INGESTER_HOST` | `leaflet-search-ingester.internal` | ingester websocket host (`/channel` protocol) |
58
+
| `INGESTER_PORT` | `2480` | ingester websocket port |
58
59
| `PORT` | `3000` | HTTP server port |
59
60
| `TURSO_URL` | - | Turso database URL (required) |
60
61
| `TURSO_TOKEN` | - | Turso auth token (required) |
+2
-1
docs/README.md
+2
-1
docs/README.md
···
77
77
- [search-architecture.md](search-architecture.md) — FTS5 details, scaling considerations, future options
78
78
- [content-extraction.md](content-extraction.md) — how content is extracted from each platform
79
79
- [api.md](api.md) — API endpoint reference
80
-
- [exclusions.md](exclusions.md) — the registry of manually excluded authors: the policy line, enforcement layers, and the evidence for each ban
80
+
- [exclusions.md](exclusions.md) — the registry of manually excluded authors: the policy line (composed vs generated), enforcement layers, and the evidence for each ban
81
+
- [spam-detection-plan.md](spam-detection-plan.md) — the labeler: pub-search autonomously labels accounts that generate documents from datasets (`machine-generated`, see [/labels](https://pub-search.waow.tech/labels)); doc is the original plan with an as-built status header
81
82
- [agent-surfaces.md](agent-surfaces.md) — adopting pub-search for agents: MCP vs HTTP API, when to use which
82
83
- [snapshot-pipeline.md](snapshot-pipeline.md) — how the keyword index ships (builder → manifest → R2 → verified adoption), what scales, and how to do production data surgery
83
84
- [scaling-plan.md](scaling-plan.md) — the plan of record: snapshot builder → R2 → verified swap → live overlay (largely executed; see status header)
+18
docs/api.md
+18
docs/api.md
···
253
253
}
254
254
```
255
255
256
+
### labeler
257
+
258
+
```
259
+
GET /api/labeler
260
+
```
261
+
262
+
read-only summary of the machine-generated labeler's state: counts by review
263
+
state plus every decided author (site, state, model reason, heuristic score,
264
+
title patterns). backs the [/labels](https://pub-search.waow.tech/labels)
265
+
page. the labels themselves are served over standard XRPC at
266
+
`labeler.pub-search.waow.tech` (`com.atproto.label.queryLabels` /
267
+
`subscribeLabels`).
268
+
269
+
`GET /admin/label?token=…&did=…&val=machine-generated&neg=0|1` (BACKFILL_TOKEN-
270
+
gated) emits or negates a label manually — the negation path is the appeal /
271
+
correction lever; it also updates the classifier's state so the account is
272
+
never re-flagged.
273
+
256
274
### health
257
275
258
276
```
+9
-5
docs/exclusions.md
+9
-5
docs/exclusions.md
···
68
68
> frequent, is a `banned_dids` turso table read at runtime so a ban becomes a
69
69
> data change instead of a deploy.
70
70
>
71
-
> the planned end-state for that "next step" is a pub-search **labeler**: a ban
72
-
> is a signed account label (`bulk-mirror`) that backs the `banned_dids` store
73
-
> and that other AT Protocol readers can subscribe to. detection feeds a
74
-
> human-reviewed queue, not an auto-banner. see
75
-
> [spam-detection-plan.md](spam-detection-plan.md).
71
+
> that labeler now EXISTS (2026-06-30): pub-search emits signed
72
+
> `machine-generated` account labels autonomously — a firehose classifier
73
+
> nominates, a model gate confirms (majority-of-3 votes), and a mislabel is
74
+
> corrected by negation (`/admin/label?neg=1`), not a review queue. labels are
75
+
> public at labeler.pub-search.waow.tech and explained at
76
+
> [pub-search.waow.tech/labels](https://pub-search.waow.tech/labels). the
77
+
> `banned_dids`-backed hard-drop is still pending (gated on the notify/appeal
78
+
> loop). see [spam-detection-plan.md](spam-detection-plan.md) for the original
79
+
> plan.
76
80
77
81
## the registry
78
82
+5
-2
docs/hydrant.md
+5
-2
docs/hydrant.md
···
1
-
# hydrant: potential tap replacement
1
+
# hydrant: potential tap replacement (HISTORICAL)
2
2
3
-
evaluated 2026-03-22. **decision: not adopting now**, but worth revisiting.
3
+
evaluated 2026-03-22. **decision: not adopting.** OBE — we replaced tap with
4
+
our own verified firehose ingester (`ingester/`) on 2026-06-09, which covers
5
+
the needs weighed here (verification, filtering, no ACK-blocking class of
6
+
bugs). kept for the evaluation record.
4
7
5
8
## what is hydrant
6
9
+8
-2
docs/spam-detection-plan.md
+8
-2
docs/spam-detection-plan.md
···
1
1
# spam / bulk-mirror detection + a pub-search labeler
2
2
3
-
status: **plan** (2026-06-29). supersedes the "next step" note in
4
-
[exclusions.md](exclusions.md) about moving bans off a compiled-in list.
3
+
status: **BUILT** (labeler live 2026-06-30; this doc is the original plan and
4
+
drifts from the as-built system). as built: the label is `machine-generated`
5
+
(renamed from `bulk-mirror`), the classifier is autonomous
6
+
(`backend/src/ingest/classifier.zig`: heuristic pre-filter → majority-of-3
7
+
model votes on co/core → emit), and the policy line evolved to **composed vs
8
+
generated** (see [exclusions.md](exclusions.md)). the public face is
9
+
[pub-search.waow.tech/labels](https://pub-search.waow.tech/labels). hard-drop
10
+
(acting on labels) is still pending the notify/appeal loop.
5
11
6
12
the design, in AT Protocol terms ([labels spec](https://atproto.com/specs/label)):
7
13
labeling **is** the primitive — there is no review queue. a labeler *emits*
+1
-1
site/stats.html
+1
-1
site/stats.html
···
125
125
</section>
126
126
127
127
<footer>
128
-
<a href="/">back</a> · <a href="/atlas.html">atlas</a> · source on <a href="https://tangled.sh/@zzstoatzz.io/leaflet-search">tangled</a> · <button type="button" class="theme-toggle" id="theme-toggle" aria-label="cycle theme"></button>
128
+
<a href="/">back</a> · <a href="/atlas.html">atlas</a> · <a href="/labels">labels</a> · source on <a href="https://tangled.sh/@zzstoatzz.io/leaflet-search">tangled</a> · <button type="button" class="theme-toggle" id="theme-toggle" aria-label="cycle theme"></button>
129
129
</footer>
130
130
</div>
131
131