zzstoatzz.io
search for standard sites
pub-search.waow.tech
search
zig
blog
atproto
feat(labeler): redraw the line — composed vs generated, judge → gemma-4-12B
The old line ('written by a person') was wrong in both directions: most
banned content WAS human-written at the source (patents by attorneys,
recall notices by regulators, transcripts of humans speaking), and
original writing by an AI belongs in the index. What actually unites
every ban is the generating process: one document per dataset record,
template + data determining the text — even when the dataset is the
account's own (prideraiser enumerates its own campaigns).
Everything now states that line consistently: the judge prompt (kept in
lockstep with scripts/judge-eval), the /labels page, the exclusions.md
policy, and the label-declaration description.
Judge → gemma-4-12B-it-8bit: 19/19 conclusive votes correct across two
validation runs on the new prompt (all misses were 503s/timeouts, never
wrong verdicts), and it's the model our own provider machine serves.
judge-eval gains coryd.dev (942-doc human) as a 4th ground-truth case,
JUDGE_TIMEOUT/JUDGE_CONCURRENCY knobs, creds validation, and reasoning-
model parsing. SCORING_VERSION 8→9 re-judges all decided authors under
the new line on deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Fable 5
date
(Jul 1, 2026, 11:02 PM -0500)
commit
24bbbf51
24bbbf511cba80eb3f392ebd692491abc829eb42
parent
e60dd346
e60dd3468ecc62a7e19d005b2b013e813ba246ef
+71
-46
+31
-22
backend/src/ingest/classifier.zig
+31
-22
backend/src/ingest/classifier.zig
···
38
38
// re-scores the whole corpus from a clean slate. v2 = precision fix (content-word
39
39
// scaffold only, curation veto). v3 = threshold 0.55→0.50. v4 = model-pass gate
40
40
// (heuristic flags → LLM confirms content is machine-generated before emit).
41
-
const SCORING_VERSION: i64 = 8; // v8: judge model → Qwen3.6-35B (3/3 on judge-eval; 7B was 0/3)
41
+
const SCORING_VERSION: i64 = 9; // v9: composed-vs-generated prompt + judge → gemma-4-12B (self-served)
42
42
43
43
// review pipeline states. The heuristic is a cheap PRE-FILTER: it never emits
44
44
// directly (titles can't tell a branded real blog from a registry mirror). It
···
57
57
// review provider: any OpenAI-compatible chat-completions endpoint. Defaults
58
58
// to co/core; override all three via env (REVIEW_API_URL / REVIEW_MODEL /
59
59
// REVIEW_API_KEY) to switch providers with a secrets change, no deploy of code.
60
-
// picked by scripts/judge-eval (2026-07-01): 3/3 known-truth accounts, every
61
-
// completed vote correct. Qwen2.5-7B (the previous judge) scored 0/3 —
62
-
// confidently inverted on all three. Slower + reasoning-style, which the
63
-
// worker tolerates (votes retry on inconclusive; latency doesn't back
64
-
// anything up — candidates just stay PENDING longer).
65
-
const DEFAULT_REVIEW_MODEL = "mlx-community/Qwen3.6-35B-A3B-4bit";
60
+
// picked by scripts/judge-eval: gemma-4-12B went 19/19 conclusive votes
61
+
// correct across two runs on the composed-vs-generated prompt (2026-07-02),
62
+
// and it's the model our own provider machine serves — reliability and
63
+
// economics we control. Qwen2.5-7B (the original judge) was 0/3, confidently
64
+
// inverted. Reasoning-style + slow is fine: the worker is a durable queue,
65
+
// candidates just stay PENDING longer.
66
+
const DEFAULT_REVIEW_MODEL = "mlx-community/gemma-4-12B-it-8bit";
66
67
const DEFAULT_REVIEW_URL = "https://console.cocore.dev/api/v1/chat/completions";
67
68
68
69
const ReviewCfg = struct {
···
631
632
const samples = try fetchVoteMaterial(allocator, did, vote);
632
633
defer allocator.free(samples);
633
634
634
-
// prompt validated against the known cases on cocore Qwen2.5-7B: the framing
635
-
// (our task: long-form human writing vs automated feeds, "coherent ≠ human")
636
-
// + concrete examples are what make a small model separate a real blog from a
637
-
// transit/catalog feed. Without them it calls everything coherent "human".
635
+
// the line is COMPOSED vs GENERATED, not human vs machine: most banned
636
+
// content was human-written at the source (patents, recall notices,
637
+
// transcripts) and original AI writing is welcome. Keep this text in
638
+
// lockstep with scripts/judge-eval PROMPT_HEAD, and re-validate any change
639
+
// there against the known-truth set before shipping (19/19 conclusive
640
+
// votes correct on gemma-4-12B + Qwen3.6-35B, 2026-07-02).
638
641
var prompt: std.Io.Writer.Allocating = .init(allocator);
639
642
defer prompt.deinit();
640
643
try prompt.writer.writeAll(
641
-
\\A search engine indexes original long-form writing by people (blogs, essays,
642
-
\\articles). It must EXCLUDE accounts that are automated feeds, catalogs, or
643
-
\\database exports — even when the text reads coherently.
644
+
\\A search engine indexes writing: documents COMPOSED by an author — a person or
645
+
\\an AI — who chose what to say. It must EXCLUDE accounts that GENERATE
646
+
\\documents from a data source: one document per database row, feed event,
647
+
\\catalog entry, or result, where a template plus the data determines the text.
644
648
\\
645
-
\\The test: did a PERSON sit and write each item as original prose, OR is an
646
-
\\automated system emitting one record per database row / feed event / catalog
647
-
\\entry / log?
649
+
\\The test is NOT whether the text is fluent, and NOT whether a human once wrote
650
+
\\the underlying material — patents, recall notices, episode summaries, and
651
+
\\transcripts were all written by people, but republishing them one-per-record
652
+
\\is still generation. The test: does each document exist because its author had
653
+
\\something to say, OR because a record exists in some dataset?
648
654
\\
649
-
\\machine=true examples: a transit-alert bot ("Red Line delayed near Roosevelt"),
650
-
\\a patent-database mirror, a TV-episode catalog (one entry per episode),
651
-
\\product-recall summaries, daily log entries, chart/stats dumps. Coherent != human.
652
-
\\machine=false examples: a personal blog (tech notes, essays, reviews), even with
653
-
\\branded/templated titles or a numbered series.
655
+
\\machine=true examples: a patent-database mirror, vehicle-recall summaries (one
656
+
\\per recall), a TV-episode catalog (one per episode), transit alerts (one per
657
+
\\service event), chart/stats dumps (one per chart-day), tournament results (one
658
+
\\per event), fundraiser announcements stamped from campaign records — even when
659
+
\\the underlying data belongs to the account itself.
660
+
\\machine=false examples: a personal blog or essay series (even branded,
661
+
\\numbered, templated-looking titles, or extremely prolific), a daily journal
662
+
\\(the date is a schedule, not a data source), original writing by an AI agent.
654
663
\\
655
664
\\Evidence from ONE account (facts, a title sample spanning its whole
656
665
\\history, and excerpts from the middle of several documents):
+12
-6
docs/exclusions.md
+12
-6
docs/exclusions.md
···
1
1
# exclusions: what we keep out of the corpus, and why
2
2
3
-
pub-search indexes **long-form writing by people**. some actors publish
4
-
valid, well-signed AT Protocol records that are nonetheless not that —
3
+
pub-search indexes **composed writing** — documents an author (a person or
4
+
an AI) wrote because they had something to say. some actors publish valid,
5
+
well-signed AT Protocol records that are nonetheless not that —
5
6
machine-generated registry mirrors, scraper bridges, bulk archives. this
6
7
file is the registry of every manual exclusion: who, when, why, and the
7
8
evidence. if we're going to editorialize, we explain ourselves.
8
9
9
10
## what qualifies (the policy line)
10
11
11
-
an author is excluded when **all** of these hold:
12
+
the line is **composed vs. generated**, not human vs. machine: most banned
13
+
content was human-written at the source (patents, recall notices,
14
+
transcripts), and original writing by an AI is welcome. an author is
15
+
excluded when **all** of these hold:
12
16
13
-
1. **no human authorship** — the records are generated from an external
14
-
database/feed, not written
17
+
1. **generated, not composed** — each document exists because a record
18
+
exists in some dataset (external or the account's own), with a template
19
+
determining the text; not because an author chose to write it
15
20
2. **bulk scale** — the volume materially distorts the corpus or a topic's
16
21
search results (rule of thumb: would be >1% of the corpus, or owns a
17
22
topic's semantic results)
···
19
24
removes noise, not voices
20
25
21
26
an author is NOT excluded for: low quality, controversial content, high
22
-
(human) volume, or self-promotion. the line is authorship, not taste.
27
+
volume, self-promotion, or being an AI. the line is how the documents come
28
+
to exist, not taste and not the author's species.
23
29
24
30
## how exclusions are enforced (the new architecture)
25
31
+18
-11
scripts/judge-eval
+18
-11
scripts/judge-eval
···
31
31
"did:plc:u5kk4h7tr4s3ntskbmhl5z7d": ("sksksketch.net", False), # human illustrator
32
32
"did:plc:5swhfspkrynnbidlkrkch3lh": ("prideraiser.org", True), # fundraiser feed bot
33
33
"did:plc:4z33k5fjzw2ew3u373pg7ku5": ("festivus (episode catalog)", True),
34
+
"did:plc:sttgf52vkk46f6yuknvqxvgh": ("coryd.dev (942-doc human)", False), # volume trap
34
35
}
35
36
36
37
DEFAULT_MODELS = [
···
47
48
48
49
COCORE_URL = "https://console.cocore.dev/api/v1/chat/completions"
49
50
50
-
PROMPT_HEAD = """A search engine indexes original long-form writing by people (blogs, essays,
51
-
articles). It must EXCLUDE accounts that are automated feeds, catalogs, or
52
-
database exports — even when the text reads coherently.
51
+
PROMPT_HEAD = """A search engine indexes writing: documents COMPOSED by an author — a person or
52
+
an AI — who chose what to say. It must EXCLUDE accounts that GENERATE
53
+
documents from a data source: one document per database row, feed event,
54
+
catalog entry, or result, where a template plus the data determines the text.
53
55
54
-
The test: did a PERSON sit and write each item as original prose, OR is an
55
-
automated system emitting one record per database row / feed event / catalog
56
-
entry / log?
56
+
The test is NOT whether the text is fluent, and NOT whether a human once wrote
57
+
the underlying material — patents, recall notices, episode summaries, and
58
+
transcripts were all written by people, but republishing them one-per-record
59
+
is still generation. The test: does each document exist because its author had
60
+
something to say, OR because a record exists in some dataset?
57
61
58
-
machine=true examples: a transit-alert bot ("Red Line delayed near Roosevelt"),
59
-
a patent-database mirror, a TV-episode catalog (one entry per episode),
60
-
product-recall summaries, daily log entries, chart/stats dumps. Coherent != human.
61
-
machine=false examples: a personal blog (tech notes, essays, reviews), even with
62
-
branded/templated titles or a numbered series.
62
+
machine=true examples: a patent-database mirror, vehicle-recall summaries (one
63
+
per recall), a TV-episode catalog (one per episode), transit alerts (one per
64
+
service event), chart/stats dumps (one per chart-day), tournament results (one
65
+
per event), fundraiser announcements stamped from campaign records — even when
66
+
the underlying data belongs to the account itself.
67
+
machine=false examples: a personal blog or essay series (even branded,
68
+
numbered, templated-looking titles, or extremely prolific), a daily journal
69
+
(the date is a schedule, not a data source), original writing by an AI agent.
63
70
64
71
Evidence from ONE account (facts, a title sample spanning its whole
65
72
history, and excerpts from the middle of several documents):
+6
-4
scripts/labeler-setup
+6
-4
scripts/labeler-setup
···
65
65
"lang": "en",
66
66
"name": "machine-generated",
67
67
"description": (
68
-
"this account's content is produced by an automated "
69
-
"system, not written by a person — e.g. registry/feed "
70
-
"mirrors, transit-alert bots, catalogs, database exports. "
71
-
"pub-search excludes these from its human-writing search."
68
+
"this account's documents are generated from a data "
69
+
"source — one per database row, feed event, or catalog "
70
+
"entry — rather than composed by an author (a person or "
71
+
"an AI). e.g. registry mirrors, transit-alert bots, "
72
+
"catalogs, database exports. pub-search excludes these "
73
+
"from search."
72
74
),
73
75
}
74
76
],
+4
-3
site/labels.html
+4
-3
site/labels.html
···
62
62
<div class="container">
63
63
<h1><a href="/" class="title">pub search</a> <span class="dim">/ labels</span></h1>
64
64
<p class="lead">
65
-
pub-search indexes original writing. Accounts that emit records rather than prose —
66
-
feeds, catalogs, database exports — get a signed
65
+
pub-search indexes writing that an author — a person or an AI — composed.
66
+
Accounts that generate documents from a dataset (one per patent, recall, chart,
67
+
episode, alert) get a signed
67
68
<a href="https://atproto.com/specs/label" target="_blank" rel="noopener"><code>machine-generated</code></a>
68
69
label and are excluded from search. The decision is automated: a pattern filter nominates,
69
70
a model on <a href="https://cocore.dev" target="_blank" rel="noopener">co/core</a> decides.
···
78
79
<div class="stage-name">pattern filter</div>
79
80
<div class="stage-value">score ≥ 0.50</div>
80
81
</div>
81
-
<div class="stage" title="a language model reads titles and excerpts sampled across the account's whole history and votes: authored writing, or mechanical output? two of three votes decide, and each vote sees a different sample. inconclusive votes retry — no verdict is forced.">
82
+
<div class="stage" title="a language model reads titles and excerpts sampled across the account's whole history and votes: composed by an author, or generated one-per-record from a dataset? two of three votes decide, and each vote sees a different sample. inconclusive votes retry — no verdict is forced.">
82
83
<div class="stage-name">model review</div>
83
84
<div class="stage-value" id="st-review">2-of-3 votes</div>
84
85
</div>