zzstoatzz.io
search for standard sites
pub-search.waow.tech
search
zig
blog
atproto
docs: builder off-box plan — complete the lambda shape (typeahead pattern)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Fable 5
date
(Jul 24, 2026, 10:17 PM -0500)
commit
e55f3f89
e55f3f89953e8ee6bbd2c4bf37ae53cc3f066548
parent
71b4e1e7
71b4e1e75972ad95eb3212d2e8604b6090ae2642
+97
+97
docs/builder-offbox-plan.md
+97
docs/builder-offbox-plan.md
···
1
+
# builder off-box: completing the lambda shape
2
+
3
+
status: **in progress** (2026-07-25). companion to [scaling-plan.md](scaling-plan.md)
4
+
(this executes its steps 2-revision and 3) and to typeahead's
5
+
`docs/typeahead-index-design.md` / the "pendulum backswing" post, which this
6
+
deliberately copies. typeahead lifted its builder off fly onto heavypad via
7
+
prefect in june 2026 (`my-prefect-server/flows/typeahead_index.py`); pub-search
8
+
adopted typeahead's snapshot pipeline but kept the builder on a fly machine
9
+
with an hourly schedule — and that residue is the recurring failure.
10
+
11
+
## why the fly builder keeps paging us
12
+
13
+
freshness currently comes ONLY from snapshot cadence (no overlay), so the
14
+
builder is on an hourly leash with a wall-clock watchdog:
15
+
16
+
- build time grew 17→40min with the corpus (full 600MB content re-page from
17
+
turso every hour); turso slow patches push past the watchdog → killed build
18
+
→ stale-freshness alert (2026-07-23, -24, -25)
19
+
- the watchdog value itself was reverted by the deploy-time reconcile script
20
+
(2026-07-24) — machine-level tuning does not survive the pipeline
21
+
- hourly full-table scans of turso are standing load the serving path feels
22
+
23
+
the lambda framing (see notes repo `architecture/serving-from-snapshots.md`):
24
+
batch cadence is load-bearing *only because the speed layer is missing*.
25
+
typeahead's answer: builder = pure batch, off-SLA, on the home box ("a
26
+
late/failed run just leaves the prior snapshot serving"), freshness = overlay.
27
+
28
+
## target
29
+
30
+
```
31
+
heavypad (home-pool prefect worker) fly
32
+
───────────────────────────────── ─────────────────────────────────
33
+
pub-search-snapshot flow (daily): backend serves search unchanged:
34
+
clone pub-search → zig build → promote watcher polls R2,
35
+
BUILDER_MODE=1 CHANNEL=prod → verifies manifest, adopts.
36
+
gated build → rclone → R2 (adoption is ~1s since sidecar
37
+
attestation)
38
+
```
39
+
40
+
- same binary, same gates (doc-count vs turso, FTS sentinel, quick_check),
41
+
same R2 bucket/channels, same manifest contract. the serving side cannot
42
+
tell where a snapshot was built — that is what makes the cutover clean.
43
+
- no watchdog pressure: prefect flow timeout is generous (hours); a slow or
44
+
failed run just leaves the previous snapshot serving. no SLA, no restart
45
+
loop, no fly scan-storm.
46
+
- cadence: hourly-equivalent freshness is NOT preserved by this phase alone;
47
+
see sequencing. overlay (phase 2) is what buys the relaxed cadence.
48
+
49
+
## sequencing (no downtime, verify before delete)
50
+
51
+
1. **flow lands** in my-prefect-server (`flows/pub_search_snapshot.py`,
52
+
modeled 1:1 on `typeahead_index.py`): clone → `cd backend && zig build
53
+
-Doptimize=ReleaseSafe` → run `pub-search` with `BUILDER_MODE=1`.
54
+
env via secret blocks: shared `turso-url`/`turso-token` (already the leaf
55
+
db — same blocks atlas uses) + new `leaflet-index-r2-*` blocks
56
+
(`INDEX_R2_{ENDPOINT,BUCKET,ACCESS_KEY_ID,SECRET_ACCESS_KEY}`, bucket
57
+
`leaflet-search-index`). `RCLONE_BWLIMIT` set like typeahead's.
58
+
2. **staging soak**: run the deployment with `BUILDER_CHANNEL=staging` from
59
+
heavypad; verify `latest.staging.json` + manifest sha/counts against a
60
+
fly-built prod manifest of the same hour.
61
+
3. **prod flip**: set `BUILDER_CHANNEL=prod` + `BUILDER_ALLOW_PROD=1` on the
62
+
deployment; watch one publish → fly promote watcher adopts it (log:
63
+
`local db: adopted pending snapshot`). fly builder machine still scheduled
64
+
at this point — pointer-last upload makes concurrent builders safe (last
65
+
writer wins a whole snapshot, never a torn one).
66
+
4. **decommission fly builder** (same day): destroy machine
67
+
`snapshot-builder-hourly`; DELETE `scripts/reconcile-snapshot-builder`,
68
+
`scripts/guard-snapshot-builder`, their workflow step + path triggers in
69
+
`.github/workflows/deploy-backend.yml`, and the builder-machine runbooks
70
+
in CLAUDE.md. builder *code* (builder.zig, r2.zig, BUILDER_* env handling)
71
+
stays — it is what heavypad runs. the watchdog stays too (flow-level belt
72
+
+ binary-level suspenders), default bumped so ad-hoc runs aren't leashed.
73
+
5. **cadence**: start at every 2h (bandwidth ~14GB/day upload at 600MB —
74
+
fine on the uplink with RCLONE_BWLIMIT; revisit vs typeahead's every-3-day
75
+
+ overlay endgame). freshness alert threshold adjusted to cadence + build
76
+
time + margin.
77
+
6. **phase 2 — live overlay** (separate design, lands independently): ingest
78
+
path projects creates/updates/deletes into a small overlay next to the
79
+
snapshot (authoritative, not a cache; compacted at adoption by
80
+
`source_watermark`). then cadence relaxes to daily and the alert measures
81
+
something the overlay has already fixed.
82
+
83
+
## R2 credential note
84
+
85
+
the fly builder used `INDEX_R2_*` app secrets (values not readable back).
86
+
heavypad needs its own: mint a fresh R2 API token scoped to
87
+
`leaflet-search-index` (account has `CLOUDFLARE_API_TOKEN` locally) and store
88
+
as prefect secret blocks. fly serving machines keep their existing secrets —
89
+
read path is untouched.
90
+
91
+
## what "solved completely" means here
92
+
93
+
the failure class was: *builder duration/health coupled to serving freshness
94
+
through a tight schedule on infrastructure that fights long batch jobs*.
95
+
after this: builds are batch work on a 24-core box with no wall-clock
96
+
opponent, publishing is best-effort with gates, and a missed run costs
97
+
nothing until the overlay (phase 2) makes even cadence mostly cosmetic.