Pensieve#

Pensieve is public memory search for ATProto repos. Give it a handle and a half-remembered phrase; it walks that person's public repo, extracts semantic artifacts from arbitrary record shapes, embeds them with Voyage, stores them in Turbopuffer, and returns a small human-readable digest with links back to the native thing when Pensieve can infer one.

Live Worker:

https://pensieve.n8-3e9.workers.dev

What Exists#

A Cloudflare Worker in src/worker.js.
A single-page frontend rendered by the Worker.
A Zig prototype in src/*.zig for CAR walking and NDJSON extraction.
Small Python smoke/demo scripts in scripts/.

The production-ish MVP is the Worker. The Zig code is useful reference and a future extraction path, but the deployed app currently uses @atcute/repo in the Worker to walk repo CARs.

How It Works#

Search is two-stage:

/api/search resolves the actor, embeds the query, and searches that actor's Turbopuffer namespace.
If the namespace is missing, the frontend calls /api/update, which streams progress while Pensieve fetches com.atproto.sync.getRepo, walks the CAR, infers artifacts, embeds up to INDEX_LIMIT records, writes them to Turbopuffer, and then runs the query.

Second and later searches for the same actor are fast because they reuse the existing Turbopuffer namespace. Re-indexing currently happens only through /api/update, which deletes and rebuilds that actor namespace.

See docs/OPERATIONS.md for deployment and secrets.

Implementation details

Pensieve has one deployed runtime: a Cloudflare Worker in src/worker.js.

browser
  |
  | /api/actors       -> typeahead.waow.tech
  | /api/search       -> resolve actor -> Voyage query embedding -> Turbopuffer
  | /api/update       -> resolve actor -> PDS getRepo -> CAR walk -> Voyage docs -> Turbopuffer
  v
Cloudflare Worker

Request Flow#

/ returns the single-page app. The frontend is inline HTML/CSS/JS generated by renderIndex().

/api/actors proxies handle search to the sibling typeahead service:

https://typeahead.waow.tech/xrpc/app.bsky.actor.searchActorsTypeahead

/api/search normalizes the actor handle, resolves it to DID + PDS with @atcute/identity-resolver, embeds the query with Voyage, queries the actor's Turbopuffer namespace with vector ANN and BM25, fuses the ranked rows, dedupes equivalent artifacts, and returns display-ready results. If both Turbopuffer queries return 404, the response sets needsIndex: true and the frontend calls /api/update.

/api/update streams newline-delimited JSON progress events. It resolves the actor, deletes the actor's Turbopuffer namespace, fetches the actor repo through com.atproto.sync.getRepo from the actor's PDS, walks the CAR with @atcute/repo, infers artifact envelopes, embeds up to INDEX_LIMIT text-bearing artifacts with Voyage, upserts rows into Turbopuffer, and emits a final query result.

Artifact Envelope#

Pensieve indexes arbitrary records by trying to infer a generic artifact:

{
  kind,
  title,
  body,
  url,
  media,
  refs,
  date,
  confidence
}

This is deliberately not a curated collection allowlist. The extraction pass walks the record, drops obvious protocol noise, scores likely title/body/url/date fields, and stores both the searchable text and display attributes in Turbopuffer.

The UI renders a small digest from the envelope: platform/app badge, artifact kind, date, duplicate count, title/body, primary native URL when known, and a details link to PDSLS for raw inspection.

Link Hierarchy#

Primary clicks should take the user to the most useful human destination:

artifact.url when the record contains one.
Known native URL builders, currently Bluesky posts and some Tangled records.
PDSLS as fallback.

PDSLS is still useful for inspecting raw records, but it should not be the main destination when Pensieve already knows a better app URL.

Cache Shape#

There is no separate cache layer. Turbopuffer namespaces are the cache:

first search for an actor can be slow because it builds the namespace
later searches for that actor are fast because /api/search reuses it
/api/update is a rebuild path and deletes the namespace first

Namespace naming:

${TURBOPUFFER_NAMESPACE_PREFIX}-${did with punctuation replaced}

Zig Prototype#

The Zig code is an earlier extractor and still useful reference:

src/repo_walk.zig streams com.atproto.sync.getRepo to a temp file, mmaps the CAR, indexes blocks, and walks the MST.
src/record_text.zig flattens records into embedding-ready text.
src/main.zig exposes extract and demo.

The deployed Worker does not call the Zig binary today.

Development#

npm install
npm run dev

Useful checks:

node --check src/worker.js
npm run dry-run

Deploy:

npm run deploy

Zig prototype:

zig build test
zig build run -- extract --actor zzstoatzz.io --out corpus.ndjson
zig build run -- demo --actor zzstoatzz.io --media --dir /tmp/pensieve-demo

Secrets#

Do not commit secrets. The deployed Worker expects these Cloudflare Worker secrets:

VOYAGE_API_KEY
TURBOPUFFER_API_KEY

The non-secret namespace prefix lives in wrangler.jsonc as TURBOPUFFER_NAMESPACE_PREFIX.

Local development can use wrangler secret put, .dev.vars, or op run with 1Password. The local .env / .dev.vars files are intentionally ignored.

Project Map#

src/worker.js        Cloudflare Worker, API, indexing, frontend HTML/CSS/JS
src/*.zig            Zig CAR extraction prototype
scripts/tpuf_demo.py Python smoke script for NDJSON -> Voyage -> Turbopuffer
wrangler.jsonc       Cloudflare Worker config
docs/                Operations notes

Contributing Notes#

Keep product copy non-jargony. Avoid surfacing "CAR", "repo", "collection", or implementation terms in the UI unless they are hidden behind an inspection affordance.
Prefer deterministic artifact extraction before adding model calls.
Keep PDSLS as a details/inspection fallback, not the primary destination when a native artifact URL is known.
Verify UI changes with a browser at desktop and mobile widths.

Configure Feed