Commits
The greedy trailing char class swallowed a wrapping "(" and trailing
".", ")", "?" from prose, so "(https://x).” was stored verbatim. Drop
the leading "(" from the pattern and post-trim trailing prose
punctuation, keeping a ")" only when it balances a "(" inside the URL
(e.g. wiki/Foo_(bar)). Adds node:test coverage and a `npm test` script.
Also adds scripts/repair-card-urls.mjs: re-runs each stored
network.cosmik.card URL through the fixed extractor and rewrites
malformed or missing-top-level-url records in place (idempotent, same
rkey). Dry-run by default.
Structure-only spec: PRICE_RULE_MONTHLY, PRICE_CHANNEL_BACKFILL,
PRICE_GUILD_BACKFILL as tunable variables, mapped to the data model and Stripe
mechanics, with revenue/margin formulas and the Stripe-fixed-fee analysis.
Prices and the open §6 decisions still TBD.
Recovers a record dropped by a pre-retry-fix transient error without
re-walking the whole channel. Fetches the one message over REST so it never
opens a second gateway session that would fight the live bot, then runs it
through the same match/extract/dispatch path the worker uses. Deterministic
rkeys keep the re-dispatch idempotent. Exits non-zero if any write still fails.
The dispatch retry loop only retried 429s; every other failure — 5xx, socket
resets like UND_ERR_SOCKET, thrown dispatches — was recorded and skipped, the
cursor advanced past it, and the job still finished "done". So a done job could
silently carry dropped records, and cursor-based resume never re-attempts them;
only a full channel re-run recovered the loss.
Now retry every non-permanent error: 429s wait out the ratelimit-reset, other
transients get exponential backoff, up to MAX_DISPATCH_RETRIES. Only a non-429
4xx is treated as permanent. Deterministic rkeys keep re-dispatch idempotent.
Errors that survive all retries are persisted for the operational repair re-run,
which remains the backstop. Upholds the "never lose a record" invariant by
construction; permission failures stay an acceptable skip.
Per-channel progress measured the fraction of the since..until calendar
window walked. With the Everything window (since_ts ~10y ago) but channels
created in 2024-2026, the cursor blows through all real content while
covering only a few percent of the span, pinning every running bar near 0
until it snaps to 100 on completion. Clamp the floor to the channel's
creation time (decoded from the channel-id snowflake, no API call) so the
percentage reflects the span that can actually hold messages.
Backfilling a wildcard rule meant picking one channel at a time and
re-running per channel. Add a fan-out: POST /rules/:id/backfill/all
enqueues one job per readable channel, drained FIFO by the existing
worker (write-pacing is process-global, so the queue stays under the
PDS rate limit). Re-running skips channels that already have a
pending/running job.
The wildcard status page becomes channel-centric: an aggregate header
(X/N channels complete, running/queued/failed, totals), a per-channel
table, live batch polling, a 'Backfill all N channels' button, and
'Cancel all'. Single-channel backfill now guards per-channel instead
of per-rule, so one channel can be re-run while a batch runs on others.
Channel-scoped rules keep their existing single-channel view.
The Backfill button only ever rendered on an existing rule's row, so a
freshly-linked identity with no rules (e.g. a second atproto identity)
showed no sign backfill existed at all. Add a disabled backfill preview
under the new-rule form (noting it enables once the rule is saved, so
you can confirm the rule is right first), and a real enabled 'Backfill
this rule' link on the edit page. Both gate on the existing allowlist.
Each identity in the atproto switcher renders the handle on its own
row with action buttons (Sign out / Switch to / Re-auth / Delete) in a
flex group beneath. The previous flex-row layout forced the handle and
2–3 buttons to share ~340px of card width, which wrapped mid-row in
ugly ways.
Inserts a jobs(user_id, did, did_handle, did_avatar) table between users
and rules. rules.user_id and write_counters.user_id move to job_id so a
single Discord account can drive multiple atproto identities, each with
its own rule set and counter ledger.
Includes a one-shot in-place migration (table-rebuild dance with
foreign_key_check) that seeds jobs from users with a DID, rewires rules
and counters, drops rows belonging to orphan no-DID users, and removes
did* columns from users.
Web UI gets an identity switcher (active pill + per-job sign-out / switch
/ delete / re-auth, plus "link another"). Bot, backfill, active-guilds,
and CLI scripts are switched over to job_id. Smoke test under
scripts/test-jobs-migration.mjs exercises the migration end-to-end.
eurosky.social's PDS sends ratelimit-policy=5000;w=3600 (5000 writes per
hour) without a Retry-After header. The worker now reads ratelimit-reset
to size the backoff sleep, defaults pacing to 60 writes/minute (3600/hr,
well under the policy and leaves room for live writes), and flushes
backfill_jobs progress per-message instead of per-page so the UI / poller
doesn't go dark during a multi-minute sleep.
- New per-rule backfill: paginates channel history ~1s/page and
replays each message through the existing match → extract →
dispatch pipeline. UI shows messages-processed / records-written
with a live progress bar, plus a Cancel button. Gated to a
single allowlisted Discord account via BACKFILL_ALLOWED_DISCORD_ID
while the feature is being shaken out.
- Destinations now use deterministic rkeys (sha256(messageId|item),
base64url, 24 chars) and putRecord, so a backfill replay
overwrites in place instead of duplicating. dispatch() returns
the count of PDS writes for accurate progress display.
- Live dispatcher threads ctx.createdAt through, so PDS records
carry the original Discord message time whether they were
written live or via backfill.
- /rules/:id/edit: edit rules in place. Form HTML + validation
extracted into shared renderRuleForm / parseRuleForm so add and
edit can't drift. Preserves rule.id (keeping backfill history
attached), created_at, and enabled flag.
discord.js silently drops messages when its websocket is unhealthy
and previously we had no listeners on Error / ShardError /
ShardDisconnect / Invalidated, so a "bot stopped processing"
incident left no trace in Railway logs to diagnose from.
Adds:
- client.on Error / Warn / ShardError / ShardDisconnect / ShardReady
/ ShardResume / ShardReconnecting / Invalidated / GuildUnavailable
in bot.js. ShardDisconnect logs the websocket close code + reason.
Invalidated exits the process so Railway restarts us into a fresh
connection (discord.js can't recover from this on its own).
- process.on unhandledRejection / uncaughtException in index.js so
stray async failures from any handler land in Railway logs with
stack traces instead of being silently swallowed.
Every event uses a "disjecta bot[...]" or "disjecta[...]" prefix so
they're greppable in Railway's log UI.
Adding a new case in the webhook handler used to require also clicking
into the Stripe dashboard to subscribe the endpoint to that event,
which is easy to forget — adding customer.deleted in the previous
commit was the immediate trigger.
EVENTS_HANDLED in stripe.js is now the single source of truth.
syncWebhookEvents() runs at startup, finds the endpoint matching
${PUBLIC_URL}/webhooks/stripe, and extends its enabled_events with
anything missing. Non-destructive: existing events stay, '*' is
respected, no endpoint match logs a clear "create one" warning.
Webhook handler now de-dupes events via a `webhook_events(event_id)`
table — Stripe retries are no-ops. Adds a `customer.deleted` handler
that nulls our stripe_customer_id reference.
Subscription teardown is consistent: any transition from active=1 to
inactive (subscription.deleted, subscription.updated to past_due/
canceled, or customer.deleted) disables every rule the user owns.
Re-subscribing flips active back on but leaves rules disabled — the
user re-enables them deliberately in the UI.
/subscribe redirects already-active users to /portal (via a tiny
GET shim) to prevent duplicate subscriptions. Replaces the raw
requireStripe() throw with a friendly 503 errorPage.
Grace window: a new GRACE_UNTIL env (ISO date) carries existing
test-period users when BILLING_DISABLED flips to 0. findMatchingRules
and active-guilds both filter on `customer_id IS NOT NULL OR now <
GRACE_UNTIL`, so the deadline takes effect to the second. UI shows
"free until YYYY-MM-DD" while the window is open.
Subscribe button label is fetched from Stripe at startup
(stripe.prices.retrieve) so changing the price in Stripe doesn't
leave the UI claiming "$2/month" forever. Falls back to plain
"Subscribe" if Stripe is unreachable.
readme: documents the billing state machine + the local
`stripe listen --forward-to ...` workflow for testing webhooks
without a public URL.
Defaulting community-bookmark as checked led to users accidentally
fanning out to it when they only meant to write Semble. Now no
lexicon is pre-selected — explicit opt-in.
Wrap the per-rule Disable/Delete forms in a flex container so they
sit side by side instead of wrapping in the narrow actions column.
The bookmark adapter is now a registry of bookmark-flavored lexicons
(community standard + Semble) rather than a single hardcoded NSID. A
rule picks any subset; each gets its own record per URL. Adding a new
target is one entry in LEXICONS.
Semble support writes network.cosmik.card. If the user picks one of
their Semble collections in the form, an afterWrite hook also writes
a network.cosmik.collectionLink with strongRefs to the card and the
collection (cid re-resolved at write time).
New /api/semble/collections endpoint lists the linked user's
network.cosmik.collection records to populate the picker. Form lazy-
loads the dropdown when Semble Card is ticked.
Old rules with destination_config=null fall back to writing only the
community lexicon, so existing bookmark rules keep working.
- Render the linked atproto handle + avatar instead of raw DID
- Gate "ready" state on the bot actually being in at least one of the
user's guilds; new "Add bot to a server" CTA when it isn't
- Use the destinations CATALOG for a typed dropdown with per-adapter
description and configSchema
- Honor billingDisabled when auto-activating users
Adds atproto-identity.js which prefers the bsky public getProfile API
(handle + CDN-hosted avatar URL) and falls back to the DID document
for non-bsky DIDs. Persists handle/avatar on the users row so the web
UI doesn't have to re-fetch on every page load.
Implied by loopback dev mode; can also be set explicitly via env. When
on, Discord+atproto-linked users are auto-activated without going
through Stripe. Lets us run a free test period before billing is wired
up end-to-end.
Each adapter now exports a `meta` block (id, name, description, modes,
configSchema) so the web UI can render a typed dropdown. Adds the
community.lexicon.bookmarks.bookmark adapter as the default named
destination. Logs each successful createRecord for atproto-record and
bookmark — previously only errors were visible, so prod writes looked
silent in Railway logs.
Closes the Discord gateway, HTTP server, and SQLite handle before exit
so Railway redeploys stop firing "deployment crashed" notifications.
Project name in package.json/lock, README, env templates, log messages,
HTML page titles/headers, atproto OAuth client_name, Discord slash command
namespace (/disject → /disjecta), DB filename default, and Railway domain
defaults in scripts/_railway-push-env.mjs.
Production resources rebuilt fresh on Railway under the new project name
"disjecta" with custom domain disjecta.portable.agency.
User-facing:
- Per-rule extract mode: URLs only or full message content
- Per-rule content filter: substring or regex (case-insensitive)
- Author input accepts @username (resolved via Discord member search) in
addition to numeric IDs, removing the developer-mode requirement
- /rules page now shows examples and explains that the bot does nothing
until a rule exists for a server
- Atmosphere unlink retained; copy now reflects routing both URLs and
message content
Loopback dev mode:
- Detect http://localhost (or 127.0.0.1) and skip atproto OAuth client
construction with a clear warning, so the rest of the app can be
exercised without an HTTPS tunnel
- /rules and rule mutations no longer require a linked DID in loopback
- Auto-activate users in loopback so rule matching actually fires without
Stripe involvement
Internals:
- match.js: messageMatches + validateMatchPattern
- discord-oauth.js: searchGuildMembers via bot REST
- db.js: ensureColumn for non-destructive migrations on existing DBs
- New columns: rules.mode, rules.match_kind, rules.match_value
- Rule POST validation tightened; bot evaluates content match per rule
after metadata SQL filter, before extraction
Dev scripts (gitignored convention not yet applied):
- scripts/_set-oauth-key.mjs: generate fresh JWK and rewrite .env in place
- scripts/_validate-env.mjs: report env health without printing secrets
Schema is no longer init-only via a separate command; CREATE TABLE IF
NOT EXISTS runs at module load on every entry point.
Bot harvests URLs from messages in any Discord server it's invited to
and routes them to per-rule destinations on the user's PDS. Web app
handles Discord+atproto OAuth, Stripe subscriptions, and rule CRUD.
Pass-through by design: only account metadata and aggregate write
counts are stored; no message content or URL data persists.
Defenses against free-rider servers: in-memory pre-filter on the hot
path, plus auto-leave for guilds with no active rules after a grace
period (default 7 days).
The greedy trailing char class swallowed a wrapping "(" and trailing
".", ")", "?" from prose, so "(https://x).” was stored verbatim. Drop
the leading "(" from the pattern and post-trim trailing prose
punctuation, keeping a ")" only when it balances a "(" inside the URL
(e.g. wiki/Foo_(bar)). Adds node:test coverage and a `npm test` script.
Also adds scripts/repair-card-urls.mjs: re-runs each stored
network.cosmik.card URL through the fixed extractor and rewrites
malformed or missing-top-level-url records in place (idempotent, same
rkey). Dry-run by default.
Recovers a record dropped by a pre-retry-fix transient error without
re-walking the whole channel. Fetches the one message over REST so it never
opens a second gateway session that would fight the live bot, then runs it
through the same match/extract/dispatch path the worker uses. Deterministic
rkeys keep the re-dispatch idempotent. Exits non-zero if any write still fails.
The dispatch retry loop only retried 429s; every other failure — 5xx, socket
resets like UND_ERR_SOCKET, thrown dispatches — was recorded and skipped, the
cursor advanced past it, and the job still finished "done". So a done job could
silently carry dropped records, and cursor-based resume never re-attempts them;
only a full channel re-run recovered the loss.
Now retry every non-permanent error: 429s wait out the ratelimit-reset, other
transients get exponential backoff, up to MAX_DISPATCH_RETRIES. Only a non-429
4xx is treated as permanent. Deterministic rkeys keep re-dispatch idempotent.
Errors that survive all retries are persisted for the operational repair re-run,
which remains the backstop. Upholds the "never lose a record" invariant by
construction; permission failures stay an acceptable skip.
Per-channel progress measured the fraction of the since..until calendar
window walked. With the Everything window (since_ts ~10y ago) but channels
created in 2024-2026, the cursor blows through all real content while
covering only a few percent of the span, pinning every running bar near 0
until it snaps to 100 on completion. Clamp the floor to the channel's
creation time (decoded from the channel-id snowflake, no API call) so the
percentage reflects the span that can actually hold messages.
Backfilling a wildcard rule meant picking one channel at a time and
re-running per channel. Add a fan-out: POST /rules/:id/backfill/all
enqueues one job per readable channel, drained FIFO by the existing
worker (write-pacing is process-global, so the queue stays under the
PDS rate limit). Re-running skips channels that already have a
pending/running job.
The wildcard status page becomes channel-centric: an aggregate header
(X/N channels complete, running/queued/failed, totals), a per-channel
table, live batch polling, a 'Backfill all N channels' button, and
'Cancel all'. Single-channel backfill now guards per-channel instead
of per-rule, so one channel can be re-run while a batch runs on others.
Channel-scoped rules keep their existing single-channel view.
The Backfill button only ever rendered on an existing rule's row, so a
freshly-linked identity with no rules (e.g. a second atproto identity)
showed no sign backfill existed at all. Add a disabled backfill preview
under the new-rule form (noting it enables once the rule is saved, so
you can confirm the rule is right first), and a real enabled 'Backfill
this rule' link on the edit page. Both gate on the existing allowlist.
Inserts a jobs(user_id, did, did_handle, did_avatar) table between users
and rules. rules.user_id and write_counters.user_id move to job_id so a
single Discord account can drive multiple atproto identities, each with
its own rule set and counter ledger.
Includes a one-shot in-place migration (table-rebuild dance with
foreign_key_check) that seeds jobs from users with a DID, rewires rules
and counters, drops rows belonging to orphan no-DID users, and removes
did* columns from users.
Web UI gets an identity switcher (active pill + per-job sign-out / switch
/ delete / re-auth, plus "link another"). Bot, backfill, active-guilds,
and CLI scripts are switched over to job_id. Smoke test under
scripts/test-jobs-migration.mjs exercises the migration end-to-end.
eurosky.social's PDS sends ratelimit-policy=5000;w=3600 (5000 writes per
hour) without a Retry-After header. The worker now reads ratelimit-reset
to size the backoff sleep, defaults pacing to 60 writes/minute (3600/hr,
well under the policy and leaves room for live writes), and flushes
backfill_jobs progress per-message instead of per-page so the UI / poller
doesn't go dark during a multi-minute sleep.
- New per-rule backfill: paginates channel history ~1s/page and
replays each message through the existing match → extract →
dispatch pipeline. UI shows messages-processed / records-written
with a live progress bar, plus a Cancel button. Gated to a
single allowlisted Discord account via BACKFILL_ALLOWED_DISCORD_ID
while the feature is being shaken out.
- Destinations now use deterministic rkeys (sha256(messageId|item),
base64url, 24 chars) and putRecord, so a backfill replay
overwrites in place instead of duplicating. dispatch() returns
the count of PDS writes for accurate progress display.
- Live dispatcher threads ctx.createdAt through, so PDS records
carry the original Discord message time whether they were
written live or via backfill.
- /rules/:id/edit: edit rules in place. Form HTML + validation
extracted into shared renderRuleForm / parseRuleForm so add and
edit can't drift. Preserves rule.id (keeping backfill history
attached), created_at, and enabled flag.
discord.js silently drops messages when its websocket is unhealthy
and previously we had no listeners on Error / ShardError /
ShardDisconnect / Invalidated, so a "bot stopped processing"
incident left no trace in Railway logs to diagnose from.
Adds:
- client.on Error / Warn / ShardError / ShardDisconnect / ShardReady
/ ShardResume / ShardReconnecting / Invalidated / GuildUnavailable
in bot.js. ShardDisconnect logs the websocket close code + reason.
Invalidated exits the process so Railway restarts us into a fresh
connection (discord.js can't recover from this on its own).
- process.on unhandledRejection / uncaughtException in index.js so
stray async failures from any handler land in Railway logs with
stack traces instead of being silently swallowed.
Every event uses a "disjecta bot[...]" or "disjecta[...]" prefix so
they're greppable in Railway's log UI.
Adding a new case in the webhook handler used to require also clicking
into the Stripe dashboard to subscribe the endpoint to that event,
which is easy to forget — adding customer.deleted in the previous
commit was the immediate trigger.
EVENTS_HANDLED in stripe.js is now the single source of truth.
syncWebhookEvents() runs at startup, finds the endpoint matching
${PUBLIC_URL}/webhooks/stripe, and extends its enabled_events with
anything missing. Non-destructive: existing events stay, '*' is
respected, no endpoint match logs a clear "create one" warning.
Webhook handler now de-dupes events via a `webhook_events(event_id)`
table — Stripe retries are no-ops. Adds a `customer.deleted` handler
that nulls our stripe_customer_id reference.
Subscription teardown is consistent: any transition from active=1 to
inactive (subscription.deleted, subscription.updated to past_due/
canceled, or customer.deleted) disables every rule the user owns.
Re-subscribing flips active back on but leaves rules disabled — the
user re-enables them deliberately in the UI.
/subscribe redirects already-active users to /portal (via a tiny
GET shim) to prevent duplicate subscriptions. Replaces the raw
requireStripe() throw with a friendly 503 errorPage.
Grace window: a new GRACE_UNTIL env (ISO date) carries existing
test-period users when BILLING_DISABLED flips to 0. findMatchingRules
and active-guilds both filter on `customer_id IS NOT NULL OR now <
GRACE_UNTIL`, so the deadline takes effect to the second. UI shows
"free until YYYY-MM-DD" while the window is open.
Subscribe button label is fetched from Stripe at startup
(stripe.prices.retrieve) so changing the price in Stripe doesn't
leave the UI claiming "$2/month" forever. Falls back to plain
"Subscribe" if Stripe is unreachable.
readme: documents the billing state machine + the local
`stripe listen --forward-to ...` workflow for testing webhooks
without a public URL.
Defaulting community-bookmark as checked led to users accidentally
fanning out to it when they only meant to write Semble. Now no
lexicon is pre-selected — explicit opt-in.
Wrap the per-rule Disable/Delete forms in a flex container so they
sit side by side instead of wrapping in the narrow actions column.
The bookmark adapter is now a registry of bookmark-flavored lexicons
(community standard + Semble) rather than a single hardcoded NSID. A
rule picks any subset; each gets its own record per URL. Adding a new
target is one entry in LEXICONS.
Semble support writes network.cosmik.card. If the user picks one of
their Semble collections in the form, an afterWrite hook also writes
a network.cosmik.collectionLink with strongRefs to the card and the
collection (cid re-resolved at write time).
New /api/semble/collections endpoint lists the linked user's
network.cosmik.collection records to populate the picker. Form lazy-
loads the dropdown when Semble Card is ticked.
Old rules with destination_config=null fall back to writing only the
community lexicon, so existing bookmark rules keep working.
- Render the linked atproto handle + avatar instead of raw DID
- Gate "ready" state on the bot actually being in at least one of the
user's guilds; new "Add bot to a server" CTA when it isn't
- Use the destinations CATALOG for a typed dropdown with per-adapter
description and configSchema
- Honor billingDisabled when auto-activating users
Each adapter now exports a `meta` block (id, name, description, modes,
configSchema) so the web UI can render a typed dropdown. Adds the
community.lexicon.bookmarks.bookmark adapter as the default named
destination. Logs each successful createRecord for atproto-record and
bookmark — previously only errors were visible, so prod writes looked
silent in Railway logs.
Project name in package.json/lock, README, env templates, log messages,
HTML page titles/headers, atproto OAuth client_name, Discord slash command
namespace (/disject → /disjecta), DB filename default, and Railway domain
defaults in scripts/_railway-push-env.mjs.
Production resources rebuilt fresh on Railway under the new project name
"disjecta" with custom domain disjecta.portable.agency.
User-facing:
- Per-rule extract mode: URLs only or full message content
- Per-rule content filter: substring or regex (case-insensitive)
- Author input accepts @username (resolved via Discord member search) in
addition to numeric IDs, removing the developer-mode requirement
- /rules page now shows examples and explains that the bot does nothing
until a rule exists for a server
- Atmosphere unlink retained; copy now reflects routing both URLs and
message content
Loopback dev mode:
- Detect http://localhost (or 127.0.0.1) and skip atproto OAuth client
construction with a clear warning, so the rest of the app can be
exercised without an HTTPS tunnel
- /rules and rule mutations no longer require a linked DID in loopback
- Auto-activate users in loopback so rule matching actually fires without
Stripe involvement
Internals:
- match.js: messageMatches + validateMatchPattern
- discord-oauth.js: searchGuildMembers via bot REST
- db.js: ensureColumn for non-destructive migrations on existing DBs
- New columns: rules.mode, rules.match_kind, rules.match_value
- Rule POST validation tightened; bot evaluates content match per rule
after metadata SQL filter, before extraction
Dev scripts (gitignored convention not yet applied):
- scripts/_set-oauth-key.mjs: generate fresh JWK and rewrite .env in place
- scripts/_validate-env.mjs: report env health without printing secrets
Schema is no longer init-only via a separate command; CREATE TABLE IF
NOT EXISTS runs at module load on every entry point.
Bot harvests URLs from messages in any Discord server it's invited to
and routes them to per-rule destinations on the user's PDS. Web app
handles Discord+atproto OAuth, Stripe subscriptions, and rule CRUD.
Pass-through by design: only account metadata and aggregate write
counts are stored; no message content or URL data persists.
Defenses against free-rider servers: in-memory pre-filter on the hot
path, plus auto-leave for guilds with no active rules after a grace
period (default 7 days).