disjecta/src/ at main · burrito.space/disjecta · Tangled

burrito.space / disjecta

0

This repository has no description

0

disjecta / src

1 folder 15 files

at main

dietrich ayala extract: trim prose punctuation/parens leaking into URLs 4w ago

backfill: proactive write pacing + 429 backoff with Retry-After

2 months ago

active-guilds.js

extract: trim prose punctuation/parens leaking into URLs The greedy trailing char class swallowed a wrapping "(" and trailing ".", ")", "?" from prose, so "(https://x).” was stored verbatim. Drop the leading "(" from the pattern and post-trim trailing prose punctuation, keeping a ")" only when it balances a "(" inside the URL (e.g. wiki/Foo_(bar)). Adds node:test coverage and a `npm test` script. Also adds scripts/repair-card-urls.mjs: re-runs each stored network.cosmik.card URL through the fixed extractor and rewrites malformed or missing-top-level-url records in place (idempotent, same rkey). Dry-run by default.

1 month ago

atproto-identity.js

resolve and cache atproto profile (handle + avatar) per user Adds atproto-identity.js which prefers the bsky public getProfile API (handle + CDN-hosted avatar URL) and falls back to the DID document for non-bsky DIDs. Persists handle/avatar on the users row so the web UI doesn't have to re-fetch on every page load.

2 months ago

backfill: retry transient dispatch errors, not just 429s The dispatch retry loop only retried 429s; every other failure — 5xx, socket resets like UND_ERR_SOCKET, thrown dispatches — was recorded and skipped, the cursor advanced past it, and the job still finished "done". So a done job could silently carry dropped records, and cursor-based resume never re-attempts them; only a full channel re-run recovered the loss. Now retry every non-permanent error: 429s wait out the ratelimit-reset, other transients get exponential backoff, up to MAX_DISPATCH_RETRIES. Only a non-429 4xx is treated as permanent. Deterministic rkeys keep re-dispatch idempotent. Errors that survive all retries are persisted for the operational repair re-run, which remains the backstop. Upholds the "never lose a record" invariant by construction; permission failures stay an acceptable skip.

1 month ago

db+web: jobs table — one Discord account → N atproto identities Inserts a jobs(user_id, did, did_handle, did_avatar) table between users and rules. rules.user_id and write_counters.user_id move to job_id so a single Discord account can drive multiple atproto identities, each with its own rule set and counter ledger. Includes a one-shot in-place migration (table-rebuild dance with foreign_key_check) that seeds jobs from users with a DID, rewires rules and counters, drops rows belonging to orphan no-DID users, and removes did* columns from users. Web UI gets an identity switcher (active pill + per-job sign-out / switch / delete / re-auth, plus "link another"). Bot, backfill, active-guilds, and CLI scripts are switched over to job_id. Smoke test under scripts/test-jobs-migration.mjs exercises the migration end-to-end.

2 months ago

initial: discord URL harvester + atproto destination Bot harvests URLs from messages in any Discord server it's invited to and routes them to per-rule destinations on the user's PDS. Web app handles Discord+atproto OAuth, Stripe subscriptions, and rule CRUD. Pass-through by design: only account metadata and aggregate write counts are stored; no message content or URL data persists. Defenses against free-rider servers: in-memory pre-filter on the hot path, plus auto-leave for guilds with no active rules after a grace period (default 7 days).

2 months ago

web+db: backfill every channel in one click for server-wide rules Backfilling a wildcard rule meant picking one channel at a time and re-running per channel. Add a fan-out: POST /rules/:id/backfill/all enqueues one job per readable channel, drained FIFO by the existing worker (write-pacing is process-global, so the queue stays under the PDS rate limit). Re-running skips channels that already have a pending/running job. The wildcard status page becomes channel-centric: an aggregate header (X/N channels complete, running/queued/failed, totals), a per-channel table, live batch polling, a 'Backfill all N channels' button, and 'Cancel all'. Single-channel backfill now guards per-channel instead of per-rule, so one channel can be re-run while a batch runs on others. Channel-scoped rules keep their existing single-channel view.

1 month ago

discord-oauth.js

add content/regex filtering, username resolution, loopback dev mode User-facing: - Per-rule extract mode: URLs only or full message content - Per-rule content filter: substring or regex (case-insensitive) - Author input accepts @username (resolved via Discord member search) in addition to numeric IDs, removing the developer-mode requirement - /rules page now shows examples and explains that the bot does nothing until a rule exists for a server - Atmosphere unlink retained; copy now reflects routing both URLs and message content Loopback dev mode: - Detect http://localhost (or 127.0.0.1) and skip atproto OAuth client construction with a clear warning, so the rest of the app can be exercised without an HTTPS tunnel - /rules and rule mutations no longer require a linked DID in loopback - Auto-activate users in loopback so rule matching actually fires without Stripe involvement Internals: - match.js: messageMatches + validateMatchPattern - discord-oauth.js: searchGuildMembers via bot REST - db.js: ensureColumn for non-destructive migrations on existing DBs - New columns: rules.mode, rules.match_kind, rules.match_value - Rule POST validation tightened; bot evaluates content match per rule after metadata SQL filter, before extraction Dev scripts (gitignored convention not yet applied): - scripts/_set-oauth-key.mjs: generate fresh JWK and rewrite .env in place - scripts/_validate-env.mjs: report env health without printing secrets Schema is no longer init-only via a separate command; CREATE TABLE IF NOT EXISTS runs at module load on every entry point.

2 months ago

extract: trim prose punctuation/parens leaking into URLs The greedy trailing char class swallowed a wrapping "(" and trailing ".", ")", "?" from prose, so "(https://x).” was stored verbatim. Drop the leading "(" from the pattern and post-trim trailing prose punctuation, keeping a ")" only when it balances a "(" inside the URL (e.g. wiki/Foo_(bar)). Adds node:test coverage and a `npm test` script. Also adds scripts/repair-card-urls.mjs: re-runs each stored network.cosmik.card URL through the fixed extractor and rewrites malformed or missing-top-level-url records in place (idempotent, same rkey). Dry-run by default.

1 month ago

bot+web: backfill rules from Discord history; edit rules in place - New per-rule backfill: paginates channel history ~1s/page and replays each message through the existing match → extract → dispatch pipeline. UI shows messages-processed / records-written with a live progress bar, plus a Cancel button. Gated to a single allowlisted Discord account via BACKFILL_ALLOWED_DISCORD_ID while the feature is being shaken out. - Destinations now use deterministic rkeys (sha256(messageId|item), base64url, 24 chars) and putRecord, so a backfill replay overwrites in place instead of duplicating. dispatch() returns the count of PDS writes for accurate progress display. - Live dispatcher threads ctx.createdAt through, so PDS records carry the original Discord message time whether they were written live or via backfill. - /rules/:id/edit: edit rules in place. Form HTML + validation extracted into shared renderRuleForm / parseRuleForm so add and edit can't drift. Preserves rule.id (keeping backfill history attached), created_at, and enabled flag.

2 months ago

initial: discord URL harvester + atproto destination Bot harvests URLs from messages in any Discord server it's invited to and routes them to per-rule destinations on the user's PDS. Web app handles Discord+atproto OAuth, Stripe subscriptions, and rule CRUD. Pass-through by design: only account metadata and aggregate write counts are stored; no message content or URL data persists. Defenses against free-rider servers: in-memory pre-filter on the hot path, plus auto-leave for guilds with no active rules after a grace period (default 7 days).

2 months ago

add content/regex filtering, username resolution, loopback dev mode User-facing: - Per-rule extract mode: URLs only or full message content - Per-rule content filter: substring or regex (case-insensitive) - Author input accepts @username (resolved via Discord member search) in addition to numeric IDs, removing the developer-mode requirement - /rules page now shows examples and explains that the bot does nothing until a rule exists for a server - Atmosphere unlink retained; copy now reflects routing both URLs and message content Loopback dev mode: - Detect http://localhost (or 127.0.0.1) and skip atproto OAuth client construction with a clear warning, so the rest of the app can be exercised without an HTTPS tunnel - /rules and rule mutations no longer require a linked DID in loopback - Auto-activate users in loopback so rule matching actually fires without Stripe involvement Internals: - match.js: messageMatches + validateMatchPattern - discord-oauth.js: searchGuildMembers via bot REST - db.js: ensureColumn for non-destructive migrations on existing DBs - New columns: rules.mode, rules.match_kind, rules.match_value - Rule POST validation tightened; bot evaluates content match per rule after metadata SQL filter, before extraction Dev scripts (gitignored convention not yet applied): - scripts/_set-oauth-key.mjs: generate fresh JWK and rewrite .env in place - scripts/_validate-env.mjs: report env health without printing secrets Schema is no longer init-only via a separate command; CREATE TABLE IF NOT EXISTS runs at module load on every entry point.

2 months ago

add BILLING_DISABLED flag for test period Implied by loopback dev mode; can also be set explicitly via env. When on, Discord+atproto-linked users are auto-activated without going through Stripe. Lets us run a free test period before billing is wired up end-to-end.

2 months ago

secret-store.js

initial: discord URL harvester + atproto destination Bot harvests URLs from messages in any Discord server it's invited to and routes them to per-rule destinations on the user's PDS. Web app handles Discord+atproto OAuth, Stripe subscriptions, and rule CRUD. Pass-through by design: only account metadata and aggregate write counts are stored; no message content or URL data persists. Defenses against free-rider servers: in-memory pre-filter on the hot path, plus auto-leave for guilds with no active rules after a grace period (default 7 days).

2 months ago

stripe: auto-sync webhook endpoint enabled_events on startup Adding a new case in the webhook handler used to require also clicking into the Stripe dashboard to subscribe the endpoint to that event, which is easy to forget — adding customer.deleted in the previous commit was the immediate trigger. EVENTS_HANDLED in stripe.js is now the single source of truth. syncWebhookEvents() runs at startup, finds the endpoint matching ${PUBLIC_URL}/webhooks/stripe, and extends its enabled_events with anything missing. Non-destructive: existing events stay, '*' is respected, no endpoint match logs a clear "create one" warning.

2 months ago

web: clamp backfill progress bar to channel creation time Per-channel progress measured the fraction of the since..until calendar window walked. With the Everything window (since_ts ~10y ago) but channels created in 2024-2026, the cursor blows through all real content while covering only a few percent of the span, pinning every running bar near 0 until it snaps to 100 on completion. Clamp the floor to the channel's creation time (decoded from the channel-id snowflake, no API call) so the percentage reflects the span that can actually hold messages.

1 month ago