Clone this repository
For self-hosted knots, clone URLs may differ based on your setup.
Lays the foundation for measuring the three highest-leverage graph
walkers identified in the bench-target survey (Tier 1 #1, #2, and the
push-time #5 from the agent's tier-2 list):
- `count_commits_by_email` (mnemosyne_protocol::ref_update) —
runs per refUpdate inside the push tx.
- `expand_commit_closure` (mnemosyne_protocol::upload_pack) —
runs per fetch/clone to expand the client's want list.
- `verify_connectivity` (mnemosyne_postgres) — runs per
non-delete ref on every push to confirm reachability.
All three are CPU-dominated worklist walks (HashSet ops, gix_object
parsing, parent iteration) that a careless refactor can regress
2-10× without breaking correctness — exactly what criterion's
baseline-comparison feature is for. The proptests in
`ref_update_walk` already catch correctness regressions; these
benches catch performance ones.
Foundation
----------
The walkers used to take either `&PgBackend` (non-mutating callers
through `Arc<PgBackend>`) or live as `PushTx` methods (mutating
through the push transaction). To bench them against in-memory
fixtures with zero DB latency in the loop, they needed to be
generic over the object source.
`mnemosyne_git::ObjectSource` (new) takes `&mut self`, so types
holding exclusive resources like `PushTx` can implement it directly.
The blanket impl `impl<T: ObjectStore + ?Sized> ObjectSource for &T`
bridges from the existing read-only `ObjectStore`, so non-mutating
callers don't pay any ergonomic tax. The cost is one extra trait
type the codebase has to remember; the win is that walker code
written once works for production (PgBackend), push paths (PushTx),
and benches/tests (in-memory store) without duplication.
Refactored:
- `count_commits_by_email` — `CommitSource` removed; now generic
over `ObjectSource` directly. (Cosmetic — same shape, broader
interface.)
- `expand_commit_closure` — was `async fn(&RepoState, ...)`, now
generic. Pulled out an `ExpansionError` enum so the source's
error and the walk's `MissingWant` failure are distinguishable
at the call site (handler maps both to `ProtocolError`).
- `verify_connectivity` — same pattern: extracted as a free
generic function alongside `PushTx::verify_connectivity`, with
a `ConnectivityError<E>` enum. The PushTx method becomes a
one-line wrapper that maps back to `PgError`.
`mnemosyne_git::MemObjectStore` is a small `HashMap`-backed
`ObjectStore` (~120 LOC, behind an `Arc<Mutex<...>>` so clones share
state) that benches and tests reach for to avoid the testcontainer
ceremony. Production binaries get the unused code stripped by LTO;
if the cost ever grows we can gate it behind a `test-support`
feature.
Benches
-------
- `mnemosyne_protocol/benches/walkers.rs` — count_commits_by_email
and expand_commit_closure against linear / fanout / mixed DAGs
of 100 commits.
- `mnemosyne_postgres/benches/connectivity.rs` — verify_connectivity
against deep (100 commits, one tree) and wide (10 commits ×
200 blobs) fixtures.
Baselines documented in each bench's docstring. `just bench` runs
all of them; the criterion baseline-comparison flow (`--save-baseline
<name>` then `--baseline <name>`) detects regressions automatically.
Pairs with the `ref_update_walk` proptests in `mnemosyne_tests` —
same algorithms, different signals (correctness vs perf).
Deferred
--------
The `walk_commits` / `last_commit_for_path` / `oid_at_path` cluster
in `mnemosyne_xrpc::repo::history` is the natural next target (agent's
Tier 3 #8 — explicitly flagged in code as 'will need batching before
this is a hot path'). Refactoring those needs `read_tree_entries` to
also be source-generic; wider blast radius than this slice warrants.
Leaving as a follow-up; the bench infrastructure here is ready to
absorb them.
Closes part of slice 1 (issue #1) — the appview-facing contract.
Adds a Postgres-backed event stream so the Tangled appview can learn
about activity on this knot without polling per-repo XRPCs. Push
handlers emit `sh.tangled.git.refUpdate` records inside the same
transaction as the ref edits they describe, and a `GET /events`
WebSocket backfills from `?cursor=` then live-tails via Postgres
`LISTEN/NOTIFY`.
Design (issue #1: knot replicas must be load-balanceable)
---------------------------------------------------------
Strictly-ordered `seq` cursor under concurrent writes is achieved by
a transaction-scoped advisory lock (`EVENTS_ADVISORY_LOCK_ID`)
acquired immediately before INSERT. The lock-hold window is just
INSERT + `pg_notify` + commit; earlier work in the push transaction
(pack ingest, ref edits) remains concurrent. When a second emission
source materializes (probably with the `sh.tangled.knot.member`
ingest in slice 3), the upgrade path is a `stream` column + per-
stream lock id — the outbox pattern remains the further-out option
if contention ever becomes a real bottleneck.
The bench (`just bench`) shows the design plateaus at ~700 ev/s
aggregate by N=8 writers and stays flat at N=32 — the expected
signature of a clean serialization point, no pathological super-
linear degradation. The stress test (`just stress`) cross-validates
the same plateau (~2 000 ev/s under warmer conditions) and the
correctness claim that every emitted event reaches subscribers
exactly once, in seq order.
Meta computation
----------------
The two-phase `count_commits_by_email` walk implements proper
`git rev-list old..new` semantics — phase 1 marks `old_oid`'s
ancestor closure as uninteresting (bounded by
`OLD_ANCESTOR_WALK_LIMIT`), phase 2 walks from `new_oid` skipping
uninteresting OIDs. Proptest caught my first "stop AT old"
implementation that visited siblings of the boundary; the
regression fixture is checked in.
`langBreakdown` is populated on `language_stats_cache` hit (free)
and omitted on miss. A post-commit cache warmer task is on the
follow-up list — until then, the first push to a fresh tree has no
`langBreakdown` in its event; subsequent pushes pick it up once
the `sh.tangled.repo.languages` XRPC has been queried.
Tests added
-----------
- `ref_update_walk` (4 proptests, 256 cases each): termination,
bounded visit, no leakage past `old_oid`, empty-on-delete, no
double-count. Walker is generic over a `CommitSource` trait so
the tests drive it against an in-memory DAG without Postgres.
- `events_websocket` (4 integration tests): backfill ordering,
live-tail via NOTIFY, cursor resume, concurrent-writers strict
monotonicity at N=16.
- `stress_events` (gated by `STRESS_LEN`, runs under `just stress`):
pumps `STRESS_EVENTS_TOTAL` events through N producers, asserts
every committed seq reaches the subscriber, reports throughput
and catch-up margin.
- `event_emission` criterion bench (runs under `just bench` against
a one-shot container): characterizes the lock-hold path's
contention curve with criterion baselines for regression
detection. Pairs with `stress_events` — same code path, different
failure modes (correctness vs perf).
Known caveats
-------------
- TID rkey collisions are theoretically possible within the same
microsecond — jacquard 0.12.0-beta.2's `Tid::now_0()` has a
documented TODO about auto-increment. Revisit when upstream fixes.
- Cache warmer is deferred; first-push langBreakdown is empty until
the lang XRPC has been queried for that tree.
- Per-stream advisory lock is deferred to YAGNI; one lock today is
fine because there's exactly one emission source (push).
Closes #2 (target/ bloat).
The 22 files in `mnemosyne_tests/tests/*.rs` were each compiling
into their own statically-linked binary, every one of which pulled
in the full crate graph (sqlx, tokio, bollard, gix, reqwest,
mnemosyne_protocol, mnemosyne_ssh, mnemosyne_xrpc, ...). Across stale
hashed copies in `target/debug/deps/`, that reached ~80 GB. After
consolidation, one binary (~115 MB) replaces the 22 (~1.4 GB+
combined per build, multiplied across stale copies).
Layout change:
tests/
integration/
main.rs <-- single binary entry point
common/ <-- shared fixtures (formerly tests/common/)
<former tests/foo.rs as modules>
proptest-regressions/<-- seeds moved here (see below)
`Cargo.toml` declares the binary via `[[test]]` and turns off
`autotests` so a stray `tests/*.rs` from a future PR can't silently
re-introduce the per-file-binary explosion.
Pre-existing nextest config bug surfaced and fixed
---------------------------------------------------
The `public-keys` test-group override was using filters of the form
`test(=mnemosyne_tests::xrpc_endpoints::list_keys_*)`. Nextest's
`test()` matcher operates on the test name *within* the binary
(`xrpc_endpoints::list_keys_*` in the consolidated layout, just
`list_keys_*` in the old per-binary layout), not the full
`<package>::<binary>::<test>` triple printed by `--list`. So the
override was matching zero tests, the group was silently empty, and
the `DELETE FROM public_keys` races described in the config's own
comment were happening on every run — they just rarely lost.
Consolidation made the race deterministic enough to fail today's
`just test` run. Filters rewritten to bare module paths and a
comment added so the next person to touch this doesn't re-break it.
Proptest regression layout
--------------------------
Proptest's default `FileFailurePersistence::SourceParallel` walks
up from the source file looking for `tests/` (a special dir name)
and places `proptest-regressions/<name>.txt` parallel to it. Under
the old top-level-file layout that resolved to
`tests/<name>.proptest-regressions`; under the new layout it
resolves to `tests/proptest-regressions/<name>.txt`. Migrated the
four existing seed files so proptest still replays them.
cargo-sweep
-----------
Added `just sweep [days=14]` (wraps `cargo sweep --time`) and
`just sweep-toolchain` (wraps `cargo sweep --installed`). Includes
`target/debug/incremental/` automatically — addresses the 15 GB
incremental cache without disabling it, so local fast-rebuild wins
are preserved.
What the issue listed but I did NOT do
--------------------------------------
`SQLX_OFFLINE=true` + `.sqlx/` (the issue's lever #2) does not
apply here: this codebase has zero `sqlx::query!()` / `query_as!()`
macro invocations — every callsite uses the runtime string forms
(`sqlx::query(...)`). The 141 stale `libsqlx_postgres` copies were
just normal incremental residue from sqlx being on the dep graph
of every test binary, which the consolidation + sweep address
directly.
Verification
------------
`just test` → 162/162 pass, 2 skipped (stress, by design).
`list_keys_*` tests now run sequentially under the `public-keys`
group (~0.03–0.1 s each in a single thread) instead of racing.
The shared Postgres testcontainer is now booted once per test run by
a nextest setup script in `.config/nextest.toml`, which writes the
host port into every test process's environment via $NEXTEST_ENV.
Tests read `MNEMOSYNE_TEST_PG_PORT` and connect — they no longer
boot their own container.
Nextest has no first-class teardown concept, so the justfile wraps
`cargo nextest run` with a bash trap that does `docker rm -f` on
the recorded container id. Trap fires on EXIT/INT/TERM, so
interrupt-mid-run also cleans up.
`cargo test` keeps working via a fallback in each crate's
`common::shared_pool()`: if `MNEMOSYNE_TEST_PG_PORT` is absent, the
old in-Rust container boot + `ctor::dtor` cleanup path kicks in.
That keeps single-test iteration with plain `cargo test` available
without forcing nextest.
Knock-on cleanups:
- `unique_repo_did` / `unique_test_did` now PID-namespaced. Nextest
runs each test in its own subprocess against the shared postgres,
so per-process counters would collide on `did:test:000...001`
across processes.
- The `list_keys` and SSH-related tests that touch the shared
`public_keys` table are pinned to a `max-threads = 1` nextest
test group so the cross-process `DELETE FROM public_keys` race
can't fire.
- `unique_test_did` lifted into `mnemosyne_postgres/tests/common`
from three per-file copies.
Rust statics never run `Drop`, so the `ContainerAsync` instances
held in `OnceCell` statics (the process-wide postgres + git+ssh
containers) leaked into the Docker daemon on every `cargo test`
run. They'd accumulate as "Exited" or "Up" rows until manually
pruned.
Fix: register a `#[ctor::dtor]` that takes the container out of a
`Mutex<Option<ContainerAsync>>`, then `rm()`s it. `rm()` consumes
the container by value, so there's no follow-on async-drop firing
on a no-longer-current runtime — which is what previously panicked
with "there is no reactor running" when calling `stop()` and
letting the borrow drop.
Trap-on-TERM in the alpine startup script so the git+ssh container
exits cleanly (code 0) instead of via SIGKILL escalation (137) —
`exec sleep infinity` would otherwise become PID 1 and ignore
SIGTERM.
Postgres integration tests in `mnemosyne_postgres/tests/*` also
converted to a shared-pool pattern. Each `#[tokio::test]` was
building its own runtime, and sqlx `PgPool` work has runtime
affinity — a pool initialized on one test's runtime hangs when used
from the other's. Switched to `common::runtime().block_on(...)` for
a single shared runtime, mirroring what `mnemosyne_tests` already
does.
Known residual: `stress_concurrent_push.rs` still uses a
per-function container that relies on `ContainerAsync`'s own Drop
to clean up. That Drop is unreliable from sync context; if the
stress test runs, it may still leak one container. Out of scope
for this commit — the bulk path (every test that uses
`common::shared_pool()`) is fixed.
#[instrument] on the four handler callbacks that delineate a session:
auth_publickey, exec_request, start_git_session, plus the spawned
git-op task inside dispatch_pending.
auth_publickey records algorithm, fingerprint, authenticated_did,
outcome (accepted | rejected_unknown_key | accept_any_mode).
exec_request records command, parsed service, outcome
(started_git_session | rejected_unknown_command | echo_accept_any_mode).
start_git_session adds repo_did + advertisement_bytes once known.
The spawned dispatch task uses tracing::info_span! + .instrument()
because tokio::spawn doesn't propagate spans automatically; without
that wrap the receive_pack / upload_pack inner spans would float as
orphans.
The eprintln on cleanup failure becomes tracing::warn! tied to the
dispatch span — the failure stays best-effort but is now queryable.
Small visibility tweak in mnemosyne_protocol: Service::as_str() is
now `pub` so the SSH instrumentation can record the service as a
span field across the crate boundary.
#[instrument] on the three smart-HTTP handlers — info_refs,
upload_pack, receive_pack. One span per HTTP request, the wide
event for what happened with that operation.
receive_pack is the richest: span fields include authenticated_did,
authorized_as (owner | collaborator | denied | denied_unauthenticated),
command_count, pack_bytes, delete_only. A query like "show me all
collaborator pushes that mutated more than 100MB of pack" becomes a
structured filter.
upload_pack records want_count, expanded_want_count (post commit-
closure walk), pack_bytes, side_band_64k. info_refs records service
+ body_bytes.
Adds tracing as a dep of mnemosyne_protocol. Existing protocol
integration tests stay green; no behavior change beyond
instrumentation.
Dispatcher::apply gets a #[tracing::instrument] span — one span per
firehose event, the wide event for "what happened with this
record." Fields: nsid, author_did, rkey, operation, time_us at
creation; outcome, skip_reason, repo_did, subject_did filled in via
Span::current().record() as the handler discovers them.
Handlers (publicKey, repo, repo.collaborator) replace their eprintln
skip-logs with tracing::warn! events tied to the dispatcher span,
plus span.record("skip_reason", ...) for the wide-event field. Now
querying "show me all rejected non-owner collaborator records"
becomes one structured filter instead of a string-search across log
lines.
Subscriber's connect / disconnect / malformed-frame eprintlns
become warn!/error! events. Infra failures from dispatcher.apply()
log at error! level since they signal "something is broken,"
distinct from per-event policy skips which are info!/warn! inside
the handler span.
Adds tracing + tracing-subscriber to the workspace. Both mnemosyne
and mnemosyne-ingest binaries now initialize a JSON-only subscriber
at startup with span events on CLOSE — the wide-event posture where
one log line per logical unit of work carries all the context
accumulated by that unit.
Default env filter mutes the ecosystem chatter we don't author
(sqlx logs every query at INFO; hyper/tower/h2/russh log
connection-level framing). Override via RUST_LOG. No eprintln
conversions in this commit — that's per-subsystem follow-up work.
Init function is duplicated verbatim across the two binaries rather
than factored into a shared crate; the duplication beats a
single-purpose crate while we have only two callers.