solpbc.org / solstone
personal memory agent
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

feat(ci): add journal_io raw-mechanic gate; migrate 4 journal-data stragglers

Add scripts/check_journal_io_mechanic.py: a static AST check that flags raw
durable-write / locked-RMW mechanics (os.replace, filesystem Path.replace,
mkstemp+manual-replace, flock LOCK_EX RMW) against owner journal data anywhere
outside solstone/think/journal_io/. Unlike the import-binding access lint, this
gate SCANS owners — they may import journal_io primitives but must carry no
hand-rolled mechanics. Seeds with an empty allowlist; wired into make ci via
install-checks alongside the access and layer-hygiene checks.

To seed the gate empty and green, migrate every remaining journal-data straggler
to journal_io (byte-identical output, equivalent locking):
- link/paths.py: state/token/totp writes -> write_json (token/totp keep 0o600)
- convey/chat_stream.py: chat.jsonl -> atomic_replace (ensure_ascii=False house
pattern); declared the L2 owner of chronicle chat/<seg>/chat.jsonl
- activities.py: _write_jsonl_records -> atomic_replace; locked_modify ->
hold_lock (drops the bare-LOCK_EX blocking RMW + max_retries retry loop)
- importers/shared.py: chronicle import.<source>/imported.jsonl -> atomic_replace
(a 4th straggler the original inventory missed; surfaced in prep)

locked_modify now raises journal_io.LockTimeout on contention; surface it in
sol's voice at the activities CLI write boundary via a new ACTIVITIES_BUSY
reason, mirroring the entities ENTITY_BUSY pattern. Tests reassert locking and
byte-fidelity invariants instead of the old flock/mkstemp internals.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

author
Jer Miller co-author
Claude Opus 4.8 (1M context) date (Jun 7, 2026, 6:01 AM -0600) commit 5bba1b93 parent c3b6bc95
+773 -135
+1
AGENTS.md
··· 195 195 | Awareness activity state (`awareness/activity_state.json`) | `solstone/think/thinking.py` | 196 196 | Identity (`identity/*.md`, `identity/history.jsonl` audit log) | `solstone/think/identity.py` | 197 197 | Todos (`facets/*/todos/*.jsonl`) | `solstone/apps/todos/todo.py` + `solstone/apps/todos/call.py` | 198 + | Chronicle chat stream (`chronicle/**/chat/<seg>/chat.jsonl`) | `solstone/convey/chat_stream.py` | 198 199 | Config (`config/journal.json`) | `solstone/think/journal_config.py` | 199 200 | Push devices (`config/push_devices.json`) | `solstone/think/push/devices.py` | 200 201 | Convey config (`config/convey.json`) | `solstone/convey/config.py` + `solstone/think/facets.py` |
+8 -1
Makefile
··· 14 14 PYTEST_BASETEMP_INIT := BASETEMP=$$(mktemp -d /var/tmp/solstone-pytest-XXXXXX); trap 'rm -rf "$$BASETEMP"' EXIT INT TERM; 15 15 PYTEST_BASETEMP_FLAG := --basetemp "$$BASETEMP" 16 16 17 - .PHONY: install uninstall test test-cov test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills dev all sandbox sandbox-stop install-pinchtab install-models parakeet-helper parakeet-helper-clean wheel-macos wheel-macos-clean verify-browser update-browser-baselines review verify verify-api update-api-baselines service-logs check-layer-hygiene check-api-conventions check-journal-io-access smoke-cogitate release release-test FORCE 17 + .PHONY: install uninstall test test-cov test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills dev all sandbox sandbox-stop install-pinchtab install-models parakeet-helper parakeet-helper-clean wheel-macos wheel-macos-clean verify-browser update-browser-baselines review verify verify-api update-api-baselines service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic smoke-cogitate release release-test FORCE 18 18 19 19 # Default target - install package in editable mode 20 20 all: install ··· 461 461 @echo "=== Running journal-io access check ===" 462 462 @$(MAKE) check-journal-io-access 463 463 @echo "" 464 + @echo "=== Running journal-io mechanic check ===" 465 + @$(MAKE) check-journal-io-mechanic 466 + @echo "" 464 467 @echo "=== Checking extras consistency ===" 465 468 @$(VENV_BIN)/python scripts/check_extras_consistency.py 466 469 @echo "" ··· 530 533 # Journal-io write-primitive access check (see AGENTS.md §7 L2) 531 534 check-journal-io-access: .installed 532 535 $(VENV_BIN)/python scripts/check_journal_io_access.py 536 + 537 + # Journal raw-mechanic check (see AGENTS.md §7 L2) 538 + check-journal-io-mechanic: .installed 539 + $(VENV_BIN)/python scripts/check_journal_io_mechanic.py 533 540 534 541 # Re-run the live four-backend integrated-façade cogitate smoke. Spawns the 535 542 # archived runner (extro `vpe/workspace/archived/`) against this venv so the
+1
scripts/check_journal_io_access.py
··· 133 133 "solstone/think/importers/cli.py", 134 134 "solstone/think/importers/documents.py", 135 135 "solstone/think/importers/shared.py", 136 + "solstone/convey/chat_stream.py", 136 137 # imports/** bundle + sync-cursor writers (local/CLI import flows). 137 138 "solstone/think/importers/plaud.py", # streamed imported-audio install. 138 139 "solstone/think/importers/sync.py",
+427
scripts/check_journal_io_mechanic.py
··· 1 + #!/usr/bin/env python3 2 + # SPDX-License-Identifier: AGPL-3.0-only 3 + # Copyright (c) 2026 sol pbc 4 + 5 + """Journal raw-mechanic lint. 6 + 7 + This check flags raw durable-write and locked read-modify-write mechanics used 8 + against owner journal data anywhere outside ``solstone.think.journal_io``. 9 + Unlike ``check_journal_io_access.py``, owner modules are intentionally scanned: 10 + owners may import journal_io primitives, but after migration they must not carry 11 + their own raw replacement or bare exclusive-flock mechanics. 12 + 13 + Flagged mechanics: 14 + 15 + D1 - ``os.replace``. Calls are resolved through ``import os`` / aliases and 16 + ``from os import replace`` bindings. Bare attribute-name matching is forbidden. 17 + 18 + D2 - ``Path.replace`` heuristic. The detector flags one-argument 19 + ``.replace(...)`` calls only when the receiver looks like a temp path: 20 + direct ``.with_suffix/.with_name/.with_stem(...)`` receiver, a receiver name 21 + containing ``tmp``/``temp``, or a receiver name bound from known temp/path 22 + creation patterns. This deliberately avoids string/date ``replace`` calls. 23 + Limit: a non-temp-named Path variable with one non-string argument is a false 24 + negative; in this repository the ``os.replace`` rule and temp-name tracking 25 + cover every real durable-write site. The temp-substring receiver-name match 26 + can also over-match names like ``template`` or ``attempt``. 27 + 28 + D3 - ``flock(LOCK_EX)``. ``fcntl.flock`` calls with a lock-mode subtree 29 + containing ``LOCK_EX`` and not ``LOCK_NB`` are flagged. ``LOCK_UN`` and 30 + ``LOCK_EX | LOCK_NB`` are not violations. 31 + 32 + The check ships green with an empty allowlist. The allowlist is keyed by 33 + ``(file, kind)`` with an allowed count so it can ratchet down, matching the 34 + existing journal_io access check. 35 + """ 36 + 37 + from __future__ import annotations 38 + 39 + import argparse 40 + import ast 41 + import sys 42 + from pathlib import Path 43 + 44 + ROOT = Path(__file__).resolve().parent.parent 45 + 46 + PATH_TEMP_METHODS: frozenset[str] = frozenset({"with_suffix", "with_name", "with_stem"}) 47 + VIOLATION_KINDS: frozenset[str] = frozenset( 48 + {"os.replace", "Path.replace", "flock(LOCK_EX)"} 49 + ) 50 + 51 + EXCLUDED_FILES: frozenset[str] = frozenset( 52 + { 53 + # Ops/runtime state and single-process guards. 54 + "solstone/think/scheduler.py", 55 + "solstone/think/supervisor.py", 56 + "solstone/think/readiness.py", 57 + "solstone/think/providers/state.py", 58 + "solstone/think/providers/local_install.py", 59 + "solstone/think/providers/mlx_install.py", 60 + "solstone/think/services/scout.py", 61 + "solstone/think/services/spl.py", 62 + "solstone/think/steward.py", 63 + "solstone/talent/steward.py", 64 + "solstone/think/install_guard.py", 65 + "solstone/think/install_models.py", 66 + "solstone/think/setup.py", 67 + "solstone/think/user_config.py", 68 + "solstone/think/voice/brain.py", 69 + "solstone/think/sync_check.py", 70 + "solstone/think/runner.py", 71 + "solstone/think/providers_cli.py", 72 + "solstone/think/start.py", 73 + "solstone/talent/daily_schedule.py", 74 + "solstone/think/routines.py", 75 + "solstone/apps/sol/maint/005_migrate_dream_to_think_schedules.py", 76 + "solstone/apps/timeline/maint/001_register_schedules.py", 77 + # App-storage and temporary upload/transcription files. 78 + "solstone/apps/import/routes.py", 79 + "solstone/apps/support/routes.py", 80 + "solstone/observe/transcribe/whisper.py", 81 + "solstone/observe/transcribe/_parakeet_coreml.py", 82 + "solstone/observe/transcribe/revai.py", 83 + "solstone/think/journal_export.py", 84 + # UI/pipeline runtime state, not owner journal content. 85 + "solstone/apps/home/routes.py", 86 + "solstone/apps/transcripts/routes.py", 87 + "solstone/think/data_state.py", 88 + "solstone/think/skills_cli.py", 89 + } 90 + ) 91 + 92 + EXCLUDED_PREFIXES: tuple[str, ...] = () 93 + 94 + # Empty by design. If this check flags owner journal-data, migrate the site. If 95 + # it flags non-owner runtime storage, classify it in EXCLUDED_FILES. 96 + ALLOWLIST: dict[tuple[str, str], int] = {} 97 + 98 + 99 + def _is_test_file(rel: Path) -> bool: 100 + return ( 101 + "tests" in rel.parts 102 + or rel.name == "conftest.py" 103 + or (rel.name.startswith("test_") and rel.suffix == ".py") 104 + ) 105 + 106 + 107 + def _is_excluded(rel: Path) -> bool: 108 + rel_str = rel.as_posix() 109 + return rel_str in EXCLUDED_FILES or any( 110 + rel_str.startswith(prefix) for prefix in EXCLUDED_PREFIXES 111 + ) 112 + 113 + 114 + def discover_modules(root: Path) -> list[Path]: 115 + """Return posix-relative scanned modules under ``solstone/``.""" 116 + scope = root / "solstone" 117 + if not scope.is_dir(): 118 + return [] 119 + 120 + found: list[Path] = [] 121 + for path in sorted(scope.rglob("*.py")): 122 + rel = path.relative_to(root) 123 + rel_str = rel.as_posix() 124 + if "__pycache__" in rel.parts: 125 + continue 126 + if rel_str.startswith("solstone/think/journal_io/"): 127 + continue 128 + if _is_test_file(rel): 129 + continue 130 + if _is_excluded(rel): 131 + continue 132 + found.append(rel) 133 + return found 134 + 135 + 136 + def _call_detail(func: ast.expr) -> str: 137 + try: 138 + return ast.unparse(func) 139 + except Exception: 140 + return "<call>" 141 + 142 + 143 + def _is_attr_call(expr: ast.AST, attrs: frozenset[str]) -> bool: 144 + return ( 145 + isinstance(expr, ast.Call) 146 + and isinstance(expr.func, ast.Attribute) 147 + and expr.func.attr in attrs 148 + ) 149 + 150 + 151 + def _is_tempfile_call( 152 + expr: ast.AST, 153 + name: str, 154 + tempfile_aliases: set[str], 155 + direct_names: set[str], 156 + ) -> bool: 157 + if not isinstance(expr, ast.Call): 158 + return False 159 + func = expr.func 160 + if isinstance(func, ast.Name): 161 + return func.id in direct_names 162 + return ( 163 + isinstance(func, ast.Attribute) 164 + and func.attr == name 165 + and isinstance(func.value, ast.Name) 166 + and func.value.id in tempfile_aliases 167 + ) 168 + 169 + 170 + def _iter_target_names(target: ast.AST) -> list[str]: 171 + if isinstance(target, ast.Name): 172 + return [target.id] 173 + if isinstance(target, (ast.Tuple, ast.List)): 174 + names: list[str] = [] 175 + for elt in target.elts: 176 + names.extend(_iter_target_names(elt)) 177 + return names 178 + return [] 179 + 180 + 181 + def _collect_bindings( 182 + tree: ast.AST, 183 + ) -> tuple[set[str], set[str], set[str], set[str], set[str]]: 184 + os_aliases: set[str] = set() 185 + os_replace_names: set[str] = set() 186 + fcntl_aliases: set[str] = set() 187 + flock_names: set[str] = set() 188 + tempfile_aliases: set[str] = set() 189 + mkstemp_names: set[str] = set() 190 + named_temp_names: set[str] = set() 191 + temp_names: set[str] = set() 192 + 193 + for node in ast.walk(tree): 194 + if isinstance(node, ast.Import): 195 + for alias in node.names: 196 + bound = alias.asname or alias.name 197 + if alias.name == "os": 198 + os_aliases.add(bound) 199 + elif alias.name == "fcntl": 200 + fcntl_aliases.add(bound) 201 + elif alias.name == "tempfile": 202 + tempfile_aliases.add(bound) 203 + elif isinstance(node, ast.ImportFrom): 204 + module = node.module or "" 205 + for alias in node.names: 206 + bound = alias.asname or alias.name 207 + if module == "os" and alias.name == "replace": 208 + os_replace_names.add(bound) 209 + elif module == "fcntl" and alias.name == "flock": 210 + flock_names.add(bound) 211 + elif module == "tempfile" and alias.name == "mkstemp": 212 + mkstemp_names.add(bound) 213 + elif module == "tempfile" and alias.name == "NamedTemporaryFile": 214 + named_temp_names.add(bound) 215 + 216 + for node in ast.walk(tree): 217 + if isinstance(node, (ast.Assign, ast.AnnAssign)): 218 + value = node.value 219 + targets = node.targets if isinstance(node, ast.Assign) else [node.target] 220 + if value is None: 221 + continue 222 + for target in targets: 223 + if _is_attr_call(value, PATH_TEMP_METHODS) or _is_tempfile_call( 224 + value, 225 + "NamedTemporaryFile", 226 + tempfile_aliases, 227 + named_temp_names, 228 + ): 229 + temp_names.update(_iter_target_names(target)) 230 + if _is_tempfile_call(value, "mkstemp", tempfile_aliases, mkstemp_names): 231 + if ( 232 + isinstance(target, (ast.Tuple, ast.List)) 233 + and len(target.elts) >= 2 234 + and isinstance(target.elts[1], ast.Name) 235 + ): 236 + temp_names.add(target.elts[1].id) 237 + elif isinstance(node, ast.With): 238 + for item in node.items: 239 + if item.optional_vars is None: 240 + continue 241 + if _is_tempfile_call( 242 + item.context_expr, 243 + "NamedTemporaryFile", 244 + tempfile_aliases, 245 + named_temp_names, 246 + ): 247 + temp_names.update(_iter_target_names(item.optional_vars)) 248 + 249 + return os_aliases, os_replace_names, fcntl_aliases, flock_names, temp_names 250 + 251 + 252 + def _is_os_replace_call( 253 + func: ast.expr, 254 + os_aliases: set[str], 255 + os_replace_names: set[str], 256 + ) -> bool: 257 + if isinstance(func, ast.Name): 258 + return func.id in os_replace_names 259 + return ( 260 + isinstance(func, ast.Attribute) 261 + and func.attr == "replace" 262 + and isinstance(func.value, ast.Name) 263 + and func.value.id in os_aliases 264 + ) 265 + 266 + 267 + def _is_temp_name(name: str, temp_names: set[str]) -> bool: 268 + lowered = name.lower() 269 + return "tmp" in lowered or "temp" in lowered or name in temp_names 270 + 271 + 272 + def _is_path_like_receiver(receiver: ast.expr, temp_names: set[str]) -> bool: 273 + if _is_attr_call(receiver, PATH_TEMP_METHODS): 274 + return True 275 + return isinstance(receiver, ast.Name) and _is_temp_name(receiver.id, temp_names) 276 + 277 + 278 + def _is_path_replace_call(node: ast.Call, temp_names: set[str]) -> bool: 279 + if not isinstance(node.func, ast.Attribute) or node.func.attr != "replace": 280 + return False 281 + if len(node.args) != 1 or node.keywords: 282 + return False 283 + if any(isinstance(arg, ast.Starred) for arg in node.args): 284 + return False 285 + arg = node.args[0] 286 + if isinstance(arg, ast.Constant) and isinstance(arg.value, str): 287 + return False 288 + return _is_path_like_receiver(node.func.value, temp_names) 289 + 290 + 291 + def _is_flock_call( 292 + func: ast.expr, 293 + fcntl_aliases: set[str], 294 + flock_names: set[str], 295 + ) -> bool: 296 + if isinstance(func, ast.Name): 297 + return func.id in flock_names 298 + return ( 299 + isinstance(func, ast.Attribute) 300 + and func.attr == "flock" 301 + and isinstance(func.value, ast.Name) 302 + and func.value.id in fcntl_aliases 303 + ) 304 + 305 + 306 + def _lock_mode_names(node: ast.AST) -> set[str]: 307 + names: set[str] = set() 308 + for child in ast.walk(node): 309 + if isinstance(child, ast.Name): 310 + names.add(child.id) 311 + elif isinstance(child, ast.Attribute): 312 + names.add(child.attr) 313 + return names 314 + 315 + 316 + def _is_bare_lock_ex(node: ast.Call) -> bool: 317 + if len(node.args) < 2: 318 + return False 319 + lock_names = _lock_mode_names(node.args[1]) 320 + return "LOCK_EX" in lock_names and "LOCK_NB" not in lock_names 321 + 322 + 323 + def scan_source(source: str, filename: str = "<source>") -> list[tuple[int, str, str]]: 324 + """Return ``(lineno, kind, detail)`` mechanic violations for source.""" 325 + tree = ast.parse(source, filename=filename) 326 + os_aliases, os_replace_names, fcntl_aliases, flock_names, temp_names = ( 327 + _collect_bindings(tree) 328 + ) 329 + 330 + findings: list[tuple[int, str, str]] = [] 331 + for node in ast.walk(tree): 332 + if not isinstance(node, ast.Call): 333 + continue 334 + 335 + if _is_os_replace_call(node.func, os_aliases, os_replace_names): 336 + findings.append((node.lineno, "os.replace", _call_detail(node.func))) 337 + continue 338 + 339 + if _is_path_replace_call(node, temp_names): 340 + findings.append((node.lineno, "Path.replace", _call_detail(node.func))) 341 + continue 342 + 343 + if _is_flock_call(node.func, fcntl_aliases, flock_names) and _is_bare_lock_ex( 344 + node 345 + ): 346 + findings.append((node.lineno, "flock(LOCK_EX)", _call_detail(node.func))) 347 + 348 + findings.sort() 349 + return findings 350 + 351 + 352 + def scan_file(path: Path) -> list[tuple[int, str, str]]: 353 + return scan_source(path.read_text(encoding="utf-8"), filename=str(path)) 354 + 355 + 356 + def count_violations(root: Path) -> dict[tuple[str, str], int]: 357 + """Map ``(posix-relpath, kind)`` -> occurrence count across the tree.""" 358 + counts: dict[tuple[str, str], int] = {} 359 + for rel in discover_modules(root): 360 + for _lineno, kind, _detail in scan_file(root / rel): 361 + key = (rel.as_posix(), kind) 362 + counts[key] = counts.get(key, 0) + 1 363 + return counts 364 + 365 + 366 + def evaluate( 367 + root: Path, 368 + allowlist: dict[tuple[str, str], int], 369 + ) -> tuple[list[str], list[str]]: 370 + """Return ``(new_violations, tracked)`` human-readable lines.""" 371 + new: list[str] = [] 372 + tracked: list[str] = [] 373 + for rel in discover_modules(root): 374 + rel_str = rel.as_posix() 375 + findings = scan_file(root / rel) 376 + by_kind: dict[str, list[int]] = {} 377 + for lineno, kind, _detail in findings: 378 + by_kind.setdefault(kind, []).append(lineno) 379 + for kind, linenos in sorted(by_kind.items()): 380 + count = len(linenos) 381 + allowed = allowlist.get((rel_str, kind), 0) 382 + if count > allowed: 383 + lines = ", ".join(str(n) for n in sorted(linenos)) 384 + new.append( 385 + f"{rel_str}: raw journal I/O mechanic {kind} " 386 + f"({count} occurrence(s), allowed {allowed}) at line(s) {lines}" 387 + ) 388 + elif allowed: 389 + tracked.append(f"{rel_str}: {count}/{allowed} {kind} (allowlisted)") 390 + return new, tracked 391 + 392 + 393 + def main(argv: list[str] | None = None) -> int: 394 + parser = argparse.ArgumentParser(description="Journal raw-mechanic lint") 395 + parser.add_argument( 396 + "--root", 397 + type=Path, 398 + default=ROOT, 399 + help="Repository root to scan (defaults to the checkout root).", 400 + ) 401 + args = parser.parse_args(argv) 402 + 403 + new, tracked = evaluate(args.root, ALLOWLIST) 404 + 405 + if tracked: 406 + print("journal-io-mechanic: known violations (allowlisted, ratcheting down):") 407 + for line in tracked: 408 + print(f" {line}") 409 + print() 410 + 411 + if new: 412 + print("journal-io-mechanic: NEW violations:", file=sys.stderr) 413 + for line in new: 414 + print(f" {line}", file=sys.stderr) 415 + print(file=sys.stderr) 416 + print( 417 + "Route durable journal writes through solstone.think.journal_io.", 418 + file=sys.stderr, 419 + ) 420 + return 1 421 + 422 + print("journal-io-mechanic: pass") 423 + return 0 424 + 425 + 426 + if __name__ == "__main__": 427 + raise SystemExit(main())
+43 -23
solstone/apps/activities/call.py
··· 15 15 16 16 import typer 17 17 18 + from solstone.convey.reasons import ACTIVITIES_BUSY 18 19 from solstone.think.activities import ( 19 20 append_activity_record, 20 21 append_edit, ··· 30 31 from solstone.think.entities.loading import load_entities 31 32 from solstone.think.entities.matching import find_matching_entity 32 33 from solstone.think.facets import get_facets, log_call_action 34 + from solstone.think.journal_io import LockTimeout 33 35 from solstone.think.utils import ( 34 36 get_sol_facet, 35 37 now_ms, ··· 466 468 note="created", 467 469 ) 468 470 469 - if not append_activity_record(resolved_facet, resolved_day, record): 471 + try: 472 + created = append_activity_record(resolved_facet, resolved_day, record) 473 + except LockTimeout: 474 + typer.echo(ACTIVITIES_BUSY.message, err=True) 475 + raise typer.Exit(1) 476 + 477 + if not created: 470 478 typer.echo(f"Error: activity already exists: {span_id}", err=True) 471 479 raise typer.Exit(1) 472 480 ··· 523 531 raise typer.Exit(1) 524 532 525 533 note_text = note or f"updated fields: {', '.join(sorted(patch))}" 526 - updated = update_activity_record( 527 - resolved_facet, 528 - resolved_day, 529 - span_id, 530 - patch, 531 - actor="cli:update", 532 - note=note_text, 533 - ) 534 + try: 535 + updated = update_activity_record( 536 + resolved_facet, 537 + resolved_day, 538 + span_id, 539 + patch, 540 + actor="cli:update", 541 + note=note_text, 542 + ) 543 + except LockTimeout: 544 + typer.echo(ACTIVITIES_BUSY.message, err=True) 545 + raise typer.Exit(1) 534 546 if updated is None: 535 547 typer.echo(f"activity not found: {span_id}", err=True) 536 548 raise typer.Exit(1) ··· 569 581 """Hide an activity record without deleting it.""" 570 582 resolved_facet = resolve_sol_facet(facet) 571 583 resolved_day = resolve_sol_day(day) 572 - record = mute_activity_record( 573 - resolved_facet, 574 - resolved_day, 575 - span_id, 576 - actor="cli:mute", 577 - reason=reason, 578 - ) 584 + try: 585 + record = mute_activity_record( 586 + resolved_facet, 587 + resolved_day, 588 + span_id, 589 + actor="cli:mute", 590 + reason=reason, 591 + ) 592 + except LockTimeout: 593 + typer.echo(ACTIVITIES_BUSY.message, err=True) 594 + raise typer.Exit(1) 579 595 if record is None: 580 596 typer.echo(f"activity not found: {span_id}", err=True) 581 597 raise typer.Exit(1) ··· 614 630 """Restore a previously hidden activity record.""" 615 631 resolved_facet = resolve_sol_facet(facet) 616 632 resolved_day = resolve_sol_day(day) 617 - record = unmute_activity_record( 618 - resolved_facet, 619 - resolved_day, 620 - span_id, 621 - actor="cli:unmute", 622 - reason=reason, 623 - ) 633 + try: 634 + record = unmute_activity_record( 635 + resolved_facet, 636 + resolved_day, 637 + span_id, 638 + actor="cli:unmute", 639 + reason=reason, 640 + ) 641 + except LockTimeout: 642 + typer.echo(ACTIVITIES_BUSY.message, err=True) 643 + raise typer.Exit(1) 624 644 if record is None: 625 645 typer.echo(f"activity not found: {span_id}", err=True) 626 646 raise typer.Exit(1)
+3 -15
solstone/convey/chat_stream.py
··· 5 5 6 6 import json 7 7 import logging 8 - import os 9 8 import sys 10 9 import threading 11 10 import time ··· 15 14 16 15 from solstone.think.callosum import callosum_send 17 16 from solstone.think.indexer.journal import index_file 17 + from solstone.think.journal_io import atomic_replace 18 18 from solstone.think.streams import update_stream, write_segment_stream 19 19 from solstone.think.utils import ( 20 20 day_path, ··· 413 413 414 414 415 415 def _write_events_file(path: Path, events: list[dict[str, Any]]) -> None: 416 - tmp_path = path.with_suffix(f".{os.getpid()}-{threading.get_ident()}.tmp") 417 - try: 418 - with open(tmp_path, "w", encoding="utf-8") as handle: 419 - for event in events: 420 - handle.write(json.dumps(event, ensure_ascii=False)) 421 - handle.write("\n") 422 - os.replace(tmp_path, path) 423 - except Exception: 424 - try: 425 - if tmp_path.exists(): 426 - tmp_path.unlink() 427 - except OSError: 428 - pass 429 - raise 416 + body = "".join(json.dumps(event, ensure_ascii=False) + "\n" for event in events) 417 + atomic_replace(path, body)
+5
solstone/convey/reasons.py
··· 316 316 "I couldn't update that entity right now because it was busy. Try again in a moment.", 317 317 503, 318 318 ) 319 + ACTIVITIES_BUSY = Reason( 320 + "activities_busy", 321 + "I couldn't update activities right now because they were busy. Try again in a moment.", 322 + 503, 323 + ) 319 324 320 325 # identity 321 326 IDENTITY_BUSY = Reason(
+17 -48
solstone/think/activities.py
··· 11 11 """ 12 12 13 13 import difflib 14 - import fcntl 15 14 import json 16 15 import logging 17 16 import os 18 - import random 19 17 import re 20 - import tempfile 21 - import time 22 18 from datetime import UTC, datetime 23 19 from pathlib import Path 24 20 from typing import Any 25 21 22 + from solstone.think.journal_io import atomic_replace, hold_lock 26 23 from solstone.think.utils import get_journal, segment_parse 27 24 28 25 logger = logging.getLogger(__name__) ··· 784 781 785 782 def _write_jsonl_records(path: Path, records: list[dict[str, Any]]) -> None: 786 783 """Atomically write JSONL entries to *path*.""" 787 - path.parent.mkdir(parents=True, exist_ok=True) 788 - fd, tmp_name = tempfile.mkstemp(dir=path.parent, suffix=".tmp") 789 - try: 790 - with os.fdopen(fd, "w", encoding="utf-8") as handle: 791 - for record in records: 792 - handle.write(json.dumps(record, ensure_ascii=False) + "\n") 793 - os.replace(tmp_name, path) 794 - except BaseException: 795 - if os.path.exists(tmp_name): 796 - os.unlink(tmp_name) 797 - raise 784 + atomic_replace( 785 + path, 786 + "".join(json.dumps(record, ensure_ascii=False) + "\n" for record in records), 787 + ) 798 788 799 789 800 790 def _fallback_activity_title(record: dict[str, Any]) -> str: ··· 835 825 modify_fn: Any, 836 826 *, 837 827 create_if_missing: bool = False, 838 - max_retries: int = 3, 839 828 ) -> None: 840 829 """Perform a locked load-modify-save cycle on a JSONL file.""" 841 - lock_path = path.parent / f"{path.name}.lock" 842 - 843 - last_error: OSError | None = None 844 - for attempt in range(max_retries): 845 - try: 846 - path.parent.mkdir(parents=True, exist_ok=True) 847 - with open(lock_path, "w", encoding="utf-8") as lock_file: 848 - fcntl.flock(lock_file, fcntl.LOCK_EX) 849 - try: 850 - existed = path.exists() 851 - if not existed and not create_if_missing: 852 - raise FileNotFoundError(path) 853 - current = _read_jsonl_records(path) if existed else [] 854 - updated = modify_fn([dict(item) for item in current]) 855 - if not isinstance(updated, list): 856 - raise TypeError("modify_fn must return list[dict]") 857 - if not existed and not updated: 858 - return 859 - if existed and updated == current: 860 - return 861 - _write_jsonl_records(path, updated) 862 - finally: 863 - fcntl.flock(lock_file, fcntl.LOCK_UN) 830 + with hold_lock(path): 831 + existed = path.exists() 832 + if not existed and not create_if_missing: 833 + raise FileNotFoundError(path) 834 + current = _read_jsonl_records(path) if existed else [] 835 + updated = modify_fn([dict(item) for item in current]) 836 + if not isinstance(updated, list): 837 + raise TypeError("modify_fn must return list[dict]") 838 + if not existed and not updated: 864 839 return 865 - except (FileNotFoundError, TypeError, ValueError): 866 - raise 867 - except OSError as exc: 868 - last_error = exc 869 - if attempt < max_retries - 1: 870 - time.sleep(random.uniform(0.05, 0.3) * (attempt + 1)) 871 - 872 - if last_error is not None: 873 - raise last_error 840 + if existed and updated == current: 841 + return 842 + _write_jsonl_records(path, updated) 874 843 875 844 876 845 def append_edit(
+1 -12
solstone/think/importers/shared.py
··· 436 436 Returns: 437 437 List of created file paths (absolute) 438 438 """ 439 - import tempfile 440 439 from collections import defaultdict 441 440 442 441 # Group entries by day (YYYYMMDD extracted from ts) ··· 505 504 lines.append(json.dumps(entry)) 506 505 content = "\n".join(lines) + "\n" 507 506 508 - # Atomic write: write to temp file, then rename 509 - fd, tmp_path = tempfile.mkstemp( 510 - dir=str(import_dir), suffix=".tmp", prefix="imported_" 511 - ) 512 - try: 513 - with os.fdopen(fd, "w", encoding="utf-8") as f: 514 - f.write(content) 515 - os.replace(tmp_path, str(out_path)) 516 - except BaseException: 517 - os.unlink(tmp_path) 518 - raise 507 + atomic_replace(out_path, content) 519 508 520 509 created.append(str(out_path)) 521 510 logger.info("Wrote %d entries to %s", len(day_entries), out_path)
+8 -31
solstone/think/link/paths.py
··· 31 31 from dataclasses import dataclass 32 32 from pathlib import Path 33 33 34 + from solstone.think.journal_io import write_json 34 35 from solstone.think.utils import get_journal 35 36 36 37 # Production spl-relay endpoint. Single source of truth — self-hosters ··· 146 147 return None 147 148 148 149 def save(self) -> None: 149 - path = state_path() 150 - path.parent.mkdir(parents=True, exist_ok=True) 151 - tmp = path.with_suffix(".json.tmp") 152 - with open(tmp, "w", encoding="utf-8") as f: 153 - json.dump( 154 - {"instance_id": self.instance_id, "home_label": self.home_label}, 155 - f, 156 - indent=2, 157 - ) 158 - f.write("\n") 159 - f.flush() 160 - os.fsync(f.fileno()) 161 - os.replace(tmp, path) 150 + write_json( 151 + state_path(), 152 + {"instance_id": self.instance_id, "home_label": self.home_label}, 153 + indent=2, 154 + ) 162 155 163 156 164 157 def load_service_token() -> str | None: ··· 178 171 def save_service_token(token: str) -> None: 179 172 """Persist the service token atomically with mode 0600.""" 180 173 path = service_token_path() 181 - path.parent.mkdir(parents=True, exist_ok=True) 182 - tmp = path.with_suffix(".json.tmp") 183 - with open(tmp, "w", encoding="utf-8") as f: 184 - json.dump({"service_token": token}, f, indent=2) 185 - f.write("\n") 186 - f.flush() 187 - os.fsync(f.fileno()) 188 - os.chmod(tmp, 0o600) 189 - os.replace(tmp, path) 174 + write_json(path, {"service_token": token}, indent=2, mode=0o600) 190 175 191 176 192 177 def generate_totp_secret() -> str: ··· 209 194 def save_totp_secret(secret: str) -> None: 210 195 """Persist the relay pairing TOTP secret atomically with mode 0600.""" 211 196 path = totp_secret_path() 212 - path.parent.mkdir(parents=True, exist_ok=True) 213 - tmp = path.with_suffix(".json.tmp") 214 - with open(tmp, "w", encoding="utf-8") as f: 215 - json.dump({"totp_secret": secret}, f, indent=2) 216 - f.write("\n") 217 - f.flush() 218 - os.fsync(f.fileno()) 219 - os.chmod(tmp, 0o600) 220 - os.replace(tmp, path) 197 + write_json(path, {"totp_secret": secret}, indent=2, mode=0o600)
+6 -2
tests/link/test_paths.py
··· 192 192 193 193 token_path = service_token_path() 194 194 assert token_path.exists() 195 - assert not any(path.name.endswith(".tmp") for path in token_path.parent.iterdir()) 195 + assert json.loads(token_path.read_text("utf-8")) == {"service_token": "tok.123"} 196 + assert token_path.stat().st_mode & 0o777 == 0o600 197 + assert {path.name for path in token_path.parent.iterdir()} == {token_path.name} 196 198 197 199 198 200 def test_load_service_token_reads_legacy_account_key( ··· 245 247 246 248 secret_path = totp_secret_path() 247 249 assert secret_path.exists() 248 - assert not any(path.name.endswith(".tmp") for path in secret_path.parent.iterdir()) 250 + assert json.loads(secret_path.read_text("utf-8")) == {"totp_secret": "SECRET"} 251 + assert secret_path.stat().st_mode & 0o777 == 0o600 252 + assert {path.name for path in secret_path.parent.iterdir()} == {secret_path.name} 249 253 250 254 251 255 def test_generate_totp_secret_shape() -> None:
+8 -3
tests/test_activities_locking.py
··· 58 58 target=worker, args=("cli:update", "first writer"), kwargs={"hold_lock": True} 59 59 ) 60 60 second = threading.Thread( 61 - target=worker, args=("cli:mute", "second writer"), kwargs={"hold_lock": False} 61 + target=worker, 62 + args=("cli:mute", "second writer café"), 63 + kwargs={"hold_lock": False}, 62 64 ) 63 65 64 66 first.start() ··· 75 77 assert len(records) == 1 76 78 assert [edit["note"] for edit in records[0]["edits"]] == [ 77 79 "first writer", 78 - "second writer", 80 + "second writer café", 79 81 ] 80 82 81 - raw_lines = record_path.read_text(encoding="utf-8").splitlines() 83 + raw_text = record_path.read_text(encoding="utf-8") 84 + assert "café" in raw_text 85 + assert "\\u00e9" not in raw_text 86 + raw_lines = raw_text.splitlines() 82 87 assert len(raw_lines) == 1 83 88 assert json.loads(raw_lines[0])["edits"][1]["actor"] == "cli:mute"
+245
tests/test_check_journal_io_mechanic.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + """Self-test for scripts/check_journal_io_mechanic.py.""" 5 + 6 + from __future__ import annotations 7 + 8 + import importlib.util 9 + import subprocess 10 + import sys 11 + from pathlib import Path 12 + 13 + import pytest 14 + 15 + REPO_ROOT = Path(__file__).resolve().parents[1] 16 + SCRIPT = REPO_ROOT / "scripts" / "check_journal_io_mechanic.py" 17 + 18 + 19 + def _load_checker(): 20 + spec = importlib.util.spec_from_file_location("check_journal_io_mechanic", SCRIPT) 21 + assert spec and spec.loader 22 + module = importlib.util.module_from_spec(spec) 23 + spec.loader.exec_module(module) 24 + return module 25 + 26 + 27 + cjm = _load_checker() 28 + 29 + 30 + BAD_OS_REPLACE = """\ 31 + # SPDX-License-Identifier: AGPL-3.0-only 32 + # Copyright (c) 2026 sol pbc 33 + import os 34 + 35 + 36 + def persist(tmp, path): 37 + os.replace(tmp, path) 38 + """ 39 + 40 + OWNER_OS_REPLACE = """\ 41 + # SPDX-License-Identifier: AGPL-3.0-only 42 + # Copyright (c) 2026 sol pbc 43 + import os 44 + 45 + 46 + def persist(tmp, path): 47 + os.replace(tmp, path) 48 + """ 49 + 50 + HOME_OS_REPLACE = """\ 51 + # SPDX-License-Identifier: AGPL-3.0-only 52 + # Copyright (c) 2026 sol pbc 53 + import os 54 + 55 + 56 + def atomic(tmp, path): 57 + os.replace(tmp, path) 58 + """ 59 + 60 + EXCLUDED_OS_REPLACE = """\ 61 + # SPDX-License-Identifier: AGPL-3.0-only 62 + # Copyright (c) 2026 sol pbc 63 + import os 64 + 65 + 66 + def persist(tmp, path): 67 + os.replace(tmp, path) 68 + """ 69 + 70 + TEST_OS_REPLACE = """\ 71 + # SPDX-License-Identifier: AGPL-3.0-only 72 + # Copyright (c) 2026 sol pbc 73 + import os 74 + 75 + 76 + def test_raw_replace(tmp_path): 77 + os.replace(tmp_path / "a", tmp_path / "b") 78 + """ 79 + 80 + 81 + def _write_file(root: Path, rel: str, content: str) -> None: 82 + path = root / rel 83 + path.parent.mkdir(parents=True, exist_ok=True) 84 + path.write_text(content, encoding="utf-8") 85 + 86 + 87 + @pytest.fixture 88 + def bad_root(tmp_path) -> Path: 89 + root = tmp_path / "bad" 90 + _write_file(root, "solstone/apps/badapp/routes.py", BAD_OS_REPLACE) 91 + return root 92 + 93 + 94 + @pytest.fixture 95 + def owner_root(tmp_path) -> Path: 96 + root = tmp_path / "owner" 97 + _write_file(root, "solstone/think/entities/saving.py", OWNER_OS_REPLACE) 98 + return root 99 + 100 + 101 + @pytest.fixture 102 + def home_root(tmp_path) -> Path: 103 + root = tmp_path / "home" 104 + _write_file(root, "solstone/think/journal_io/custom.py", HOME_OS_REPLACE) 105 + return root 106 + 107 + 108 + @pytest.fixture 109 + def excluded_root(tmp_path) -> Path: 110 + root = tmp_path / "excluded" 111 + _write_file(root, "solstone/think/scheduler.py", EXCLUDED_OS_REPLACE) 112 + return root 113 + 114 + 115 + @pytest.fixture 116 + def test_root(tmp_path) -> Path: 117 + root = tmp_path / "test" 118 + _write_file(root, "solstone/apps/badapp/test_raw.py", TEST_OS_REPLACE) 119 + return root 120 + 121 + 122 + def _run(root: Path) -> subprocess.CompletedProcess: 123 + return subprocess.run( 124 + [sys.executable, str(SCRIPT), "--root", str(root)], 125 + capture_output=True, 126 + text=True, 127 + ) 128 + 129 + 130 + @pytest.mark.parametrize( 131 + ("source", "kind"), 132 + [ 133 + ( 134 + "import os\n\ndef persist(tmp, path):\n os.replace(tmp, path)\n", 135 + "os.replace", 136 + ), 137 + ( 138 + "import os as ops\n\ndef persist(tmp, path):\n ops.replace(tmp, path)\n", 139 + "os.replace", 140 + ), 141 + ( 142 + "from os import replace as swap\n\ndef persist(tmp, path):\n swap(tmp, path)\n", 143 + "os.replace", 144 + ), 145 + ("def persist(tmp, path):\n tmp.replace(path)\n", "Path.replace"), 146 + ( 147 + "def persist(dest):\n dest.with_suffix('.tmp').replace(dest)\n", 148 + "Path.replace", 149 + ), 150 + ( 151 + "import tempfile\n\n" 152 + "def persist(path):\n" 153 + " fd, tmp_path = tempfile.mkstemp(dir=path.parent)\n" 154 + " tmp_path.replace(path)\n", 155 + "Path.replace", 156 + ), 157 + ( 158 + "import fcntl\n\ndef lock(f):\n fcntl.flock(f, fcntl.LOCK_EX)\n", 159 + "flock(LOCK_EX)", 160 + ), 161 + ( 162 + "from fcntl import flock, LOCK_EX\n\ndef lock(f):\n flock(f, LOCK_EX)\n", 163 + "flock(LOCK_EX)", 164 + ), 165 + ], 166 + ) 167 + def test_scan_source_flags_raw_mechanics(source: str, kind: str) -> None: 168 + findings = cjm.scan_source(source) 169 + assert [finding[1] for finding in findings] == [kind] 170 + 171 + 172 + @pytest.mark.parametrize( 173 + "source", 174 + [ 175 + "def clean(s):\n return s.replace('a', 'b')\n", 176 + "def clean(dt, tz):\n return dt.replace(tzinfo=tz)\n", 177 + "def clean(path):\n return path.name.replace('_', ' ')\n", 178 + "import fcntl\n\ndef lock(f):\n fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)\n", 179 + "import fcntl\n\ndef unlock(f):\n fcntl.flock(f, fcntl.LOCK_UN)\n", 180 + ( 181 + "import tempfile\n" 182 + "from solstone.think.journal_io import install_file\n\n" 183 + "def persist(dest):\n" 184 + " with tempfile.NamedTemporaryFile(delete=False) as tmp:\n" 185 + " tmp.write(b'ok')\n" 186 + " install_file(tmp.name, dest)\n" 187 + ), 188 + ], 189 + ) 190 + def test_scan_source_ignores_false_positive_surfaces(source: str) -> None: 191 + assert cjm.scan_source(source) == [] 192 + 193 + 194 + def test_bad_module_exits_one_and_names_file_and_kind(bad_root): 195 + result = _run(bad_root) 196 + assert result.returncode == 1, result.stdout + result.stderr 197 + assert "solstone/apps/badapp/routes.py" in result.stderr 198 + assert "os.replace" in result.stderr 199 + 200 + 201 + def test_owner_path_is_scanned(owner_root): 202 + result = _run(owner_root) 203 + assert result.returncode == 1, result.stdout + result.stderr 204 + assert "solstone/think/entities/saving.py" in result.stderr 205 + assert "os.replace" in result.stderr 206 + 207 + 208 + def test_journal_io_home_is_skipped(home_root): 209 + result = _run(home_root) 210 + assert result.returncode == 0, result.stdout + result.stderr 211 + assert "journal-io-mechanic: pass" in result.stdout 212 + 213 + 214 + def test_excluded_ops_path_is_skipped(excluded_root): 215 + result = _run(excluded_root) 216 + assert result.returncode == 0, result.stdout + result.stderr 217 + assert "journal-io-mechanic: pass" in result.stdout 218 + 219 + 220 + def test_test_modules_are_skipped(test_root): 221 + result = _run(test_root) 222 + assert result.returncode == 0, result.stdout + result.stderr 223 + assert "journal-io-mechanic: pass" in result.stdout 224 + 225 + 226 + def test_ratchet_by_file_kind_count(bad_root): 227 + new, tracked = cjm.evaluate(bad_root, {}) 228 + assert new 229 + assert tracked == [] 230 + 231 + counts = cjm.count_violations(bad_root) 232 + new_exact, tracked_exact = cjm.evaluate(bad_root, counts) 233 + assert new_exact == [] 234 + assert tracked_exact 235 + 236 + key = next(iter(counts)) 237 + ratcheted = dict(counts) 238 + ratcheted[key] = counts[key] - 1 239 + new_over, _ = cjm.evaluate(bad_root, ratcheted) 240 + assert new_over 241 + 242 + 243 + def test_repo_tree_is_green(): 244 + result = _run(REPO_ROOT) 245 + assert result.returncode == 0, result.stdout + result.stderr