solstone Diagnostic Guide#

Quick reference for debugging and diagnosing issues. For detailed specifications, see linked documentation.

Quick Health Check#

# Check if supervisor services are running
pgrep -af "sol:observer|sol:sense|sol:supervisor"

# Check Callosum socket exists
ls -la journal/health/callosum.sock

# Check for stuck agents (should be empty or short-lived)
ls journal/talents/*/*_active.jsonl 2>/dev/null

Healthy state:

• All three processes running
• callosum.sock exists
• supervisor.status events show no stale heartbeats
• No _active.jsonl files older than a few minutes

Diagnostic Commands#

Use the diagnostic command that matches the question:

• sol doctor — is the CLI usable on this machine? This universal battery is safe on a journal-less or repo-less box.
• journal doctor — is this journal host healthy, and what should be fixed? This is the health diagnosis view.
• make preflight — can a fresh source checkout get ready before .venv or uv exist?
• journal health — what live supervisor status is being reported right now?

journal-prefixed commands, including journal doctor and journal setup, require a journal-host install because the journal executable ships in the solstone-journal distribution, not in the thin sol client.

sol doctor runs four checks:

journal doctor runs the journal-host battery:

host_dependencies fix guidance is: Reinstall the journal host stack: pip install --upgrade solstone-journal | uv tool install --upgrade solstone-journal | pipx install --force solstone-journal. On an NVIDIA host use solstone-journal-cuda instead — never install both.

journal doctor is role-aware. If there is no local journal directory or no installed service, folder and service checks emit skip (no local journal or no local journal service) rather than failing. Invalid service config, service identity mismatch, crash loops, systemd failed state, and journal-sync conflicts are blocker failures. An installed service with no supervisor socket is a warning when the OS unit is not failed. Host dependency and feature checks report missing journal-host packaging pieces directly.

On macOS, supervisor_conflict fails when journal.app is running while the legacy org.solpbc.solstone LaunchAgent is installed or loaded, or when a foreign persistent LaunchAgent targets /Applications/solstone.app. The proven legacy remediation is journal service uninstall; foreign launcher findings include one-line remove foreign launchers targeting /Applications/solstone.app commands for the matching plists. In a proven conflict, other diagnoses stay visible but their action strings point back to resolving the supervisor conflict first, so the report does not mix service creation, restart, setup, upgrade, or deletion advice with the conflict fix. If the topology or foreign-launcher scan is incomplete rather than proven, only service lifecycle actions are withheld until it can be determined.

journal setup step 1 runs journal doctor --readiness: the client readiness checks (python_version, sol_importable, local_bin_sol_reachable, stale_alias_symlink, disk_space, journal_dir_writable) plus host_dependencies, default_stt_ready, feature:pdf-import, feature:pdf-export, and feature:whisper. It does not run runtime service, sync, config-dir, or launchd checks. A blocker failure still stops setup early; feature advisories stay advisory and include the exact extra-install command.

make preflight runs scripts/preflight.py, the stdlib-only source-checkout readiness battery that is valid before .venv/uv exist: python_version, uv_installed, venv_consistent, local_bin_sol_reachable, disk_space, and config_dir_readable. It shares probe primitives with doctor through solstone/think/probe.py, but its behavior is unchanged.

Service Architecture#

The supervisor (journal supervisor) manages these services:

Cortex (agent execution) connects to Callosum but runs independently via journal cortex.

See CALLOSUM.md for message protocol and CORTEX.md for agent system.

Log Locations#

Symlink structure: Journal-level symlinks point to current day's logs. Day-level symlinks point to current process instance (by ref).

# Tail current observer log
tail -f journal/health/observer.log

# Find today's logs
ls -la journal/$(date +%Y%m%d)/health/

Health Signals#

Health uses a fail-fast model: observers exit if they detect problems, and supervisor restarts them. Health is simply whether the observer is running and sending status events.

journal doctor also runs the observer_ingest_health advisory check. It warns when the journal has recorded an active observer ingest rejection, but never blocks. Remediation is to update or restart the observer, then confirm a valid upload clears the active rejection.

Both signals track the same thing: is the observer alive and communicating? If the observer has capture problems (e.g., screencast files not growing), it exits gracefully and supervisor restarts it.

Staleness threshold: 60 seconds (configurable via --threshold).

Callosum Status Events#

Services emit periodic status to Callosum. Most emit every 5 seconds when active:

• observe.status - Capture state (screencast, audio, activity)
• cortex.status - Running agents list
• supervisor.status - Service health, stale heartbeats

The native observe.status event also carries a diagnostics-only health beacon with the allowlisted fields name, stream_type, version, uptime, last_successful_sync, pending_queue_depth, recent_error_count, and last_error_reason. It is emitted at startup and every 5 seconds, including when healthy-idle, contains no captured content or file paths, and is distinct from platform upload observer beacons and journal-detected ingest rejections.

The supervisor checks for observe.status event freshness and includes stale_heartbeats in its own status.

See CALLOSUM.md Tract Registry for event schemas.

Reading Agent Files#

Location: journal/talents/

File states:

• {name}/{timestamp}_active.jsonl - Agent currently running
• {name}/{timestamp}.jsonl - Agent completed

Event sequence (JSONL, one event per line):

1. request - Initial spawn request (prompt, provider, name)
2. start - Agent began execution (model info)
3. tool_start/tool_end - Tool calls (paired by call_id)
4. thinking - Model reasoning (if supported)
5. finish or error - Final result or failure

# View an agent's final result
jq -r 'select(.event=="finish") | .result' journal/talents/default/1234567890123.jsonl

# List agents in today's journal-day index with their prompts
for id in $(jq -r '.use_id' journal/talents/$(date +%Y%m%d).jsonl 2>/dev/null); do
  f=$(find journal/agents -maxdepth 2 -path "*/${id}.jsonl" -print -quit)
  [ -n "$f" ] || continue
  echo "=== $(basename "$f") ==="
  head -1 "$f" | jq -r '.prompt[:80]'
done

See CORTEX.md for complete event schemas and agent configuration.

Common Issues#

Observer not capturing#

# Check observer log for errors
tail -50 journal/health/observer.log | grep -i error

# Check if observer is emitting status (supervisor.status will show stale_heartbeats)
# Health is derived from solstone.observe.status Callosum events

Causes: DBus issues, screencast permissions, audio device unavailable.

Agent appears stuck#

# Find active agents
ls -la journal/talents/*/*_active.jsonl

# Check last event in active agent
tail -1 journal/talents/*/*_active.jsonl | jq .

Causes: Backend timeout, tool hanging, network issues.

No Callosum events#

# Verify socket exists
ls -la journal/health/callosum.sock

# Check supervisor is running
pgrep -af sol:supervisor

Causes: Supervisor not started, socket path permissions.

Processing backlog#

# Check sense log for queue status
grep -i "queue" journal/health/sense.log | tail -10

Causes: Slow transcription, describe API rate limits.

Useful Commands#

# Watch all service logs
tail -f journal/health/*.log

# Count entries in today's journal-day index by status
echo "Completed: $([ -f journal/talents/$(date +%Y%m%d).jsonl ] && wc -l < journal/talents/$(date +%Y%m%d).jsonl || echo 0)"
echo "Running: $(ls journal/talents/*/*_active.jsonl 2>/dev/null | wc -l)"

# Find agents that errored on today's local execution day
jq -r --arg today "$(date +%Y%m%d)" '
  (.ts | select(type != "boolean") | tonumber?) as $ts
  | select($ts > 0)
  | select((($ts / 1000) | localtime | strftime("%Y%m%d")) == $today)
  | select(.status == "error")
  | .use_id
' journal/talents/????????.jsonl 2>/dev/null

# Check token usage for today
wc -l journal/tokens/$(date +%Y%m%d).jsonl

# Find errors in today's logs
grep -i error journal/$(date +%Y%m%d)/health/*.log

# Watch Callosum events in real-time
socat - UNIX-CONNECT:journal/health/callosum.sock

Recovery Playbooks#

Unfinalized MOV Files (Missing moov Atom)#

Symptoms: journal describe fails with av.error.InvalidDataError: Invalid data found when processing input. Sense logs show describe failed ... exit code 1 and Segment observed with errors ... ['describe exit 1'].

Diagnosis: The .mov file has ftyp + wide + mdat atoms but is missing the moov atom. The mdat size is 0 (extends-to-EOF). This means the screen recorder (solstone-macos native app) never finalized the file — it wrote video frames but crashed or was interrupted before writing the metadata index.

Known trigger: screen sharing active during solstone-macos native app capture causes AVAssetWriter finalization to be skipped (missing endSession() call in VideoWriter.swift).

# Confirm the issue — should report "moov atom not found"
ffprobe -v error journal/YYYYMMDD/STREAM/SEGMENT/center_1_screen.mov

# Inspect atom structure (moov should be present but isn't)
python3 -c "
import struct, os, sys
path = sys.argv[1]
size = os.path.getsize(path)
pos = 0
with open(path, 'rb') as f:
    while pos < size:
        f.seek(pos)
        header = f.read(8)
        if len(header) < 8: break
        atom_size, atom_type = struct.unpack('>I4s', header)
        atom_type = atom_type.decode('ascii', errors='replace')
        flag = '  [extends-to-EOF]' if atom_size == 0 else ''
        if atom_size == 0: atom_size = size - pos
        print(f'  {atom_type:6s} {atom_size:>12,} bytes{flag}')
        pos += atom_size
" /path/to/broken.mov

Recovery: Extract HEVC parameter sets (VPS/SPS/PPS) from a working sibling file's hvcC box, convert the broken file's length-prefixed NALUs to Annex B format, and remux with ffmpeg.

Prerequisites: a good .mov from the same stream/session (same codec settings), Python 3, ffmpeg.

# Step 1: Extract VPS/SPS/PPS from a good reference file
python3 -c "
import struct, os, sys

def find_atom(data, name, offset=0):
    pos = offset
    while pos < len(data) - 8:
        size = struct.unpack('>I', data[pos:pos+4])[0]
        atype = data[pos+4:pos+8]
        if size < 8: break
        if atype == name: return pos, size
        if atype in (b'moov', b'trak', b'mdia', b'minf', b'stbl'):
            result = find_atom(data, name, pos + 8)
            if result: return result
        pos += size
    return None

with open(sys.argv[1], 'rb') as f:
    data = f.read()
pos, size = find_atom(data, b'stsd')
stsd = data[pos:pos+size]
hvcc_off = stsd.find(b'hvcC')
hvcc_size = struct.unpack('>I', stsd[hvcc_off-4:hvcc_off])[0]
cfg = stsd[hvcc_off-4+8:hvcc_off-4+hvcc_size]
offset = 23
with open('/tmp/hevc_params.bin', 'wb') as pf:
    for i in range(cfg[22]):
        num = struct.unpack('>H', cfg[offset+1:offset+3])[0]
        offset += 3
        for j in range(num):
            nalu_len = struct.unpack('>H', cfg[offset:offset+2])[0]
            pf.write(b'\x00\x00\x00\x01')
            pf.write(cfg[offset+2:offset+2+nalu_len])
            offset += 2 + nalu_len
print('Wrote parameter sets to /tmp/hevc_params.bin')
" /path/to/good_reference.mov

# Step 2: Convert broken file to Annex B and remux
python3 -c "
import struct, os, subprocess, sys

src, dst, seg_duration = sys.argv[1], sys.argv[2], int(sys.argv[3])
fsize = os.path.getsize(src)
mdat_offset = 36  # ftyp(20) + wide(8) + mdat_header(8)

with open('/tmp/hevc_params.bin', 'rb') as pf:
    params = pf.read()

annex_b = '/tmp/recovery_raw.h265'
frame_count = 0
with open(src, 'rb') as fin, open(annex_b, 'wb') as fout:
    fout.write(params)
    fin.seek(mdat_offset)
    bytes_read = 0
    mdat_size = fsize - mdat_offset
    while bytes_read < mdat_size - 4:
        lb = fin.read(4)
        if len(lb) < 4: break
        nalu_len = struct.unpack('>I', lb)[0]
        if nalu_len <= 0 or nalu_len > mdat_size - bytes_read: break
        nalu_data = fin.read(nalu_len)
        if len(nalu_data) < nalu_len: break
        nal_type = (nalu_data[0] >> 1) & 0x3f
        if nal_type < 32: frame_count += 1
        fout.write(b'\x00\x00\x00\x01')
        fout.write(nalu_data)
        bytes_read += 4 + nalu_len

fps = f'{frame_count}/{seg_duration}'
print(f'{frame_count} frames, {fps} fps')
subprocess.run(['ffmpeg', '-y', '-v', 'warning', '-r', fps,
    '-f', 'hevc', '-i', annex_b, '-c', 'copy',
    '-movflags', '+faststart', '-tag:v', 'hvc1', dst], check=True)
os.unlink(annex_b)
print(f'Recovered: {dst}')
" /path/to/broken.mov /path/to/recovered.mov DURATION_SECS

# Step 3: Verify recovery
ffprobe -v error -show_streams /path/to/recovered.mov
# Should show codec_name=hevc, correct width/height/duration

# Step 4: Replace original and re-run describe
cp /path/to/recovered.mov /path/to/broken.mov
journal describe /path/to/broken.mov -v

Notes:

• The segment duration (DURATION_SECS) comes from the segment folder name (HHMMSS_LEN — LEN is duration in seconds)
• The reference file must be from the same stream/session so codec parameters match
• PyAV (used by journal describe) bundles its own HEVC decoder, so this works even if system ffmpeg lacks one
• After recovery, run journal indexer if you need the new screen extracts searchable

See Also#

• logs.md - Journal logs, health files, and event formats
• CORTEX.md - Agent system, events, configuration
• CALLOSUM.md - Message bus protocol