solpbc.org / solstone
personal memory agent
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

feat(providers): install nvattest appraiser with honest attestation reasons

Install the pinned sol pbc nvattest appraiser into the journal provider cache, pass its bundled CA bundle to the GPU appraiser, and surface nvattest-specific confidential attestation states through the brain and Thinking payload instead of collapsing them to generic rejection or stale states.

Proven by tests:
- linux-x86_64 archive pin resolves to the exact sol pbc URL/sha256/version; unsupported platform yields a typed outcome with no network attempt; sha mismatch refuses and leaves no partial tree; valid sidecar is a no-op; the old NVIDIA 1.2.2 sidecar upgrades in place to 1.2.2-sol.1.
- a fresh verify triggers ensure-install before attestation (demonstrated red on the unpatched tree: `AttributeError: module ... has no attribute 'ensure_nvattest_installed'`).
- the nvattest argv carries `--ca-bundle <dir>/share/ca/ca-bundle.pem`; a tree missing it yields `nvattest_integrity_failed` rather than an argv without the flag.
- all five nvattest reasons arrive intact in `active_lane.confidential_attestation` at the payload level with `nvattest_install_in_progress` presenting as `verifying` and the rest as `failed` (demonstrated red on the unpatched tree: the reason collapsed to `state: stale`).
- concurrent ensure-install collapses to one install via a non-blocking cache-local lock; lock contention presents as in-progress, never failure.
- with no confidential provenance, no install is attempted and `cache/providers/nvattest/` is not created.
- GPU-leg appraisal failures log a fixed-width 16-hex stderr fingerprint plus length; raw stderr is never logged.
- `make ci` green: 14530 passed, 16 skipped.

Explicitly deferred, not proven here:
- live end-to-end verify against the production engine (VPE-direct, post-ship). No real nvattest binary, no real GPU evidence, and no real engine was exercised — every appraisal in the suite is a mocked subprocess.
- macOS/aarch64 artifacts (the platform map takes a one-line data addition once published).
- owner-facing copy for the new in-progress / install-failed / unsupported-platform conditions (VPX owns the experience pass; this lode only extends the data).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

author
Jer Miller co-author
Claude Opus 4.8 (1M context) date (Jul 21, 2026, 11:01 PM -0600) commit dd64fa1a parent c8c078c8
+885 -77
+1
AGENTS.md
··· 250 250 | Provider install leases (`health/providers/{local,parakeet}.lease`) | `solstone/think/providers/install_lease.py` | 251 251 | Provider runtime health and retry-token records (`health/providers/runtime/{local,parakeet}.json`, `health/providers/runtime/{local,parakeet}.retry-token.json`, `health/providers/runtime/{local,parakeet}.operation.lock`) | `solstone/think/providers/runtime_health.py` | 252 252 | Provider artifact manifests (`cache/providers/**/.solstone-provider-manifest.json`, `cache/providers/local/mlx/**/*.manifest.json`) | `solstone/think/providers/artifact_proof.py` | 253 + | nvattest appraiser cache (`cache/providers/nvattest/**`) | `solstone/think/providers/nvattest_install.py` | 253 254 | Media offload ledger (`health/offload/<YYYYMMDD>.jsonl`) | `solstone/think/offload_ledger.py` | 254 255 | Parakeet server placement record (`health/parakeet-cpp.placement`) | `solstone/think/providers/parakeet_server.py` | 255 256 | Hosted backup binding (`backup/hosted/binding.json`) | `solstone/think/backup/hosted.py` |
+1
CHANGELOG.md
··· 15 15 ### Fixed 16 16 17 17 - a model reply that comes back empty is now a failure everywhere: key and model checks, thinking-engine health, and daily analysis all agree, so one path no longer accepts a blank answer another path rejects. 18 + - confidential processing now gets its nvattest appraiser on first use when the machine supports it, and Thinking names appraiser setup or integrity problems instead of showing a generic attestation rejection. 18 19 - choosing an exact Gemini model from Thinking's advisory now remembers it for when confidential processing is off, while sol keeps thinking with confidential processing now. 19 20 20 21 ## [0.9.0] - 2026-07-19
+2
scripts/check_journal_io_access.py
··· 149 149 # Provider install status, proof cache, and artifact manifests. 150 150 "solstone/think/providers/artifact_proof.py", 151 151 "solstone/think/providers/install_state.py", 152 + # Provider cache-local nvattest artifacts and install single-flight lock. 153 + "solstone/think/providers/nvattest_install.py", 152 154 "solstone/think/providers/runtime_health.py", 153 155 "solstone/think/schedule_config.py", 154 156 "solstone/think/push/devices.py",
+7 -2
scripts/spp_ratls_loopback_e2e.py
··· 92 92 parser.add_argument( 93 93 "--nvattest-dir", 94 94 type=Path, 95 - help="nvattest install root containing bin/nvattest and lib/ (requires --real)", 95 + help=( 96 + "nvattest install root containing bin/nvattest, lib/, and " 97 + "share/ca/ca-bundle.pem (requires --real)" 98 + ), 96 99 ) 97 100 parser.add_argument( 98 101 "--upstream-port", ··· 145 148 try: 146 149 locate_nvattest(args.nvattest_dir) 147 150 except GpuAppraisalError: 148 - parser.error("--nvattest-dir must contain bin/nvattest and lib/") 151 + parser.error( 152 + "--nvattest-dir must contain bin/nvattest, lib/, and share/ca/ca-bundle.pem" 153 + ) 149 154 150 155 return RealModeConfig( 151 156 nvattest_dir=args.nvattest_dir.resolve(),
+37
solstone/apps/thinking/tests/test_confidential_attestation_payload.py
··· 476 476 477 477 478 478 @pytest.mark.parametrize( 479 + ("reason", "aggregate", "component_status", "expected_state"), 480 + [ 481 + ("nvattest_install_failed", "unhealthy", "failed", "failed"), 482 + ("nvattest_platform_unsupported", "blocked", "blocked", "failed"), 483 + ("nvattest_unavailable", "blocked", "blocked", "failed"), 484 + ("nvattest_integrity_failed", "unhealthy", "failed", "failed"), 485 + ("nvattest_install_in_progress", "blocked", "blocked", "verifying"), 486 + ], 487 + ) 488 + def test_build_brain_presentation_preserves_nvattest_reason( 489 + monkeypatch, 490 + reason: str, 491 + aggregate: str, 492 + component_status: str, 493 + expected_state: str, 494 + ): 495 + inspection = _inspection( 496 + aggregate=aggregate, 497 + reason=reason, 498 + record=_record( 499 + lane_prerequisites=_component(component_status, reason), 500 + generate=_component("not_attempted", reason), 501 + cogitate=_component("not_attempted", reason), 502 + ), 503 + ) 504 + 505 + presentation = _build_presentation(monkeypatch, inspection, configured=True) 506 + 507 + assert presentation["confidential_attestation"] == { 508 + "state": expected_state, 509 + "reason": reason, 510 + "observed_at": NOW_ISO, 511 + "expires_at": None, 512 + } 513 + 514 + 515 + @pytest.mark.parametrize( 479 516 "inspection", 480 517 [ 481 518 _inspection(aggregate="checking", reason="brain_check_in_progress"),
+35 -4
solstone/think/brain_cli.py
··· 67 67 68 68 LOG = logging.getLogger("solstone.think.brain_cli") 69 69 70 + _SPP_ATTESTATION_FAILURE_REASON_TO_BRAIN_REASON = { 71 + "gateway_unreachable": "attestation_not_verified", 72 + "nvattest_install_in_progress": "nvattest_install_in_progress", 73 + "nvattest_platform_unsupported": "nvattest_platform_unsupported", 74 + "nvattest_unavailable": "nvattest_unavailable", 75 + "nvattest_install_failed": "nvattest_install_failed", 76 + "nvattest_integrity_failed": "nvattest_integrity_failed", 77 + "tls_handshake_failed": "attestation_rejected", 78 + "proof_http_failed": "attestation_rejected", 79 + "certificate_invalid": "attestation_rejected", 80 + "certificate_extension_missing": "attestation_rejected", 81 + "certificate_extension_not_critical": "attestation_rejected", 82 + "certificate_extension_invalid": "attestation_rejected", 83 + "certificate_evidence_invalid": "attestation_rejected", 84 + "nonce_mismatch": "attestation_rejected", 85 + "spki_mismatch": "attestation_rejected", 86 + "cpu_verification_failed": "attestation_rejected", 87 + "gpu_nonce_mismatch": "attestation_rejected", 88 + "gpu_appraisal_failed": "attestation_rejected", 89 + "composite_appraisal_failed": "attestation_rejected", 90 + "exporter_proof_invalid": "attestation_rejected", 91 + "exporter_mismatch": "attestation_rejected", 92 + "exporter_quote_failed": "attestation_rejected", 93 + "endpoint_invalid": "attestation_rejected", 94 + "unexpected_error": "attestation_rejected", 95 + } 96 + 70 97 RefreshOutcome = Literal["busy", "stale_expected_fingerprint", "lost_fence"] 71 98 _REFRESH_EXIT_3: frozenset[str] = frozenset( 72 99 {"busy", "stale_expected_fingerprint", "lost_fence"} ··· 405 432 406 433 state = spp.get_attestation_state() 407 434 if state.failure is not None: 408 - reason = ( 409 - "attestation_not_verified" 410 - if state.failure.kind == "unreachable" 411 - else "attestation_rejected" 435 + reason = _SPP_ATTESTATION_FAILURE_REASON_TO_BRAIN_REASON.get( 436 + state.failure.reason_code 412 437 ) 438 + if reason is None: 439 + LOG.warning( 440 + "event=spp_attestation_reason_unmapped raw_reason=%s", 441 + state.failure.reason_code, 442 + ) 443 + reason = "attestation_rejected" 413 444 return _failed_component(now, reason), reason 414 445 if state.session is None: 415 446 return _failed_component(
+19
solstone/think/brain_health.py
··· 450 450 "observed_at": observed_at, 451 451 "expires_at": expires_at, 452 452 } 453 + if reason == "nvattest_install_in_progress": 454 + return { 455 + "state": "verifying", 456 + "reason": reason, 457 + "observed_at": observed_at, 458 + "expires_at": expires_at, 459 + } 460 + if reason in { 461 + "nvattest_platform_unsupported", 462 + "nvattest_unavailable", 463 + "nvattest_install_failed", 464 + "nvattest_integrity_failed", 465 + }: 466 + return { 467 + "state": "failed", 468 + "reason": reason, 469 + "observed_at": observed_at, 470 + "expires_at": expires_at, 471 + } 453 472 if reason == "attestation_not_verified": 454 473 return { 455 474 "state": "unreachable",
+17 -2
solstone/think/providers/brain_state.py
··· 106 106 "local_runtime_not_ready", 107 107 "local_artifact_not_ready", 108 108 "attestation_not_verified", 109 + "nvattest_install_in_progress", 110 + "nvattest_platform_unsupported", 111 + "nvattest_unavailable", 109 112 "provider_key_invalid", 110 113 "model_not_found", 111 114 "provider_quota_exceeded", ··· 118 121 "cogitate_terminal_error", 119 122 "attestation_rejected", 120 123 "attestation_expired", 124 + "nvattest_install_failed", 125 + "nvattest_integrity_failed", 121 126 "local_server_unhealthy", 122 127 "configuration_invalid", 123 128 "fingerprint_key_unavailable", ··· 152 157 "local_runtime_not_ready": "blocked", 153 158 "local_artifact_not_ready": "blocked", 154 159 "attestation_not_verified": "blocked", 160 + "nvattest_install_in_progress": "blocked", 161 + "nvattest_platform_unsupported": "blocked", 162 + "nvattest_unavailable": "blocked", 155 163 "provider_key_invalid": "unhealthy", 156 164 "model_not_found": "unhealthy", 157 165 "provider_quota_exceeded": "unhealthy", ··· 164 172 "cogitate_terminal_error": "unhealthy", 165 173 "attestation_rejected": "unhealthy", 166 174 "attestation_expired": "unhealthy", 175 + "nvattest_install_failed": "unhealthy", 176 + "nvattest_integrity_failed": "unhealthy", 167 177 "local_server_unhealthy": "unhealthy", 168 178 "configuration_invalid": "unknown", 169 179 "fingerprint_key_unavailable": "unknown", ··· 201 211 "attestation_not_verified", 202 212 "attestation_rejected", 203 213 "attestation_expired", 214 + "nvattest_install_in_progress", 215 + "nvattest_platform_unsupported", 216 + "nvattest_unavailable", 217 + "nvattest_install_failed", 218 + "nvattest_integrity_failed", 204 219 "local_server_unhealthy", 205 220 "local_runtime_state_invalid", 206 221 "local_runtime_state_unavailable", ··· 266 281 _EVIDENCE_ALLOWED_REASON_CODES = frozenset().union( 267 282 *BRAIN_EVIDENCE_REASON_CODES.values() 268 283 ) 269 - if len(_EVIDENCE_ALLOWED_REASON_CODES) != 26: 270 - raise RuntimeError("brain evidence reason partition must contain 26 reasons") 284 + if len(_EVIDENCE_ALLOWED_REASON_CODES) != 31: 285 + raise RuntimeError("brain evidence reason partition must contain 31 reasons") 271 286 if len(BRAIN_PROJECTION_ONLY_REASON_CODES) != 10: 272 287 raise RuntimeError("brain projection-only reason partition must contain 10 reasons") 273 288 if _EVIDENCE_ALLOWED_REASON_CODES & BRAIN_PROJECTION_ONLY_REASON_CODES:
+142 -20
solstone/think/providers/nvattest_install.py
··· 11 11 import hashlib 12 12 import json 13 13 import os 14 + import platform 14 15 import shutil 15 16 import stat 17 + import sys 16 18 import tempfile 17 19 from dataclasses import dataclass 18 20 from pathlib import Path 21 + from typing import Literal 19 22 23 + from solstone.think.journal_io import LockTimeout 24 + from solstone.think.journal_io.locking import hold_lock 20 25 from solstone.think.providers.rfdetr_install import ( 21 26 RfdetrInstallError, 22 27 ) ··· 26 31 from solstone.think.utils import get_journal 27 32 28 33 SPP_NVATTEST_DIR_ENV = "SPP_NVATTEST_DIR" 29 - NVATTEST_VERSION = "1.2.2" 30 - NVATTEST_ARCHIVE_NAME = "libnvat-linux-x86_64-1.2.2.1780962352-archive.tar.xz" 31 - NVATTEST_ARCHIVE_URL = ( 32 - "https://developer.download.nvidia.com/compute/nvat/redist/libnvat/" 33 - f"linux-x86_64/{NVATTEST_ARCHIVE_NAME}" 34 - ) 35 - NVATTEST_ARCHIVE_SHA256 = ( 36 - "3f10da6fca794b7e3025c6645447947ec8bc45bcfde5b5b1d23241c7115630db" 37 - ) 38 34 SIDECAR_NAME = ".nvattest-install.json" 35 + CA_BUNDLE_RELATIVE_PATH = Path("share") / "ca" / "ca-bundle.pem" 36 + ENSURE_LOCK_TIMEOUT_S = 0.1 37 + ENSURE_LOCK_POLL_INTERVAL_S = 0.02 38 + 39 + NvattestArchiveKey = Literal["linux-x86_64"] 40 + NvattestEnsureStatus = Literal[ 41 + "already_installed", 42 + "installed", 43 + "install_in_flight", 44 + "install_failed", 45 + "platform_unsupported", 46 + ] 39 47 40 48 41 49 class NvattestInstallError(RuntimeError): ··· 75 83 ) 76 84 77 85 78 - NVATTEST_ARCHIVE_SPEC = NvattestArchiveSpec( 79 - version=NVATTEST_VERSION, 80 - url=NVATTEST_ARCHIVE_URL, 81 - archive_name=NVATTEST_ARCHIVE_NAME, 82 - sha256=NVATTEST_ARCHIVE_SHA256, 83 - ) 86 + @dataclass(frozen=True, slots=True) 87 + class NvattestEnsureResult: 88 + status: NvattestEnsureStatus 89 + nvattest_dir: Path | None = None 90 + reason_code: str | None = None 91 + detail: str | None = None 92 + 93 + 94 + NVATTEST_ARCHIVES: dict[NvattestArchiveKey, NvattestArchiveSpec] = { 95 + "linux-x86_64": NvattestArchiveSpec( 96 + version="1.2.2-sol.1", 97 + url=( 98 + "https://updates.solstone.app/providers/nvattest/" 99 + "libnvat-linux-x86_64-1.2.2-sol.1-archive.tar.xz" 100 + ), 101 + archive_name="libnvat-linux-x86_64-1.2.2-sol.1-archive.tar.xz", 102 + sha256="60ef75d1873e7129f03ea80d107d92b2ef216d2a8815958617b30d9c721d474a", 103 + ), 104 + } 105 + 106 + 107 + def nvattest_archive_key( 108 + os_name: str | None = None, 109 + arch: str | None = None, 110 + ) -> NvattestArchiveKey | None: 111 + if os_name is None: 112 + os_name = "linux" if sys.platform.startswith("linux") else sys.platform 113 + if arch is None: 114 + arch = platform.machine() 115 + normalized_arch = arch.lower() 116 + if os_name == "linux" and normalized_arch in {"amd64", "x64", "x86_64"}: 117 + return "linux-x86_64" 118 + return None 119 + 120 + 121 + def resolve_nvattest_archive_spec( 122 + archive_key: NvattestArchiveKey | None = None, 123 + ) -> NvattestArchiveSpec: 124 + resolved = archive_key or nvattest_archive_key() 125 + if resolved is None: 126 + raise NvattestInstallError( 127 + "platform_unsupported", 128 + "nvattest archive unsupported on this platform", 129 + ) 130 + return NVATTEST_ARCHIVES[resolved] 84 131 85 132 86 133 def cache_root(journal_path: str | Path | None = None) -> Path: ··· 103 150 return cache_root(journal_path) 104 151 105 152 153 + def ensure_nvattest_installed( 154 + *, 155 + explicit_override: str | Path | None = None, 156 + journal_path: str | Path | None = None, 157 + spec: NvattestArchiveSpec | None = None, 158 + lock_timeout: float = ENSURE_LOCK_TIMEOUT_S, 159 + ) -> NvattestEnsureResult: 160 + """Ensure the journal-cache nvattest install is ready without blocking peers.""" 161 + 162 + nvattest_dir = resolve_nvattest_dir( 163 + explicit_override, 164 + journal_path=journal_path, 165 + ) 166 + if explicit_override is not None or os.environ.get(SPP_NVATTEST_DIR_ENV): 167 + # Override layout validation stays in nvgpu.binary so appraiser reasons 168 + # still traverse binary -> composite -> ratls instead of install plumbing. 169 + return NvattestEnsureResult( 170 + status="already_installed", 171 + nvattest_dir=nvattest_dir, 172 + ) 173 + 174 + try: 175 + resolved_spec = spec or resolve_nvattest_archive_spec() 176 + except NvattestInstallError as exc: 177 + return NvattestEnsureResult( 178 + status="platform_unsupported", 179 + reason_code=exc.reason_code, 180 + detail=str(exc), 181 + ) 182 + 183 + try: 184 + with hold_lock( 185 + _install_lock_path(journal_path), 186 + timeout=lock_timeout, 187 + poll_interval=ENSURE_LOCK_POLL_INTERVAL_S, 188 + ): 189 + if _installed(nvattest_dir, resolved_spec): 190 + return NvattestEnsureResult( 191 + status="already_installed", 192 + nvattest_dir=nvattest_dir, 193 + ) 194 + try: 195 + installed = install_nvattest( 196 + spec=resolved_spec, 197 + journal_path=journal_path, 198 + ) 199 + except NvattestInstallError as exc: 200 + return NvattestEnsureResult( 201 + status="install_failed", 202 + nvattest_dir=nvattest_dir, 203 + reason_code=exc.reason_code, 204 + detail=str(exc), 205 + ) 206 + return NvattestEnsureResult(status="installed", nvattest_dir=installed) 207 + except LockTimeout: 208 + return NvattestEnsureResult( 209 + status="install_in_flight", 210 + nvattest_dir=nvattest_dir, 211 + reason_code="install-in-progress", 212 + ) 213 + 214 + 106 215 def install_nvattest( 107 216 *, 108 217 force: bool = False, 109 - spec: NvattestArchiveSpec = NVATTEST_ARCHIVE_SPEC, 218 + spec: NvattestArchiveSpec | None = None, 110 219 journal_path: str | Path | None = None, 111 220 ) -> Path: 112 221 """Download, verify, and install nvattest into the journal provider cache.""" 113 222 223 + spec = spec or resolve_nvattest_archive_spec() 114 224 root = cache_root(journal_path) 115 225 if not force and _installed(root, spec): 116 226 return root ··· 138 248 139 249 140 250 def _has_runtime_layout(root: Path) -> bool: 141 - return (root / "bin" / "nvattest").is_file() and (root / "lib").is_dir() 251 + return ( 252 + (root / "bin" / "nvattest").is_file() 253 + and (root / "lib").is_dir() 254 + and (root / CA_BUNDLE_RELATIVE_PATH).is_file() 255 + ) 142 256 143 257 144 258 def _installed(root: Path, spec: NvattestArchiveSpec) -> bool: ··· 161 275 journal_path: str | Path | None = None, 162 276 ) -> Path: 163 277 return cache_root(journal_path) / ".downloads" / spec.archive_name 278 + 279 + 280 + def _install_lock_path(journal_path: str | Path | None = None) -> Path: 281 + return cache_root(journal_path) / ".install" 164 282 165 283 166 284 def _sha256_file(path: Path) -> str: ··· 234 352 for path in extract_dir.rglob("nvattest") 235 353 if path.is_file() 236 354 and path.parent.name == "bin" 237 - and (path.parent.parent / "lib").is_dir() 355 + and _has_runtime_layout(path.parent.parent) 238 356 ] 239 357 if len(matches) != 1: 240 358 raise NvattestInstallError( ··· 247 365 def _install_extracted_tree(source: Path, root: Path) -> None: 248 366 binary = source / "bin" / "nvattest" 249 367 lib_dir = source / "lib" 250 - if not binary.is_file() or not lib_dir.is_dir(): 368 + ca_bundle = source / CA_BUNDLE_RELATIVE_PATH 369 + if not binary.is_file() or not lib_dir.is_dir() or not ca_bundle.is_file(): 251 370 raise NvattestInstallError( 252 371 "archive_layout_invalid", 253 - "extracted archive must contain bin/nvattest and lib/", 372 + ( 373 + "extracted archive must contain bin/nvattest, lib/, " 374 + "and share/ca/ca-bundle.pem" 375 + ), 254 376 ) 255 377 256 378 root.mkdir(parents=True, exist_ok=True)
+6 -1
solstone/think/services/spp_attest/composite.py
··· 29 29 log = logging.getLogger(__name__) 30 30 31 31 _GPU_REASONS = frozenset( 32 - {"nvattest_unavailable", "gpu_nonce_mismatch", "gpu_appraisal_failed"} 32 + { 33 + "nvattest_unavailable", 34 + "nvattest_integrity_failed", 35 + "gpu_nonce_mismatch", 36 + "gpu_appraisal_failed", 37 + } 33 38 ) 34 39 35 40
+76 -7
solstone/think/services/spp_attest/nvgpu/appraise.py
··· 5 5 6 6 from __future__ import annotations 7 7 8 + import hashlib 8 9 import json 10 + import logging 9 11 import subprocess 10 12 import tempfile 11 13 from pathlib import Path ··· 24 26 from solstone.think.services.spp_attest.nvgpu.evidence import to_nvattest_evidence 25 27 from solstone.think.services.spp_attest.snp import AppraisalStep 26 28 from solstone.think.services.spp_attest.tlv import GpuEnvelope 29 + 30 + log = logging.getLogger(__name__) 27 31 28 32 29 33 def appraise_gpu_leg( ··· 59 63 handle.write(json.dumps(evidence, sort_keys=True)) 60 64 handle.write("\n") 61 65 62 - command = build_nvattest_attest_command( 63 - nvattest_dir=nvattest_dir, 64 - evidence_file=evidence_path, 65 - owner_nonce=owner_nonce, 66 - rim_store=rim_store, 67 - rim_dir=rim_dir, 68 - ) 66 + try: 67 + command = build_nvattest_attest_command( 68 + nvattest_dir=nvattest_dir, 69 + evidence_file=evidence_path, 70 + owner_nonce=owner_nonce, 71 + rim_store=rim_store, 72 + rim_dir=rim_dir, 73 + ) 74 + except GpuAppraisalError as exc: 75 + _log_gpu_appraisal_failure( 76 + exc.reason, 77 + exception_class=type(exc).__name__, 78 + stderr=None, 79 + ) 80 + raise 69 81 try: 70 82 completed = subprocess.run( 71 83 command.argv, ··· 75 87 check=False, 76 88 ) 77 89 except OSError as exc: 90 + _log_gpu_appraisal_failure( 91 + "nvattest_unavailable", 92 + exception_class=type(exc).__name__, 93 + stderr=None, 94 + ) 78 95 raise GpuAppraisalError("nvattest_unavailable") from exc 79 96 80 97 try: 81 98 stdout_obj = parse_nvattest_stdout(completed.stdout) 82 99 except ValueError as exc: 100 + _log_gpu_appraisal_failure( 101 + "gpu_appraisal_failed", 102 + returncode=completed.returncode, 103 + stderr=completed.stderr, 104 + ) 83 105 raise GpuAppraisalError("gpu_appraisal_failed") from exc 84 106 85 107 decision = classify_nvattest_result( ··· 88 110 owner_nonce=owner_nonce, 89 111 ) 90 112 if not isinstance(decision, NvattestAcceptance): 113 + _log_gpu_appraisal_failure( 114 + decision.reason, 115 + returncode=completed.returncode, 116 + stderr=completed.stderr, 117 + ) 91 118 raise GpuAppraisalError(decision.reason) 92 119 93 120 steps = [ ··· 111 138 steps=steps, 112 139 ) 113 140 except ValueError as exc: 141 + _log_gpu_appraisal_failure( 142 + "gpu_appraisal_failed", 143 + returncode=completed.returncode, 144 + stderr=completed.stderr, 145 + ) 114 146 raise GpuAppraisalError("gpu_appraisal_failed") from exc 115 147 finally: 116 148 if evidence_path is not None: ··· 119 151 120 152 def _ok(name: str, detail: str) -> AppraisalStep: 121 153 return AppraisalStep(name=name, status="ok", detail=detail) 154 + 155 + 156 + def _log_gpu_appraisal_failure( 157 + reason_code: str, 158 + *, 159 + stderr: str | bytes | None, 160 + returncode: object | None = None, 161 + exception_class: str | None = None, 162 + ) -> None: 163 + stderr_bytes = _stderr_bytes(stderr) 164 + digest = hashlib.sha256(stderr_bytes).hexdigest()[:16] 165 + if exception_class is not None: 166 + log.warning( 167 + "event=nvattest_gpu_appraisal_failed reason=%s exception=%s " 168 + "stderr_len=%d stderr_sha256=%s", 169 + reason_code, 170 + exception_class, 171 + len(stderr_bytes), 172 + digest, 173 + ) 174 + return 175 + log.warning( 176 + "event=nvattest_gpu_appraisal_failed reason=%s returncode=%s " 177 + "stderr_len=%d stderr_sha256=%s", 178 + reason_code, 179 + returncode, 180 + len(stderr_bytes), 181 + digest, 182 + ) 183 + 184 + 185 + def _stderr_bytes(stderr: str | bytes | None) -> bytes: 186 + if stderr is None: 187 + return b"" 188 + if isinstance(stderr, bytes): 189 + return stderr 190 + return stderr.encode("utf-8", "surrogateescape")
+11 -4
solstone/think/services/spp_attest/nvgpu/binary.py
··· 12 12 from solstone.think.services.spp_attest.nvgpu.errors import GpuAppraisalError 13 13 from solstone.think.services.spp_attest.tlv import SPDM_NONCE_SIZE 14 14 15 + CA_BUNDLE_RELATIVE_PATH = Path("share") / "ca" / "ca-bundle.pem" 16 + 15 17 16 18 @dataclass(frozen=True, slots=True) 17 19 class NvattestCommand: ··· 19 21 env: dict[str, str] 20 22 21 23 22 - def locate_nvattest(nvattest_dir: Path) -> tuple[Path, Path]: 23 - """Return the nvattest binary and lib directory under an injected install dir.""" 24 + def locate_nvattest(nvattest_dir: Path) -> tuple[Path, Path, Path]: 25 + """Return the nvattest binary, lib directory, and CA bundle.""" 24 26 25 27 root = nvattest_dir.resolve() 26 28 binary = root / "bin" / "nvattest" 27 29 lib_dir = root / "lib" 30 + ca_bundle = root / CA_BUNDLE_RELATIVE_PATH 28 31 if not root.is_dir(): 29 32 raise GpuAppraisalError("nvattest_unavailable") 30 33 if not binary.is_file(): 31 34 raise GpuAppraisalError("nvattest_unavailable") 32 35 if not lib_dir.is_dir(): 33 36 raise GpuAppraisalError("nvattest_unavailable") 34 - return binary, lib_dir 37 + if not ca_bundle.is_file(): 38 + raise GpuAppraisalError("nvattest_integrity_failed") 39 + return binary, lib_dir, ca_bundle 35 40 36 41 37 42 def build_nvattest_attest_command( ··· 53 58 if rim_store == "remote" and rim_dir is not None: 54 59 raise ValueError("rim_dir is only valid when rim_store == 'dir'") 55 60 56 - binary, lib_dir = locate_nvattest(nvattest_dir) 61 + binary, lib_dir, ca_bundle = locate_nvattest(nvattest_dir) 57 62 argv = [ 58 63 str(binary), 59 64 "--format", ··· 69 74 "local", 70 75 "--rim-store", 71 76 rim_store, 77 + "--ca-bundle", 78 + str(ca_bundle), 72 79 ] 73 80 if rim_dir is not None: 74 81 argv.extend(["--rim-dir", str(rim_dir)])
+1
solstone/think/services/spp_attest/nvgpu/errors.py
··· 11 11 12 12 GpuAppraisalReason = Literal[ 13 13 "nvattest_unavailable", 14 + "nvattest_integrity_failed", 14 15 "gpu_nonce_mismatch", 15 16 "gpu_appraisal_failed", 16 17 ]
+2
solstone/think/services/spp_attest/ratls/verify.py
··· 126 126 code = "gpu_nonce_mismatch" 127 127 elif "nvattest_unavailable" in reason: 128 128 code = "nvattest_unavailable" 129 + elif "nvattest_integrity_failed" in reason: 130 + code = "nvattest_integrity_failed" 129 131 elif "gpu_appraisal_failed" in reason: 130 132 code = "gpu_appraisal_failed" 131 133 else:
+82 -22
solstone/think/services/spp_transport.py
··· 12 12 import threading 13 13 import time 14 14 from datetime import datetime, timezone 15 + from pathlib import Path 15 16 from typing import Any, Literal 16 17 from urllib.parse import urlsplit 17 18 18 19 from OpenSSL import SSL 19 20 20 21 from solstone.think.models import AttestationFailedError, AttestationStaleError 21 - from solstone.think.providers.nvattest_install import resolve_nvattest_dir 22 + from solstone.think.providers.nvattest_install import ( 23 + ensure_nvattest_installed, 24 + resolve_nvattest_dir, 25 + ) 22 26 from solstone.think.services import spp 23 27 from solstone.think.services.spp_attest.cadence import AttestationSession 24 28 from solstone.think.services.spp_attest.composite import verify_composite ··· 75 79 ) 76 80 77 81 82 + def _nvattest_prerequisite_failed( 83 + kind: Literal["failed", "unreachable"], 84 + reason_code: str, 85 + ) -> None: 86 + with _LOCK: 87 + _teardown_locked() 88 + _attestation_failed(kind, reason_code) 89 + 90 + 91 + def _ensure_nvattest_for_attestation(block: dict[str, Any]) -> Path | None: 92 + if spp.confidential_provenance() is None: 93 + return None 94 + result = ensure_nvattest_installed(explicit_override=block.get("nvattest_dir")) 95 + if result.status in {"already_installed", "installed"}: 96 + return result.nvattest_dir 97 + if result.status == "install_in_flight": 98 + # Another process owns appraiser acquisition; evidence has not been rejected. 99 + _nvattest_prerequisite_failed("unreachable", "nvattest_install_in_progress") 100 + if result.status == "platform_unsupported": 101 + # This host cannot acquire the appraiser archive, so attestation cannot pass. 102 + _nvattest_prerequisite_failed("failed", "nvattest_platform_unsupported") 103 + if result.status == "install_failed": 104 + # Local appraiser acquisition failed before evidence verification could run. 105 + _nvattest_prerequisite_failed("failed", "nvattest_install_failed") 106 + _nvattest_prerequisite_failed("failed", "unexpected_error") 107 + 108 + 78 109 def _endpoint_from_block(block: dict[str, Any]) -> RatlsEndpoint: 79 110 endpoint_url = str(block.get("endpoint_url") or "") 80 111 parsed = urlsplit(endpoint_url) ··· 174 205 thread.start() 175 206 176 207 177 - def _establish_channel_locked(block: dict[str, Any], now: datetime) -> AttestedChannel: 208 + def _establish_channel_locked( 209 + block: dict[str, Any], 210 + now: datetime, 211 + *, 212 + nvattest_dir: Path | None = None, 213 + ) -> AttestedChannel: 178 214 try: 179 215 endpoint = _endpoint_from_block(block) 180 216 return establish_attested_channel( 181 217 endpoint, 182 218 owner_nonce=secrets.token_bytes(OWNER_NONCE_BYTES), 183 - nvattest_dir=resolve_nvattest_dir(block.get("nvattest_dir")), 219 + nvattest_dir=nvattest_dir 220 + if nvattest_dir is not None 221 + else resolve_nvattest_dir(block.get("nvattest_dir")), 184 222 now=now, 185 223 composite_verifier=verify_composite, 186 224 monotonic_now=time.monotonic, ··· 202 240 _attestation_failed("failed", "unexpected_error") 203 241 204 242 205 - def _establish_and_record_locked(block: dict[str, Any], now: datetime) -> None: 206 - channel = _establish_channel_locked(block, now) 243 + def _establish_and_record_locked( 244 + block: dict[str, Any], 245 + now: datetime, 246 + *, 247 + nvattest_dir: Path | None = None, 248 + ) -> None: 249 + channel = _establish_channel_locked(block, now, nvattest_dir=nvattest_dir) 207 250 _start_listener_locked() 208 251 _POOL.append(channel) 209 252 spp.record_attestation_verified( ··· 216 259 ) 217 260 218 261 262 + def _reuse_or_raise_stale_locked(now: datetime) -> bool: 263 + state = spp.get_attestation_state() 264 + if ( 265 + state.session is not None 266 + and state.session.status(now) == "verified" 267 + and _transport_live_locked() 268 + ): 269 + return True 270 + if ( 271 + state.session is not None 272 + and state.session.status(now) != "verified" 273 + and _transport_live_locked() 274 + ): 275 + _teardown_locked() 276 + raise AttestationStaleError( 277 + "the confidential attestation cadence lapsed (attestation_stale)" 278 + ) 279 + return False 280 + 281 + 219 282 def verify_confidential_attestation(block: dict[str, Any]) -> None: 220 283 global _CONFIDENTIAL_BLOCK 221 284 222 285 now = datetime.now(timezone.utc) 223 286 with _LOCK: 224 287 _CONFIDENTIAL_BLOCK = dict(block) 225 - state = spp.get_attestation_state() 226 - if ( 227 - state.session is not None 228 - and state.session.status(now) == "verified" 229 - and _transport_live_locked() 230 - ): 288 + if _reuse_or_raise_stale_locked(now): 231 289 return 232 - if ( 233 - state.session is not None 234 - and state.session.status(now) != "verified" 235 - and _transport_live_locked() 236 - ): 237 - _teardown_locked() 238 - raise AttestationStaleError( 239 - "the confidential attestation cadence lapsed (attestation_stale)" 240 - ) 241 290 242 - _establish_and_record_locked(block, now) 291 + nvattest_dir = _ensure_nvattest_for_attestation(block) 292 + with _LOCK: 293 + _CONFIDENTIAL_BLOCK = dict(block) 294 + if _reuse_or_raise_stale_locked(now): 295 + return 296 + _establish_and_record_locked(block, now, nvattest_dir=nvattest_dir) 243 297 244 298 245 299 def confidential_egress_base_url(endpoint_base_url: str) -> str: ··· 295 349 if block is None: 296 350 return 297 351 now = datetime.now(timezone.utc) 352 + try: 353 + nvattest_dir = _ensure_nvattest_for_attestation(block) 354 + except AttestationFailedError: 355 + return 298 356 with _LOCK: 299 357 _teardown_locked() 300 358 spp.clear_attestation_state() 301 359 _CONFIDENTIAL_BLOCK = dict(block) 302 360 try: 303 - _establish_and_record_locked(block, now) 361 + _establish_and_record_locked(block, now, nvattest_dir=nvattest_dir) 304 362 except AttestationFailedError: 305 363 return 306 364 ··· 320 378 if _CONFIDENTIAL_BLOCK is None: 321 379 return _activate_channel_locked(channel) 322 380 if channel is None: 381 + # Public entry points run ensure-install before the first verified session; 382 + # this locked refill only opens an extra channel and must not download. 323 383 channel = _establish_channel_locked(_CONFIDENTIAL_BLOCK, now) 324 384 return _activate_channel_locked(channel) 325 385
+13 -5
tests/services/test_spp_attest_composite.py
··· 69 69 (root / "bin").mkdir(parents=True) 70 70 (root / "bin" / "nvattest").write_text("#!/bin/sh\n", encoding="utf-8") 71 71 (root / "lib").mkdir() 72 + (root / "share" / "ca").mkdir(parents=True) 73 + (root / "share" / "ca" / "ca-bundle.pem").write_text("ca\n", encoding="utf-8") 72 74 return root 73 75 74 76 ··· 345 347 _assert_owner_message_safe(exc_info.value) 346 348 347 349 348 - def test_verify_composite_rejects_gpu_unavailable_without_cpu_only_pass( 350 + @pytest.mark.parametrize( 351 + "reason_code", 352 + [ 353 + "nvattest_unavailable", 354 + "nvattest_integrity_failed", 355 + ], 356 + ) 357 + def test_verify_composite_rejects_gpu_appraiser_prerequisite_without_cpu_only_pass( 349 358 tmp_path: Path, 359 + reason_code: str, 350 360 ) -> None: 351 361 def unavailable_gpu_appraiser( 352 362 _envelope: GpuEnvelope, ··· 355 365 nvattest_dir: Path, 356 366 ) -> GpuAppraisal: 357 367 assert nvattest_dir 358 - raise GpuAppraisalError("nvattest_unavailable") 368 + raise GpuAppraisalError(reason_code) 359 369 360 370 with pytest.raises(AttestationFailedError) as exc_info: 361 371 verify_composite( ··· 368 378 gpu_appraiser=unavailable_gpu_appraiser, 369 379 ) 370 380 371 - assert exc_info.value.detail == ( 372 - "the GPU leg rejected the evidence (nvattest_unavailable)" 373 - ) 381 + assert exc_info.value.detail == f"the GPU leg rejected the evidence ({reason_code})" 374 382 _assert_owner_message_safe(exc_info.value)
+53
tests/services/test_spp_attest_nvgpu.py
··· 4 4 from __future__ import annotations 5 5 6 6 import base64 7 + import hashlib 7 8 import json 9 + import logging 8 10 import subprocess 9 11 from copy import deepcopy 10 12 from pathlib import Path ··· 51 53 (root / "bin").mkdir(parents=True, exist_ok=True) 52 54 (root / "bin" / "nvattest").write_text("#!/bin/sh\n", encoding="utf-8") 53 55 (root / "lib").mkdir(exist_ok=True) 56 + (root / "share" / "ca").mkdir(parents=True, exist_ok=True) 57 + (root / "share" / "ca" / "ca-bundle.pem").write_text("ca\n", encoding="utf-8") 54 58 return root 55 59 56 60 ··· 360 364 assert marker not in str(exc_info.value) 361 365 362 366 367 + def test_gpu_appraisal_failure_log_uses_bounded_stderr_digest( 368 + tmp_path: Path, 369 + monkeypatch: pytest.MonkeyPatch, 370 + caplog: pytest.LogCaptureFixture, 371 + ) -> None: 372 + stderr = "collector detail\n" * 5000 373 + caplog.set_level(logging.WARNING, logger=appraise_module.log.name) 374 + 375 + with pytest.raises(GpuAppraisalError): 376 + _run_appraisal_with_stdout( 377 + monkeypatch, 378 + tmp_path, 379 + "not json", 380 + stderr=stderr, 381 + ) 382 + 383 + messages = [ 384 + record.getMessage() 385 + for record in caplog.records 386 + if record.name == appraise_module.log.name 387 + and "event=nvattest_gpu_appraisal_failed" in record.getMessage() 388 + ] 389 + assert len(messages) == 1 390 + message = messages[0] 391 + fingerprint = message.rsplit("stderr_sha256=", 1)[1].split()[0] 392 + assert len(fingerprint) == 16 393 + assert fingerprint == hashlib.sha256(stderr.encode("utf-8")).hexdigest()[:16] 394 + assert f"stderr_len={len(stderr.encode('utf-8'))}" in message 395 + assert len(message) < 180 396 + assert "collector detail" not in message 397 + 398 + 363 399 def test_bool_false_returncode_rejects( 364 400 tmp_path: Path, 365 401 monkeypatch: pytest.MonkeyPatch, ··· 409 445 410 446 assert command.env["SPP_NVATTEST_PARENT_SENTINEL"] == "kept" 411 447 assert command.env["LD_LIBRARY_PATH"] == str(nvattest_dir / "lib") 448 + assert command.argv[command.argv.index("--ca-bundle") + 1] == str( 449 + nvattest_dir / "share" / "ca" / "ca-bundle.pem" 450 + ) 451 + 452 + 453 + def test_nvattest_command_rejects_missing_ca_bundle(tmp_path: Path) -> None: 454 + nvattest_dir = _fake_nvattest_dir(tmp_path) 455 + (nvattest_dir / "share" / "ca" / "ca-bundle.pem").unlink() 456 + 457 + with pytest.raises(GpuAppraisalError) as exc_info: 458 + build_nvattest_attest_command( 459 + nvattest_dir=nvattest_dir, 460 + evidence_file=tmp_path / "evidence.json", 461 + owner_nonce=_owner_nonce(), 462 + ) 463 + 464 + assert exc_info.value.reason == "nvattest_integrity_failed" 412 465 413 466 414 467 def test_nvattest_command_uses_absolute_install_paths_not_path_or_python_namespace(
+5
tests/services/test_spp_attest_purity.py
··· 165 165 (nvattest_dir / "bin").mkdir(parents=True) 166 166 (nvattest_dir / "bin" / "nvattest").write_text("#!/bin/sh\n", encoding="utf-8") 167 167 (nvattest_dir / "lib").mkdir() 168 + (nvattest_dir / "share" / "ca").mkdir(parents=True) 169 + (nvattest_dir / "share" / "ca" / "ca-bundle.pem").write_text( 170 + "ca\n", 171 + encoding="utf-8", 172 + ) 168 173 envelope = decode_gpu_envelope((FIXTURE_DIR / "gpu-envelope.tlv").read_bytes()) 169 174 owner_nonce = bytes.fromhex((FIXTURE_DIR / "nonce.hex").read_text().strip()) 170 175 observed: list[Path] = []
+20 -5
tests/services/test_spp_attest_ratls_verify.py
··· 220 220 assert exc_info.value.reason_code == "spki_mismatch" 221 221 222 222 223 - def test_verify_certificate_evidence_maps_composite_failure(tmp_path: Path) -> None: 223 + @pytest.mark.parametrize( 224 + ("detail", "expected_reason"), 225 + [ 226 + ( 227 + "the CPU leg rejected evidence (cpu_verification_failed)", 228 + "cpu_verification_failed", 229 + ), 230 + ( 231 + "the GPU leg rejected evidence (nvattest_integrity_failed)", 232 + "nvattest_integrity_failed", 233 + ), 234 + ], 235 + ) 236 + def test_verify_certificate_evidence_maps_composite_failure( 237 + tmp_path: Path, 238 + detail: str, 239 + expected_reason: str, 240 + ) -> None: 224 241 nonce = b"n" * 32 225 242 key, spki = _key_and_spki() 226 243 evidence = _evidence(nonce, spki) 227 244 228 245 def composite_verifier(_bundle, **_kwargs): 229 - raise AttestationFailedError( 230 - "the CPU leg rejected evidence (cpu_verification_failed)" 231 - ) 246 + raise AttestationFailedError(detail) 232 247 233 248 with pytest.raises(RatlsVerificationError) as exc_info: 234 249 verify_certificate_evidence( ··· 239 254 composite_verifier=composite_verifier, 240 255 ) 241 256 242 - assert exc_info.value.reason_code == "cpu_verification_failed" 257 + assert exc_info.value.reason_code == expected_reason 243 258 244 259 245 260 def test_verify_exporter_proof_binds_quote_to_exporter(monkeypatch) -> None:
+2
tests/services/test_spp_ratls_loopback_e2e.py
··· 21 21 (root / "bin").mkdir(parents=True) 22 22 (root / "bin" / "nvattest").write_text("#!/bin/sh\n", encoding="utf-8") 23 23 (root / "lib").mkdir() 24 + (root / "share" / "ca").mkdir(parents=True) 25 + (root / "share" / "ca" / "ca-bundle.pem").write_text("ca\n", encoding="utf-8") 24 26 return root 25 27 26 28
+98 -1
tests/services/test_spp_transport.py
··· 7 7 import time 8 8 from datetime import datetime, timedelta, timezone 9 9 from pathlib import Path 10 + from types import SimpleNamespace 10 11 from unittest.mock import Mock 11 12 12 13 import pytest ··· 59 60 60 61 61 62 @pytest.fixture(autouse=True) 62 - def _clear_transport_state(): 63 + def _clear_transport_state(monkeypatch: pytest.MonkeyPatch): 64 + monkeypatch.setattr( 65 + spp_transport, 66 + "ensure_nvattest_installed", 67 + lambda **_kwargs: SimpleNamespace( 68 + status="already_installed", 69 + nvattest_dir=Path("/tmp/solstone-nvattest-test"), 70 + reason_code=None, 71 + detail=None, 72 + ), 73 + ) 63 74 spp.delete_attestation_state() 64 75 spp_transport.teardown_confidential_transport() 65 76 yield ··· 331 342 "cpu_verification_failed", 332 343 "gpu_nonce_mismatch", 333 344 "nvattest_unavailable", 345 + "nvattest_integrity_failed", 334 346 "gpu_appraisal_failed", 335 347 "composite_appraisal_failed", 336 348 "exporter_proof_invalid", ··· 393 405 assert failure.reason_code == reason_code 394 406 395 407 408 + @pytest.mark.parametrize( 409 + ("status", "kind", "reason_code"), 410 + [ 411 + ("install_in_flight", "unreachable", "nvattest_install_in_progress"), 412 + ("platform_unsupported", "failed", "nvattest_platform_unsupported"), 413 + ("install_failed", "failed", "nvattest_install_failed"), 414 + ], 415 + ) 416 + def test_verify_confidential_attestation_records_nvattest_install_prerequisites( 417 + tmp_path: Path, 418 + monkeypatch: pytest.MonkeyPatch, 419 + status: str, 420 + kind: str, 421 + reason_code: str, 422 + ) -> None: 423 + block = _write_confidential_config(tmp_path, monkeypatch) 424 + establish = Mock(side_effect=AssertionError("verify should not run")) 425 + monkeypatch.setattr(spp_transport, "establish_attested_channel", establish) 426 + monkeypatch.setattr( 427 + spp_transport, 428 + "ensure_nvattest_installed", 429 + lambda **_kwargs: SimpleNamespace( 430 + status=status, 431 + nvattest_dir=None, 432 + reason_code=None, 433 + detail=None, 434 + ), 435 + ) 436 + 437 + with pytest.raises(AttestationFailedError): 438 + spp_transport.verify_confidential_attestation(block) 439 + 440 + establish.assert_not_called() 441 + failure = spp.get_attestation_state().failure 442 + assert failure is not None 443 + assert failure.kind == kind 444 + assert failure.reason_code == reason_code 445 + 446 + 396 447 def test_recheck_confidential_attestation_records_success( 397 448 tmp_path: Path, 398 449 monkeypatch: pytest.MonkeyPatch, ··· 417 468 assert block["endpoint_url"] == "https://spp.example.test:9443" 418 469 419 470 471 + def test_recheck_confidential_attestation_ensures_nvattest_before_verify( 472 + tmp_path: Path, 473 + monkeypatch: pytest.MonkeyPatch, 474 + ) -> None: 475 + block = _write_confidential_config(tmp_path, monkeypatch) 476 + _patch_listener(monkeypatch) 477 + nvattest_dir = tmp_path / "cache" / "providers" / "nvattest" 478 + ensured: list[dict[str, object]] = [] 479 + 480 + def fake_ensure_nvattest_installed(**kwargs): 481 + ensured.append(kwargs) 482 + return SimpleNamespace( 483 + status="already_installed", 484 + nvattest_dir=nvattest_dir, 485 + reason_code=None, 486 + detail=None, 487 + ) 488 + 489 + def fake_establish(_endpoint, **kwargs): 490 + assert ensured 491 + assert kwargs["nvattest_dir"] == nvattest_dir 492 + return _FakeChannel(object()) 493 + 494 + monkeypatch.setattr( 495 + spp_transport, "ensure_nvattest_installed", fake_ensure_nvattest_installed 496 + ) 497 + monkeypatch.setattr(spp_transport, "establish_attested_channel", fake_establish) 498 + 499 + spp_transport.recheck_confidential_attestation() 500 + 501 + assert ensured == [{"explicit_override": None}] 502 + assert spp.get_attestation_state().failure is None 503 + assert block["endpoint_url"] == "https://spp.example.test:9443" 504 + 505 + 420 506 def test_recheck_confidential_attestation_fails_closed_and_preserves_last_verified( 421 507 tmp_path: Path, 422 508 monkeypatch: pytest.MonkeyPatch, ··· 441 527 442 528 443 529 def test_recheck_confidential_attestation_off_is_noop( 530 + tmp_path: Path, 444 531 monkeypatch: pytest.MonkeyPatch, 445 532 ) -> None: 533 + monkeypatch.setenv("SOLSTONE_JOURNAL", str(tmp_path)) 534 + config_dir = tmp_path / "config" 535 + config_dir.mkdir(parents=True) 536 + (config_dir / "journal.json").write_text("{}", encoding="utf-8") 446 537 establish = Mock(side_effect=AssertionError("attestation attempted")) 447 538 monkeypatch.setattr(spp_transport, "establish_attested_channel", establish) 539 + monkeypatch.setattr( 540 + spp_transport, 541 + "ensure_nvattest_installed", 542 + Mock(side_effect=AssertionError("nvattest ensure attempted")), 543 + ) 448 544 449 545 spp_transport.recheck_confidential_attestation() 450 546 451 547 establish.assert_not_called() 548 + assert not (tmp_path / "cache" / "providers" / "nvattest").exists()
+59
tests/test_brain_cli.py
··· 6 6 import argparse 7 7 import hashlib 8 8 import json 9 + import logging 9 10 import sys 10 11 from datetime import datetime, timezone 11 12 from pathlib import Path ··· 405 406 }, 406 407 NOW, 407 408 journal_path=journal, 409 + ) 410 + 411 + 412 + @pytest.mark.parametrize( 413 + ("raw_reason", "expected_reason", "expected_status"), 414 + [ 415 + ("nvattest_install_in_progress", "nvattest_install_in_progress", "blocked"), 416 + ("nvattest_platform_unsupported", "nvattest_platform_unsupported", "blocked"), 417 + ("nvattest_unavailable", "nvattest_unavailable", "blocked"), 418 + ("nvattest_install_failed", "nvattest_install_failed", "failed"), 419 + ("nvattest_integrity_failed", "nvattest_integrity_failed", "failed"), 420 + ("gateway_unreachable", "attestation_not_verified", "blocked"), 421 + ], 422 + ) 423 + def test_spp_prerequisite_maps_failure_reason_code( 424 + monkeypatch: pytest.MonkeyPatch, 425 + raw_reason: str, 426 + expected_reason: str, 427 + expected_status: str, 428 + ) -> None: 429 + from solstone.think.services import spp 430 + 431 + spp.delete_attestation_state() 432 + monkeypatch.setattr( 433 + "solstone.think.services.spp_transport.recheck_confidential_attestation", 434 + lambda: spp.record_attestation_failed("failed", raw_reason), 435 + ) 436 + try: 437 + component, reason = brain_cli._spp_prerequisite(NOW) 438 + finally: 439 + spp.delete_attestation_state() 440 + 441 + assert reason == expected_reason 442 + assert component["reason_code"] == expected_reason 443 + assert component["status"] == expected_status 444 + 445 + 446 + def test_spp_prerequisite_warns_and_fails_closed_on_unmapped_reason( 447 + monkeypatch: pytest.MonkeyPatch, 448 + caplog: pytest.LogCaptureFixture, 449 + ) -> None: 450 + from solstone.think.services import spp 451 + 452 + spp.delete_attestation_state() 453 + monkeypatch.setattr( 454 + "solstone.think.services.spp_transport.recheck_confidential_attestation", 455 + lambda: spp.record_attestation_failed("failed", "new_raw_reason"), 456 + ) 457 + caplog.set_level(logging.WARNING, logger=brain_cli.LOG.name) 458 + try: 459 + component, reason = brain_cli._spp_prerequisite(NOW) 460 + finally: 461 + spp.delete_attestation_state() 462 + 463 + assert reason == "attestation_rejected" 464 + assert component["reason_code"] == "attestation_rejected" 465 + assert ( 466 + "event=spp_attestation_reason_unmapped raw_reason=new_raw_reason" in caplog.text 408 467 ) 409 468 410 469
+6 -1
tests/test_brain_state.py
··· 257 257 "local_runtime_not_ready", 258 258 "local_artifact_not_ready", 259 259 "attestation_not_verified", 260 + "nvattest_install_in_progress", 261 + "nvattest_platform_unsupported", 262 + "nvattest_unavailable", 260 263 "provider_key_invalid", 261 264 "model_not_found", 262 265 "provider_quota_exceeded", ··· 269 272 "cogitate_terminal_error", 270 273 "attestation_rejected", 271 274 "attestation_expired", 275 + "nvattest_install_failed", 276 + "nvattest_integrity_failed", 272 277 "local_server_unhealthy", 273 278 "configuration_invalid", 274 279 "fingerprint_key_unavailable", ··· 307 312 assert set(BRAIN_REASON_TO_AGGREGATE) == BRAIN_REASON_CODES 308 313 assert set(BRAIN_REASON_TO_AGGREGATE.values()) <= BRAIN_AGGREGATE_STATES 309 314 evidence_reasons = frozenset().union(*BRAIN_EVIDENCE_REASON_CODES.values()) 310 - assert len(evidence_reasons) == 26 315 + assert len(evidence_reasons) == 31 311 316 assert len(BRAIN_PROJECTION_ONLY_REASON_CODES) == 10 312 317 assert evidence_reasons | BRAIN_PROJECTION_ONLY_REASON_CODES == BRAIN_REASON_CODES 313 318 assert not (evidence_reasons & BRAIN_PROJECTION_ONLY_REASON_CODES)
+190 -3
tests/test_nvattest_install.py
··· 7 7 import json 8 8 import shutil 9 9 import tarfile 10 + from collections.abc import Iterator 11 + from contextlib import contextmanager 10 12 from pathlib import Path 11 13 12 14 import pytest 13 15 16 + from solstone.think.journal_io import LockTimeout 14 17 from solstone.think.providers import nvattest_install 18 + 19 + 20 + def test_linux_x86_64_archive_pin_is_exact() -> None: 21 + spec = nvattest_install.NVATTEST_ARCHIVES["linux-x86_64"] 22 + expected_url = ( 23 + "https://updates.solstone.app/providers/nvattest/" 24 + "libnvat-linux-x86_64-1.2.2-sol.1-archive.tar.xz" 25 + ) 26 + legacy_sha = "3f10da6fca794b7e3025c6645447947ec8bc45bcfde5b5b1d23241c7115630db" 27 + 28 + assert spec.version == "1.2.2-sol.1" 29 + assert spec.url == expected_url 30 + assert ( 31 + spec.sha256 32 + == "60ef75d1873e7129f03ea80d107d92b2ef216d2a8815958617b30d9c721d474a" 33 + ) 34 + source = Path(nvattest_install.__file__).read_text(encoding="utf-8") 35 + assert "developer.download.nvidia.com" not in source 36 + assert legacy_sha not in source 15 37 16 38 17 39 def test_install_nvattest_reinstalls_partial_cache_without_sidecar( ··· 84 106 (root / "bin").mkdir(parents=True) 85 107 (root / "bin" / "nvattest").write_text("installed\n", encoding="utf-8") 86 108 (root / "lib").mkdir() 109 + (root / "share" / "ca").mkdir(parents=True) 110 + (root / "share" / "ca" / "ca-bundle.pem").write_text("ca\n", encoding="utf-8") 87 111 spec = nvattest_install.NvattestArchiveSpec( 88 112 version="1.0.0", 89 113 url="https://example.invalid/nvattest.tar.gz", ··· 104 128 assert nvattest_install.install_nvattest(spec=spec, journal_path=tmp_path) == root 105 129 106 130 107 - def _fixture_spec(tmp_path: Path) -> nvattest_install.NvattestArchiveSpec: 131 + def test_install_nvattest_upgrades_old_nvidia_sidecar( 132 + tmp_path: Path, 133 + monkeypatch: pytest.MonkeyPatch, 134 + ) -> None: 135 + root = nvattest_install.cache_root(tmp_path) 136 + (root / "bin").mkdir(parents=True) 137 + (root / "bin" / "nvattest").write_text("old\n", encoding="utf-8") 138 + (root / "lib").mkdir() 139 + (root / "share" / "ca").mkdir(parents=True) 140 + (root / "share" / "ca" / "ca-bundle.pem").write_text("old-ca\n", encoding="utf-8") 141 + (root / nvattest_install.SIDECAR_NAME).write_text( 142 + json.dumps( 143 + { 144 + "archive_sha256": ( 145 + "3f10da6fca794b7e3025c6645447947ec8bc45bcfde5b5b1d23241c7115630db" 146 + ), 147 + "version": "1.2.2", 148 + } 149 + ) 150 + + "\n", 151 + encoding="utf-8", 152 + ) 153 + spec = _fixture_spec(tmp_path, version="1.2.2-sol.1") 154 + calls: list[Path] = [] 155 + 156 + def fake_download(_url: str, dest: Path, _expected_sha256: str) -> None: 157 + calls.append(dest) 158 + dest.parent.mkdir(parents=True, exist_ok=True) 159 + shutil.copy2(tmp_path / spec.archive_name, dest) 160 + 161 + monkeypatch.setattr(nvattest_install, "_download_file", fake_download) 162 + 163 + nvattest_install.install_nvattest(spec=spec, journal_path=tmp_path) 164 + 165 + assert len(calls) == 1 166 + assert (root / "bin" / "nvattest").read_text(encoding="utf-8") == "new\n" 167 + sidecar = json.loads( 168 + (root / nvattest_install.SIDECAR_NAME).read_text(encoding="utf-8") 169 + ) 170 + assert sidecar["version"] == "1.2.2-sol.1" 171 + 172 + 173 + def test_install_nvattest_rejects_hash_mismatch_without_partial_tree( 174 + tmp_path: Path, 175 + monkeypatch: pytest.MonkeyPatch, 176 + ) -> None: 177 + class FakeResponse: 178 + def __enter__(self): 179 + return self 180 + 181 + def __exit__(self, *_exc_info) -> bool: 182 + return False 183 + 184 + def raise_for_status(self) -> None: 185 + return None 186 + 187 + def iter_bytes(self): 188 + yield b"not the pinned archive" 189 + 190 + def fake_stream(*_args, **_kwargs): 191 + return FakeResponse() 192 + 193 + monkeypatch.setattr("httpx.stream", fake_stream) 194 + spec = nvattest_install.NvattestArchiveSpec( 195 + version="1.2.2-sol.1", 196 + url="https://example.invalid/nvattest.tar.xz", 197 + archive_name="nvattest.tar.xz", 198 + sha256="0" * 64, 199 + ) 200 + 201 + with pytest.raises(nvattest_install.NvattestInstallError) as exc_info: 202 + nvattest_install.install_nvattest(spec=spec, journal_path=tmp_path) 203 + 204 + assert exc_info.value.reason_code == "sha256_mismatch" 205 + root = nvattest_install.cache_root(tmp_path) 206 + assert not (root / "bin").exists() 207 + assert not (root / "lib").exists() 208 + assert not (root / "share").exists() 209 + assert not (root / ".downloads" / "nvattest.tar.xz").exists() 210 + assert not (root / ".downloads" / "nvattest.tar.xz.tmp").exists() 211 + 212 + 213 + def test_ensure_nvattest_unsupported_platform_does_not_touch_cache( 214 + tmp_path: Path, 215 + monkeypatch: pytest.MonkeyPatch, 216 + ) -> None: 217 + monkeypatch.setattr(nvattest_install, "nvattest_archive_key", lambda: None) 218 + 219 + result = nvattest_install.ensure_nvattest_installed(journal_path=tmp_path) 220 + 221 + assert result.status == "platform_unsupported" 222 + assert result.reason_code == "platform_unsupported" 223 + assert not nvattest_install.cache_root(tmp_path).exists() 224 + 225 + 226 + def test_ensure_nvattest_lock_timeout_is_in_flight( 227 + tmp_path: Path, 228 + monkeypatch: pytest.MonkeyPatch, 229 + ) -> None: 230 + @contextmanager 231 + def fake_hold_lock(path: Path, *, timeout: float, **_kwargs) -> Iterator[None]: 232 + raise LockTimeout(path, timeout) 233 + yield 234 + 235 + monkeypatch.setattr(nvattest_install, "hold_lock", fake_hold_lock) 236 + 237 + result = nvattest_install.ensure_nvattest_installed(journal_path=tmp_path) 238 + 239 + assert result.status == "install_in_flight" 240 + assert result.reason_code == "install-in-progress" 241 + 242 + 243 + def test_ensure_nvattest_override_skips_cache_download( 244 + tmp_path: Path, 245 + monkeypatch: pytest.MonkeyPatch, 246 + ) -> None: 247 + override = tmp_path / "override" 248 + monkeypatch.setenv(nvattest_install.SPP_NVATTEST_DIR_ENV, str(override)) 249 + monkeypatch.setattr( 250 + nvattest_install, 251 + "_download_file", 252 + lambda *_args, **_kwargs: pytest.fail("download should not run"), 253 + ) 254 + 255 + result = nvattest_install.ensure_nvattest_installed(journal_path=tmp_path) 256 + 257 + assert result.status == "already_installed" 258 + assert result.nvattest_dir == override 259 + assert not nvattest_install.cache_root(tmp_path).exists() 260 + 261 + 262 + def test_install_nvattest_accepts_wrapped_archive( 263 + tmp_path: Path, 264 + monkeypatch: pytest.MonkeyPatch, 265 + ) -> None: 266 + spec = _fixture_spec(tmp_path, wrapped=True) 267 + 268 + def fake_download(_url: str, dest: Path, _expected_sha256: str) -> None: 269 + dest.parent.mkdir(parents=True, exist_ok=True) 270 + shutil.copy2(tmp_path / spec.archive_name, dest) 271 + 272 + monkeypatch.setattr(nvattest_install, "_download_file", fake_download) 273 + 274 + installed = nvattest_install.install_nvattest(spec=spec, journal_path=tmp_path) 275 + 276 + assert (installed / "bin" / "nvattest").read_text(encoding="utf-8") == "new\n" 277 + 278 + 279 + def _fixture_spec( 280 + tmp_path: Path, 281 + *, 282 + version: str = "9.9.9", 283 + wrapped: bool = False, 284 + ) -> nvattest_install.NvattestArchiveSpec: 108 285 archive_name = "nvattest-fixture.tar.gz" 109 286 source = tmp_path / "source" / "nvattest-fixture" 110 287 (source / "bin").mkdir(parents=True) ··· 113 290 (source / "lib" / "libnvat.so.1").write_text("library\n", encoding="utf-8") 114 291 (source / "lib" / "libnvat.so").symlink_to("libnvat.so.1") 115 292 (source / "LICENSE").write_text("license\n", encoding="utf-8") 293 + (source / "share" / "ca").mkdir(parents=True) 294 + (source / "share" / "ca" / "ca-bundle.pem").write_text("ca\n", encoding="utf-8") 295 + (source / "share" / "THIRD_PARTY_NOTICES.md").write_text( 296 + "notices\n", 297 + encoding="utf-8", 298 + ) 116 299 117 300 archive_path = tmp_path / archive_name 118 301 with tarfile.open(archive_path, "w:gz") as archive: 119 - archive.add(source, arcname=source.name) 302 + if wrapped: 303 + archive.add(source, arcname=source.name) 304 + else: 305 + for child in source.iterdir(): 306 + archive.add(child, arcname=child.name) 120 307 return nvattest_install.NvattestArchiveSpec( 121 - version="9.9.9", 308 + version=version, 122 309 url="https://example.invalid/nvattest-fixture.tar.gz", 123 310 archive_name=archive_name, 124 311 sha256=hashlib.sha256(archive_path.read_bytes()).hexdigest(),