solpbc.org / solstone
personal memory agent
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

feat(schemas): pre-send provider schema preparation, bounds ratchet, local eval harness

Foundation for the KG schema enrichment arc. Canonical *.schema.json files
may now carry maxItems/maxLength generation bounds; providers that cannot
accept a given keyword receive a deterministic, pre-send reduced copy. No
shipped schema is bounded in this lode — the framework is proven by fixture
and by the guardrail ratchet.

Entry point: solstone/think/schema_prep.py::prepare_provider_schema
Eval harness: make eval-schemas

D1 — per-provider support matrix (deviates from a blanket cloud strip).
Current provider docs say OpenAI and Google ACCEPT minItems/maxItems/
minimum/maximum; only minLength/maxLength are outside their supported
subset. Anthropic rejects numeric/string/array constraints beyond minItems
0/1, so all six bounds are stripped there. pattern is never stripped for any
provider. $schema/$comment are stripped for all three cloud providers. local
and any unregistered provider name receive a canonical deep copy; prep never
raises on an unknown provider. STRICT_UNSUPPORTED_KEYWORDS carries the doc
citations and is the single source of truth — the portability test imports it
rather than restating the list. Because today's 22 discovered generation
schemas carry zero bounds, every outgoing payload is byte-identical to before
this change; a parametrized test asserts that over all schemas x all
providers.

D2 — the transform drops only validation-only keywords, recursively. It is
pure, deep-copying, idempotent, and structure-preserving: required,
additionalProperties, type/nullability, enum, const, pattern, format, items
and descriptions are never touched. No runtime try-then-fallback was added;
c78f9ff5 deleted exactly that, and the reduction here is deterministic and
pre-send. local.py::_normalize_schema_patterns stays where it is as a
llama.cpp transport detail.

D4 — response validation stays CANONICAL. All three models.py call sites
(generate, generate_with_result, agenerate) hand the prepared schema to the
provider and keep validating the response against the canonical schema.
HAZARD for follow-on lodes: for anthropic, bounds are stripped from the
request but still enforced on the response, so once shipped schemas carry
maxItems/maxLength an over-long Anthropic response raises
SchemaValidationError. That is a deliberate fail-loudly choice (CLAUDE.md
§8), and it is covered by test_generate_validates_response_against_canonical_schema.

Also in this commit:
- tests/test_schema_strict_portability.py rewritten. violations() now checks
only the structural rules prep does not and must not fix (root object,
additionalProperties:false, all properties required, no oneOf). The
banned-key assertion moved to the provider-facing schema, per provider, so
maxItems legitimately survives for openai/google. Zero-facet hydration
guard retained (hydrate, then prep). Negative self-check strengthened.
- solstone/think/schema_bounds.py::unbounded_nodes — reusable deny-by-default
helper: arrays without maxItems, and free-text strings (no enum/const/
pattern/format) without maxLength.
- scripts/check_schema_bounds.py — ratchet wired into make install-checks,
following the check_call_http_only.py evaluate()->(new, stale, tracked)
idiom. 22 reasoned allowlist entries; stale entries that no longer violate
fail the gate so the allowlist cannot rot. daily_schedule.schema.json is
already clean and is deliberately absent. Each follow-on lode deletes its
entry.
- tests/eval_schemas.py + make eval-schemas — opt-in local strict-JSON eval
over tests/fixtures/schema_eval/cases.jsonl, including a deliberately
bounded case. Reports per-case schema-validity and content-preservation,
writes results atomically to the gitignored tmp/schema-eval/, and exits
non-zero naming `journal install-provider local` when the local model is
not ready. Not collected by pytest; scoring lives in
solstone/think/schema_eval.py with offline unit tests.

Existing test_local/test_models/test_anthropic/test_openai/test_google/
test_sense_schema/test_schedule_schema pass unmodified. No external network
call in any collected test; no new pytest marker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

author
Jer Miller co-author
Claude Opus 4.8 (1M context) date (Jul 8, 2026, 9:13 PM -0600) commit a4609311 parent 47e70a84
+901 -41
+11 -1
Makefile
··· 14 14 PYTEST_BASETEMP_INIT := BASETEMP=$$(mktemp -d /var/tmp/solstone-pytest-XXXXXX); trap 'rm -rf "$$BASETEMP"' EXIT INT TERM; 15 15 PYTEST_BASETEMP_FLAG := --basetemp "$$BASETEMP" 16 16 17 - .PHONY: install uninstall test test-cov test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills render-packaging openapi check-openapi contract check-contract dev all sandbox sandbox-stop install-models parakeet-helper parakeet-helper-clean wheel-macos wheel-macos-clean verify verify-api update-api-baselines service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic check-call-http-only check-tools-http-only check-access-imports-clean check-convey-bind-imports-clean check-thin-base-install check-cogitate-prompts smoke-cogitate release release-test FORCE 17 + .PHONY: install uninstall test test-cov test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills render-packaging openapi check-openapi contract check-contract dev all sandbox sandbox-stop install-models parakeet-helper parakeet-helper-clean wheel-macos wheel-macos-clean verify verify-api update-api-baselines eval-schemas service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic check-call-http-only check-tools-http-only check-access-imports-clean check-convey-bind-imports-clean check-schema-bounds check-thin-base-install check-cogitate-prompts smoke-cogitate release release-test FORCE 18 18 19 19 # Default target - install package in editable mode 20 20 all: install ··· 230 230 $(MAKE) sandbox-stop; \ 231 231 exit $$RESULT 232 232 233 + eval-schemas: .installed 234 + $(VENV_BIN)/python tests/eval_schemas.py 235 + 233 236 # Regenerate API baseline files. By default uses the deterministic Flask 234 237 # test-client path (frozen time). For sandbox-only endpoints (graph, search, 235 238 # badge-count, updated-days), pass SANDBOX=1 to regenerate from the live ··· 410 413 @echo "=== Running call-http-only check ===" 411 414 @$(MAKE) check-call-http-only 412 415 @echo "" 416 + @echo "=== Running schema-bounds check ===" 417 + @$(MAKE) check-schema-bounds 418 + @echo "" 413 419 @echo "=== Running tools-http-only check ===" 414 420 @$(MAKE) check-tools-http-only 415 421 @echo "" ··· 511 517 # sol call HTTP-only gate (call.py reaches the journal only over HTTP) 512 518 check-call-http-only: .installed 513 519 $(VENV_BIN)/python scripts/check_call_http_only.py 520 + 521 + # Generation schema bounds ratchet 522 + check-schema-bounds: .installed 523 + $(VENV_BIN)/python scripts/check_schema_bounds.py 514 524 515 525 # Built-in sol call tools HTTP-only gate 516 526 check-tools-http-only: .installed
+14
docs/PROVIDERS.md
··· 102 102 103 103 **Important:** Providers should gracefully ignore unsupported parameters rather than raising errors. 104 104 105 + ### Structured-output schema preparation 106 + 107 + Canonical generation schemas may contain local grammar bounds such as 108 + `maxItems` and `maxLength`. The wrapper in `solstone/think/models.py` prepares a 109 + provider-facing copy with `solstone/think/schema_prep.py` before calling a 110 + provider. `STRICT_UNSUPPORTED_KEYWORDS` is the single support matrix for strict 111 + cloud providers; local receives a canonical copy so llama.cpp can turn bounds 112 + into grammar constraints. Response validation still uses the canonical schema. 113 + 114 + Use `make check-schema-bounds` to run the bounds ratchet for canonical schemas. 115 + Use `make eval-schemas` to run the opt-in local llama.cpp structured-output 116 + eval harness; it requires `journal install-provider local` and a running local 117 + server via `journal start` or `journal service start`. 118 + 105 119 ## run_cogitate() 106 120 107 121 Handles tool-calling execution.
+160
scripts/check_schema_bounds.py
··· 1 + #!/usr/bin/env python3 2 + # SPDX-License-Identifier: AGPL-3.0-only 3 + # Copyright (c) 2026 sol pbc 4 + 5 + """Ratcheting guard for generation-schema array and free-text bounds.""" 6 + 7 + from __future__ import annotations 8 + 9 + import argparse 10 + import json 11 + import sys 12 + from pathlib import Path 13 + from typing import Any 14 + 15 + ROOT = Path(__file__).resolve().parent.parent 16 + 17 + if str(ROOT) not in sys.path: 18 + sys.path.insert(0, str(ROOT)) 19 + 20 + from solstone.apps.timeline.rollup import build_rollup_schema # noqa: E402 21 + from solstone.think.schema_bounds import unbounded_nodes # noqa: E402 22 + 23 + ALLOWLIST: dict[str, str] = { 24 + "build_rollup_schema(3)": "documents follow-on lode", 25 + "solstone/apps/entities/talent/detection.schema.json": ( 26 + "entity_observer follow-on lode" 27 + ), 28 + "solstone/apps/entities/talent/entities_review.schema.json": ( 29 + "entity_observer follow-on lode" 30 + ), 31 + "solstone/apps/entities/talent/entity_observer.schema.json": ( 32 + "entity_observer follow-on lode" 33 + ), 34 + "solstone/apps/timeline/talent/segment_summary.schema.json": ( 35 + "documents follow-on lode" 36 + ), 37 + "solstone/observe/categories/meeting.schema.json": "screen follow-on lode", 38 + "solstone/observe/describe.schema.json": "screen follow-on lode", 39 + "solstone/observe/enrich.schema.json": ( 40 + "unbounded pending KG schema enrichment arc" 41 + ), 42 + "solstone/observe/extract.schema.json": "screen follow-on lode", 43 + "solstone/observe/transcribe/gemini.schema.json": ( 44 + "unbounded pending KG schema enrichment arc" 45 + ), 46 + "solstone/talent/chat.schema.json": "messaging follow-on lode", 47 + "solstone/talent/participation.schema.json": "calendar follow-on lode", 48 + "solstone/talent/participation_entry.schema.json": "calendar follow-on lode", 49 + "solstone/talent/pulse.schema.json": "morning_briefing follow-on lode", 50 + "solstone/talent/schedule.schema.json": "calendar follow-on lode", 51 + "solstone/talent/sense.schema.json": ("unbounded pending KG schema enrichment arc"), 52 + "solstone/talent/speaker_attribution.schema.json": ( 53 + "unbounded pending KG schema enrichment arc" 54 + ), 55 + "solstone/talent/steward.schema.json": "morning_briefing follow-on lode", 56 + "solstone/talent/story.schema.json": "story follow-on lode", 57 + "solstone/think/detect_created.schema.json": ( 58 + "unbounded pending KG schema enrichment arc" 59 + ), 60 + "solstone/think/detect_transcript_json.schema.json": ( 61 + "unbounded pending KG schema enrichment arc" 62 + ), 63 + "solstone/think/detect_transcript_segment.schema.json": ( 64 + "unbounded pending KG schema enrichment arc" 65 + ), 66 + } 67 + 68 + 69 + def discover_schemas(root: Path) -> dict[str, dict[str, Any]]: 70 + """Return generation schemas keyed by stable schema id.""" 71 + discovered: dict[str, dict[str, Any]] = {} 72 + for path in sorted((root / "solstone").glob("**/*.schema.json")): 73 + schema = json.loads(path.read_text(encoding="utf-8")) 74 + if isinstance(schema.get("x-journal-contract"), dict): 75 + continue 76 + discovered[path.relative_to(root).as_posix()] = schema 77 + discovered["build_rollup_schema(3)"] = build_rollup_schema(3) 78 + return discovered 79 + 80 + 81 + def evaluate( 82 + root: Path, allowlist: dict[str, str] 83 + ) -> tuple[list[str], list[str], list[str]]: 84 + """Return ``(new, stale, tracked)`` human-readable lines.""" 85 + schemas = discover_schemas(root) 86 + live = {} 87 + for schema_id, schema in schemas.items(): 88 + hits = unbounded_nodes(schema) 89 + if hits: 90 + live[schema_id] = hits 91 + 92 + new: list[str] = [] 93 + stale: list[str] = [] 94 + tracked: list[str] = [] 95 + 96 + for schema_id in sorted(set(live) | set(allowlist)): 97 + hits = live.get(schema_id, []) 98 + reason = allowlist.get(schema_id) 99 + if hits and reason is None: 100 + joined = ", ".join(hits) 101 + new.append( 102 + f"{schema_id}: {len(hits)} unbounded node(s): {joined} - " 103 + "add generation bounds or add a temporary allowlist reason." 104 + ) 105 + elif not hits and reason is not None: 106 + stale.append( 107 + f"{schema_id}: allowlisted but now clean - delete the entry " 108 + "from check_schema_bounds.py." 109 + ) 110 + elif hits and reason is not None: 111 + tracked.append( 112 + f"{schema_id}: {len(hits)} unbounded node(s) ({reason}; allowlisted)" 113 + ) 114 + 115 + return new, stale, tracked 116 + 117 + 118 + def main(argv: list[str] | None = None) -> int: 119 + parser = argparse.ArgumentParser(description="generation schema bounds lint") 120 + parser.add_argument( 121 + "--root", 122 + type=Path, 123 + default=ROOT, 124 + help="Repository root to scan (defaults to the checkout root).", 125 + ) 126 + args = parser.parse_args(argv) 127 + 128 + new, stale, tracked = evaluate(args.root, ALLOWLIST) 129 + 130 + if tracked: 131 + print("schema-bounds: known unbounded schemas (allowlisted):") 132 + for line in tracked: 133 + print(f" {line}") 134 + print() 135 + 136 + if new or stale: 137 + if new: 138 + print("schema-bounds: NEW violations:", file=sys.stderr) 139 + for line in new: 140 + print(f" {line}", file=sys.stderr) 141 + print(file=sys.stderr) 142 + if stale: 143 + print("schema-bounds: STALE allowlist entries:", file=sys.stderr) 144 + for line in stale: 145 + print(f" {line}", file=sys.stderr) 146 + print(file=sys.stderr) 147 + print( 148 + "Generation schemas need maxItems on arrays and maxLength on " 149 + "free-text strings; remove stale allowlist entries as schemas are " 150 + "bounded.", 151 + file=sys.stderr, 152 + ) 153 + return 1 154 + 155 + print("schema-bounds: pass") 156 + return 0 157 + 158 + 159 + if __name__ == "__main__": 160 + raise SystemExit(main())
+7 -3
solstone/think/models.py
··· 15 15 import frontmatter 16 16 from jsonschema import Draft202012Validator 17 17 18 + from solstone.think.schema_prep import prepare_provider_schema 18 19 from solstone.think.utils import get_config, get_journal 19 20 20 21 logger = logging.getLogger(__name__) ··· 1298 1299 1299 1300 # Get provider module via registry (raises ValueError for unknown providers) 1300 1301 provider_mod = get_provider_module(provider) 1302 + provider_schema = prepare_provider_schema(json_schema, provider) 1301 1303 1302 1304 timeout_s = DEFAULT_PROVIDER_TIMEOUT_S if timeout_s is None else timeout_s 1303 1305 ··· 1310 1312 max_output_tokens=max_output_tokens, 1311 1313 system_instruction=system_instruction, 1312 1314 json_output=json_output, 1313 - json_schema=json_schema, 1315 + json_schema=provider_schema, 1314 1316 thinking_budget=thinking_budget, 1315 1317 timeout_s=timeout_s, 1316 1318 **kwargs, ··· 1511 1513 _reject_local_cloud_model_override(provider, model_override) 1512 1514 1513 1515 provider_mod = get_provider_module(provider) 1516 + provider_schema = prepare_provider_schema(json_schema, provider) 1514 1517 1515 1518 timeout_s = DEFAULT_PROVIDER_TIMEOUT_S if timeout_s is None else timeout_s 1516 1519 ··· 1522 1525 max_output_tokens=max_output_tokens, 1523 1526 system_instruction=system_instruction, 1524 1527 json_output=json_output, 1525 - json_schema=json_schema, 1528 + json_schema=provider_schema, 1526 1529 thinking_budget=thinking_budget, 1527 1530 timeout_s=timeout_s, 1528 1531 **kwargs, ··· 1619 1622 1620 1623 # Get provider module via registry (raises ValueError for unknown providers) 1621 1624 provider_mod = get_provider_module(provider) 1625 + provider_schema = prepare_provider_schema(json_schema, provider) 1622 1626 1623 1627 timeout_s = DEFAULT_PROVIDER_TIMEOUT_S if timeout_s is None else timeout_s 1624 1628 ··· 1631 1635 max_output_tokens=max_output_tokens, 1632 1636 system_instruction=system_instruction, 1633 1637 json_output=json_output, 1634 - json_schema=json_schema, 1638 + json_schema=provider_schema, 1635 1639 thinking_budget=thinking_budget, 1636 1640 timeout_s=timeout_s, 1637 1641 **kwargs,
+46
solstone/think/schema_bounds.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + """Guard helpers for generation-schema bounds.""" 5 + 6 + from __future__ import annotations 7 + 8 + from typing import Any 9 + 10 + 11 + def _has_type(node: dict[str, Any], schema_type: str) -> bool: 12 + value = node.get("type") 13 + if value == schema_type: 14 + return True 15 + return isinstance(value, list) and schema_type in value 16 + 17 + 18 + def unbounded_nodes(schema: dict[str, Any]) -> list[str]: 19 + """Return paths for generation nodes missing local grammar bounds. 20 + 21 + Violations: 22 + - array nodes (``type == "array"`` or nullable/list-valued type containing 23 + ``"array"``) without ``maxItems`` 24 + - free-text string nodes (``"string"`` in type) that have none of 25 + ``enum``, ``const``, ``pattern``, or ``format`` and no ``maxLength`` 26 + """ 27 + found: list[str] = [] 28 + 29 + def walk(node: Any, path: str) -> None: 30 + if isinstance(node, dict): 31 + if _has_type(node, "array") and "maxItems" not in node: 32 + found.append(path) 33 + if ( 34 + _has_type(node, "string") 35 + and "maxLength" not in node 36 + and not {"enum", "const", "pattern", "format"} & set(node) 37 + ): 38 + found.append(path) 39 + for key, value in node.items(): 40 + walk(value, f"{path}/{key}") 41 + elif isinstance(node, list): 42 + for index, value in enumerate(node): 43 + walk(value, f"{path}[{index}]") 44 + 45 + walk(schema, "$") 46 + return found
+58
solstone/think/schema_eval.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + """Pure scoring helpers for structured-output schema evals.""" 5 + 6 + from __future__ import annotations 7 + 8 + import json 9 + from typing import Any, Sequence 10 + 11 + from jsonschema import Draft202012Validator 12 + 13 + 14 + def schema_validity(text: str, schema: dict[str, Any]) -> dict[str, Any]: 15 + """Validate response text against ``schema`` and return a serializable result.""" 16 + try: 17 + parsed = json.loads(text) 18 + except json.JSONDecodeError as exc: 19 + return { 20 + "valid": False, 21 + "errors": [ 22 + { 23 + "path": "$", 24 + "message": f"Invalid JSON: {exc.msg}", 25 + } 26 + ], 27 + } 28 + 29 + errors = [] 30 + for error in sorted( 31 + Draft202012Validator(schema).iter_errors(parsed), 32 + key=lambda item: list(item.path), 33 + ): 34 + path = "$" 35 + for part in error.path: 36 + if isinstance(part, int): 37 + path += f"[{part}]" 38 + else: 39 + path += f"/{part}" 40 + errors.append({"path": path, "message": error.message}) 41 + 42 + return {"valid": not errors, "errors": errors} 43 + 44 + 45 + def content_preservation(text: str, expect_contains: Sequence[str]) -> dict[str, Any]: 46 + """Score case-insensitive substring preservation in response text.""" 47 + needles = list(expect_contains) 48 + if not needles: 49 + return {"fraction": 1.0, "found": [], "missing": []} 50 + 51 + haystack = text.lower() 52 + found = [needle for needle in needles if needle.lower() in haystack] 53 + missing = [needle for needle in needles if needle.lower() not in haystack] 54 + return { 55 + "fraction": len(found) / len(needles), 56 + "found": found, 57 + "missing": missing, 58 + }
+109
solstone/think/schema_prep.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + """Provider-facing JSON Schema preparation for strict structured output.""" 5 + 6 + from __future__ import annotations 7 + 8 + import copy 9 + import logging 10 + from typing import Any 11 + 12 + logger = logging.getLogger(__name__) 13 + 14 + # Provider structured-output support references: 15 + # - OpenAI: https://developers.openai.com/api/docs/guides/structured-outputs 16 + # supports pattern/format/minimum/maximum/minItems/maxItems; minLength and 17 + # maxLength are not listed in the supported subset. 18 + # - Anthropic: https://platform.claude.com/docs/en/build-with-claude/structured-outputs 19 + # rejects numeric/string/array constraints beyond minItems values 0 or 1; 20 + # strip minItems wholesale here for deterministic simplicity. Pattern is 21 + # documented as supported. 22 + # - Google: https://ai.google.dev/gemini-api/docs/structured-output 23 + # supports string format, number minimum/maximum, and array minItems/maxItems. 24 + STRICT_UNSUPPORTED_KEYWORDS: dict[str, frozenset[str]] = { 25 + "openai": frozenset({"$schema", "$comment", "minLength", "maxLength"}), 26 + "google": frozenset({"$schema", "$comment", "minLength", "maxLength"}), 27 + "anthropic": frozenset( 28 + { 29 + "$schema", 30 + "$comment", 31 + "minLength", 32 + "maxLength", 33 + "minItems", 34 + "maxItems", 35 + "minimum", 36 + "maximum", 37 + } 38 + ), 39 + } 40 + 41 + # Hazard: cloud-provider reductions are request-only. Canonical response 42 + # validation in models.py still enforces stripped bounds, so an Anthropic 43 + # response that overruns future canonical maxItems/maxLength bounds will fail 44 + # loudly with SchemaValidationError. 45 + 46 + 47 + def unsupported_keyword_hits(schema: dict[str, Any] | None, provider: str) -> list[str]: 48 + """Return JSON-pointer-ish paths for keywords unsupported by ``provider``.""" 49 + if schema is None: 50 + return [] 51 + 52 + unsupported = STRICT_UNSUPPORTED_KEYWORDS.get(provider, frozenset()) 53 + if not unsupported: 54 + return [] 55 + 56 + found: list[str] = [] 57 + 58 + def walk(node: Any, path: str) -> None: 59 + if isinstance(node, dict): 60 + for key, value in node.items(): 61 + child_path = f"{path}/{key}" 62 + if key in unsupported: 63 + found.append(child_path) 64 + walk(value, child_path) 65 + elif isinstance(node, list): 66 + for index, value in enumerate(node): 67 + walk(value, f"{path}[{index}]") 68 + 69 + walk(schema, "$") 70 + return found 71 + 72 + 73 + def prepare_provider_schema( 74 + schema: dict[str, Any] | None, provider: str 75 + ) -> dict[str, Any] | None: 76 + """Return a provider-facing copy of ``schema`` with unsupported keys removed.""" 77 + if schema is None: 78 + return None 79 + 80 + prepared = copy.deepcopy(schema) 81 + unsupported = STRICT_UNSUPPORTED_KEYWORDS.get(provider, frozenset()) 82 + if not unsupported: 83 + return prepared 84 + 85 + removed: list[str] = [] 86 + 87 + def walk(node: Any, path: str) -> None: 88 + if isinstance(node, dict): 89 + for key in list(node): 90 + child_path = f"{path}/{key}" 91 + if key in unsupported: 92 + node.pop(key) 93 + removed.append(child_path) 94 + continue 95 + walk(node[key], child_path) 96 + elif isinstance(node, list): 97 + for index, value in enumerate(node): 98 + walk(value, f"{path}[{index}]") 99 + 100 + walk(prepared, "$") 101 + 102 + if removed: 103 + logger.debug( 104 + "Removed %d unsupported JSON Schema keyword(s) for provider %s", 105 + len(removed), 106 + provider, 107 + ) 108 + 109 + return prepared
+142
tests/eval_schemas.py
··· 1 + #!/usr/bin/env python3 2 + # SPDX-License-Identifier: AGPL-3.0-only 3 + # Copyright (c) 2026 sol pbc 4 + 5 + """Opt-in local structured-output schema eval harness.""" 6 + 7 + from __future__ import annotations 8 + 9 + import argparse 10 + import json 11 + import sys 12 + from pathlib import Path 13 + from typing import Any 14 + 15 + ROOT = Path(__file__).resolve().parents[1] 16 + 17 + if str(ROOT) not in sys.path: 18 + sys.path.insert(0, str(ROOT)) 19 + 20 + from solstone.think.models import generate_with_result # noqa: E402 21 + from solstone.think.providers.local import LocalProviderError # noqa: E402 22 + from solstone.think.schema_eval import ( # noqa: E402 23 + content_preservation, 24 + schema_validity, 25 + ) 26 + 27 + DEFAULT_CASES = ROOT / "tests" / "fixtures" / "schema_eval" / "cases.jsonl" 28 + DEFAULT_OUT = ROOT / "tmp" / "schema-eval" 29 + LOCAL_NOT_READY = ( 30 + "Local schema eval requires the bundled local provider. Run " 31 + "`journal install-provider local`, then start it with `journal start` " 32 + "(or `journal service start` for an installed service)." 33 + ) 34 + 35 + 36 + def _resolve_path(path: Path) -> Path: 37 + if path.is_absolute(): 38 + return path 39 + return ROOT / path 40 + 41 + 42 + def load_cases(path: Path) -> list[dict[str, Any]]: 43 + cases: list[dict[str, Any]] = [] 44 + with path.open(encoding="utf-8") as handle: 45 + for lineno, line in enumerate(handle, start=1): 46 + stripped = line.strip() 47 + if not stripped: 48 + continue 49 + case = json.loads(stripped) 50 + if "schema_path" in case: 51 + schema_path = _resolve_path(Path(case["schema_path"])) 52 + case["schema"] = json.loads(schema_path.read_text(encoding="utf-8")) 53 + cases.append(case) 54 + return cases 55 + 56 + 57 + def run_case(case: dict[str, Any]) -> dict[str, Any]: 58 + result = generate_with_result( 59 + contents=case["input"], 60 + context="schema.eval", 61 + provider="local", 62 + temperature=0.0, 63 + max_output_tokens=512, 64 + system_instruction=case["system_instruction"], 65 + json_output=True, 66 + json_schema=case["schema"], 67 + ) 68 + text = result["text"] 69 + return { 70 + "name": case["name"], 71 + "text": text, 72 + "schema_validity": schema_validity(text, case["schema"]), 73 + "content_preservation": content_preservation( 74 + text, case.get("expect_contains", []) 75 + ), 76 + "finish_reason": result.get("finish_reason"), 77 + "model": result.get("model"), 78 + "usage": result.get("usage"), 79 + } 80 + 81 + 82 + def _atomic_write_text(path: Path, text: str) -> None: 83 + tmp = path.with_suffix(path.suffix + ".tmp") 84 + tmp.write_text(text, encoding="utf-8") 85 + tmp.replace(path) 86 + 87 + 88 + def write_outputs(out_dir: Path, results: list[dict[str, Any]]) -> None: 89 + out_dir.mkdir(parents=True, exist_ok=True) 90 + jsonl = "".join(json.dumps(result, sort_keys=True) + "\n" for result in results) 91 + _atomic_write_text(out_dir / "results.jsonl", jsonl) 92 + 93 + valid_count = sum(1 for result in results if result["schema_validity"]["valid"]) 94 + average_content = ( 95 + sum(result["content_preservation"]["fraction"] for result in results) 96 + / len(results) 97 + if results 98 + else 1.0 99 + ) 100 + summary = ( 101 + f"cases: {len(results)}\n" 102 + f"schema_valid: {valid_count}/{len(results)}\n" 103 + f"average_content_preservation: {average_content:.3f}\n" 104 + ) 105 + _atomic_write_text(out_dir / "summary.txt", summary) 106 + 107 + 108 + def main(argv: list[str] | None = None) -> int: 109 + parser = argparse.ArgumentParser(description="local structured schema eval") 110 + parser.add_argument("--cases", type=Path, default=DEFAULT_CASES) 111 + parser.add_argument("--out", type=Path, default=DEFAULT_OUT) 112 + args = parser.parse_args(argv) 113 + 114 + cases = load_cases(_resolve_path(args.cases)) 115 + results: list[dict[str, Any]] = [] 116 + for case in cases: 117 + try: 118 + results.append(run_case(case)) 119 + except LocalProviderError as exc: 120 + if exc.reason_code == "local_model_not_ready": 121 + print(LOCAL_NOT_READY, file=sys.stderr) 122 + return 2 123 + raise 124 + 125 + out_dir = _resolve_path(args.out) 126 + write_outputs(out_dir, results) 127 + 128 + valid_count = sum(1 for result in results if result["schema_validity"]["valid"]) 129 + average_content = ( 130 + sum(result["content_preservation"]["fraction"] for result in results) 131 + / len(results) 132 + if results 133 + else 1.0 134 + ) 135 + print(f"schema eval wrote {out_dir}") 136 + print(f"schema-valid: {valid_count}/{len(results)}") 137 + print(f"average content preservation: {average_content:.3f}") 138 + return 0 139 + 140 + 141 + if __name__ == "__main__": 142 + raise SystemExit(main())
+3
tests/fixtures/schema_eval/cases.jsonl
··· 1 + {"name":"bounded_snacks","schema":{"type":"object","properties":{"snacks":{"type":"array","maxItems":2,"items":{"type":"string","maxLength":20}}},"required":["snacks"],"additionalProperties":false},"system_instruction":"Return only JSON matching the schema. Extract the requested facts from the input.","input":"Extract up to two snack names from this list: apple, banana, cherry, date, elderberry. Prefer apple and banana.","expect_contains":["apple","banana"]} 2 + {"name":"contact_card","schema":{"type":"object","properties":{"name":{"type":"string","maxLength":60},"role":{"type":"string","maxLength":80},"summary":{"type":"string","maxLength":160}},"required":["name","role","summary"],"additionalProperties":false},"system_instruction":"Return only JSON matching the schema. Preserve the named person and role.","input":"Mira Patel is the incident coordinator. She is tracking database failover status for the evening handoff.","expect_contains":["Mira Patel","incident coordinator","database failover"]} 3 + {"name":"decision_flags","schema":{"type":"object","properties":{"decision":{"type":"string","maxLength":120},"approved":{"type":"boolean"},"tags":{"type":"array","maxItems":3,"items":{"type":"string","enum":["billing","security","launch","support"]}}},"required":["decision","approved","tags"],"additionalProperties":false},"system_instruction":"Return only JSON matching the schema. Capture the decision and select applicable tags.","input":"The team approved delaying the launch until the security review finishes. Billing and support are not involved.","expect_contains":["delaying","security","launch"]}
+79
tests/test_schema_bounds.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + from __future__ import annotations 5 + 6 + from typing import Any 7 + 8 + import pytest 9 + 10 + from solstone.think.schema_bounds import unbounded_nodes 11 + 12 + 13 + def _schema(property_schema: dict[str, Any]) -> dict[str, Any]: 14 + return { 15 + "type": "object", 16 + "properties": {"field": property_schema}, 17 + "required": ["field"], 18 + "additionalProperties": False, 19 + } 20 + 21 + 22 + def test_bounded_array_passes() -> None: 23 + assert unbounded_nodes(_schema({"type": "array", "maxItems": 3})) == [] 24 + 25 + 26 + def test_unbounded_array_fails() -> None: 27 + assert unbounded_nodes(_schema({"type": "array"})) == ["$/properties/field"] 28 + 29 + 30 + @pytest.mark.parametrize( 31 + "property_schema", 32 + [ 33 + {"type": "string", "enum": ["a"]}, 34 + {"type": "string", "const": "a"}, 35 + {"type": "string", "pattern": "^[a-z]+$"}, 36 + {"type": "string", "format": "date-time"}, 37 + ], 38 + ) 39 + def test_constrained_strings_pass(property_schema: dict[str, Any]) -> None: 40 + assert unbounded_nodes(_schema(property_schema)) == [] 41 + 42 + 43 + def test_bare_string_fails() -> None: 44 + assert unbounded_nodes(_schema({"type": "string"})) == ["$/properties/field"] 45 + 46 + 47 + def test_bounded_string_passes() -> None: 48 + assert unbounded_nodes(_schema({"type": "string", "maxLength": 80})) == [] 49 + 50 + 51 + def test_nested_arrays_and_objects_report_paths() -> None: 52 + schema = { 53 + "type": "object", 54 + "properties": { 55 + "groups": { 56 + "type": "array", 57 + "items": { 58 + "type": "object", 59 + "properties": { 60 + "title": {"type": ["string", "null"]}, 61 + "codes": { 62 + "type": ["array", "null"], 63 + "maxItems": 3, 64 + "items": {"type": "string", "pattern": "^[A-Z]+$"}, 65 + }, 66 + }, 67 + "required": ["title", "codes"], 68 + "additionalProperties": False, 69 + }, 70 + } 71 + }, 72 + "required": ["groups"], 73 + "additionalProperties": False, 74 + } 75 + 76 + assert unbounded_nodes(schema) == [ 77 + "$/properties/groups", 78 + "$/properties/groups/items/properties/title", 79 + ]
+71
tests/test_schema_eval.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + from __future__ import annotations 5 + 6 + from solstone.think.schema_eval import content_preservation, schema_validity 7 + 8 + 9 + def test_schema_validity_accepts_valid_response() -> None: 10 + result = schema_validity( 11 + '{"items": ["alpha"]}', 12 + { 13 + "type": "object", 14 + "properties": { 15 + "items": {"type": "array", "items": {"type": "string"}}, 16 + }, 17 + "required": ["items"], 18 + "additionalProperties": False, 19 + }, 20 + ) 21 + 22 + assert result == {"valid": True, "errors": []} 23 + 24 + 25 + def test_schema_validity_reports_invalid_json() -> None: 26 + result = schema_validity("{", {"type": "object"}) 27 + 28 + assert result["valid"] is False 29 + assert result["errors"][0]["path"] == "$" 30 + 31 + 32 + def test_schema_validity_reports_schema_errors() -> None: 33 + result = schema_validity( 34 + '{"items": ["alpha", "beta"]}', 35 + { 36 + "type": "object", 37 + "properties": { 38 + "items": { 39 + "type": "array", 40 + "maxItems": 1, 41 + "items": {"type": "string"}, 42 + }, 43 + }, 44 + "required": ["items"], 45 + "additionalProperties": False, 46 + }, 47 + ) 48 + 49 + assert result["valid"] is False 50 + assert result["errors"][0]["path"] == "$/items" 51 + 52 + 53 + def test_content_preservation_scores_case_insensitive_matches() -> None: 54 + result = content_preservation( 55 + '{"summary": "Alpha and beta are present."}', 56 + ["alpha", "BETA", "gamma"], 57 + ) 58 + 59 + assert result == { 60 + "fraction": 2 / 3, 61 + "found": ["alpha", "BETA"], 62 + "missing": ["gamma"], 63 + } 64 + 65 + 66 + def test_content_preservation_empty_needles_passes() -> None: 67 + assert content_preservation("anything", []) == { 68 + "fraction": 1.0, 69 + "found": [], 70 + "missing": [], 71 + }
+168
tests/test_schema_prep.py
··· 1 + # SPDX-License-Identifier: AGPL-3.0-only 2 + # Copyright (c) 2026 sol pbc 3 + 4 + from __future__ import annotations 5 + 6 + import copy 7 + import json 8 + from pathlib import Path 9 + from types import SimpleNamespace 10 + from typing import Any 11 + from unittest.mock import MagicMock, patch 12 + 13 + import pytest 14 + 15 + from solstone.apps.timeline.rollup import build_rollup_schema 16 + from solstone.think.models import SchemaValidationError, generate 17 + from solstone.think.schema_prep import prepare_provider_schema 18 + 19 + REPO_ROOT = Path(__file__).resolve().parents[1] 20 + 21 + 22 + @pytest.fixture 23 + def bounded_schema() -> dict[str, Any]: 24 + return { 25 + "type": "object", 26 + "properties": { 27 + "labels": { 28 + "type": "array", 29 + "maxItems": 2, 30 + "items": { 31 + "type": "string", 32 + "maxLength": 12, 33 + "pattern": "^[a-z]+$", 34 + "enum": ["alpha", "beta"], 35 + }, 36 + } 37 + }, 38 + "required": ["labels"], 39 + "additionalProperties": False, 40 + } 41 + 42 + 43 + def _discover_schemas() -> tuple[dict[str, Any], ...]: 44 + discovered: list[dict[str, Any]] = [] 45 + for path in sorted((REPO_ROOT / "solstone").glob("**/*.schema.json")): 46 + schema = json.loads(path.read_text(encoding="utf-8")) 47 + if isinstance(schema.get("x-journal-contract"), dict): 48 + continue 49 + discovered.append(schema) 50 + discovered.append(build_rollup_schema(3)) 51 + return tuple(discovered) 52 + 53 + 54 + def test_local_receives_canonical_copy(bounded_schema: dict[str, Any]) -> None: 55 + prepared = prepare_provider_schema(bounded_schema, "local") 56 + 57 + assert prepared == bounded_schema 58 + assert prepared is not bounded_schema 59 + 60 + 61 + @pytest.mark.parametrize("provider", ["openai", "google"]) 62 + def test_openai_and_google_keep_array_bounds_and_strip_length_bounds( 63 + bounded_schema: dict[str, Any], provider: str 64 + ) -> None: 65 + prepared = prepare_provider_schema(bounded_schema, provider) 66 + 67 + labels = prepared["properties"]["labels"] # type: ignore[index] 68 + item = labels["items"] 69 + assert labels["maxItems"] == 2 70 + assert "maxLength" not in item 71 + assert item["pattern"] == "^[a-z]+$" 72 + assert item["enum"] == ["alpha", "beta"] 73 + 74 + 75 + def test_anthropic_strips_array_and_length_bounds( 76 + bounded_schema: dict[str, Any], 77 + ) -> None: 78 + prepared = prepare_provider_schema(bounded_schema, "anthropic") 79 + 80 + labels = prepared["properties"]["labels"] # type: ignore[index] 81 + item = labels["items"] 82 + assert "maxItems" not in labels 83 + assert "maxLength" not in item 84 + assert item["pattern"] == "^[a-z]+$" 85 + assert item["enum"] == ["alpha", "beta"] 86 + 87 + 88 + @pytest.mark.parametrize("provider", ["local", "openai", "google", "anthropic", "fake"]) 89 + def test_prepare_provider_schema_is_pure_and_idempotent( 90 + bounded_schema: dict[str, Any], provider: str 91 + ) -> None: 92 + original = copy.deepcopy(bounded_schema) 93 + prepared = prepare_provider_schema(bounded_schema, provider) 94 + 95 + assert bounded_schema == original 96 + assert prepare_provider_schema(prepared, provider) == prepared 97 + 98 + 99 + def test_none_and_unknown_provider_passthrough( 100 + bounded_schema: dict[str, Any], 101 + ) -> None: 102 + assert prepare_provider_schema(None, "openai") is None 103 + 104 + prepared = prepare_provider_schema(bounded_schema, "fake") 105 + assert prepared == bounded_schema 106 + assert prepared is not bounded_schema 107 + 108 + 109 + @pytest.mark.parametrize("provider", ["local", "openai", "google", "anthropic"]) 110 + @pytest.mark.parametrize("schema", _discover_schemas()) 111 + def test_current_shipped_schemas_are_byte_identical_after_prep( 112 + schema: dict[str, Any], provider: str 113 + ) -> None: 114 + assert prepare_provider_schema(schema, provider) == schema 115 + 116 + 117 + def _patched_generate( 118 + provider: str, schema: dict[str, Any], response_text: str 119 + ) -> MagicMock: 120 + """Run ``generate`` against a stubbed provider module; return its mock.""" 121 + run_generate = MagicMock( 122 + return_value={"text": response_text, "finish_reason": "stop"} 123 + ) 124 + provider_module = SimpleNamespace(run_generate=run_generate) 125 + 126 + with ( 127 + patch( 128 + "solstone.think.models.resolve_provider", 129 + return_value=(provider, "model"), 130 + ), 131 + patch( 132 + "solstone.think.providers.get_provider_module", 133 + return_value=provider_module, 134 + ), 135 + ): 136 + generate("hello", "test.context", json_schema=schema) 137 + 138 + return run_generate 139 + 140 + 141 + def test_generate_sends_reduced_schema_to_anthropic( 142 + bounded_schema: dict[str, Any], 143 + ) -> None: 144 + run_generate = _patched_generate("anthropic", bounded_schema, '{"labels": []}') 145 + 146 + sent = run_generate.call_args.kwargs["json_schema"] 147 + assert sent == prepare_provider_schema(bounded_schema, "anthropic") 148 + assert "maxItems" not in sent["properties"]["labels"] 149 + assert bounded_schema["properties"]["labels"]["maxItems"] == 2 150 + 151 + 152 + def test_generate_sends_canonical_schema_to_local( 153 + bounded_schema: dict[str, Any], 154 + ) -> None: 155 + run_generate = _patched_generate("local", bounded_schema, '{"labels": []}') 156 + 157 + assert run_generate.call_args.kwargs["json_schema"] == bounded_schema 158 + 159 + 160 + def test_generate_validates_response_against_canonical_schema( 161 + bounded_schema: dict[str, Any], 162 + ) -> None: 163 + """D4: bounds stripped from the anthropic request are still enforced on the 164 + response. A 3-item answer overruns the canonical maxItems:2 and fails loudly.""" 165 + overrun = '{"labels": ["alpha", "beta", "alpha"]}' 166 + 167 + with pytest.raises(SchemaValidationError): 168 + _patched_generate("anthropic", bounded_schema, overrun)
+33 -37
tests/test_schema_strict_portability.py
··· 1 1 # SPDX-License-Identifier: AGPL-3.0-only 2 2 # Copyright (c) 2026 sol pbc 3 3 4 - """Offline CI gate for req_bfbdbux6 strict schema portability.""" 4 + """Offline CI gate for strict structured-output schema portability.""" 5 5 6 6 from __future__ import annotations 7 7 ··· 12 12 import pytest 13 13 14 14 from solstone.apps.timeline.rollup import build_rollup_schema 15 + from solstone.think.schema_prep import ( 16 + prepare_provider_schema, 17 + unsupported_keyword_hits, 18 + ) 15 19 from solstone.think.talent import hydrate_runtime_enums 16 20 17 21 REPO_ROOT = Path(__file__).resolve().parents[1] 18 - BANNED_KEYS = frozenset( 19 - { 20 - "$schema", 21 - "$comment", 22 - "minLength", 23 - "maxLength", 24 - "minItems", 25 - "maxItems", 26 - "minimum", 27 - "maximum", 28 - } 29 - ) 22 + STRICT_PROVIDERS = ("openai", "anthropic", "google") 30 23 31 24 32 25 def _discover_schemas() -> tuple[tuple[str, dict[str, Any]], ...]: ··· 56 49 def walk(node: Any, path: str) -> None: 57 50 if isinstance(node, dict): 58 51 for key in node: 59 - if key in BANNED_KEYS: 60 - found.append(f"{path}: banned key {key!r}") 61 52 if key == "oneOf": 62 53 found.append(f"{path}: banned key 'oneOf'") 63 54 ··· 80 71 return found 81 72 82 73 83 - def banned_key_hits(schema: dict[str, Any]) -> list[str]: 84 - found: list[str] = [] 85 - 86 - def walk(node: Any, path: str) -> None: 87 - if isinstance(node, dict): 88 - for key, value in node.items(): 89 - if key in BANNED_KEYS: 90 - found.append(f"{path}: banned key {key!r}") 91 - walk(value, f"{path}/{key}") 92 - elif isinstance(node, list): 93 - for index, value in enumerate(node): 94 - walk(value, f"{path}[{index}]") 95 - 96 - walk(schema, "$") 97 - return found 98 - 99 - 100 74 @pytest.mark.parametrize( 101 75 ("schema_id", "schema"), 102 76 [pytest.param(schema_id, schema, id=schema_id) for schema_id, schema in SCHEMAS], ··· 108 82 assert schema_violations == [], f"{schema_id}: {schema_violations}" 109 83 110 84 85 + @pytest.mark.parametrize("provider", STRICT_PROVIDERS) 86 + @pytest.mark.parametrize( 87 + ("schema_id", "schema"), 88 + [pytest.param(schema_id, schema, id=schema_id) for schema_id, schema in SCHEMAS], 89 + ) 90 + def test_prepared_schemas_have_no_provider_unsupported_keywords( 91 + schema_id: str, schema: dict[str, Any], provider: str 92 + ) -> None: 93 + prepared = prepare_provider_schema(schema, provider) 94 + 95 + assert prepared is not None 96 + assert unsupported_keyword_hits(prepared, provider) == [], schema_id 97 + 98 + 111 99 @pytest.mark.parametrize( 112 100 "schema_path", 113 101 [ ··· 123 111 124 112 hydrated = hydrate_runtime_enums(schema) 125 113 126 - assert banned_key_hits(hydrated) == [] 114 + for provider in STRICT_PROVIDERS: 115 + prepared = prepare_provider_schema(hydrated, provider) 116 + assert prepared is not None 117 + assert unsupported_keyword_hits(prepared, provider) == [] 127 118 128 119 129 120 @pytest.mark.parametrize( ··· 131 122 [ 132 123 { 133 124 "type": "object", 134 - "$comment": "bad", 135 125 "properties": { 136 - "a": {"type": "array", "minItems": 1}, 126 + "a": {"type": "array"}, 137 127 "b": {"type": "string"}, 128 + "c": {"oneOf": [{"type": "string"}, {"type": "integer"}]}, 138 129 }, 139 130 "required": ["a"], 140 - "additionalProperties": False, 141 131 } 142 132 ], 143 133 ) 144 134 def test_strict_portability_guard_rejects_bad_schema(schema: dict[str, Any]) -> None: 145 - assert violations(schema) 135 + schema_violations = violations(schema) 136 + 137 + assert any( 138 + "object missing additionalProperties:false" in v for v in schema_violations 139 + ) 140 + assert any("properties not required" in v for v in schema_violations) 141 + assert any("banned key 'oneOf'" in v for v in schema_violations)