solpbc.org / solstone
personal memory agent
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

fix(think): strip maxItems for the google provider

On the google BYO lane, litellm passes our JSON Schema verbatim as Gemini's responseJsonSchema. Gemini compiles maxItems into a bounded decoding grammar with an undocumented, additive, whole-schema budget; over budget it returns a bare 400 INVALID_ARGUMENT naming no keyword, path, or limit. 7 of the 8 shipped schemas carrying maxItems are rejected as-is; all 8 pass stripped. Because sense is the floor talent (schedule: segment, priority: 5), its 400 meant no segment talent ran at all on google while brain health still read "sol can think".

The root cause is that the whole-schema grammar budget is additive across every bounded array and scales with N x item complexity, not a per-keyword or per-array limit. Tuning individual maxItems values is therefore not the fix, no schema value was changed, and minItems is free.

The change adds one keyword to the google row of STRICT_UNSUPPORTED_KEYWORDS, corrects the provider comments, and updates the tests that previously asserted the broken behavior.

Residual risk: stripping maxItems from the google request means Gemini is no longer grammar-constrained at generation time, while models.py still validates the response against the canonical schema and x-truncate has no array path. The failure mode changes shape rather than disappearing: before, 400 INVALID_ARGUMENT on every segment, always; after, a segment dense enough to produce more than 96 entities raises SchemaValidationError instead of being grammar-clamped. That is a strictly better trade, a loud, specific, occasional failure replacing a silent, total one, and it is the posture Anthropic already ships under.

Second-order: this makes the entire segment talent chain reachable on the google lane for the first time. Entities, facets, screen, speaker attribution, and segment summaries have never executed there.

author
Jer Miller date (Jul 25, 2026, 10:45 AM -0600) commit 8e2ec263 parent b0d2436c
+135 -13
+23 -5
solstone/think/schema_prep.py
··· 25 25 # strip minItems wholesale here for deterministic simplicity. Pattern is 26 26 # documented as supported. 27 27 # - Google: https://ai.google.dev/gemini-api/docs/structured-output 28 - # supports string format, number minimum/maximum, and array minItems/maxItems. 28 + # lists minItems/maxItems as supported, and the API does *accept* both -- but 29 + # responseJsonSchema compiles maxItems into a bounded decoding grammar whose 30 + # expansion has an undocumented WHOLE-SCHEMA budget. Over budget the API 31 + # returns a bare 400 INVALID_ARGUMENT naming no keyword, path, or limit. The 32 + # cost is additive across every bounded array and scales with N x item 33 + # complexity. Measured 2026-07-25 on gemini-3.5-flash (also 3.5-pro and 34 + # 2.5-flash): sense.schema.json's entities array alone passes at maxItems<=27 35 + # and fails at 28; entities=27 plus speakers=16 fails though each passes 36 + # alone; 7 of the 8 shipped schemas carrying maxItems are rejected outright 37 + # and all 8 pass with it stripped. minItems is free. So strip maxItems. 29 38 STRICT_UNSUPPORTED_KEYWORDS: dict[str, frozenset[str]] = { 30 39 "openai": frozenset( 31 40 {"$schema", "$comment", "minLength", "maxLength", SCHEMA_TRUNCATE_KEY} 32 41 ), 33 42 "google": frozenset( 34 - {"$schema", "$comment", "minLength", "maxLength", SCHEMA_TRUNCATE_KEY} 43 + { 44 + "$schema", 45 + "$comment", 46 + "minLength", 47 + "maxLength", 48 + "maxItems", 49 + SCHEMA_TRUNCATE_KEY, 50 + } 35 51 ), 36 52 "anthropic": frozenset( 37 53 { ··· 49 65 } 50 66 51 67 # Hazard: cloud-provider reductions are request-only. Canonical response 52 - # validation in models.py still enforces stripped bounds and annotations, so an 53 - # Anthropic response that overruns future canonical maxItems/maxLength bounds 68 + # validation in models.py still enforces stripped bounds and annotations, so a 69 + # Google or Anthropic response that overruns canonical maxItems/maxLength bounds 54 70 # raises on generate/agenerate or records invalid canonical validation on 55 71 # advisory result paths unless an honored annotation truncates that specific 56 - # instance path first. 72 + # instance path first. Google is the live case after the deliberate request 73 + # strip: the segment chain runs on Gemini with shipped sense.schema.json 74 + # maxItems bounds such as entities:96. 57 75 58 76 59 77 def unsupported_keyword_hits(schema: dict[str, Any] | None, provider: str) -> list[str]:
+112 -8
tests/test_schema_prep.py
··· 68 68 assert prepared is not bounded_schema 69 69 70 70 71 - @pytest.mark.parametrize("provider", ["openai", "google"]) 72 - def test_openai_and_google_keep_array_bounds_and_strip_length_bounds( 73 - bounded_schema: dict[str, Any], provider: str 71 + def test_openai_keeps_array_bounds_and_strips_length_bounds( 72 + bounded_schema: dict[str, Any], 74 73 ) -> None: 75 - prepared = prepare_provider_schema(bounded_schema, provider) 74 + prepared = prepare_provider_schema(bounded_schema, "openai") 76 75 77 76 labels = prepared["properties"]["labels"] # type: ignore[index] 78 77 item = labels["items"] ··· 80 79 assert "maxLength" not in item 81 80 assert item["pattern"] == "^[a-z]+$" 82 81 assert item["enum"] == ["alpha", "beta"] 82 + 83 + 84 + def test_google_strips_maxitems_and_length_bounds_but_keeps_supported_constraints() -> ( 85 + None 86 + ): 87 + schema = { 88 + "$schema": "https://json-schema.org/draft/2020-12/schema", 89 + "$comment": "provider prep strips unsupported annotations", 90 + "type": "object", 91 + "properties": { 92 + "labels": { 93 + "type": "array", 94 + "minItems": 1, 95 + "maxItems": 2, 96 + "items": { 97 + "type": "string", 98 + "minLength": 1, 99 + "maxLength": 12, 100 + "pattern": "^[a-z]+$", 101 + "enum": ["alpha", "beta"], 102 + SCHEMA_TRUNCATE_KEY: True, 103 + }, 104 + }, 105 + "score": {"type": "number", "minimum": 0, "maximum": 10}, 106 + }, 107 + "required": ["labels", "score"], 108 + "additionalProperties": False, 109 + } 110 + 111 + prepared = prepare_provider_schema(schema, "google") 112 + 113 + assert "$schema" not in prepared 114 + assert "$comment" not in prepared 115 + labels = prepared["properties"]["labels"] # type: ignore[index] 116 + item = labels["items"] 117 + assert "maxItems" not in labels 118 + assert labels["minItems"] == 1 119 + assert "minLength" not in item 120 + assert "maxLength" not in item 121 + assert SCHEMA_TRUNCATE_KEY not in item 122 + assert item["pattern"] == "^[a-z]+$" 123 + assert item["enum"] == ["alpha", "beta"] 124 + assert prepared["properties"]["score"] == { 125 + "type": "number", 126 + "minimum": 0, 127 + "maximum": 10, 128 + } 129 + assert prepared["required"] == ["labels", "score"] 130 + assert prepared["additionalProperties"] is False 83 131 84 132 85 133 def test_anthropic_strips_array_and_length_bounds( ··· 293 341 "solstone/apps/entities/talent/entity_observer.schema.json", 294 342 ) 295 343 344 + # Shipped schemas that currently carry canonical maxItems. This is provider-prep 345 + # coverage, not the schema-bounds ratchet tuple above. 346 + SHIPPED_SCHEMAS_WITH_MAX_ITEMS = ( 347 + "solstone/talent/sense.schema.json", 348 + "solstone/talent/screen.schema.json", 349 + "solstone/talent/documents.schema.json", 350 + "solstone/talent/story.schema.json", 351 + "solstone/apps/entities/talent/entity_observer.schema.json", 352 + "solstone/observe/categories/calendar.schema.json", 353 + "solstone/observe/categories/messaging.schema.json", 354 + "solstone/talent/morning_briefing.schema.json", 355 + ) 356 + 296 357 297 358 def _load_shipped_schema(relative_path: str) -> dict[str, Any]: 298 359 return json.loads((REPO_ROOT / relative_path).read_text(encoding="utf-8")) ··· 329 390 assert prepare_provider_schema(schema, "local") == schema 330 391 331 392 332 - @pytest.mark.parametrize("provider", ["openai", "google"]) 333 393 @pytest.mark.parametrize("relative_path", BOUNDED_SCHEMAS) 334 - def test_bounded_schemas_lose_only_maxlength_for_openai_and_google( 335 - relative_path: str, provider: str 394 + def test_bounded_schemas_lose_only_maxlength_for_openai( 395 + relative_path: str, 336 396 ) -> None: 337 - prepared = prepare_provider_schema(_load_shipped_schema(relative_path), provider) 397 + prepared = prepare_provider_schema(_load_shipped_schema(relative_path), "openai") 338 398 339 399 assert _keyword_paths(prepared, "maxLength") == [] 340 400 assert _keyword_paths(prepared, "maxItems") 341 401 342 402 343 403 @pytest.mark.parametrize("relative_path", BOUNDED_SCHEMAS) 404 + def test_bounded_schemas_lose_size_bounds_for_google(relative_path: str) -> None: 405 + prepared = prepare_provider_schema(_load_shipped_schema(relative_path), "google") 406 + 407 + assert _keyword_paths(prepared, "maxLength") == [] 408 + assert _keyword_paths(prepared, "maxItems") == [] 409 + 410 + 411 + @pytest.mark.parametrize("relative_path", BOUNDED_SCHEMAS) 344 412 def test_bounded_schemas_lose_every_size_bound_for_anthropic( 345 413 relative_path: str, 346 414 ) -> None: ··· 350 418 assert _keyword_paths(prepared, "maxItems") == [] 351 419 352 420 421 + @pytest.mark.parametrize("relative_path", SHIPPED_SCHEMAS_WITH_MAX_ITEMS) 422 + def test_google_strips_maxitems_from_every_shipped_schema_that_has_them( 423 + relative_path: str, 424 + ) -> None: 425 + """The maxLength check is conditional because sense carries no maxLength.""" 426 + canonical = _load_shipped_schema(relative_path) 427 + canonical_max_length_paths = _keyword_paths(canonical, "maxLength") 428 + assert _keyword_paths(canonical, "maxItems") 429 + 430 + prepared = prepare_provider_schema(canonical, "google") 431 + 432 + assert _keyword_paths(prepared, "maxItems") == [] 433 + if canonical_max_length_paths: 434 + assert _keyword_paths(prepared, "maxLength") == [] 435 + 436 + 353 437 def _patched_generate( 354 438 provider: str, schema: dict[str, Any], response_text: str 355 439 ) -> MagicMock: ··· 385 469 assert bounded_schema["properties"]["labels"]["maxItems"] == 2 386 470 387 471 472 + def test_generate_sends_reduced_schema_to_google( 473 + bounded_schema: dict[str, Any], 474 + ) -> None: 475 + run_generate = _patched_generate("google", bounded_schema, '{"labels": []}') 476 + 477 + sent = run_generate.call_args.kwargs["json_schema"] 478 + assert sent == prepare_provider_schema(bounded_schema, "google") 479 + assert "maxItems" not in sent["properties"]["labels"] 480 + assert bounded_schema["properties"]["labels"]["maxItems"] == 2 481 + 482 + 388 483 def test_generate_sends_canonical_schema_to_local( 389 484 bounded_schema: dict[str, Any], 390 485 ) -> None: ··· 402 497 403 498 with pytest.raises(SchemaValidationError): 404 499 _patched_generate("anthropic", bounded_schema, overrun) 500 + 501 + 502 + def test_generate_validates_google_response_against_canonical_schema( 503 + bounded_schema: dict[str, Any], 504 + ) -> None: 505 + overrun = '{"labels": ["alpha", "beta", "alpha"]}' 506 + 507 + with pytest.raises(SchemaValidationError): 508 + _patched_generate("google", bounded_schema, overrun)