bot/docs/internal/ at main · zzstoatzz.io/bot · Tangled

zzstoatzz.io / bot

0

a digital entity named phi that roams bsky phi.zzstoatzz.io

0

bot / docs / internal

1 file

at main

zzstoatzz measure the prompt cache instead of asserting it 2d ago

measure the prompt cache instead of asserting it phi's caching setup (1h tool defs + instructions, 5m messages, and `memoize_per_run` so a network-fetched context block can't shift the prefix mid-run) was carefully reasoned and never verified. a regression in any of it is invisible: the run still succeeds, just at full price. CacheObservingModel wraps the model and reads the provider's own cache_read_tokens / cache_write_tokens off each response. pydantic-ai 1.x has no after-model-request hook, so the model wrapper is the seam — the approach is borrowed from pydantic-ai-harness's CacheStabilityMonitor, which needs the 2.x capabilities API. two things it can see that nothing else could: - mid-run collapse: history is append-only within a run, so each request should read back at least what the last one cached. a drop below half the established prefix warns once and latches until a healthy read-back re-stabilizes. marks are per (provider, model) so the haiku sub-passes don't false-positive the main model. - cross-run carry: whether a run's first request reads back the 1h tool+instruction prefix, which is the entire premise of choosing 1h. observation is best-effort — it can never take a run down — and end_run sits in a finally, since a failed run still spent input tokens. /api/cache serves the rolling window; the cockpit's /operator page renders one stacked bar per run (read / written at a premium / paid in full) with collapse flags and a carry marker. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2 days ago