Fix structured terminal honesty follow-ups
Enforce the structured terminal invariant everywhere a request can finish: any terminal GenerateCompletionInfo for a request with parameters.structuredOutput != nil now carries a non-nil StructuredOutputResult. This closes the remaining .cancelled + nil holes in cancelledBatchStream/cancelledGenerationStream for shutdown, acceleration rejection, and raw native-MTP submit rejection; the public BatchEngine.generate defensive no-info wrapper; solo fast-path setup failure including non-StructuredOutputError throws; and Evaluate.generateLoopTask cancellation/general iterator-construction catches. Non-structured requests still report nil at the same sites.
Fix the llguidance poison bypass in commit(tokenID:). Stop tokens were handled before the poison check, so a poisoned box receiving an accepting EOS could return success instead of the original poison reason. Stop-token handling now runs under the lock after throwIfPoisonedLocked(), preserving healthy accepting EOS success and pre-acceptance stop-token dead-end behavior.
Restore stop-string streaming semantics. Stop-string detection no longer forces detokenizer.flush() on every token, which had bypassed the 24-character holdback for all stop-string requests including structuredOutput == nil. Structured stop detection uses a separate accumulation buffer, and the scheduler yield is narrowed so ordinary stop-string streaming remains unchanged.
Update tests and probe coverage. The penalty-before-grammar test now exercises the split PreparedLogitProcessors path instead of putting a grammar processor inside CompositeLogitProcessor. The probe gate is VMLX_STRUCTURED_OUTPUT_PROBE=1 for all four operator modes, and mask-timing now reports warmed p50/p95 keys (empty_p50_ms, empty_p95_ms, prefix64_p50_ms, prefix64_p95_ms) from individual samples, with labels for lockstep vs constraint-only quantities. There are no timing assertions.
Docs now state the structuredOutput terminal contract and all rejected decode paths: speculative, native-MTP, and block-diffusion.
Executable coverage added: structured vs non-structured contrast at cancellation/rejection sites; poisoned box plus accepting EOS commit fails with the original poison reason; ordinary stop-string requests keep unchanged chunk boundaries and a nil structured terminal field. Focused suites are now 17 + 19 tests.
Validation on this tree: swift build -c release completed; swift test --filter StructuredOutputFocusedTests passed 17 tests; swift test --filter StructuredOutputTerminalFocusedTests passed 19 tests; git diff --check was clean. Broad swift test was not run to green because it deadlocks on the pre-existing FocusedMLXTestSupport process-wide POSIX semaphore /vmlx_mlx_lock at a suite boundary, with no assertion failure; this is pre-existing harness/environment behavior and not caused by this change.
Fix structured terminal honesty follow-ups
Enforce the structured terminal invariant everywhere a request can finish: any terminal GenerateCompletionInfo for a request with parameters.structuredOutput != nil now carries a non-nil StructuredOutputResult. This closes the remaining .cancelled + nil holes in cancelledBatchStream/cancelledGenerationStream for shutdown, acceleration rejection, and raw native-MTP submit rejection; the public BatchEngine.generate defensive no-info wrapper; solo fast-path setup failure including non-StructuredOutputError throws; and Evaluate.generateLoopTask cancellation/general iterator-construction catches. Non-structured requests still report nil at the same sites.
Fix the llguidance poison bypass in commit(tokenID:). Stop tokens were handled before the poison check, so a poisoned box receiving an accepting EOS could return success instead of the original poison reason. Stop-token handling now runs under the lock after throwIfPoisonedLocked(), preserving healthy accepting EOS success and pre-acceptance stop-token dead-end behavior.
Restore stop-string streaming semantics. Stop-string detection no longer forces detokenizer.flush() on every token, which had bypassed the 24-character holdback for all stop-string requests including structuredOutput == nil. Structured stop detection uses a separate accumulation buffer, and the scheduler yield is narrowed so ordinary stop-string streaming remains unchanged.
Update tests and probe coverage. The penalty-before-grammar test now exercises the split PreparedLogitProcessors path instead of putting a grammar processor inside CompositeLogitProcessor. The probe gate is VMLX_STRUCTURED_OUTPUT_PROBE=1 for all four operator modes, and mask-timing now reports warmed p50/p95 keys (empty_p50_ms, empty_p95_ms, prefix64_p50_ms, prefix64_p95_ms) from individual samples, with labels for lockstep vs constraint-only quantities. There are no timing assertions.
Docs now state the structuredOutput terminal contract and all rejected decode paths: speculative, native-MTP, and block-diffusion.
Executable coverage added: structured vs non-structured contrast at cancellation/rejection sites; poisoned box plus accepting EOS commit fails with the original poison reason; ordinary stop-string requests keep unchanged chunk boundaries and a nil structured terminal field. Focused suites are now 17 + 19 tests.
Validation on this tree: swift build -c release completed; swift test --filter StructuredOutputFocusedTests passed 17 tests; swift test --filter StructuredOutputTerminalFocusedTests passed 19 tests; git diff --check was clean. Broad swift test was not run to green because it deadlocks on the pre-existing FocusedMLXTestSupport process-wide POSIX semaphore /vmlx_mlx_lock at a suite boundary, with no assertion failure; this is pre-existing harness/environment behavior and not caused by this change.
Deliver the prose that precedes a tool call in full
Reported from agent runs on Ornith: the assistant's visible sentence cut
off mid-word right before every tool call — "…finalize the Pipe function
in the right plac", "Let me clean up the smoke test and". Reproduced
4/8 turns at temperature 0 on ornith-1.0-9b-mxfp8 ("…removing the
temporary director"). The missing tail ranged from two characters to a
whole clause.
Layer-by-layer tracing (raw tokens -> detokenizer -> reasoning parser ->
tool processor -> stop matcher -> wire) found two independent drops, both
on the boundary between the answer text and the tool-call envelope:
1. `ToolCallProcessor.processEOS` discarded `leadingTextBeforeToolCall`
whenever the buffered envelope PARSED. The prose sharing a chunk with
the envelope's start tag is stashed there, and Qwen3.5-family models
stop generating right after `</tool_call>` — the envelope's last
characters routinely arrive only in the end-of-stream flush, so the
call completes in `processEOS`, not on the streaming end-tag path
(which already returned the stash). Successful parse now surfaces the
held prose through the same `visibleLeadingTextBeforeToolCall` helper
the streaming path uses.
2. `TextToolTokenLoopHandler` emitted the stop-string matcher's held tail
AFTER the `.toolCall` event (the matcher drain sits at the end of
`onGenerationEnd`). Consumers deliberately suppress all text once a
tool call lands — the no-leak guard for post-tool prose — so a tail
emitted after the event is silently dropped even though it precedes
the call in the model's output. Live trace: the engine's parser
returned "…directory at /tmp/smoke.\n\n" in full, but everything past
the matcher's holdback point died downstream. `emitRouted` now drains
the matcher before any `.toolCall` event goes out. A stop string can
no longer straddle the call boundary; stops match the assistant's
answer span, which the call legitimately ends.
After both: 0/7 truncations on the same repro (previously 4/8), sentences
complete on both sides of every tool call, on ornith-1.0-9b-mxfp8 and
lfm2.5-8b-a1b-mxfp8; tool calls still parse with correct arguments and
nothing leaks after them.
New tests pin the invariants byte-faithfully to the live chunk shapes:
ProseBeforeToolCallEOSTests (EOS-completed envelope returns the prose;
mid-stream completion unchanged; failed parses still flush; single-chunk
envelopes) and ToolCallStopMatcherOrderingTests (matcher-held text
precedes the `.toolCall` event; nothing trails it).
The pre-existing failure in ToolCallEdgeCasesTests ("LFM2 processor
accepts observed OpenAI tool_call JSON envelope", visible "\n") fails
identically on main and is untouched by this change.
Make the memory-safety level reach the prefix-cache store, and stop overcounting it
Two defects, both in the guard added by #139/#140.
The "Safety Level" slider did nothing. `VMLXMemorySafetySettings.slider` was a
stored field that no resolver ever read -- every one of them switches on `mode` --
so a host that wired its slider to it (osaurus does) shipped a control that changed
nothing, while the resolved-plan readout printed the new level beside the old,
still-in-force caps. `slider` is now a projection of `mode`: same five choices, in
`allCases` order, so the two controls cannot disagree and the slider does what it
has always claimed to. Decoding keeps `mode` authoritative, so an existing user's
level does not shift under them on upgrade -- only the number shown for it gets
honest. The 0...4 validation error is gone because it is no longer expressible.
And the store budget ignored the level entirely: it gated a KV copy against raw
physical RAM and a hard-coded margin, identically whether the user asked for
Performance or Strict. It now spends a share of *free headroom* set by the level.
A share of the increment, deliberately -- not a ceiling on total memory. The load
caps are an MLX graph-scheduling guideline, not a hard resident limit, so a large
pack legitimately runs above them (Hy3 sits at ~96 GB under the default 0.70 cap
on a 128 GB Mac); budgeting the store against `0.70 * physical` would put
`activeBytes` over the line before a single KV byte and refuse *every* store,
silently killing the prefix cache for exactly the models whose re-prefill hurts
most. `CacheStoreBudget.policy` carries the level down to the decode loop, which
has no settings handle -- the same channel `TiedHeadQuantizationPolicy.current`
already uses.
The materialization factor was also wrong. It claimed three copies -- a deep copy,
a host `Data` extract, and the disk-store cache -- and none of the three survive
contact with the code: `KVCacheSimple.copy()` takes full-range slices that share
the source buffer, `extractLayerData` returns references, `makeDiskStoreCache`
returns the snapshot unchanged on the raw path, and `mlx_save_safetensors` streams
to the file descriptor rather than building a host `Data`. What is left is the
`contiguous` pass before the write, worth at most one compact copy. The reporter's
own footprint series said so all along: a +23 GiB excursion against a 23 GiB cache.
At 3x the guard refused stores that fit, quietly costing cache hits it was never
meant to cost.
The panic that motivated the guard is still refused, the 8 GB host still caches,
and a 16k-token conversation under a 94 GiB resident pack still caches. Strict and
Performance now reach different verdicts on the same host, model and cache, which
is the entire point.
Baseline: the two `VMLXMemorySafetySettingsTests` failures on this branch
(`defaultMaxKVSize` 65536 != 8192, 16384 != 4096) reproduce identically on clean
main and are untouched by this change.
extro.rook.host