extro.rook.host
Commits
The committed dylib was checksum-verified but not reproducible. Rebuilding it
produced different bytes every time, so the manifest only proved the file had
not been edited -- not that it came from the pinned source.
Four things leaked the build environment into the artifact:
- Cargo ran via --manifest-path from the vmlx working directory, so rustup
resolved the *default* toolchain instead of llguidance's rust-toolchain.toml.
The old artifact carries the stable rustc hash, proving the 1.95.0 pin was
never honored. Cargo now runs with its working directory inside the clone,
which makes the upstream pin load-bearing; we never name the version
ourselves.
- Absolute paths were baked in: 77 copies of the cargo registry path and the
physical build root. RUSTFLAGS now remaps the clone's physical path (via
pwd -P: $TMPDIR is a symlink and rustc records the resolved path) to
/llguidance, and CARGO_HOME to /cargo.
- LC_UUID varied per link. The cause was not the linker being random: with
debug = 1 from the workspace release profile, ld builds a debug map whose
N_OSO stabs hold absolute .o paths and hashes it into the UUID *before*
strip -S -x erases those strings -- so the strings came out clean while the
UUID still encoded where the build ran. -C debuginfo=0 removes the debug map
and makes the UUID a hash of path-independent content. We deliberately do not
pass -no_uuid: ld refuses to link against a dylib with no LC_UUID.
- The install name is now set at link time. Left alone, ld records the absolute
output path as LC_ID_DYLIB, and that string's length sizes the Mach-O header
padding, shifting every section address by the length of the build root.
install_name_tool rewrites the string but cannot un-shift the sections, so
fixing it after the fact still left a build-root-dependent layout.
The recipe is now one build_staged_dylib function, and the new
--check-reproducible mode runs that same function in two build roots of
different path lengths and fails unless the results are byte-identical. A repro
check that reimplemented the recipe would prove nothing.
The regenerated dylib is byte-identical across three independent roots.
Info.plist, llguidance.h, and module.modulemap are unchanged. The pin, features,
and triple remain in exactly one place.
The top-level README's consumer install section now notes that macOS app bundles must embed and codesign libllguidance.dylib with Xcode Embed & Sign, while command-line SwiftPM builds need no extra setup because the install name is @rpath/libllguidance.dylib and SwiftPM's default @loader_path rpath resolves it.
Follow-up to acf1842.
Replaces the committed 18,405,824-byte static archive (libllguidance-stripped.a) with a stripped 2,957,200-byte dynamic library (libllguidance.dylib) built from the same pin (0384f3f6aab6cebe8abf9b74db0079b96f5837ef), same target triple, same default cargo features, still --locked.
install_name is @rpath/libllguidance.dylib; SwiftPM copies the dylib next to the executable and its default @loader_path rpath resolves it, so Package.swift needs no linker settings, rpath, or unsafeFlags -- the binaryTarget declaration is unchanged.
regenerate-llguidance.sh now stages/strips the dylib, sets the install name, and asserts it via otool -D before creating the xcframework.
App-bundle consumers must now embed and codesign libllguidance.dylib (Xcode Embed & Sign does this for a dynamic binary target); documented in Vendors/README.md.
Verified: verify-llguidance.sh passes, StructuredOutputProbe builds and launches, StructuredOutputFocusedTests 17/17 and StructuredOutputTerminalFocusedTests 19/19 pass.
Preserve the existing Tangled/Rook branch lineage while adopting the final upstream-integrated, audited implementation tree.
StructuredOutputProbe text and throughput modes already apply the chat template through container.prepare, but they never passed an explicit non-thinking chat-template context. Qwen3 templates branch on enable_thinking, leaving constrained runs unrepresentative: Qwen3 0.6B could loop legal JSON whitespace to maxTokens, and Qwen3.5 could route JSON tokens as reasoning with empty visible output.
Pass the same non-thinking additionalContext to text mode and to both throughput inputs so constrained and unconstrained throughput remain apples-to-apples. Also account for reasoning chunks with additive stdout keys, reasoning_output and constrained_reasoning_output, so empty visible output is self-diagnosing.
Enforce the structured terminal invariant everywhere a request can finish: any terminal GenerateCompletionInfo for a request with parameters.structuredOutput != nil now carries a non-nil StructuredOutputResult. This closes the remaining .cancelled + nil holes in cancelledBatchStream/cancelledGenerationStream for shutdown, acceleration rejection, and raw native-MTP submit rejection; the public BatchEngine.generate defensive no-info wrapper; solo fast-path setup failure including non-StructuredOutputError throws; and Evaluate.generateLoopTask cancellation/general iterator-construction catches. Non-structured requests still report nil at the same sites.
Fix the llguidance poison bypass in commit(tokenID:). Stop tokens were handled before the poison check, so a poisoned box receiving an accepting EOS could return success instead of the original poison reason. Stop-token handling now runs under the lock after throwIfPoisonedLocked(), preserving healthy accepting EOS success and pre-acceptance stop-token dead-end behavior.
Restore stop-string streaming semantics. Stop-string detection no longer forces detokenizer.flush() on every token, which had bypassed the 24-character holdback for all stop-string requests including structuredOutput == nil. Structured stop detection uses a separate accumulation buffer, and the scheduler yield is narrowed so ordinary stop-string streaming remains unchanged.
Update tests and probe coverage. The penalty-before-grammar test now exercises the split PreparedLogitProcessors path instead of putting a grammar processor inside CompositeLogitProcessor. The probe gate is VMLX_STRUCTURED_OUTPUT_PROBE=1 for all four operator modes, and mask-timing now reports warmed p50/p95 keys (empty_p50_ms, empty_p95_ms, prefix64_p50_ms, prefix64_p95_ms) from individual samples, with labels for lockstep vs constraint-only quantities. There are no timing assertions.
Docs now state the structuredOutput terminal contract and all rejected decode paths: speculative, native-MTP, and block-diffusion.
Executable coverage added: structured vs non-structured contrast at cancellation/rejection sites; poisoned box plus accepting EOS commit fails with the original poison reason; ordinary stop-string requests keep unchanged chunk boundaries and a nil structured terminal field. Focused suites are now 17 + 19 tests.
Validation on this tree: swift build -c release completed; swift test --filter StructuredOutputFocusedTests passed 17 tests; swift test --filter StructuredOutputTerminalFocusedTests passed 19 tests; git diff --check was clean. Broad swift test was not run to green because it deadlocks on the pre-existing FocusedMLXTestSupport process-wide POSIX semaphore /vmlx_mlx_lock at a suite boundary, with no assertion failure; this is pre-existing harness/environment behavior and not caused by this change.
GenerateCompletionInfo now carries a trailing defaulted structuredOutput: StructuredOutputResult?. nil means ordinary non-structured generation and never means unknown. Structured terminal results are .accepted when the final matcher state is accepting, .incomplete(.stopString/.maxTokens) for stop-string or length truncation before acceptance, .failed(StructuredOutputError) for concrete matcher/native failures, and .cancelled for real task/user cancellation. GenerateStopReason is unchanged so downstream exhaustive switches still compile; structured failure is distinguished from ordinary cancellation by structuredOutput being non-nil. Cancellation takes precedence over an accepting prefix.
This intentionally breaks the old public GenerateParameters.processor(tokenizer:stopTokenIDs:) entry point and replaces it with prepareLogitProcessors(tokenizer:stopTokenIDs:) -> PreparedLogitProcessors. That split returns penalties/user processors separately from the structured grammar handle, so the grammar processor is never hidden inside CompositeLogitProcessor again. Penalties still run before the grammar mask on every TokenIterator and BatchEngine path.
Approved core change: StructuredOutputLlguidance gains an attributed per-slot batch surface, computeBatchMasksAttributed / batchProcessAttributed, returning Result values per slot while still making exactly one llg_par_compute_mask call per step. It records each slot's failure independently, allowing BatchEngine to fail only the affected slot with no second mask compute. The public throwing batchProcess remains as a thin wrapper over the attributed surface.
During integration, a production deadlock was fixed in that attributed core path: record() re-acquired the non-recursive NSLock already held across all boxes in computeBatchMasksAttributed. recordLocked() now follows the existing poisonLocked/reconcileLocked convention for callers that already hold the box lock.
Executable tests prove preflight before queue admission for one success and four failure modes; schema compile count exactly once per request; TokenIterator one-token lookahead staging so an emitted token is never retroactively reclassified by a lookahead dead end; two or more structured BatchEngine slots batching into one native call with maxStepCount >= 2 and independent per-slot state; one slot's dead end failing alone while its neighbor completes; mixed structured and ordinary slots; EOS masked until accepting and accepting EOS reporting .accepted; stop-string and maxTokens before acceptance reporting .incomplete; structured and ordinary cancellation distinguishable; penalties before the grammar mask; structuredOutput == nil unchanged with terminal field nil; and speculative, native-MTP, and block-diffusion paths still rejecting structured output.
The VPE operator session still must prove the live opt-in probe modes, all gated behind VMLX_STRUCTURED_OUTPUT_PROBE=1 and never part of swift test: constrained text generation on Qwen3-0.6B-4bit; constrained image-input generation on the local Gemma VLM with a synthetic image; warmed empty-prefix and 64+-token mask timings against the real ~150k-token Qwen tokenizer; and constrained versus unconstrained decode throughput for a representative ~100-token object schema.
Validation on this exact tree:
- swift build -c release: Build complete.
- swift test --filter StructuredOutputFocusedTests: 17 tests passed.
- swift test --filter StructuredOutputTerminalFocusedTests: 15 tests passed, all executed.
- swift-format lint --configuration .swift-format on all changed files: clean; format --in-place produced no changes.
The broad swift test suite was not run to green. It deadlocks on a pre-existing process-wide harness semaphore at a suite boundary, FocusedMLXTestSupport.swift:106-118, named POSIX semaphore /vmlx_mlx_lock, with no assertion failure. This is an existing environmental/harness issue, not caused by this change.
Audit follow-ups on the llguidance constrained-decoding core.
Poison the box when constraint and matcher can no longer be proven identical.
commit() advances both native states in sequence, and the C API has no
rollback, so if exactly one of them advances -- or if the masks or stop flags
diverge -- the box is now permanently poisoned and every later mask or commit
fails with the original reason. Previously it threw once and then kept
decoding from divergent grammar state, which is precisely the failure the
shadow matcher exists to prevent.
Fix the batch locking order. computeBatchMasks sorted boxes by
ObjectIdentifier.hashValue, which is not a total order (two distinct boxes can
collide), so the deadlock-avoidance argument did not hold. It now sorts by
ObjectIdentifier itself, which is Comparable and orders by address, and it
deduplicates: `lock` is a non-recursive NSLock, so passing the same processor
twice would previously have deadlocked it against itself. Covered by a new
test.
Make the tokenizer-lifetime test actually test something. It asserted that the
callback owner outlives the caller, but the box also stored the tokenizer, so
the fixture stayed alive regardless -- the assertion could not fail. The box no
longer stores the unused `tokenizer` property, and the fixture now uses a
vocabulary distinct from every other test so it seeds its own entry in the
content-hashed native tokenizer cache. The test drops the processor entirely
and then proves the cached entry's retained owner still keeps the Swift
tokenizer alive and the callback usable. It failed before this change.
Also: StructuredOutputBatchInstrumentation is now internal rather than public
(tests reach it via @testable; it was never meant to be API), and the unused
BatchSlot.status field is removed -- accepting/dead-end state is reported by
batchStatuses(), which reads each slot's shadow matcher.
Not addressed here by design: mid-decode dead ends are recorded on lastError
and fail closed to an all -inf mask, but are not yet raised as generation
errors by TokenIterator/BatchEngine. Wiring those terminal semantics is the
next lode's scope.
17 focused tests green against the real engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prototype in StructuredOutput.swift masked logits by decoding every
candidate token and re-running a recursive whole-prefix JSON matcher over the
entire generated text. On a 151,669-token Qwen tokenizer that cost 414 ms for
the empty-prefix mask and 7.66 s after 65 tokens. Replace it with an
incremental core backed by llguidance (MIT), vendored as a committed,
network-free macOS-arm64 SwiftPM binary input.
Warmed empty-prefix mask on that same Qwen tokenizer is now 0.017 ms
(constraint-only 0.009 ms; the shadow matcher below roughly doubles it).
Measured with tools/StructuredOutputProbe; the VPE session owns the
full-corpus perf numbers.
Native design
-------------
Masks come from an LlgConstraint and are computed through llguidance's native
batch entry point, llg_par_compute_mask, with one LlgConstraintStep per
request -- the single-request path submits one step, the batch path submits N
in one call. The constraint API alone cannot distinguish an accepting
completion from a dead end (both return rc=0/is_stop=true/mask=NULL), so each
request also owns a synchronized LlgMatcher shadow that supplies
accepting/stopped/dead-end classification. Both advance on the same sampled
token under one lock, and any divergence in mask bits or stop state fails
closed. A dead end is reported as an error, never as a silent EOS. The
duplicate grammar state is a deliberate correctness cost.
llguidance's constraint API requires compute_mask() before every
commit_token(); the upstream assertion guarding this is commented out, so
violating it corrupts constraint state silently while the matcher advances
correctly. The box tracks mask freshness and enforces that invariant itself
rather than leaving it as a rule callers must remember.
Schema preflight
----------------
llguidance's strict mode is not strict enough: with lenient=false it rejects
known-unimplemented keywords (uniqueItems) but silently ignores arbitrary
unknown ones ({"foo": 1} compiles). Since no unsupported keyword may be
silently dropped, a Swift allowlist -- mirroring the pin's IMPLEMENTED[27] and
META_AND_ANNOTATIONS[15] -- runs before llguidance. Malformed, empty,
unsupported, contradictory, and impossible schemas are all rejected at
preflight, before generation admission. Impossible schemas (enum disjoint from
type, minimum > maximum, minItems > maxItems) are rejected by llguidance at
compile time, so this is a real preflight and not a mid-decode discovery.
Tokenizer metadata
------------------
New optional StructuredOutputTokenizerSource protocol vends tokenizer.json
contents plus an EOS id, rather than widening the Tokenizer protocol (which
would source-break every conformer) or reaching into VMLXTokenizers' internal
byte-decoder table. The metadata is optional: a model without tokenizer.json
still loads and generates and only fails when structured output is requested.
The tokenize callback is exact (approximate greedy is not correctness-safe)
and its owner is retained for the lifetime of the native tokenizer, which is
cached process-wide by SHA-256 of the tokenizer JSON.
Named invariant: logitsWidth >= vocabSize. Padded ids past the tokenizer
vocabulary are masked off; a narrower logits width fails closed.
Removed
-------
JSONSchemaConstraint, JSONSchemaNode, JSONSchemaCompiler, PrefixMatchResult,
the recursive matcher, and the per-candidate vocabulary scan are deleted, not
flagged off. JSONSchemaConstraint was public, so this is a public source break
-- it *is* the recursive matcher, and its semantics were actively unsound
(forced alphabetical key order, rejected optional properties outright, no
bounds support); preserving it would have meant shipping a second,
contradictory schema engine. StructuredOutputPrefixStatus survives and now
reports matcher state (.complete = accepting, .invalid = dead end). The
StructuredOutput(jsonSchema:), GenerateParameters.structuredOutput, and
processor(tokenizer:stopTokenIDs:) contracts are unchanged for callers.
Decode paths that reject structured output today still reject it identically;
TokenIterator/BatchEngine terminal semantics are untouched (next lode).
Note llguidance emits object properties in schema declaration order, where the
prototype forced alphabetical order.
Vendored artifact
-----------------
Vendors/llguidance/CLlguidance.xcframework -- stripped static lib 18,405,824
bytes (17.6 MiB), the repo's first committed binary vendor. A local
binaryTarget(path:) needs no checksum arg and no unsafeFlags; a plain C target
plus a committed .a provably requires unsafeFlags(["-L", ...]), so it was not
used. The binary target is declared only under #if os(macOS), so the Linux
manifest still loads and builds; native code sits behind #if
canImport(CLlguidance) and reports .nativeUnavailable elsewhere.
Regenerate: scripts/regenerate-llguidance.sh (sole source of the pin,
cargo features, and target triple -- docs never restate the SHA)
Verify: scripts/verify-llguidance.sh (offline; hashes every file in
the xcframework and fails on a single flipped byte)
Trimming cargo features saved only 2.7% (17,915,344 bytes), so defaults are
kept; rayon is required by llg_par_compute_mask.
Tests: 16 focused tests against the real engine, covering constraint/matcher
lockstep, dead end vs completion, proof that one native batch call receives 2
steps, callback-owner lifetime, packed-mask-to-MLX conversion incl. padded
tail, composite ordering, concurrent matcher isolation, and every named schema
edge. Timings are never asserted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RunBench: stop silently ignoring BENCH_COMPILE_DECODE on the batch path
The perf bench accepted `BENCH_COMPILE_DECODE=1` and dropped it. The only thing
that ever set compile in `runPerfBench` was `enableCompiledDecode = useTokenIterator`,
so a `BENCH_COMPILE_DECODE=1` run on the default `path=batch` measured the EAGER path
and printed it as compiled. Every "compiled decode is worth N%" figure taken from this
bench on the batch path was eager-vs-eager.
A flag a harness accepts and drops is worse than one it rejects: it manufactures a
comparison nobody ran, and it is impossible to catch by reading the output, because
the output looks exactly like the run you asked for.
With the flag actually reaching the parameters it claims to set, compiled decode turns
out to be strongly model-dependent -- and in one case a large REGRESSION:
model eager compiled effect
gemma-4-E2B-it-MXFP4 125.9 167.1 +33%
Llama-3.2-1B-4bit 521.2 383.0 -26%
Hy3-JANG_2K 10.8 10.8 ~0%
(median of 3, 256 tok, temp 0, quiet M5 Max; Hy3 at 128 tok / materialized.)
That matters beyond the bench: osaurus exposes compiled decode as a single global
"Decode Perf" toggle. Its sign changes by model, so a global switch is the wrong shape
for it -- but nobody could see that while the measurement harness was blind.
No runtime behaviour changes; this is bench-only.
Make the memory-safety level reach the prefix-cache store, and stop overcounting it
Two defects, both in the guard added by #139/#140.
The "Safety Level" slider did nothing. `VMLXMemorySafetySettings.slider` was a
stored field that no resolver ever read -- every one of them switches on `mode` --
so a host that wired its slider to it (osaurus does) shipped a control that changed
nothing, while the resolved-plan readout printed the new level beside the old,
still-in-force caps. `slider` is now a projection of `mode`: same five choices, in
`allCases` order, so the two controls cannot disagree and the slider does what it
has always claimed to. Decoding keeps `mode` authoritative, so an existing user's
level does not shift under them on upgrade -- only the number shown for it gets
honest. The 0...4 validation error is gone because it is no longer expressible.
And the store budget ignored the level entirely: it gated a KV copy against raw
physical RAM and a hard-coded margin, identically whether the user asked for
Performance or Strict. It now spends a share of *free headroom* set by the level.
A share of the increment, deliberately -- not a ceiling on total memory. The load
caps are an MLX graph-scheduling guideline, not a hard resident limit, so a large
pack legitimately runs above them (Hy3 sits at ~96 GB under the default 0.70 cap
on a 128 GB Mac); budgeting the store against `0.70 * physical` would put
`activeBytes` over the line before a single KV byte and refuse *every* store,
silently killing the prefix cache for exactly the models whose re-prefill hurts
most. `CacheStoreBudget.policy` carries the level down to the decode loop, which
has no settings handle -- the same channel `TiedHeadQuantizationPolicy.current`
already uses.
The materialization factor was also wrong. It claimed three copies -- a deep copy,
a host `Data` extract, and the disk-store cache -- and none of the three survive
contact with the code: `KVCacheSimple.copy()` takes full-range slices that share
the source buffer, `extractLayerData` returns references, `makeDiskStoreCache`
returns the snapshot unchanged on the raw path, and `mlx_save_safetensors` streams
to the file descriptor rather than building a host `Data`. What is left is the
`contiguous` pass before the write, worth at most one compact copy. The reporter's
own footprint series said so all along: a +23 GiB excursion against a 23 GiB cache.
At 3x the guard refused stores that fit, quietly costing cache hits it was never
meant to cost.
The panic that motivated the guard is still refused, the 8 GB host still caches,
and a 16k-token conversation under a 94 GiB resident pack still caches. Strict and
Performance now reach different verdicts on the same host, model and cache, which
is the entire point.
Baseline: the two `VMLXMemorySafetySettingsTests` failures on this branch
(`defaultMaxKVSize` 65536 != 8192, 16384 != 4096) reproduce identically on clean
main and are untouched by this change.
Scale the cache-store safety margin to the host, and pin Hy3's stamp demotion
Two follow-ups to the cache-store guard.
1. The flat 4 GiB safety margin silently disabled the prefix cache on small Macs.
The margin is right on a 128 GB host and nonsense on an 8 GB one: with a 4.5 GiB
model resident, `active + 4 GiB` already exceeds 8 GiB, so the guard refused every
store — even one whose copies would have fit comfortably — on exactly the machines
whose users most notice a slow re-prefill. The margin now scales with the host
(an eighth of RAM, capped at the original 4 GiB), so a 128 GB Mac behaves exactly
as before and an 8 GB Mac keeps caching. A 16 GB host with a 32k context is still
refused, because three copies of that KV genuinely do not fit.
2. Pin the demotion that keeps Hunyuan v3's markers out of the visible answer.
The shipped HY3 bundles stamp `reasoning_parser: "qwen3"` — a plain-`<think>`
parser that can never match `</think:opensource>`. `shouldIgnoreReasoningStamp`
demotes that bogus stamp back to the model-type resolver, and that demotion is the
only thing standing between the user and a marker leak: honour the stamp and the
model's whole answer, raw markers and all, lands in the visible reply. The alias
work made the parser correct; nothing pinned the wiring that reaches it. Now the
whole chain is pinned — the real bundle's capabilities in, a parser that actually
closes on the model's markers out.
Never let saving the prefix cache take the host down, and stop swallowing Hunyuan answers
Two independent correctness fixes plus the guard that ties them together.
1. The prefix-cache store could kill the machine.
An external report had a 128 GB Mac kernel-panic under Llama-3.3-70B-8bit at
64K. Their own footprint time series exonerated prefill — it tracked
expectation the whole way — and put the +23 GiB spike immediately after
generation, in the cache-store window. That is exactly what the store does:
it materialises the KV cache several times over (a full snapshot copy, a host
`Data` extract, then another copy for the disk-store cache), at the precise
moment memory is already at its high-water mark. Nothing measured anything
first, so a store that could not possibly fit was attempted anyway. macOS does
not jetsam a plain user process out of the way, so page reclaim stalls,
watchdogd starves, and the kernel panics.
CacheStoreBudget measures the cache before any copy is made and skips the
store when the copies would not fit. A skipped store costs one slower request;
an unchecked store costs the machine. It is wired into all four store paths —
Evaluate, BatchEngine, BlockDiffusion (both the prompt store and the
per-boundary loop, which re-checks because one affordable store does not prove
N more are), and NativeMTP.
makeDiskStoreCache also duplicated the whole KV for nothing: the copy exists
only to protect the snapshot from maybeQuantizeKVCache's in-place mutation,
which is a no-op when nothing will be quantized — the exact path the prompt
boundary store takes. That copy is now elided.
2. Hunyuan v3 packs with a bare marker lost their entire answer.
HY3's chat template writes every protocol marker through one variable
(`'<think{}>'.format(HYTK)`). The open-source packs set it to `:opensource`;
the preview conversions leave it empty and emit a bare `</think>`. Our parser
knew only the suffixed spelling and starts inside reasoning, so on a bare pack
the close marker never matched: the model's whole reply — tool calls included —
was routed to reasoning and the user saw an empty answer. The tool-call parser
already accepted both spellings; only the reasoning side was blind.
ReasoningParser now takes marker aliases (empty by default, so every other
family is untouched), carried through forPrompt and sized into the partial-tag
holdback. A match that could still grow into a longer spelling is held back
rather than consumed, so no alias can close a block early and leak the rest of
a marker as visible text.
3. unclosedReasoning was read before the tail it depends on had been parsed.
The detokenizer holds back 24 characters; `</think:opensource>` is 20, so the
close marker was still sitting in the holdback when the flag was read. The flag
is now captured between the detokenizer flush and the parser flush — after the
held tail has been pumped through, before `flush()` unconditionally clears the
state.
Tests: the focused suite crashed on main (Index out of range, from a Hy3 parse
that returned nothing) and is now clean; the stale expectations it encoded —
that hy_v3 stamps to think_xml, which would not match the suffixed markers at
all — are corrected to the real contract.
Materialize near-RAM-scale packs instead of mmap so weights stay resident
Production loads default to mmap-backed safetensors, whose file-backed
pages are reclaimable under memory pressure. For a pack whose weights
approach physical memory (Hy3-JANG_2K: 94 GB on a 128 GB host), macOS
cannot keep the mapping hot next to KV, apps, and the file cache, so
decode refaults experts from SSD on every step: the model crawls, times
out on real turns, and never shows up as used RAM.
resolvedMemorySafetyPlan now disables mmap and raises the load fraction
to weights + headroom when a bundle's weights exceed 55% of physical
memory AND still fit (<= 86%). Packs that cannot fit at all keep mmap
streaming as their only viable mode. Gated behind an
allowsNearRAMScaleMaterialization profile flag: performance, balanced,
and safeAuto opt in; strict and diagnostic modes and any user-pinned
load fraction are never overridden.
Hy3: fp32 activations into the quantized lm_head (reference parity)
The reference runtime (jang_tools/hy3/model.py) honors the config's
enable_lm_head_fp32 by casting activations to fp32 ahead of the quantized
head matmul; the Swift path delegated to QuantizedLinear at the incoming
dtype. One [1, V] matmul per decode step, so the cost is negligible, and
greedy near-ties at the logit head now match the reference. Surfaced by a
cross-runtime parity review of the official-HY3 port (vmlx #134).
Support official Hunyuan v3 (hy_v3) end to end
The official Hunyuan v3 release differs from the Hy3-preview this runtime
was built against in three load-bearing ways: the conversion ships JANG
affine routed experts instead of JANGTQ codebooks, the chat template
builds every special-token string with Python `str.format` (which the
Jinja engine could not evaluate, so no request could even render), and
every protocol marker gained an `:opensource` suffix that neither the
reasoning nor the tool parser recognised.
Runtime:
- `Hy3AffineMoE`: the same sigmoid+expert-bias gate and always-on shared
expert as the preview MoE, with a standard `SwitchGLU` expert bank that
the affine quantize walk wraps from the per-module bits in
`config.quantization` (gate 2 / up 2 / down 3 on JANG_2K). Selected per
pack via `usesTurboQuantRoutedExperts` (weight_format / mxtq markers);
JANGTQ preview packs keep the TurboQuant path untouched. Verified on
the real 94 GB Hy3-JANG_2K: loads, generates coherently across turns
with exact recall, and compiled vs eager decode produce identical
output.
Template:
- Vendored Jinja learns the Python `str.format` subset (auto-numbered
`{}`, indexed `{0}`, named `{key}`, `{{`/`}}` escapes; format specs are
rejected rather than mis-rendered). Before this, all eight template
smoke scenarios failed with `Cannot call non-function value`; after,
8/8 render, including tools and reasoning history.
- `Hy3ReasoningTemplateContext` maps the request surface onto the
template's strict `reasoning_effort` contract (`no_think`/`low`/`high`;
the template raise_exceptions on anything else): explicit efforts are
clamped (`max` → `high`, `medium` → `low`), `enable_thinking` — which
the template itself ignores — maps true → `high`, false → `no_think`.
Applied in `LLMModelFactory.prepare` and in the bench template smoke.
Parsers:
- Reasoning: hy_v3 stamps resolve to `<think:opensource>` /
`</think:opensource>` with the existing prompt-tail start-state logic —
the template pre-fills an OPEN think in high/low and a CLOSED empty
pair in no_think, so tail detection is what keeps no_think answers out
of reasoning_content.
- Converted bundles stamp the generic `reasoning_parser=qwen3`, which can
never match the official markers; `JangLoader` demotes that stamp back
to the model-type resolver for the hy_v3 family (same treatment the
LFM2 stale stamps already get), and `reasoningStampFromModelType`
returns a dedicated `hy_v3` stamp.
- Tool calls: `HunyuanToolCallParser` accepts the `:opensource` wrapper
via tag aliases, collapses the suffix before body parsing, and the
partial-content check accepts a buffer cut mid-suffix
(`<tool_call:opens`), so streamed envelopes never flush as text. The
preview bare-marker format still parses.
Tests: Hy3OfficialMarkersTests (7 — official envelope whole and 7-char
chunked, preview back-compat, marker resolution, no_think/open-think tail
start states, reasoning/content split), JinjaStringFormatTests (7), and
the existing Hy3 no-leak/dispatch suites updated from preview to official
markers; 31/31 across the five Hy3 suites plus the tool-processor fuzz
suite green.
Deliver the prose that precedes a tool call in full
Reported from agent runs on Ornith: the assistant's visible sentence cut
off mid-word right before every tool call — "…finalize the Pipe function
in the right plac", "Let me clean up the smoke test and". Reproduced
4/8 turns at temperature 0 on ornith-1.0-9b-mxfp8 ("…removing the
temporary director"). The missing tail ranged from two characters to a
whole clause.
Layer-by-layer tracing (raw tokens -> detokenizer -> reasoning parser ->
tool processor -> stop matcher -> wire) found two independent drops, both
on the boundary between the answer text and the tool-call envelope:
1. `ToolCallProcessor.processEOS` discarded `leadingTextBeforeToolCall`
whenever the buffered envelope PARSED. The prose sharing a chunk with
the envelope's start tag is stashed there, and Qwen3.5-family models
stop generating right after `</tool_call>` — the envelope's last
characters routinely arrive only in the end-of-stream flush, so the
call completes in `processEOS`, not on the streaming end-tag path
(which already returned the stash). Successful parse now surfaces the
held prose through the same `visibleLeadingTextBeforeToolCall` helper
the streaming path uses.
2. `TextToolTokenLoopHandler` emitted the stop-string matcher's held tail
AFTER the `.toolCall` event (the matcher drain sits at the end of
`onGenerationEnd`). Consumers deliberately suppress all text once a
tool call lands — the no-leak guard for post-tool prose — so a tail
emitted after the event is silently dropped even though it precedes
the call in the model's output. Live trace: the engine's parser
returned "…directory at /tmp/smoke.\n\n" in full, but everything past
the matcher's holdback point died downstream. `emitRouted` now drains
the matcher before any `.toolCall` event goes out. A stop string can
no longer straddle the call boundary; stops match the assistant's
answer span, which the call legitimately ends.
After both: 0/7 truncations on the same repro (previously 4/8), sentences
complete on both sides of every tool call, on ornith-1.0-9b-mxfp8 and
lfm2.5-8b-a1b-mxfp8; tool calls still parse with correct arguments and
nothing leaks after them.
New tests pin the invariants byte-faithfully to the live chunk shapes:
ProseBeforeToolCallEOSTests (EOS-completed envelope returns the prose;
mid-stream completion unchanged; failed parses still flush; single-chunk
envelopes) and ToolCallStopMatcherOrderingTests (matcher-held text
precedes the `.toolCall` event; nothing trails it).
The pre-existing failure in ToolCallEdgeCasesTests ("LFM2 processor
accepts observed OpenAI tool_call JSON envelope", visible "\n") fails
identically on main and is untouched by this change.
Capture the hybrid cross-turn cache boundary during prefill
Hybrid-SSM models (Ornith/Qwen3.6 GatedDeltaNet, Nemotron-H Mamba-2, LFM2,
ZAYA CCA) reuse prefill across chat turns via a "gen-suffix-stripped"
boundary: the prompt truncated just before the last turn-start token. The
next turn replaces the trailing generation prompt with the assistant's
reply, so the full-prompt key never matches again, but that boundary does.
Hybrid cache state is path-dependent, so the boundary cannot be produced by
trimming the finished prompt cache. `storeCacheAfterGeneration` therefore
rebuilt it by replaying the entire stripped prefix through the model — a
second full prefill — and it runs before `generateLoopTask` yields `.info`,
so `finish_reason` and `[DONE]` waited on it. The response stream stayed
open long after the last token was delivered.
Measured on ornith-1.0-9b-mxfp8 (M5 Max), last content chunk to
finish_reason, max_tokens=16 so generation cannot explain it:
478 tok prompt 0.42 s
1,882 tok prompt 1.01 s
5,626 tok prompt 2.58 s
14,050 tok prompt 6.58 s
Dense gemma-4-12b is 0.17 s at the same context; it does not take this
path. `sample` during the window shows KVCache.update, KVCache.makeMask,
MetalAllocator::malloc — a real forward pass, not serialization. Reported
as: "when it finishes the query it doesn't always stop, it hangs around for
a bit then closes, no looping, just open connection, no response".
`BatchEngine.finishSlot` already yields `.info` before its store, for
exactly this reason. The solo TokenIterator path never got the same
treatment.
Rather than reorder (the `.info` yield deliberately sits behind the MLX
drain, so a consumer's post-tool decode cannot open a command encoder while
this task still holds one), take the boundary out of the prefill that
already passes through it: split `model.prepare` at the turn-start token
and copy the cache there. Same tokens, same forwards, same order — no extra
compute. One cache copy is retained until the store, alongside the prompt
snapshot already retained.
The split goes through `model.prepare` rather than the remainder it returns,
because Qwen35 — ornith and qwen3.6, the models that show the bug — consumes
the whole prompt itself and returns `.logits`. It also slices the mask
alongside the tokens: `Qwen3VLProcessor` attaches an all-ones `[1, T]` mask,
and skipping masked inputs would skip every prompt long enough to matter.
There is deliberately no re-derive fallback. Capture fails only when the
boundary lies inside the prefix this turn restored from cache — in which
case a boundary at least that long is already stored, and it is the one the
next turn matches — or on DSV4's pool cache, which cannot hold it anyway.
No post-generation forward pass remains on this path.
`CacheCoordinator.canPersistBoundaries` skips the whole thing when neither
cache tier is enabled; previously the boundary was computed and discarded.
After: 14k-token tail 6.58 s -> 0.30 s; lfm2.5-8b-a1b-mxfp8 1.68 s -> 0.09 s;
dense gemma unchanged. Cross-turn reuse still fires (growing-chat TTFT
0.90 s -> 0.27 s on turns 2-3) and multi-turn recall stays exact.
The comment claiming the re-derive "runs post-.info ... so it never adds to
the current turn's TTFT or token rate" was false, and is corrected.
NativeMTPTokenIterator and BatchEngine keep the forced re-derive for now;
neither stalls the client, and BatchEngine needs concurrency proof before
the same change lands there.
Fix gemma4 tool-call leak from stray/trailing '<' desyncing tag detection
Two live gemma-4-E2B-qat leaks in the osaurus native tool harness dropped a
spec-correct `<|tool_call>call:name{...}<tool_call|>` envelope into the visible
message, and the call never became a tool_calls entry. Root cause is the tagged
start-candidate check in ToolCallProcessor.processTaggedChunk, which decides
whether a chunk routes to the tagged path or the gemma bare-call fallback:
1. It only inspected the FIRST '<' in the chunk. Ordinary prose the model
echoes from the user — e.g. `DOUBLED=<product*2>` — put a false '<' ahead of
a real `<|tool_call>` in the same chunk. partialMatch failed on the false
'<', so the chunk fell to the bare-call fallback, which then matched the
`call:` INSIDE the wrapper and streamed the envelope out as text.
2. A chunk ending in a lone '<' (the detokenizer splitting right before
`|tool_call>`) was rejected by the `suffix.count > 1` guard, so the bare-call
fallback flushed the buffered leading text WITH the trailing '<', then leaked
the tag piecemeal over the following chunks.
Fix: scan every '<' for a start-tag candidate, and — scoped to the bare-call
fallback family (gemma-4 only) — treat a lone trailing '<' as a candidate so it
routes to the tagged path and is held in the potential-tool-call buffer until
the next chunk resolves it. `firstTag` already locates a complete tag past any
false '<' and preserves the leading prose. Other tagged formats (DSML
request_tool XML streams `<invoke>`/`<target>` char-by-char) keep the length
guard, so their behavior is unchanged.
Adds regression tests, including a byte-faithful full-pump reproduction from the
captured live leak: both fail before the change (envelope leaks, no tool call)
and pass after. Only manifests when the false/lone '<' and the real tag fall in
one chunk or across a chunk boundary, matching how the live detokenizer emits
the tail.
Activate ZAYA tool-aware template from source_model.architecture
Guard split-subscript traps across all SSM/hybrid mixers
Default-on gen-suffix-stripped SSM cache reuse for hybrid models
The gen-suffix-stripped store's correctness comment referenced
osaurusagent-35b, which has been deleted. Re-proved cache-ON == cache-OFF
byte-identically (temp=0, fresh disk cache, differential ground-truth) on
qwen-agentworld-35b-a3b MXFP8 and qwen3.6-35b-a3b MXFP8 (GatedDeltaNet MoE),
nemotron-omni-nano (Mamba-2); correctly inert on dense gemma-4-e2b.
Comment-only; no behavior change.
`shouldUseZayaToolAwareTemplate` / `shouldUseZayaVLToolAwareTemplate` gated
solely on `capabilities.family` being `zaya1*`. Bundles that stamp only
`source_model.architecture = zaya1_vl` and omit `capabilities.family` (e.g.
ZAYA1-VL JANGTQ) therefore fell through: the tool-aware template never
materialized, the bare role-only chat template was used, and the model —
given no tool-call format — leaked raw `<tool_call><function>...` XML as
visible assistant text (the parser expects `<zyphra_tool_call>`, so nothing
matched).
The gate's own doc comment says the check is meant to be "data-driven, not
family-name driven." Honor that: recognize ZAYA from `capabilities.family`
OR `source_model.architecture`, keeping the parser / think_in_template /
supports_tools contract unchanged.
Additive and low-risk: bundles that already set `capabilities.family` are
unaffected (family is still checked first); non-ZAYA bundles never match the
`zaya1*` idents. Verified live: with the fix, ZAYA1-VL-8B-JANGTQ4 activates
the tool-aware template (emits `<zyphra_tool_response>` and passes single
tool-call cases) instead of leaking the hallucinated `<response><tool_call>`
format. (The JANGTQ4 quant is still too weak to follow the format on every
multi-step turn — a separate quantization issue, not this template gate.)
PR #123 replaced the process-aborting `split(...)[i]` traps with count
guards in NemotronH, but the identical pattern — a `split` that records an
MLX error and returns an empty vector, then gets subscripted, trapping the
process before the recorded error can surface — survives untouched in every
sibling mixer. Complete the sweep with the same guard/bail idiom (return the
input so the enclosing withError scope throws the real diagnostic at exit;
the guard only trips when split already recorded an error).
15 guards across 8 files:
- Qwen35.swift — GatedDeltaNet conv split (q/k/v) + query/gate split.
Backs every qwen3.5 / 3.6 / Ornith / osaurusagent / Holo3 model.
- LFM2.swift / LFM2MoE.swift — B/C/x in_proj split (LFM2.5 Top Pick).
- BailingMoe.swift — q/k/v split (Ling-2.6 Flash).
- FalconH1.swift — in_proj + conv split.
- GraniteMoeHybrid.swift — in_proj + conv split.
- Qwen3Next.swift — query/gate, conv, qkvz (4-way), and ba splits.
- Jamba.swift — ssmStep deltaBC + processSequence xz splits.
Defensive-depth only — these fire solely on a malformed / quant-mismatched
checkpoint (a split whose width disagrees with the config). Correctly
converted bundles never hit them. Builds clean.
Hybrid-SSM chat models (Qwen3.5/3.6 & Ornith GatedDeltaNet, Nemotron-H
Mamba-2, ZAYA CCA, LFM2) never reused prefill across growing chat turns:
the stored prompt boundary ends in the chat template's generation-prompt
suffix (`<|im_start|>assistant\n`, …), which the next turn replaces with
the assistant reply + following user turn, so the full-prompt key never
matches as a prefix. Every turn recomputed the entire context.
Fix: after generation, also store the *generation-prompt-stripped* prompt
— everything before the final turn-start token — which the next turn DOES
contain as an exact prefix. Clean SSM/GatedDeltaNet state at that position
comes from the re-derive path (enableSSMReDerive), not the live
post-generation state (ahead by the gen suffix). KV is exact-sliceable.
Wired identically across all three generation paths: solo TokenIterator
(Evaluate), MTP speculation (NativeMTPTokenIterator), and BatchEngine.
Default ON for hybrid models (isHybrid), disable with
VMLX_HYBRID_STRIPPED_STORE=0. Text-only (skips on hasMediaContent so VL
turns are unaffected). The re-derive runs after the response is delivered,
so it never adds to the current turn's TTFT or token rate. Disk-restore
now restores SSM state for GatedDeltaNet ArraysCache (slotCount-based, so
a fresh all-nil cache receives the restored state) even at fmtV>=2.
Supporting plumbing: genPromptSuffixTokens computed at load by diffing the
chat template with/without add_generation_prompt (ModelContainer), threaded
through CacheCoordinator; boundary snapshots gain an allowDiskBackedRederive
/ forceRederive path that bypasses the path-dependent-cache skip guard
(safe post evalLock).
Proven live cache-ON == cache-OFF coherent on osaurusagent-35b-a3b
(GatedDeltaNet MoE, ssm=60) and Qwen3.6-27B-MXFP8-CRACK-MTP (D3
speculation, ssm=96): growing turns HIT the stripped boundary with SSM
restored and stay coherent. Dense / sliding-window models are excluded
(they already reuse via the post-answer boundary).
Fix production crash traps: Qwen3VL position ids, error-array subscripts, compiled-closure failures
Correct compiled-decode 'minimal impact' comment with measured data
The prior comment claimed disabling compiled decode had 'minimal' impact
(small GELU/SwiGLU/softcap fusions). That is wrong: compile(shapeless:)
fuses the whole single-slot decode step, and local benchmarks show a
substantial decode-throughput gain from enabling it. State the effect as
a ratio range rather than a single absolute, since RunBench direct decode
and the full server path differ substantially in absolute tok/s. Keep the
gate off by default — it is disabled for #1173 model-switch safety, not
because it is cheap.
* Extract nested-XML tool args for ZAYA rows
Some ZAYA rows emit a fully nested XML tool body —
<function>file_write
<parameter><name>path</name><value>out.txt</value></parameter>
— instead of the attribute form <function=name><parameter=key>value</parameter>
this parser was written for. The attribute scan finds no <parameter=, so every
argument is dropped: live file_write parses but 'path' is reported missing
(invalid_tool_arguments), and the <function>name (no =) variant fails the
function regex entirely and leaks the raw envelope as visible text.
Add a nested-form parse path, gated on the distinctive <name>/<value> markers so
the attribute path is left completely untouched (no regression). It reuses the
same value post-processing and schema validation, and handles both <function>name
and <function=name> openers.
ZayaNestedToolArgTests: red without the fix (path -> nil), green with it, plus
attribute-style and plain-prose regression guards.
* Test zaya nested-XML tool-arg extraction (values, closed-tag, no-fabrication)
Locks the live-captured JANGTQ4 wire shapes the nested parser handles:
- interleaved <type> tag between <name> and <value>
- closed <function>name</function> opener (tool_choice:required capture)
- names/types with NO <value> must not fabricate args (auto-mode skeleton)
---------
Co-authored-by: Eric <eric@erics-m5-max2.tailcd5195.ts.net>
Stream incremental tool-call envelope progress (Generation.toolCallProgress)
Local models that write files through a tool produce long silent gaps in
streaming consumers: while the model generates a tool-call envelope, the
tool-call processor correctly buffers it (so a partial envelope never leaks
into visible text) and the stream emits nothing until the parsed call is
complete. For a large file write that is tens of seconds with no events, so a
UI cannot preview the code as it streams.
Add a scoped progress channel:
- Generation.toolCallProgress(String): raw envelope-text deltas emitted while
a committed tool call is being collected. The complete parsed call still
arrives as a single .toolCall when the envelope closes, so .toolCall remains
the only actionable tool event; consumers with a default branch are
unaffected (additive case).
- ToolCallProcessor.collectingToolCallText: the raw buffer of the call being
collected. Deliberately scoped — nil in .potentialToolCall (an ambiguous
prefix that may still revert to plain text and be re-emitted as a visible
chunk) and nil for strip-only processors (whose parsed call is discarded, so
a progress delta would never be followed by a .toolCall). It is not a general
text tap.
- routeGenerationText diffs the collecting buffer around each processed chunk,
so every streaming path (token loop, BatchEngine, SpecDec) gains progress
events with no per-family parser change; a call that completes or reverts
within a chunk reports no growth.
Adds ToolCallProgressRoutingTests (multi-chunk envelope -> ordered deltas then
one call, plain prose -> no progress, strip-only -> no progress) and wires it
into the focused-tests source list.
NormConventionResolver: unrecognized norm_convention falls back to the vote
Serialize GPU-stream drivers + model-code guards (fix Metal concurrency crash class)
The Qwen3.6 loader-propagation test still asserted the pre-refactor helper
usesQwenPlusOneNormConvention, which 18f803c9 replaced with the shared
deterministic NormConventionResolver.shouldApplyPlusOneShift. The assertion
has been failing since that refactor merged. Repoint it at the current symbol
so the test again pins the real metadata->shift wiring.
The resolver treated ANY declared norm_convention string as authoritative
(`return usesPlusOne(explicit)`), but only three exact markers map to the
(1+weight) shift — so any other non-empty value (a converter typo, or a
descriptive string like "rms_norm") silently returned no-shift. A raw/JangQ
qwen3_5 bundle carrying such a declaration would load with its RMSNorm weights
unshifted (mean ~0, never lifted to ~1) and produce degraded/garbage output.
Only a RECOGNIZED (1+weight) marker is now authoritative; a present-but-
unrecognized declaration defers to the order-independent majority vote, which
reads the bundle's actual storage state. Adds NormConventionResolverTests
covering the recognized-marker, no-declaration, unrecognized-declaration, and
mixed-precedence cases.
Fixes an empty-array crash (Sentry 1A residual): when input.image.pixels is
present but the bundle loaded neither vision_tower nor vision_embedder
(text/audio-only Gemma4 or a partial checkpoint), the per-image loop appended
nothing and featuresList[0] / concatenated([]) at the scatter step crashed.
Guard with a typed VLMError.processing instead of aborting.
MLX is not thread-safe: mlx_eval/mlx_async_eval run gpu::eval inline on the
calling thread, encoding into and committing the shared Metal command buffer,
and the C++ stream mutex only guards the stream-map lookup (not the encoder).
Since evalLock was dropped from the eval hot path, a request's prefill
async_eval could run concurrently with a teardown/unload Stream.synchronize()
on the same command buffer.
Live-reproduced SIGABRT (disconnect-during-cold-load + retry): 'Completed
handler provided after commit' in addCompletedHandler, and AGX 'command
encoder is already encoding'. Two threads: unload->Stream.gpu.synchronize()
committing the buffer while startSoloFastPath->TokenIterator.prepare->async_eval
added a completion handler to it.
Fix: hold evalLock across the brief CPU-side encode+commit in eval/asyncEval/
MLXArray.eval (Stream.synchronize/clearCache already take it). The GPU executes
asynchronously after the lock releases, so CPU->GPU pipeline parallelism is
preserved; the lock is uncontended in steady-state single-producer decode and
only serializes during teardown/cancel/cross-model.
Also guard compiled activations (safeGeluApproximate/compiledSwiGLU/compiledGeGLU)
with CompiledDecodeTrace.isActive to avoid an illegal nested compile inside the
compiled-decode trace, and drain the stream on generateLoopTask cancel/error
early-exits.
Validated live: disconnect repro 35/35 clean (was crash at iter 1-2), concurrent
GPU 15/15 clean, decode 76.6 tok/s (no regression), multiturn coherent.
Live ZAYA / Gemma-4 AppleScript rows emit stray
</parameter></function></zyphra_tool_call> runs with no matching opener
after several agent-loop steps (MODEL_ISSUES_TRIAGE Issue 3). The
tool-call state machine only enters collection on an OPEN tag, so those
protocol closers streamed to the user as literal text.
Adds an opt-in orphanStripTags registry to ToolCallParser (default
empty, no behavior change for other formats) and teaches
ToolCallProcessor to strip registered orphan closers — including runs,
chunk-split partials, and closers embedded mid-buffer — while
unregistered tag-looking prose like </div> still flushes verbatim.
XMLFunctionParser and Gemma4ToolCallParser register their closers.
Live ZAYA / Gemma-4 AppleScript rows emit stray
</parameter></function></zyphra_tool_call> runs with no matching opener
after several agent-loop steps (MODEL_ISSUES_TRIAGE Issue 3). The
tool-call state machine only enters collection on an OPEN tag, so those
protocol closers streamed to the user as literal text.
Adds an opt-in orphanStripTags registry to ToolCallParser (default
empty, no behavior change for other formats) and teaches
ToolCallProcessor to strip registered orphan closers — including runs,
chunk-split partials, and closers embedded mid-buffer — while
unregistered tag-looking prose like </div> still flushes verbatim.
XMLFunctionParser and Gemma4ToolCallParser register their closers.
The committed dylib was checksum-verified but not reproducible. Rebuilding it
produced different bytes every time, so the manifest only proved the file had
not been edited -- not that it came from the pinned source.
Four things leaked the build environment into the artifact:
- Cargo ran via --manifest-path from the vmlx working directory, so rustup
resolved the *default* toolchain instead of llguidance's rust-toolchain.toml.
The old artifact carries the stable rustc hash, proving the 1.95.0 pin was
never honored. Cargo now runs with its working directory inside the clone,
which makes the upstream pin load-bearing; we never name the version
ourselves.
- Absolute paths were baked in: 77 copies of the cargo registry path and the
physical build root. RUSTFLAGS now remaps the clone's physical path (via
pwd -P: $TMPDIR is a symlink and rustc records the resolved path) to
/llguidance, and CARGO_HOME to /cargo.
- LC_UUID varied per link. The cause was not the linker being random: with
debug = 1 from the workspace release profile, ld builds a debug map whose
N_OSO stabs hold absolute .o paths and hashes it into the UUID *before*
strip -S -x erases those strings -- so the strings came out clean while the
UUID still encoded where the build ran. -C debuginfo=0 removes the debug map
and makes the UUID a hash of path-independent content. We deliberately do not
pass -no_uuid: ld refuses to link against a dylib with no LC_UUID.
- The install name is now set at link time. Left alone, ld records the absolute
output path as LC_ID_DYLIB, and that string's length sizes the Mach-O header
padding, shifting every section address by the length of the build root.
install_name_tool rewrites the string but cannot un-shift the sections, so
fixing it after the fact still left a build-root-dependent layout.
The recipe is now one build_staged_dylib function, and the new
--check-reproducible mode runs that same function in two build roots of
different path lengths and fails unless the results are byte-identical. A repro
check that reimplemented the recipe would prove nothing.
The regenerated dylib is byte-identical across three independent roots.
Info.plist, llguidance.h, and module.modulemap are unchanged. The pin, features,
and triple remain in exactly one place.
The top-level README's consumer install section now notes that macOS app bundles must embed and codesign libllguidance.dylib with Xcode Embed & Sign, while command-line SwiftPM builds need no extra setup because the install name is @rpath/libllguidance.dylib and SwiftPM's default @loader_path rpath resolves it.
Follow-up to acf1842.
Replaces the committed 18,405,824-byte static archive (libllguidance-stripped.a) with a stripped 2,957,200-byte dynamic library (libllguidance.dylib) built from the same pin (0384f3f6aab6cebe8abf9b74db0079b96f5837ef), same target triple, same default cargo features, still --locked.
install_name is @rpath/libllguidance.dylib; SwiftPM copies the dylib next to the executable and its default @loader_path rpath resolves it, so Package.swift needs no linker settings, rpath, or unsafeFlags -- the binaryTarget declaration is unchanged.
regenerate-llguidance.sh now stages/strips the dylib, sets the install name, and asserts it via otool -D before creating the xcframework.
App-bundle consumers must now embed and codesign libllguidance.dylib (Xcode Embed & Sign does this for a dynamic binary target); documented in Vendors/README.md.
Verified: verify-llguidance.sh passes, StructuredOutputProbe builds and launches, StructuredOutputFocusedTests 17/17 and StructuredOutputTerminalFocusedTests 19/19 pass.
StructuredOutputProbe text and throughput modes already apply the chat template through container.prepare, but they never passed an explicit non-thinking chat-template context. Qwen3 templates branch on enable_thinking, leaving constrained runs unrepresentative: Qwen3 0.6B could loop legal JSON whitespace to maxTokens, and Qwen3.5 could route JSON tokens as reasoning with empty visible output.
Pass the same non-thinking additionalContext to text mode and to both throughput inputs so constrained and unconstrained throughput remain apples-to-apples. Also account for reasoning chunks with additive stdout keys, reasoning_output and constrained_reasoning_output, so empty visible output is self-diagnosing.
Enforce the structured terminal invariant everywhere a request can finish: any terminal GenerateCompletionInfo for a request with parameters.structuredOutput != nil now carries a non-nil StructuredOutputResult. This closes the remaining .cancelled + nil holes in cancelledBatchStream/cancelledGenerationStream for shutdown, acceleration rejection, and raw native-MTP submit rejection; the public BatchEngine.generate defensive no-info wrapper; solo fast-path setup failure including non-StructuredOutputError throws; and Evaluate.generateLoopTask cancellation/general iterator-construction catches. Non-structured requests still report nil at the same sites.
Fix the llguidance poison bypass in commit(tokenID:). Stop tokens were handled before the poison check, so a poisoned box receiving an accepting EOS could return success instead of the original poison reason. Stop-token handling now runs under the lock after throwIfPoisonedLocked(), preserving healthy accepting EOS success and pre-acceptance stop-token dead-end behavior.
Restore stop-string streaming semantics. Stop-string detection no longer forces detokenizer.flush() on every token, which had bypassed the 24-character holdback for all stop-string requests including structuredOutput == nil. Structured stop detection uses a separate accumulation buffer, and the scheduler yield is narrowed so ordinary stop-string streaming remains unchanged.
Update tests and probe coverage. The penalty-before-grammar test now exercises the split PreparedLogitProcessors path instead of putting a grammar processor inside CompositeLogitProcessor. The probe gate is VMLX_STRUCTURED_OUTPUT_PROBE=1 for all four operator modes, and mask-timing now reports warmed p50/p95 keys (empty_p50_ms, empty_p95_ms, prefix64_p50_ms, prefix64_p95_ms) from individual samples, with labels for lockstep vs constraint-only quantities. There are no timing assertions.
Docs now state the structuredOutput terminal contract and all rejected decode paths: speculative, native-MTP, and block-diffusion.
Executable coverage added: structured vs non-structured contrast at cancellation/rejection sites; poisoned box plus accepting EOS commit fails with the original poison reason; ordinary stop-string requests keep unchanged chunk boundaries and a nil structured terminal field. Focused suites are now 17 + 19 tests.
Validation on this tree: swift build -c release completed; swift test --filter StructuredOutputFocusedTests passed 17 tests; swift test --filter StructuredOutputTerminalFocusedTests passed 19 tests; git diff --check was clean. Broad swift test was not run to green because it deadlocks on the pre-existing FocusedMLXTestSupport process-wide POSIX semaphore /vmlx_mlx_lock at a suite boundary, with no assertion failure; this is pre-existing harness/environment behavior and not caused by this change.
GenerateCompletionInfo now carries a trailing defaulted structuredOutput: StructuredOutputResult?. nil means ordinary non-structured generation and never means unknown. Structured terminal results are .accepted when the final matcher state is accepting, .incomplete(.stopString/.maxTokens) for stop-string or length truncation before acceptance, .failed(StructuredOutputError) for concrete matcher/native failures, and .cancelled for real task/user cancellation. GenerateStopReason is unchanged so downstream exhaustive switches still compile; structured failure is distinguished from ordinary cancellation by structuredOutput being non-nil. Cancellation takes precedence over an accepting prefix.
This intentionally breaks the old public GenerateParameters.processor(tokenizer:stopTokenIDs:) entry point and replaces it with prepareLogitProcessors(tokenizer:stopTokenIDs:) -> PreparedLogitProcessors. That split returns penalties/user processors separately from the structured grammar handle, so the grammar processor is never hidden inside CompositeLogitProcessor again. Penalties still run before the grammar mask on every TokenIterator and BatchEngine path.
Approved core change: StructuredOutputLlguidance gains an attributed per-slot batch surface, computeBatchMasksAttributed / batchProcessAttributed, returning Result values per slot while still making exactly one llg_par_compute_mask call per step. It records each slot's failure independently, allowing BatchEngine to fail only the affected slot with no second mask compute. The public throwing batchProcess remains as a thin wrapper over the attributed surface.
During integration, a production deadlock was fixed in that attributed core path: record() re-acquired the non-recursive NSLock already held across all boxes in computeBatchMasksAttributed. recordLocked() now follows the existing poisonLocked/reconcileLocked convention for callers that already hold the box lock.
Executable tests prove preflight before queue admission for one success and four failure modes; schema compile count exactly once per request; TokenIterator one-token lookahead staging so an emitted token is never retroactively reclassified by a lookahead dead end; two or more structured BatchEngine slots batching into one native call with maxStepCount >= 2 and independent per-slot state; one slot's dead end failing alone while its neighbor completes; mixed structured and ordinary slots; EOS masked until accepting and accepting EOS reporting .accepted; stop-string and maxTokens before acceptance reporting .incomplete; structured and ordinary cancellation distinguishable; penalties before the grammar mask; structuredOutput == nil unchanged with terminal field nil; and speculative, native-MTP, and block-diffusion paths still rejecting structured output.
The VPE operator session still must prove the live opt-in probe modes, all gated behind VMLX_STRUCTURED_OUTPUT_PROBE=1 and never part of swift test: constrained text generation on Qwen3-0.6B-4bit; constrained image-input generation on the local Gemma VLM with a synthetic image; warmed empty-prefix and 64+-token mask timings against the real ~150k-token Qwen tokenizer; and constrained versus unconstrained decode throughput for a representative ~100-token object schema.
Validation on this exact tree:
- swift build -c release: Build complete.
- swift test --filter StructuredOutputFocusedTests: 17 tests passed.
- swift test --filter StructuredOutputTerminalFocusedTests: 15 tests passed, all executed.
- swift-format lint --configuration .swift-format on all changed files: clean; format --in-place produced no changes.
The broad swift test suite was not run to green. It deadlocks on a pre-existing process-wide harness semaphore at a suite boundary, FocusedMLXTestSupport.swift:106-118, named POSIX semaphore /vmlx_mlx_lock, with no assertion failure. This is an existing environmental/harness issue, not caused by this change.
Audit follow-ups on the llguidance constrained-decoding core.
Poison the box when constraint and matcher can no longer be proven identical.
commit() advances both native states in sequence, and the C API has no
rollback, so if exactly one of them advances -- or if the masks or stop flags
diverge -- the box is now permanently poisoned and every later mask or commit
fails with the original reason. Previously it threw once and then kept
decoding from divergent grammar state, which is precisely the failure the
shadow matcher exists to prevent.
Fix the batch locking order. computeBatchMasks sorted boxes by
ObjectIdentifier.hashValue, which is not a total order (two distinct boxes can
collide), so the deadlock-avoidance argument did not hold. It now sorts by
ObjectIdentifier itself, which is Comparable and orders by address, and it
deduplicates: `lock` is a non-recursive NSLock, so passing the same processor
twice would previously have deadlocked it against itself. Covered by a new
test.
Make the tokenizer-lifetime test actually test something. It asserted that the
callback owner outlives the caller, but the box also stored the tokenizer, so
the fixture stayed alive regardless -- the assertion could not fail. The box no
longer stores the unused `tokenizer` property, and the fixture now uses a
vocabulary distinct from every other test so it seeds its own entry in the
content-hashed native tokenizer cache. The test drops the processor entirely
and then proves the cached entry's retained owner still keeps the Swift
tokenizer alive and the callback usable. It failed before this change.
Also: StructuredOutputBatchInstrumentation is now internal rather than public
(tests reach it via @testable; it was never meant to be API), and the unused
BatchSlot.status field is removed -- accepting/dead-end state is reported by
batchStatuses(), which reads each slot's shadow matcher.
Not addressed here by design: mid-decode dead ends are recorded on lastError
and fail closed to an all -inf mask, but are not yet raised as generation
errors by TokenIterator/BatchEngine. Wiring those terminal semantics is the
next lode's scope.
17 focused tests green against the real engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prototype in StructuredOutput.swift masked logits by decoding every
candidate token and re-running a recursive whole-prefix JSON matcher over the
entire generated text. On a 151,669-token Qwen tokenizer that cost 414 ms for
the empty-prefix mask and 7.66 s after 65 tokens. Replace it with an
incremental core backed by llguidance (MIT), vendored as a committed,
network-free macOS-arm64 SwiftPM binary input.
Warmed empty-prefix mask on that same Qwen tokenizer is now 0.017 ms
(constraint-only 0.009 ms; the shadow matcher below roughly doubles it).
Measured with tools/StructuredOutputProbe; the VPE session owns the
full-corpus perf numbers.
Native design
-------------
Masks come from an LlgConstraint and are computed through llguidance's native
batch entry point, llg_par_compute_mask, with one LlgConstraintStep per
request -- the single-request path submits one step, the batch path submits N
in one call. The constraint API alone cannot distinguish an accepting
completion from a dead end (both return rc=0/is_stop=true/mask=NULL), so each
request also owns a synchronized LlgMatcher shadow that supplies
accepting/stopped/dead-end classification. Both advance on the same sampled
token under one lock, and any divergence in mask bits or stop state fails
closed. A dead end is reported as an error, never as a silent EOS. The
duplicate grammar state is a deliberate correctness cost.
llguidance's constraint API requires compute_mask() before every
commit_token(); the upstream assertion guarding this is commented out, so
violating it corrupts constraint state silently while the matcher advances
correctly. The box tracks mask freshness and enforces that invariant itself
rather than leaving it as a rule callers must remember.
Schema preflight
----------------
llguidance's strict mode is not strict enough: with lenient=false it rejects
known-unimplemented keywords (uniqueItems) but silently ignores arbitrary
unknown ones ({"foo": 1} compiles). Since no unsupported keyword may be
silently dropped, a Swift allowlist -- mirroring the pin's IMPLEMENTED[27] and
META_AND_ANNOTATIONS[15] -- runs before llguidance. Malformed, empty,
unsupported, contradictory, and impossible schemas are all rejected at
preflight, before generation admission. Impossible schemas (enum disjoint from
type, minimum > maximum, minItems > maxItems) are rejected by llguidance at
compile time, so this is a real preflight and not a mid-decode discovery.
Tokenizer metadata
------------------
New optional StructuredOutputTokenizerSource protocol vends tokenizer.json
contents plus an EOS id, rather than widening the Tokenizer protocol (which
would source-break every conformer) or reaching into VMLXTokenizers' internal
byte-decoder table. The metadata is optional: a model without tokenizer.json
still loads and generates and only fails when structured output is requested.
The tokenize callback is exact (approximate greedy is not correctness-safe)
and its owner is retained for the lifetime of the native tokenizer, which is
cached process-wide by SHA-256 of the tokenizer JSON.
Named invariant: logitsWidth >= vocabSize. Padded ids past the tokenizer
vocabulary are masked off; a narrower logits width fails closed.
Removed
-------
JSONSchemaConstraint, JSONSchemaNode, JSONSchemaCompiler, PrefixMatchResult,
the recursive matcher, and the per-candidate vocabulary scan are deleted, not
flagged off. JSONSchemaConstraint was public, so this is a public source break
-- it *is* the recursive matcher, and its semantics were actively unsound
(forced alphabetical key order, rejected optional properties outright, no
bounds support); preserving it would have meant shipping a second,
contradictory schema engine. StructuredOutputPrefixStatus survives and now
reports matcher state (.complete = accepting, .invalid = dead end). The
StructuredOutput(jsonSchema:), GenerateParameters.structuredOutput, and
processor(tokenizer:stopTokenIDs:) contracts are unchanged for callers.
Decode paths that reject structured output today still reject it identically;
TokenIterator/BatchEngine terminal semantics are untouched (next lode).
Note llguidance emits object properties in schema declaration order, where the
prototype forced alphabetical order.
Vendored artifact
-----------------
Vendors/llguidance/CLlguidance.xcframework -- stripped static lib 18,405,824
bytes (17.6 MiB), the repo's first committed binary vendor. A local
binaryTarget(path:) needs no checksum arg and no unsafeFlags; a plain C target
plus a committed .a provably requires unsafeFlags(["-L", ...]), so it was not
used. The binary target is declared only under #if os(macOS), so the Linux
manifest still loads and builds; native code sits behind #if
canImport(CLlguidance) and reports .nativeUnavailable elsewhere.
Regenerate: scripts/regenerate-llguidance.sh (sole source of the pin,
cargo features, and target triple -- docs never restate the SHA)
Verify: scripts/verify-llguidance.sh (offline; hashes every file in
the xcframework and fails on a single flipped byte)
Trimming cargo features saved only 2.7% (17,915,344 bytes), so defaults are
kept; rayon is required by llg_par_compute_mask.
Tests: 16 focused tests against the real engine, covering constraint/matcher
lockstep, dead end vs completion, proof that one native batch call receives 2
steps, callback-owner lifetime, packed-mask-to-MLX conversion incl. padded
tail, composite ordering, concurrent matcher isolation, and every named schema
edge. Timings are never asserted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The perf bench accepted `BENCH_COMPILE_DECODE=1` and dropped it. The only thing
that ever set compile in `runPerfBench` was `enableCompiledDecode = useTokenIterator`,
so a `BENCH_COMPILE_DECODE=1` run on the default `path=batch` measured the EAGER path
and printed it as compiled. Every "compiled decode is worth N%" figure taken from this
bench on the batch path was eager-vs-eager.
A flag a harness accepts and drops is worse than one it rejects: it manufactures a
comparison nobody ran, and it is impossible to catch by reading the output, because
the output looks exactly like the run you asked for.
With the flag actually reaching the parameters it claims to set, compiled decode turns
out to be strongly model-dependent -- and in one case a large REGRESSION:
model eager compiled effect
gemma-4-E2B-it-MXFP4 125.9 167.1 +33%
Llama-3.2-1B-4bit 521.2 383.0 -26%
Hy3-JANG_2K 10.8 10.8 ~0%
(median of 3, 256 tok, temp 0, quiet M5 Max; Hy3 at 128 tok / materialized.)
That matters beyond the bench: osaurus exposes compiled decode as a single global
"Decode Perf" toggle. Its sign changes by model, so a global switch is the wrong shape
for it -- but nobody could see that while the measurement harness was blind.
No runtime behaviour changes; this is bench-only.
Two defects, both in the guard added by #139/#140.
The "Safety Level" slider did nothing. `VMLXMemorySafetySettings.slider` was a
stored field that no resolver ever read -- every one of them switches on `mode` --
so a host that wired its slider to it (osaurus does) shipped a control that changed
nothing, while the resolved-plan readout printed the new level beside the old,
still-in-force caps. `slider` is now a projection of `mode`: same five choices, in
`allCases` order, so the two controls cannot disagree and the slider does what it
has always claimed to. Decoding keeps `mode` authoritative, so an existing user's
level does not shift under them on upgrade -- only the number shown for it gets
honest. The 0...4 validation error is gone because it is no longer expressible.
And the store budget ignored the level entirely: it gated a KV copy against raw
physical RAM and a hard-coded margin, identically whether the user asked for
Performance or Strict. It now spends a share of *free headroom* set by the level.
A share of the increment, deliberately -- not a ceiling on total memory. The load
caps are an MLX graph-scheduling guideline, not a hard resident limit, so a large
pack legitimately runs above them (Hy3 sits at ~96 GB under the default 0.70 cap
on a 128 GB Mac); budgeting the store against `0.70 * physical` would put
`activeBytes` over the line before a single KV byte and refuse *every* store,
silently killing the prefix cache for exactly the models whose re-prefill hurts
most. `CacheStoreBudget.policy` carries the level down to the decode loop, which
has no settings handle -- the same channel `TiedHeadQuantizationPolicy.current`
already uses.
The materialization factor was also wrong. It claimed three copies -- a deep copy,
a host `Data` extract, and the disk-store cache -- and none of the three survive
contact with the code: `KVCacheSimple.copy()` takes full-range slices that share
the source buffer, `extractLayerData` returns references, `makeDiskStoreCache`
returns the snapshot unchanged on the raw path, and `mlx_save_safetensors` streams
to the file descriptor rather than building a host `Data`. What is left is the
`contiguous` pass before the write, worth at most one compact copy. The reporter's
own footprint series said so all along: a +23 GiB excursion against a 23 GiB cache.
At 3x the guard refused stores that fit, quietly costing cache hits it was never
meant to cost.
The panic that motivated the guard is still refused, the 8 GB host still caches,
and a 16k-token conversation under a 94 GiB resident pack still caches. Strict and
Performance now reach different verdicts on the same host, model and cache, which
is the entire point.
Baseline: the two `VMLXMemorySafetySettingsTests` failures on this branch
(`defaultMaxKVSize` 65536 != 8192, 16384 != 4096) reproduce identically on clean
main and are untouched by this change.
Two follow-ups to the cache-store guard.
1. The flat 4 GiB safety margin silently disabled the prefix cache on small Macs.
The margin is right on a 128 GB host and nonsense on an 8 GB one: with a 4.5 GiB
model resident, `active + 4 GiB` already exceeds 8 GiB, so the guard refused every
store — even one whose copies would have fit comfortably — on exactly the machines
whose users most notice a slow re-prefill. The margin now scales with the host
(an eighth of RAM, capped at the original 4 GiB), so a 128 GB Mac behaves exactly
as before and an 8 GB Mac keeps caching. A 16 GB host with a 32k context is still
refused, because three copies of that KV genuinely do not fit.
2. Pin the demotion that keeps Hunyuan v3's markers out of the visible answer.
The shipped HY3 bundles stamp `reasoning_parser: "qwen3"` — a plain-`<think>`
parser that can never match `</think:opensource>`. `shouldIgnoreReasoningStamp`
demotes that bogus stamp back to the model-type resolver, and that demotion is the
only thing standing between the user and a marker leak: honour the stamp and the
model's whole answer, raw markers and all, lands in the visible reply. The alias
work made the parser correct; nothing pinned the wiring that reaches it. Now the
whole chain is pinned — the real bundle's capabilities in, a parser that actually
closes on the model's markers out.
Two independent correctness fixes plus the guard that ties them together.
1. The prefix-cache store could kill the machine.
An external report had a 128 GB Mac kernel-panic under Llama-3.3-70B-8bit at
64K. Their own footprint time series exonerated prefill — it tracked
expectation the whole way — and put the +23 GiB spike immediately after
generation, in the cache-store window. That is exactly what the store does:
it materialises the KV cache several times over (a full snapshot copy, a host
`Data` extract, then another copy for the disk-store cache), at the precise
moment memory is already at its high-water mark. Nothing measured anything
first, so a store that could not possibly fit was attempted anyway. macOS does
not jetsam a plain user process out of the way, so page reclaim stalls,
watchdogd starves, and the kernel panics.
CacheStoreBudget measures the cache before any copy is made and skips the
store when the copies would not fit. A skipped store costs one slower request;
an unchecked store costs the machine. It is wired into all four store paths —
Evaluate, BatchEngine, BlockDiffusion (both the prompt store and the
per-boundary loop, which re-checks because one affordable store does not prove
N more are), and NativeMTP.
makeDiskStoreCache also duplicated the whole KV for nothing: the copy exists
only to protect the snapshot from maybeQuantizeKVCache's in-place mutation,
which is a no-op when nothing will be quantized — the exact path the prompt
boundary store takes. That copy is now elided.
2. Hunyuan v3 packs with a bare marker lost their entire answer.
HY3's chat template writes every protocol marker through one variable
(`'<think{}>'.format(HYTK)`). The open-source packs set it to `:opensource`;
the preview conversions leave it empty and emit a bare `</think>`. Our parser
knew only the suffixed spelling and starts inside reasoning, so on a bare pack
the close marker never matched: the model's whole reply — tool calls included —
was routed to reasoning and the user saw an empty answer. The tool-call parser
already accepted both spellings; only the reasoning side was blind.
ReasoningParser now takes marker aliases (empty by default, so every other
family is untouched), carried through forPrompt and sized into the partial-tag
holdback. A match that could still grow into a longer spelling is held back
rather than consumed, so no alias can close a block early and leak the rest of
a marker as visible text.
3. unclosedReasoning was read before the tail it depends on had been parsed.
The detokenizer holds back 24 characters; `</think:opensource>` is 20, so the
close marker was still sitting in the holdback when the flag was read. The flag
is now captured between the detokenizer flush and the parser flush — after the
held tail has been pumped through, before `flush()` unconditionally clears the
state.
Tests: the focused suite crashed on main (Index out of range, from a Hy3 parse
that returned nothing) and is now clean; the stale expectations it encoded —
that hy_v3 stamps to think_xml, which would not match the suffixed markers at
all — are corrected to the real contract.
Production loads default to mmap-backed safetensors, whose file-backed
pages are reclaimable under memory pressure. For a pack whose weights
approach physical memory (Hy3-JANG_2K: 94 GB on a 128 GB host), macOS
cannot keep the mapping hot next to KV, apps, and the file cache, so
decode refaults experts from SSD on every step: the model crawls, times
out on real turns, and never shows up as used RAM.
resolvedMemorySafetyPlan now disables mmap and raises the load fraction
to weights + headroom when a bundle's weights exceed 55% of physical
memory AND still fit (<= 86%). Packs that cannot fit at all keep mmap
streaming as their only viable mode. Gated behind an
allowsNearRAMScaleMaterialization profile flag: performance, balanced,
and safeAuto opt in; strict and diagnostic modes and any user-pinned
load fraction are never overridden.
The reference runtime (jang_tools/hy3/model.py) honors the config's
enable_lm_head_fp32 by casting activations to fp32 ahead of the quantized
head matmul; the Swift path delegated to QuantizedLinear at the incoming
dtype. One [1, V] matmul per decode step, so the cost is negligible, and
greedy near-ties at the logit head now match the reference. Surfaced by a
cross-runtime parity review of the official-HY3 port (vmlx #134).
The official Hunyuan v3 release differs from the Hy3-preview this runtime
was built against in three load-bearing ways: the conversion ships JANG
affine routed experts instead of JANGTQ codebooks, the chat template
builds every special-token string with Python `str.format` (which the
Jinja engine could not evaluate, so no request could even render), and
every protocol marker gained an `:opensource` suffix that neither the
reasoning nor the tool parser recognised.
Runtime:
- `Hy3AffineMoE`: the same sigmoid+expert-bias gate and always-on shared
expert as the preview MoE, with a standard `SwitchGLU` expert bank that
the affine quantize walk wraps from the per-module bits in
`config.quantization` (gate 2 / up 2 / down 3 on JANG_2K). Selected per
pack via `usesTurboQuantRoutedExperts` (weight_format / mxtq markers);
JANGTQ preview packs keep the TurboQuant path untouched. Verified on
the real 94 GB Hy3-JANG_2K: loads, generates coherently across turns
with exact recall, and compiled vs eager decode produce identical
output.
Template:
- Vendored Jinja learns the Python `str.format` subset (auto-numbered
`{}`, indexed `{0}`, named `{key}`, `{{`/`}}` escapes; format specs are
rejected rather than mis-rendered). Before this, all eight template
smoke scenarios failed with `Cannot call non-function value`; after,
8/8 render, including tools and reasoning history.
- `Hy3ReasoningTemplateContext` maps the request surface onto the
template's strict `reasoning_effort` contract (`no_think`/`low`/`high`;
the template raise_exceptions on anything else): explicit efforts are
clamped (`max` → `high`, `medium` → `low`), `enable_thinking` — which
the template itself ignores — maps true → `high`, false → `no_think`.
Applied in `LLMModelFactory.prepare` and in the bench template smoke.
Parsers:
- Reasoning: hy_v3 stamps resolve to `<think:opensource>` /
`</think:opensource>` with the existing prompt-tail start-state logic —
the template pre-fills an OPEN think in high/low and a CLOSED empty
pair in no_think, so tail detection is what keeps no_think answers out
of reasoning_content.
- Converted bundles stamp the generic `reasoning_parser=qwen3`, which can
never match the official markers; `JangLoader` demotes that stamp back
to the model-type resolver for the hy_v3 family (same treatment the
LFM2 stale stamps already get), and `reasoningStampFromModelType`
returns a dedicated `hy_v3` stamp.
- Tool calls: `HunyuanToolCallParser` accepts the `:opensource` wrapper
via tag aliases, collapses the suffix before body parsing, and the
partial-content check accepts a buffer cut mid-suffix
(`<tool_call:opens`), so streamed envelopes never flush as text. The
preview bare-marker format still parses.
Tests: Hy3OfficialMarkersTests (7 — official envelope whole and 7-char
chunked, preview back-compat, marker resolution, no_think/open-think tail
start states, reasoning/content split), JinjaStringFormatTests (7), and
the existing Hy3 no-leak/dispatch suites updated from preview to official
markers; 31/31 across the five Hy3 suites plus the tool-processor fuzz
suite green.
Reported from agent runs on Ornith: the assistant's visible sentence cut
off mid-word right before every tool call — "…finalize the Pipe function
in the right plac", "Let me clean up the smoke test and". Reproduced
4/8 turns at temperature 0 on ornith-1.0-9b-mxfp8 ("…removing the
temporary director"). The missing tail ranged from two characters to a
whole clause.
Layer-by-layer tracing (raw tokens -> detokenizer -> reasoning parser ->
tool processor -> stop matcher -> wire) found two independent drops, both
on the boundary between the answer text and the tool-call envelope:
1. `ToolCallProcessor.processEOS` discarded `leadingTextBeforeToolCall`
whenever the buffered envelope PARSED. The prose sharing a chunk with
the envelope's start tag is stashed there, and Qwen3.5-family models
stop generating right after `</tool_call>` — the envelope's last
characters routinely arrive only in the end-of-stream flush, so the
call completes in `processEOS`, not on the streaming end-tag path
(which already returned the stash). Successful parse now surfaces the
held prose through the same `visibleLeadingTextBeforeToolCall` helper
the streaming path uses.
2. `TextToolTokenLoopHandler` emitted the stop-string matcher's held tail
AFTER the `.toolCall` event (the matcher drain sits at the end of
`onGenerationEnd`). Consumers deliberately suppress all text once a
tool call lands — the no-leak guard for post-tool prose — so a tail
emitted after the event is silently dropped even though it precedes
the call in the model's output. Live trace: the engine's parser
returned "…directory at /tmp/smoke.\n\n" in full, but everything past
the matcher's holdback point died downstream. `emitRouted` now drains
the matcher before any `.toolCall` event goes out. A stop string can
no longer straddle the call boundary; stops match the assistant's
answer span, which the call legitimately ends.
After both: 0/7 truncations on the same repro (previously 4/8), sentences
complete on both sides of every tool call, on ornith-1.0-9b-mxfp8 and
lfm2.5-8b-a1b-mxfp8; tool calls still parse with correct arguments and
nothing leaks after them.
New tests pin the invariants byte-faithfully to the live chunk shapes:
ProseBeforeToolCallEOSTests (EOS-completed envelope returns the prose;
mid-stream completion unchanged; failed parses still flush; single-chunk
envelopes) and ToolCallStopMatcherOrderingTests (matcher-held text
precedes the `.toolCall` event; nothing trails it).
The pre-existing failure in ToolCallEdgeCasesTests ("LFM2 processor
accepts observed OpenAI tool_call JSON envelope", visible "\n") fails
identically on main and is untouched by this change.
Hybrid-SSM models (Ornith/Qwen3.6 GatedDeltaNet, Nemotron-H Mamba-2, LFM2,
ZAYA CCA) reuse prefill across chat turns via a "gen-suffix-stripped"
boundary: the prompt truncated just before the last turn-start token. The
next turn replaces the trailing generation prompt with the assistant's
reply, so the full-prompt key never matches again, but that boundary does.
Hybrid cache state is path-dependent, so the boundary cannot be produced by
trimming the finished prompt cache. `storeCacheAfterGeneration` therefore
rebuilt it by replaying the entire stripped prefix through the model — a
second full prefill — and it runs before `generateLoopTask` yields `.info`,
so `finish_reason` and `[DONE]` waited on it. The response stream stayed
open long after the last token was delivered.
Measured on ornith-1.0-9b-mxfp8 (M5 Max), last content chunk to
finish_reason, max_tokens=16 so generation cannot explain it:
478 tok prompt 0.42 s
1,882 tok prompt 1.01 s
5,626 tok prompt 2.58 s
14,050 tok prompt 6.58 s
Dense gemma-4-12b is 0.17 s at the same context; it does not take this
path. `sample` during the window shows KVCache.update, KVCache.makeMask,
MetalAllocator::malloc — a real forward pass, not serialization. Reported
as: "when it finishes the query it doesn't always stop, it hangs around for
a bit then closes, no looping, just open connection, no response".
`BatchEngine.finishSlot` already yields `.info` before its store, for
exactly this reason. The solo TokenIterator path never got the same
treatment.
Rather than reorder (the `.info` yield deliberately sits behind the MLX
drain, so a consumer's post-tool decode cannot open a command encoder while
this task still holds one), take the boundary out of the prefill that
already passes through it: split `model.prepare` at the turn-start token
and copy the cache there. Same tokens, same forwards, same order — no extra
compute. One cache copy is retained until the store, alongside the prompt
snapshot already retained.
The split goes through `model.prepare` rather than the remainder it returns,
because Qwen35 — ornith and qwen3.6, the models that show the bug — consumes
the whole prompt itself and returns `.logits`. It also slices the mask
alongside the tokens: `Qwen3VLProcessor` attaches an all-ones `[1, T]` mask,
and skipping masked inputs would skip every prompt long enough to matter.
There is deliberately no re-derive fallback. Capture fails only when the
boundary lies inside the prefix this turn restored from cache — in which
case a boundary at least that long is already stored, and it is the one the
next turn matches — or on DSV4's pool cache, which cannot hold it anyway.
No post-generation forward pass remains on this path.
`CacheCoordinator.canPersistBoundaries` skips the whole thing when neither
cache tier is enabled; previously the boundary was computed and discarded.
After: 14k-token tail 6.58 s -> 0.30 s; lfm2.5-8b-a1b-mxfp8 1.68 s -> 0.09 s;
dense gemma unchanged. Cross-turn reuse still fires (growing-chat TTFT
0.90 s -> 0.27 s on turns 2-3) and multi-turn recall stays exact.
The comment claiming the re-derive "runs post-.info ... so it never adds to
the current turn's TTFT or token rate" was false, and is corrected.
NativeMTPTokenIterator and BatchEngine keep the forced re-derive for now;
neither stalls the client, and BatchEngine needs concurrency proof before
the same change lands there.
Two live gemma-4-E2B-qat leaks in the osaurus native tool harness dropped a
spec-correct `<|tool_call>call:name{...}<tool_call|>` envelope into the visible
message, and the call never became a tool_calls entry. Root cause is the tagged
start-candidate check in ToolCallProcessor.processTaggedChunk, which decides
whether a chunk routes to the tagged path or the gemma bare-call fallback:
1. It only inspected the FIRST '<' in the chunk. Ordinary prose the model
echoes from the user — e.g. `DOUBLED=<product*2>` — put a false '<' ahead of
a real `<|tool_call>` in the same chunk. partialMatch failed on the false
'<', so the chunk fell to the bare-call fallback, which then matched the
`call:` INSIDE the wrapper and streamed the envelope out as text.
2. A chunk ending in a lone '<' (the detokenizer splitting right before
`|tool_call>`) was rejected by the `suffix.count > 1` guard, so the bare-call
fallback flushed the buffered leading text WITH the trailing '<', then leaked
the tag piecemeal over the following chunks.
Fix: scan every '<' for a start-tag candidate, and — scoped to the bare-call
fallback family (gemma-4 only) — treat a lone trailing '<' as a candidate so it
routes to the tagged path and is held in the potential-tool-call buffer until
the next chunk resolves it. `firstTag` already locates a complete tag past any
false '<' and preserves the leading prose. Other tagged formats (DSML
request_tool XML streams `<invoke>`/`<target>` char-by-char) keep the length
guard, so their behavior is unchanged.
Adds regression tests, including a byte-faithful full-pump reproduction from the
captured live leak: both fail before the change (envelope leaks, no tool call)
and pass after. Only manifests when the false/lone '<' and the real tag fall in
one chunk or across a chunk boundary, matching how the live detokenizer emits
the tail.
The gen-suffix-stripped store's correctness comment referenced
osaurusagent-35b, which has been deleted. Re-proved cache-ON == cache-OFF
byte-identically (temp=0, fresh disk cache, differential ground-truth) on
qwen-agentworld-35b-a3b MXFP8 and qwen3.6-35b-a3b MXFP8 (GatedDeltaNet MoE),
nemotron-omni-nano (Mamba-2); correctly inert on dense gemma-4-e2b.
Comment-only; no behavior change.
`shouldUseZayaToolAwareTemplate` / `shouldUseZayaVLToolAwareTemplate` gated
solely on `capabilities.family` being `zaya1*`. Bundles that stamp only
`source_model.architecture = zaya1_vl` and omit `capabilities.family` (e.g.
ZAYA1-VL JANGTQ) therefore fell through: the tool-aware template never
materialized, the bare role-only chat template was used, and the model —
given no tool-call format — leaked raw `<tool_call><function>...` XML as
visible assistant text (the parser expects `<zyphra_tool_call>`, so nothing
matched).
The gate's own doc comment says the check is meant to be "data-driven, not
family-name driven." Honor that: recognize ZAYA from `capabilities.family`
OR `source_model.architecture`, keeping the parser / think_in_template /
supports_tools contract unchanged.
Additive and low-risk: bundles that already set `capabilities.family` are
unaffected (family is still checked first); non-ZAYA bundles never match the
`zaya1*` idents. Verified live: with the fix, ZAYA1-VL-8B-JANGTQ4 activates
the tool-aware template (emits `<zyphra_tool_response>` and passes single
tool-call cases) instead of leaking the hallucinated `<response><tool_call>`
format. (The JANGTQ4 quant is still too weak to follow the format on every
multi-step turn — a separate quantization issue, not this template gate.)
PR #123 replaced the process-aborting `split(...)[i]` traps with count
guards in NemotronH, but the identical pattern — a `split` that records an
MLX error and returns an empty vector, then gets subscripted, trapping the
process before the recorded error can surface — survives untouched in every
sibling mixer. Complete the sweep with the same guard/bail idiom (return the
input so the enclosing withError scope throws the real diagnostic at exit;
the guard only trips when split already recorded an error).
15 guards across 8 files:
- Qwen35.swift — GatedDeltaNet conv split (q/k/v) + query/gate split.
Backs every qwen3.5 / 3.6 / Ornith / osaurusagent / Holo3 model.
- LFM2.swift / LFM2MoE.swift — B/C/x in_proj split (LFM2.5 Top Pick).
- BailingMoe.swift — q/k/v split (Ling-2.6 Flash).
- FalconH1.swift — in_proj + conv split.
- GraniteMoeHybrid.swift — in_proj + conv split.
- Qwen3Next.swift — query/gate, conv, qkvz (4-way), and ba splits.
- Jamba.swift — ssmStep deltaBC + processSequence xz splits.
Defensive-depth only — these fire solely on a malformed / quant-mismatched
checkpoint (a split whose width disagrees with the config). Correctly
converted bundles never hit them. Builds clean.
Hybrid-SSM chat models (Qwen3.5/3.6 & Ornith GatedDeltaNet, Nemotron-H
Mamba-2, ZAYA CCA, LFM2) never reused prefill across growing chat turns:
the stored prompt boundary ends in the chat template's generation-prompt
suffix (`<|im_start|>assistant\n`, …), which the next turn replaces with
the assistant reply + following user turn, so the full-prompt key never
matches as a prefix. Every turn recomputed the entire context.
Fix: after generation, also store the *generation-prompt-stripped* prompt
— everything before the final turn-start token — which the next turn DOES
contain as an exact prefix. Clean SSM/GatedDeltaNet state at that position
comes from the re-derive path (enableSSMReDerive), not the live
post-generation state (ahead by the gen suffix). KV is exact-sliceable.
Wired identically across all three generation paths: solo TokenIterator
(Evaluate), MTP speculation (NativeMTPTokenIterator), and BatchEngine.
Default ON for hybrid models (isHybrid), disable with
VMLX_HYBRID_STRIPPED_STORE=0. Text-only (skips on hasMediaContent so VL
turns are unaffected). The re-derive runs after the response is delivered,
so it never adds to the current turn's TTFT or token rate. Disk-restore
now restores SSM state for GatedDeltaNet ArraysCache (slotCount-based, so
a fresh all-nil cache receives the restored state) even at fmtV>=2.
Supporting plumbing: genPromptSuffixTokens computed at load by diffing the
chat template with/without add_generation_prompt (ModelContainer), threaded
through CacheCoordinator; boundary snapshots gain an allowDiskBackedRederive
/ forceRederive path that bypasses the path-dependent-cache skip guard
(safe post evalLock).
Proven live cache-ON == cache-OFF coherent on osaurusagent-35b-a3b
(GatedDeltaNet MoE, ssm=60) and Qwen3.6-27B-MXFP8-CRACK-MTP (D3
speculation, ssm=96): growing turns HIT the stripped boundary with SSM
restored and stay coherent. Dense / sliding-window models are excluded
(they already reuse via the post-answer boundary).
The prior comment claimed disabling compiled decode had 'minimal' impact
(small GELU/SwiGLU/softcap fusions). That is wrong: compile(shapeless:)
fuses the whole single-slot decode step, and local benchmarks show a
substantial decode-throughput gain from enabling it. State the effect as
a ratio range rather than a single absolute, since RunBench direct decode
and the full server path differ substantially in absolute tok/s. Keep the
gate off by default — it is disabled for #1173 model-switch safety, not
because it is cheap.
* Extract nested-XML tool args for ZAYA rows
Some ZAYA rows emit a fully nested XML tool body —
<function>file_write
<parameter><name>path</name><value>out.txt</value></parameter>
— instead of the attribute form <function=name><parameter=key>value</parameter>
this parser was written for. The attribute scan finds no <parameter=, so every
argument is dropped: live file_write parses but 'path' is reported missing
(invalid_tool_arguments), and the <function>name (no =) variant fails the
function regex entirely and leaks the raw envelope as visible text.
Add a nested-form parse path, gated on the distinctive <name>/<value> markers so
the attribute path is left completely untouched (no regression). It reuses the
same value post-processing and schema validation, and handles both <function>name
and <function=name> openers.
ZayaNestedToolArgTests: red without the fix (path -> nil), green with it, plus
attribute-style and plain-prose regression guards.
* Test zaya nested-XML tool-arg extraction (values, closed-tag, no-fabrication)
Locks the live-captured JANGTQ4 wire shapes the nested parser handles:
- interleaved <type> tag between <name> and <value>
- closed <function>name</function> opener (tool_choice:required capture)
- names/types with NO <value> must not fabricate args (auto-mode skeleton)
---------
Co-authored-by: Eric <eric@erics-m5-max2.tailcd5195.ts.net>
Local models that write files through a tool produce long silent gaps in
streaming consumers: while the model generates a tool-call envelope, the
tool-call processor correctly buffers it (so a partial envelope never leaks
into visible text) and the stream emits nothing until the parsed call is
complete. For a large file write that is tens of seconds with no events, so a
UI cannot preview the code as it streams.
Add a scoped progress channel:
- Generation.toolCallProgress(String): raw envelope-text deltas emitted while
a committed tool call is being collected. The complete parsed call still
arrives as a single .toolCall when the envelope closes, so .toolCall remains
the only actionable tool event; consumers with a default branch are
unaffected (additive case).
- ToolCallProcessor.collectingToolCallText: the raw buffer of the call being
collected. Deliberately scoped — nil in .potentialToolCall (an ambiguous
prefix that may still revert to plain text and be re-emitted as a visible
chunk) and nil for strip-only processors (whose parsed call is discarded, so
a progress delta would never be followed by a .toolCall). It is not a general
text tap.
- routeGenerationText diffs the collecting buffer around each processed chunk,
so every streaming path (token loop, BatchEngine, SpecDec) gains progress
events with no per-family parser change; a call that completes or reverts
within a chunk reports no growth.
Adds ToolCallProgressRoutingTests (multi-chunk envelope -> ordered deltas then
one call, plain prose -> no progress, strip-only -> no progress) and wires it
into the focused-tests source list.
The Qwen3.6 loader-propagation test still asserted the pre-refactor helper
usesQwenPlusOneNormConvention, which 18f803c9 replaced with the shared
deterministic NormConventionResolver.shouldApplyPlusOneShift. The assertion
has been failing since that refactor merged. Repoint it at the current symbol
so the test again pins the real metadata->shift wiring.
The resolver treated ANY declared norm_convention string as authoritative
(`return usesPlusOne(explicit)`), but only three exact markers map to the
(1+weight) shift — so any other non-empty value (a converter typo, or a
descriptive string like "rms_norm") silently returned no-shift. A raw/JangQ
qwen3_5 bundle carrying such a declaration would load with its RMSNorm weights
unshifted (mean ~0, never lifted to ~1) and produce degraded/garbage output.
Only a RECOGNIZED (1+weight) marker is now authoritative; a present-but-
unrecognized declaration defers to the order-independent majority vote, which
reads the bundle's actual storage state. Adds NormConventionResolverTests
covering the recognized-marker, no-declaration, unrecognized-declaration, and
mixed-precedence cases.
Fixes an empty-array crash (Sentry 1A residual): when input.image.pixels is
present but the bundle loaded neither vision_tower nor vision_embedder
(text/audio-only Gemma4 or a partial checkpoint), the per-image loop appended
nothing and featuresList[0] / concatenated([]) at the scatter step crashed.
Guard with a typed VLMError.processing instead of aborting.
MLX is not thread-safe: mlx_eval/mlx_async_eval run gpu::eval inline on the
calling thread, encoding into and committing the shared Metal command buffer,
and the C++ stream mutex only guards the stream-map lookup (not the encoder).
Since evalLock was dropped from the eval hot path, a request's prefill
async_eval could run concurrently with a teardown/unload Stream.synchronize()
on the same command buffer.
Live-reproduced SIGABRT (disconnect-during-cold-load + retry): 'Completed
handler provided after commit' in addCompletedHandler, and AGX 'command
encoder is already encoding'. Two threads: unload->Stream.gpu.synchronize()
committing the buffer while startSoloFastPath->TokenIterator.prepare->async_eval
added a completion handler to it.
Fix: hold evalLock across the brief CPU-side encode+commit in eval/asyncEval/
MLXArray.eval (Stream.synchronize/clearCache already take it). The GPU executes
asynchronously after the lock releases, so CPU->GPU pipeline parallelism is
preserved; the lock is uncontended in steady-state single-producer decode and
only serializes during teardown/cancel/cross-model.
Also guard compiled activations (safeGeluApproximate/compiledSwiGLU/compiledGeGLU)
with CompiledDecodeTrace.isActive to avoid an illegal nested compile inside the
compiled-decode trace, and drain the stream on generateLoopTask cancel/error
early-exits.
Validated live: disconnect repro 35/35 clean (was crash at iter 1-2), concurrent
GPU 15/15 clean, decode 76.6 tok/s (no regression), multiturn coherent.
Live ZAYA / Gemma-4 AppleScript rows emit stray
</parameter></function></zyphra_tool_call> runs with no matching opener
after several agent-loop steps (MODEL_ISSUES_TRIAGE Issue 3). The
tool-call state machine only enters collection on an OPEN tag, so those
protocol closers streamed to the user as literal text.
Adds an opt-in orphanStripTags registry to ToolCallParser (default
empty, no behavior change for other formats) and teaches
ToolCallProcessor to strip registered orphan closers — including runs,
chunk-split partials, and closers embedded mid-buffer — while
unregistered tag-looking prose like </div> still flushes verbatim.
XMLFunctionParser and Gemma4ToolCallParser register their closers.
Live ZAYA / Gemma-4 AppleScript rows emit stray
</parameter></function></zyphra_tool_call> runs with no matching opener
after several agent-loop steps (MODEL_ISSUES_TRIAGE Issue 3). The
tool-call state machine only enters collection on an OPEN tag, so those
protocol closers streamed to the user as literal text.
Adds an opt-in orphanStripTags registry to ToolCallParser (default
empty, no behavior change for other formats) and teaches
ToolCallProcessor to strip registered orphan closers — including runs,
chunk-split partials, and closers embedded mid-buffer — while
unregistered tag-looking prose like </div> still flushes verbatim.
XMLFunctionParser and Gemma4ToolCallParser register their closers.