# The VZ/slirp data plane: a measurement study A user-mode TCP gateway (slirp, terminating guest TCP on host sockets) feeding a macOS 27 `VZCustomVirtioDevice` virtio-net edge, driven toward 100 Gbit/s on a single-VM iperf3 benchmark. This document is the campaign's experiment record, written the way the work actually proceeded: an assumption, the experiment that tested it, the numbers, and what the result forced next. Negative results are recorded so the next session does not re-run a week of dead ends. Every number includes the host state under which it was measured, because on this machine that context is part of the result. Headline (2026-07-21, quiet-host records): receive 85.0 Gbit/s and transmit 68.9 Gbit/s, byte-exact with zero guest drops, from a starting point of 21.7 / 32.8 on the file-handle transport. Full BIG TCP moves the transmit operating point from P4 to P2. The current adaptive-loan path reproduced at 80.44--83.59 Gbit/s (median 81.90) over five six-second P2 legs and two boots on macOS 27.0 (26A5388g); all integrity counters were clean. Those runs carried explicit host-load warnings and therefore establish repeatability, not a new quiet absolute. The matching P16 receive reproduction reached 64.07 Gbit/s while `mediaanalysisd` was active and likewise does not replace 85.0. In a later, still busier interleaved RX window, switching the raw-pointer placement call to `writeData:length:error:` beat the adjacent `writeBuffer:exactLength:error:` controls by 25--42% (E28). Current findings, with detail kept in the experiment entries below: - **Operating point:** four shards, four queue pairs, full 524280-byte GSO/TSO; P16 for receive and P2 for transmit. These are now the bench defaults; an explicit `STREAMS` value still applies the same count to both directions (E15, E18). The configuration search is closed: E39's promotion brackets rejected every measured opt-in and combination against these defaults, so further gains are structural work (E20's split, E37's fix, frame economy), not knob settings. - **Receive:** the raw-pointer `writeData:length:error:` placement selector is now used after two bracketed wins over `writeBuffer:exactLength:error:`; both arms used the same reusable OCaml buffer and kept one payload call per frame. The remaining VZ cost is the mandatory framework copy plus per-element return. No write-side guest-buffer view exists, and enlarging the splice read buffer to the full BIG TCP ceiling regresses without increasing actual frame sizes. Preserving headroom-backed payload slices removes the preceding OCaml staging copy for most GSO frames. Host sockets stay on `Eio.Flow.single_read`; the Unix-fd bypass was removed after producing too many 64 KiB frames, and repeated nonblocking reads regressed against the adjacent Eio control. An Eio-native Cstruct ring now carries socket reads to VZ as external slices; a loaded-host triple improved mean RX about 2%, but remains opt-in pending a quiet bracket (E12, E25, E28, E29, E33-E35). Combining two ready host reads per flow is still opt-in until the quiet bracket in E31 decides whether it should become the default. Four single-queue devices can also expose four independent VZ provider queues as one logical guest port; this improved a clean P16 comparison from 73.25 to 76.65 Gbit/s, while eight devices regressed to 54.88 (E32). Stacking the receive opt-ins does not compose: the E31+E32+E35 trio measured 16-30% below the default in two independent busy windows while each opt-in alone was neutral-to-positive, and both ring-containing pairs lost (E38) -- promotion candidates are devices-only and devices plus two-read. - **Transmit:** guest-memory loans remove the drain allocation/copy and improve P1, but holding external leaves across partial writes regresses P2. The current policy attempts a direct write once, copies the unwritten suffix after a partial, and temporarily copies under backpressure. `writev` remains the measured blocker; separate writer executors and pre-copy offload both delay credits and regress throughput (E23--E27). The guest's TSQ allowance is now a knob (E36; the pinned kernel's default is 4 MiB, eight full frames, not the 1 MiB E26 assumed). E20's overlap landed as the systhread write offload and won P1 in two interleaved brackets (+3 to +12 percent against adjacent controls, neutral at P2); it is now the DEFAULT, with TCP_WRITE_OFFLOAD=false as the inline-write control arm (E42). The transmit watchdog (E37) was localised to a lost avail kick after a pair quiesces (E40) and is FIXED by the bounded Tx_pull walk: eighteen resumption cycles ran clean while the stranded counter caught seven lost kicks (E41). - **Integrity gates:** every reported winner requires zero guest drops/errors, zero VZ refusals/write failures, and loans = OCaml releases = provider-queue returns with zero outstanding or malformed loans. - **Compatibility:** the modern gateway also lifts the macOS 26 file-handle transport to 68.5 receive / 48.4 transmit in a busy window (E21); the current adaptive path still needs the same re-baseline there. ## 1. System under test ``` host (macOS 27, M5 Max: 6 P-cores + 12 E-cores, 18 total) +--------------------------------------------------------------+ | iperf3 client -P N | | | loopback TCP, one connection per stream | | published port (accept -> dup fd -> owning shard domain) | | | | | slirp gateway (OCaml 5, Eio) | | main domain: accept, DHCP/DNS/ARP, UDP/ICMP, control | | N shard domains: per-flow TCP core + host-socket splice | | + emit lane (device writes) + ingest lane (TX drains) | | | | | VZCustomVirtioDevice lanes | | one device / N pairs, or N devices / one pair | | one serial provider queue per device (Apple contract) | +------|-------------------------------------------------------+ | guest memory writes + per-buffer returnToQueue | +------|-------------------------------------------------------+ | guest VM: 8 vCPUs, LinuxKit, stock virtio_net driver | | MTU 65521, BIG TCP applied by the in-guest agent (rtnetlink)| | iperf3 3.21 server (threaded) | +--------------------------------------------------------------+ ``` | Component | Value | |---|---| | Host | M5 Max, 6 performance + 12 efficiency cores, macOS 27.0 (26A5378n) | | Device | `VZCustomVirtioDevice` (dynamic Objective-C; SDK headers lag the OS) | | Guest kernel | LinuxKit 6.12.59 (4 KiB pages), then pinned 6.18.39 (16 KiB pages, full 524280-byte virtio BIG TCP) | | Guest iperf3 | 3.21 (one thread per stream) | | Gateway | slirp: pure TCP core per shard, Eio edge, OCaml 5 domains | | Boot | EFI (GRUB ESP) -- required for per-queue MSI vectors in the guest | The device constraint that shapes everything: Apple delivers every queue of one custom device on that device's serial dispatch queue, and returning receive buffers is a per-buffer call whose cost reaches Mach IPC. Most experiments below are ways of taking work off that queue and off that per-buffer path. E32 tests several devices, each with its own provider queue. ## 2. Method The harness is `bench/vz_bench.sh`: build the pinned guest image, boot it through the gateway, measure both directions with `iperf3 -J` through a published port, and snapshot three counter planes (gateway, device, guest) at each phase boundary. Rules, each earned by a wrong conclusion it now prevents: - **One change, one measurement, revert on regression.** Every reverted experiment leaves its number in a comment at the site it would be re-tried from. - **Same-window A/B only.** Run-to-run spread on this host reaches 27% for one unchanged configuration (section 5); numbers from different noise windows are not comparable. Interleave A/B/A when a window cannot hold. - **Check the machine first.** `ps aux | awk '$3>60'` before trusting a run; `mediaanalysisd`, `corespotlightd`, and `system_installd` each cost 1.5-4 cores unannounced. - **Counters are the benchmark; profilers are for stack identity.** Both native sampling and full span capture perturb this workload by double-digit percentages (section 5). - **A discriminating experiment beats a plausible theory.** shards=2 on one queue pair (45 Gbit/s) versus shards=2 on two pairs (34) killed an interrupt-affinity theory in three minutes. - **Do not turn an unfixed bug into a product limitation.** A host/guest counter mismatch is a localisation result: preserve the reproducer, inspect both sides' capacity assumptions, and require a zero-drop rerun before disabling a protocol feature by default. (This rule reopened large-GSO receive after Experiment 2 had written it off; Experiment 12.) ## 3. The iteration log Each entry: the assumption going in, the experiment that tested it, the result, and what it forced next. ### E1. "The serial file-handle edge is the wall" - **Assumption.** The macOS 26 file-handle attachment (one datagram socketpair, serial reads) caps both directions. - **Experiment.** Implement the macOS 27 custom virtio device; same guest, same gateway. - **Result.** Transmit rose (32.8 to 38.3) but receive COLLAPSED to 1.8 under the first feature set (E2 explains why); even after E2's fix the custom device trailed the file-handle path on receive (13.2 vs 21.7) while beating it on transmit: it pays a cross-process notification per receive buffer that Apple's in-process device does not. - **Iterate.** The receive cost scales with buffers, so the buffer economy is the next subject (E2), and everything serial must come off the notification thread (E4, E5). ### E2. "Jumbo frames need mergeable receive buffers" - **Assumption.** VIRTIO_NET_F_MRG_RXBUF is how a 64 KiB frame fits guest buffers. - **Experiment.** Offer MRG_RXBUF at jumbo MTU; count buffers per frame and guest drops. - **Result.** Collapse: a 65 KiB frame spanned ~43 posted buffers, each with its own return; receive fell to 1.8 Gbit/s. Offering GUEST_CSUM|GUEST_TSO *without* MRG_RXBUF puts the driver in big-packets mode (one >= 65562-byte chain per frame): 1.8 to 13.2 immediately. - **Iterate.** One element per frame became an invariant to defend (`vz.rx_multi_frames`, `vz.rx_max_buffers` counters). The "mergeable is structurally unusable" conclusion stood until E12 re-examined it as a bug rather than a law. ### E3. "The main domain's funnels serialise everything" - **Assumption.** One effects fiber, one emitter fiber, and one recv fiber on the main domain cap receive regardless of shard count (adding shards made it *slower*). - **Experiment.** Give each shard its own device emit lane, then its own host sockets (accept on main, dup the fd, import on the shard), then fd-based mailboxes instead of `Eio.Stream` handoff. - **Result (busy host).** Receive 23.3 -> 41.8 (lanes) -> 65.9 (per-shard sockets, 4 shards); the cvwait-dominated profile (10,500 wait vs 2,000 work samples) confirmed the removed stage was coordination, not CPU. - **Iterate.** With the main domain out of the fast path, the next serial stages were the pair lock (E4) and the notification thread (E5). ### E4. "The virtio pair lock is costing a syscall-scale wait per frame" - **Assumption.** Four shards sharing two queue pairs contend the pair lock on every device write. - **Experiment.** Sample the lock wait (1-in-64 frames); match `--virtio-queue-pairs` to the shard count. - **Result (same window).** Lock wait 7.4 us -> 0.12 us per frame; receive 65.9 -> 74.1. Sweeping further: 6/6 and 8/8 REGRESS in both directions. ``` Figure 1. Receive by shard/pair count. Rows mix epochs; each is labelled with host state and whether shard ingest (E5) was active. 1sh/2qp ########## 31.4 (busy, pre-ingest) 2/2 ################ 56.3 (busy, with ingest) 4/4 ######################### 85.0 (quiet) <- operating point 5/5 ##################### 71.8 (quiet) 6/6 #################### 69.9 (quiet) 8/8 ############### 52.3 (busy) ``` - **Iterate.** The 4-shard peak is a core-budget law: four shard domains + main + the serial thread + the guest's vCPUs fill six P-cores, and a shard pushed onto E-cores caps every flow pinned to it. Config space is exhausted at 4/4; further gains must reduce per-frame work. ### E5. "Apple's serial thread is the transmit ceiling" - **Assumption.** Guest-to-host is capped because read + allocate + callback + return for every frame of every pair runs on the one mandatory serial dispatch queue (profiled at ~87% of one core). - **Experiment.** Shard-drained ingest: the notification handler only pokes a per-pair pipe; the owning shard drains the ring, reads each payload into its own domain's heap, and routes frames through the existing mailboxes. (`--virtio-shard-ingest`, A/B-able.) - **Result (same busy window).** Transmit at four pairs 52.2 -> 63.8; receive 74.1 -> 78.5 (guest ACKs during receive also stop relaying through the main domain). Extra pairs no longer cost the serial thread. - **Iterate.** Transmit now plateaus near 64-69 regardless of lane count -- the ceiling moved into per-shard work and the guest sender. Stream sweep to separate those: ``` Figure 2. Stream count at 4/4 RECEIVE (quiet) TRANSMIT (best-of-window) P8 ################### 75.1 P4 ################# 68.6-70.7 P16 ##################### 85.0 P8 ################ 63.8 P24 #################### 81.3 P16 ############### 60.6 P32 ################## 70.4 P24 ############## 55.0 ``` Transmit peaks at one flow per shard and degrades with more, so it is not window-starved; E9/E10 test the acknowledgement theories. ### E6. "GC pacing is a free win" - **Assumption.** 25% of wall in GC (profiled; the busiest shard domain half-collecting) means lazier pacing buys throughput; the pure-core bench agrees enthusiastically. - **Experiment.** `space_overhead` sweep on the pure core; then the same knob end to end in one quiet window; then bigger minor arenas (`s=4M`). ``` Figure 3. The bound-versus-measure reversal pure core (GB/s) end to end (Gbit/s, same window) o=120 ####### 1.38 o=120 RX 72.9 TX 63.1 o=200 ########### 2.21 o=400 RX 76.7 TX 48.2 (35.8 at P4) o=400 ############## 2.87 o=800 ############### 2.94 s=4M RX 64.9 TX flat o=1600 ############## 2.80 (baseline RX 84.8) ``` - **Result.** The pure core doubles at o=400; end to end the same setting collapses transmit 24% (44% at P4) -- it has no acknowledgement loop, and batchier major slices jitter ACK delivery until the guest sender's window starves. Bigger minor arenas kill receive the same way (longer individual pauses bite the near-saturated side). Defaults stand. - **Iterate.** Never accept a GC-tuning win from a benchmark whose latency loop is missing. The honest fix is fewer allocated bytes: E7. ### E7. "Recycling payload buffers removes the GC share" - **Assumption.** One payload-sized string per unit of payload (~9 GB/s through the major heap at 70 Gbit/s) is the GC share's cause; a ring of fixed buffers recycled at the guest-ack watermark removes it. The payload's lifetime provably ends at the ack (the send queue drops it). - **Experiment.** `Tcp_eio.Payload_ring` (contract pinned test-first: recycling order, ack-gated reuse, exhaustion fallback, flow-aligned accounting), wired behind `Splice.v ?recycle_payload`; three clean e2e runs; then the ring in isolation (`bench/splice_bench.exe`: socketpair feeder, cache-line consumer, bounded ack lag, four domains). - **Result.** The mechanism works: major cycles 2849 -> 448, GC 25% -> 11% of wall. End to end it is 22% SLOWER (62.3-62.9 vs 84.8-85.0). In isolation it is neutral-to-faster with collections halved (5.56 vs 5.43 Gbit/s). A later full-domain `obs` A/B made the interaction concrete. Expanding the ring from eight slots to the complete 4 MiB credit window cut sampled allocation 56.1 -> 6.44 GB and GC 3.09 -> 0.52 s, but receive still fell 50.67 -> 36.65 Gbit/s in the same profiled configuration. Sampled VZ payload placement rose 13.4 -> 22.6 us per 128 KiB frame: the recycled 64 MiB working set makes the downstream guest-memory copy colder. Byte-exactness and zero guest drops held throughout. - **Iterate.** The ring ships gated off. Removing an allocation is not a win while the mandatory following copy gets slower; the successor must change the payload copy path or copy count, not merely the ownership lifetime. Getting even this far required fixing the profiler (E8). ### E8. "The allocation profiler tells the truth" - **Assumption.** memtrace attributes the load window. - **Experiment.** Profile a 100 GB multi-domain run; get a 2 KB boot-only trace at any sampling rate. - **Result.** memtrace sampled domain 0 only -- every shard-domain sample was deliberately dropped by the tracer. Fixed in observe: per-domain writers over the shared fd (the vendored multicore trace format was already domain-complete), one domain-tagged file, consumers unchanged. Found on the way: `Domain.DLS.get` segfaults inside memprof callbacks (its backing-array growth is not re-entrant); the per-domain lookup is an atomic slot array. With full-domain data, the wait ledger closed: the dominant receive wait is unnamed domain idle -- shards are source-starved between mailbox batches, pointing the remaining receive headroom at frame size (E12) and the payload copy path, not locks. - **Iterate.** Every allocation conclusion drawn before this fix saw domain 0 only; E7's attribution used the fixed tracer. ### E9. "ACK latency limits transmit" - **Assumption.** A computed ACK waits up to a 64-input worker batch (~2 ms) before its lane emission; cutting that latency lifts transmit. - **Experiment.** Emit each input's outputs as it is handled instead of after the batch. - **Result.** Transmit COLLAPSED 63.1 -> 42.4 (16.1 at P4). Back-to-back lane sends let the device coalesce guest interrupts across the burst; interleaved sends raise one interrupt per ACK and the sending guest drowns. Reverted; the why lives at the emission site. - **Iterate.** Interrupt RATE beats ACK latency. Test the rate directly: E10. ### E10. "ACK interrupt rate limits transmit, then" - **Assumption.** Halving the gateway's ACK frames (every 4th jumbo segment instead of every 2nd) halves the sending guest's receive interrupts and lifts transmit. - **Experiment.** A stretch-acknowledgement probe in the pure core (reverted after measurement), quiet-window A/B. A byproduct: nothing in the 109-test suite pinned RFC 5681's every-second-segment rule -- it passed unchanged under stretch-4. The pin test now exists. - **Result.** Neutral (receive +1 inside noise, transmit flat). The transmit limit is not acknowledgement mechanics at all. - **Iterate.** What remains for transmit is per-shard work and the guest sender itself. The "tso_max 64 KiB: no BIG TCP transmit" premise this entry closed with was true on 6.12 and is FALSE on the patched 6.18.39 (uniboot/kernel/6.18.x, virtio-net publishes its BIG TCP TSO capacity). The then-current 4 KiB-page guest reported TsoMaxSize 184320 and a counter-read transmit leg carried 405718 frames averaging 125.3 KB (max 131004). E13 supersedes that image with the 16 KiB-page kernel and verifies the full 524280-byte limit; the historical result remains the evidence that first falsified the 64 KiB premise. ### E11. "The host sockets starve the splices" - **Assumption.** Receive-side shards are source-starved (E8's wait ledger) because per-flow socket buffers run dry (reads average 160 KB against a 256 KB buffer); bigger SO_RCVBUF/SO_SNDBUF fixes it. - **Experiment.** 4 MB both ways on every proxied host socket, interleaved A/B/A. - **Result.** Neutral (widen 72.5/65.8 vs no-widen 73.2, same windows). macOS autotuning already covers it; the 160 KB average is how loopback TCP delivers. - **Iterate.** Starvation is producer pacing plus per-byte copy count -- the payload copy path (E7's successor), and fewer, larger frames: E12. ### E12. "Large receive frames are structurally impossible" (falsified) - **Assumption (inherited from E2).** Mergeable buffers cap at one page, so frames above one big-packets chain (~64 KiB) cannot work. - **Experiment.** Treat E2's drops as an unlocalised bug instead of a law. Vary kernel and framing independently; read host and guest counters together. | Configuration | Buffers/frame (max) | Guest drops | Verdict | |---|---|---|---| | 256 KiB GSO + MRG_RXBUF, 6.12.59 | 34 (83) | 427 (2.6%) | TCP collapse, 1.6-2.2 Gbit/s | | 128 KiB GSO + MRG_RXBUF, 6.12.59 | 22 (84) | 160 | broken | | 128 KiB GSO + MRG_RXBUF, 6.18.39 (`MAX_SKB_FRAGS=45`) | -- | 429 | still broken | | 64 KiB control + MRG_RXBUF, 6.18.39 | -- | 210 | mergeable delivery itself is the bug | | 128 KiB GSO, no MRG_RXBUF, 6.18.39 | **1 (1)** | **0** | correct | - **Result.** The host wrote every frame successfully in every row (`vz.refused=0`, `vz.stalled=0`, `vz.rx_write_failures=0`); only the guest drop delta localised the fault, and it tracked mergeable delivery, not frame size. A **non-mergeable receive element is a scatter-gather chain, not a page**: with `CONFIG_MAX_SKB_FRAGS=45` one chain carries 128 KiB. The corrected E12 configuration (128 KiB GSO, mergeable off, 6.18.39) delivers every frame in exactly one element, zero drops. E13 later raises the default to the full 524280-byte limit. Combined-tree check (busy host): 95% of receive frames took the GSO path (799,345 of 839,008, max 131,004 bytes), zero drops, zero multi-buffer frames, average frame ~125 KB -- halving every per-frame serial cost. - **Iterate.** The open measurement: this configuration on a quiet host, both directions. On the busy host, four matched shards/pairs reached 61.56 Gbit/s with zero drops (one shard: 28.55); a later busy run (mediaanalysisd at 218%) read RX 76.28 / TX 59.69 at four streams with the same clean structure across 612k frames -- one buffer per frame, zero drops, 95% GSO frames, 99.9% GSO bytes, 125 KB average. The default six-second native sample attributed 67 of 91 receive-send samples to `writeBuffer:exactLength`/`memmove` and 13 to `returnToQueue`; aligned counters measured 14.2 us payload write, 2.4 us return, and only 0.23 us queue-lock wait per sampled frame. The next receive blocker is therefore the two payload copies plus framework return, not mergeable buffering or queue contention. ### E13. "Full BIG TCP and fewer allocations are the remaining TX win" - **Assumption.** Once the pinned 6.18.39 guest advertises Linux's full 524280-byte GSO/TSO size, the remaining TX plateau is the two payload-sized allocations seen in the full-domain trace: one in the VZ ring drain and one in the host-socket splice writer. - **Experiment.** Boot the 16 KiB-page kernel and require the guest agent to report `GsoMaxSize=GroMaxSize=TsoMaxSize=524280`. Capture TX with the default six-second native sample, VZ/guest/control counters, runtime events, and multi-domain memtrace. Resolve the allocation call tree, retain a splice writer buffer after its high-water mark grows, then run both allocation and unprofiled same-window A/Bs. Finally compare 4/4/P4 with 8/8/P8 and time the active native pull path one frame in 64. - **Result: the allocation diagnosis was real, but not the throughput ceiling.** The baseline allocated 89.86 GB. Two sites were almost exactly equal: `Tcp_eio.Splice.write_batch` crossed its 1 MiB target by one full GSO frame and allocated a throwaway Bigarray every batch (43.43 GB), while `Vz.Custom_virtio_net.drain_tx` allocated the incoming OCaml string (43.38 GB). Retaining a geometrically grown writer buffer removed the first site completely and cut sampled allocation to 50.76 GB (-43.5%). The new leader is the unavoidable-at-present drain string, 47.14 GB / 92.9% of allocation. The unprofiled pairs were 44.21 -> 55.26 and 65.03 -> 65.27 Gbit/s. Their disagreement is the host-noise warning in concrete form: the allocation win is proved; no wall-clock win is claimed. The post-change profiled run reached 50.82 Gbit/s (the earlier profiled baseline was 46.57), but profiled throughput is not used for the verdict. At the four-lane operating point, an unprofiled 62.25 Gbit/s run carried 62.38 GB in 126838 frames (about 492 KiB/frame), byte-exact with zero guest drops. Sampled native means per frame were: | Pull phase | Mean | |---|---:| | VZ payload read/copy | 14.6 us | | OCaml string allocation | 4.2 us | | list descriptor construction | 1.4 us | | `returnToQueue` | 2.1 us | | pair-lock wait | 0.09 us | Thus the complete measured pull is about 22.3 us per 492 KiB frame, only roughly 12% of a stream's elapsed budget. The native sample agrees: on each busy shard, `drain_tx` was 34--50 of 435 samples (8--11%), while blocked host `writev` was 120--140 (28--32%). Removing drain allocation is still useful, but cannot arithmetically turn 62--65 into 100. Eight pairs/shards/streams REGRESSED to 41.66 Gbit/s. It had no loss and a sampled 83 ns pair-lock mean, but placed the guest plus eight OCaml workers beyond this host's useful core budget. A separate macOS loopback control was also kernel-bound: iperf3 P4 reached 40.20 Gbit/s with default 128 KiB writes and 42.05 Gbit/s in reverse with 1 MiB writes, at about 90--112% system CPU. This is context, not a gateway ceiling (the gateway itself exceeds it by batching differently), but it corroborates the socket-output attribution. - **Iterate.** TX's next blocker is the host-socket delivery path, not virtio locks, queue count, or the removed splice allocation. E11 already showed that merely widening `SO_SNDBUF`/`SO_RCVBUF` is neutral, so do not repeat it. Before changing the splice again, E14 maps the loopback TCP ceiling. Only if that parameter sweep leaves headroom does the next experiment reduce host-kernel crossings/copies or change how a batch reaches the socket while preserving the large write and ACK cadence. The VZ pull allocation remains a secondary, bounded-owned-buffer target; it must be measured after the socket change rather than advertised as the route to 100. ### E14. "The loopback TCP sink leaves room for 100" - **Assumption.** The 62--65 Gbit/s TX plateau is gateway work that can be recovered by changing socket batching. This assumes, but does not yet prove, that macOS loopback TCP plus iperf3 can consume 100 Gbit/s with some write size and stream count. - **Experiment.** With no VM, sweep IPv4 loopback iperf3 over write sizes 128 KiB, 512 KiB, and 1 MiB crossed with 1, 2, 4, and 8 streams. Every point used `-t 6 -O 1 -J`; the P1 size axis was repeated. iperf3 3.21 rejects the requested 4 MiB block with `maximum = 1048576 bytes`, so no different workload is substituted for that row. Host load before the sweep was 6.31/6.53/11.44; WindowServer used 44% CPU and two unrelated agents about 12--13% each, so these are explicitly non-quiet results. - **Result.** Gbit/s from sender summaries; every cell had zero retransmits: | Write size | P1 | P2 | P4 | P8 | |---|---:|---:|---:|---:| | 128 KiB | 90.35 (repeat 88.40) | 60.46 | 39.13 | 38.73 | | 512 KiB | 97.87 (repeat 98.37) | 60.69 | 40.76 | 38.52 | | 1 MiB | **99.54 (repeat 99.12)** | 61.97 | 40.52 | 38.57 | Thus macOS loopback is not materially capped below the 100 Gbit/s target; the earlier P4 controls measured a poor stream-count/write-size combination, not the loopback ceiling. The surprising constraint is cross-flow scaling: one large writer is about 2.5x P4. The gateway exceeding raw P4 at 62--69 means iperf's client write schedule is not an exact model of the splice, but adding flows clearly is not free. - **Iterate.** Keep the loopback socket as the TX target. Before changing the batch target, sweep the full-GSO gateway at fixed 4/4 over P1/P2/P4 and expose logical batches, `Eio.Flow.single_write` calls, partial writes, and bytes per call from the OCaml splice. This separates macOS cross-flow contention from per-shard pure-core/device cost and says whether a larger logical batch would actually remove a kernel crossing. E11's socket-buffer widening stays closed. ### E15. "Full GSO has the same P4 optimum as 128 KiB GSO" (falsified) - **Assumption.** Four flows remain optimal because one flow per shard uses all four data-plane domains, and 524280-byte guest frames reduce every per-frame cost equally. - **Experiment.** Rebuild the guest agent and declarative Uniboot image, then require `GsoMaxSize=GroMaxSize=TsoMaxSize=524280`. In one VM boot at fixed four shards/four pairs, run P1/P2/P4 for six seconds each. Take aligned guest, VZ, and new OCaml splice-write counter deltas around every leg. A first run against the reused disk reported guest GSO 131072 and was rejected as a full-GSO result; it remains the historical control below. - **Result.** Full GSO moves the optimum to P2 and produces the first clean aggregate above 80 Gbit/s in this campaign: | Guest GSO | Streams | TX Gbit/s | Batches | `single_write` | Partials | Calls/batch | |---:|---:|---:|---:|---:|---:|---:| | 128 KiB (stale-image control) | 1 | 42.09 | 26833 | 27013 | 180 | 1.01 | | 128 KiB (stale-image control) | 2 | 67.82 | 43216 | 43302 | 86 | 1.00 | | 128 KiB (stale-image control) | 4 | 58.77 | 37661 | 153780 | 116119 | 4.08 | | 524280 B | 1 | 56.17 | 26968 | 27016 | 48 | 1.00 | | 524280 B | **2** | **81.41** | 39229 | 47310 | 8081 | 1.21 | | 524280 B | 4 | 63.52 | 30879 | 187635 | 156756 | 6.08 | The full-GSO P2 leg moved 61.13 GB in 117764 frames (about 519 KiB/frame). All full-GSO legs had zero guest drops/errors, zero VZ refusal deltas, and zero VZ stall deltas. P4 is not core-starved in the abstract: it is host-socket backpressure made concrete, with 5.08 partial calls added per logical batch. P1 avoids that amplification but leaves three shard domains idle and reaches only 56.17. P2 balances the two ceilings. - **Iterate.** Expose and sweep the OCaml logical batch target at P2. Judge it on actual `single_write` count: widening is useful only if calls fall, since P4 proves a larger pending write can fragment into many kernel crossings. Publish per-pair TX distribution and repeat the winner before promoting 81.41 to the quiet baseline; this run had no `>60% CPU` warning but is one window. ### E16. "Fewer socket writes make full-GSO P2 faster" (falsified) - **Assumption.** Raising the splice's 1 MiB logical write target to 2 or 4 MiB reduces kernel crossings enough to recover the remaining P2 TX gap. - **Experiment.** Keep the E15 full-GSO image, four shards/four pairs, and P2. Sweep the new `--tcp-write-batch-bytes` target over 1, 2, and 4 MiB, taking splice counter deltas around each six-second leg. Repeat 1 and 2 MiB in the same noisy host window so the comparison does not depend on the absolute. - **Result.** Larger batches did reduce actual writes, but consistently lost throughput: | Target | TX Gbit/s | Batches | `single_write` | Partials | Calls/batch | |---:|---:|---:|---:|---:|---:| | 1 MiB | **81.41** | 39229 | 47310 | 8081 | 1.21 | | 1 MiB repeat | **84.24** | 40395 | 45168 | 4773 | 1.12 | | 2 MiB | 79.70 | 24132 | 30835 | 6703 | 1.28 | | 2 MiB repeat | 80.52 | 24034 | 32709 | 8675 | 1.36 | | 4 MiB | 78.89 | 14257 | 23366 | 9109 | 1.64 | The direct same-window comparison is 84.24 versus 80.52 Gbit/s. All legs had zero guest drops and zero VZ refusal/stall deltas. A batch can exceed its target because the final queued rope is not split merely to hit it: the 1 MiB repeat reached a 4.71 MB logical batch, while 4 MiB reached 6.81 MB and a 4.91 MB largest actual write. Therefore syscall count is not the limiting scalar. Larger aggregation worsens pacing/backpressure latency despite doing less crossing work. - **Iterate.** Keep 1 MiB as the default and close batch widening on macOS. Profile the winning P2/1 MiB configuration for stack identity, then publish per-pair TX bytes/frames to distinguish flow placement from owning-shard work. Treat the native sample as attribution only because it perturbs throughput. ### E17. "Aggregate waiting identifies the P2 TX blocker" (falsified) - **Assumption.** A full `obs run` of the E16 winner can use the aggregate scheduler split to name the limiting wait. The old report did exactly that: it called the run I/O-bound because 71% of average domain capacity waited, and promoted 5.72 s of `sleep` under `accept` to the headline. - **Experiment.** Run four pairs/shards, P2, full GSO, and the 1 MiB write target for six seconds with the default six-second native sample, guest/VZ/ splice counters, runtime events, and multidomain memtrace. Reconstruct the scheduler critical path and compare it with per-thread native call graphs. Add a reporter regression containing one continuously running worker and three longer-idle listener domains. - **Result.** The old headline was wrong: all 9.559 s of the reconstructed critical path was running one worker fiber; the listener sleeps were off path. `obs report` now keeps the 71% wait in the multidomain capacity budget but reports a CPU critical path, does not emit the `accept` syscall finding, and does not mislabel suspended `accept` span time as CPU self-time. The workload itself moved 37.33 GB through 74,519 guest frames, about 501 KiB/frame, with zero VZ refusals, supply misses, or stall deltas. The splice made 25,068 writes for 24,790 logical batches (1.011 calls/batch, 278 partial calls). On each of the two hot shard threads, macOS `sample(1)` found `writev` at the leaf in 34--36% of snapshots and the VZ drain in 11--13%; the serial VZ notification queue appeared in only nine snapshots. Because `sample(1)` includes blocked threads, `writev` is evidence for the socket delivery/backpressure boundary, not a claim that the syscall burns that share of CPU. Memtrace still attributes 94.33% of allocation to `Vz.Custom_virtio_net.drain_tx`. Aligned samples measured roughly 20.3 us to read/copy a frame, 9.8 us allocation, 4.6 us descriptor construction, and 5.3 us return. That VZ read-and-return cost is real but remains secondary to the socket leaf in the native stacks. The profiler held the first five measured seconds to 46--48 Gbit/s; after it detached the final second reached 63.0, so the profile contributes attribution, not a replacement throughput number. - **Iterate.** The next blocker is now narrow but not yet split: host-socket delivery on each owning shard can be limited by the iperf3 client endpoint, kernel pacing/backpressure, or serialization of socket wait with shard core work. In the next quiet P2 run, publish per-pair bytes/frames and sample the host iperf3 client at the same time. If the client pins a core, replace the benchmark sink before gateway engineering. Otherwise measure per-shard `writev` off-CPU duration and test decoupling socket delivery from core/device drain. Do not lead with the bounded VZ buffer pool: this trace gives it only the smaller measured share. ### E18. "P2 wins by flow placement, or the client is pinned" (both falsified) - **Assumption.** E17 left three suspects for the P2 transmit plateau: a favorable hash/XPS placement of the two flows across pairs, a saturated iperf3 client endpoint, or serialization inside the shard/kernel socket path. - **Experiment.** Publish per-queue-pair TX/RX frame and byte counters from the device (always on; they ride the control-socket snapshot the bench already takes) and poll the host iperf3 client's per-thread CPU at 1 Hz with `ps -M`. One boot, four shards/pairs, full GSO: three identical P2 legs then one P4 leg. Busy host (a concurrent repo-wide test build), so every claim is same-window comparative; unprofiled. - **Result.** P2 reproduced 79.11 / 81.36 / 80.36 Gbit/s -- the E15/E16 winner was not a one-off. Placement across the three legs was pairs 0+1 (50.3/49.7%), pairs 1+2 (42/42%, 8% straying during a mid-run queue migration), and pairs 2+3 (43/42%): the flows never collided on one pair, and throughput was insensitive to which pairs carried them. The P4 leg spread exactly 25.0% per pair and still fell to 60.90. The client's busiest thread reached 45--46% of a core at P2 (three threads, ~91% total) and 29% at P4. Per-pair sums matched the global counters exactly, and every leg averaged 511 KiB per frame (full BIG TCP confirmed). - **Iterate.** Placement luck and the benchmark endpoint are both eliminated. P4's loss with perfect balance and an idle client points at the socket path itself, so measure the sink with no gateway at all: E19. ### E19. "The loopback sink scales with flows" (falsified: aggregate decays) - **Assumption.** Implicit since E14: adding streams adds sink capacity, so the gateway's stream optimum is about the gateway. - **Experiment.** Fully independent iperf3 pairs -- separate server processes, separate client processes, distinct ports, one flow each, `-l 1M` -- measured concurrently in one window: one, two, then three pairs, plus a same-window single-flow control. - **Result.** One flow 96.8; two flows 31.9 + 31.9 = 63.8; three flows 14.9 x 3 = 44.7; E14's single-process -P4 (40.5) and -P8 (38.6) complete the curve. Aggregate DECAYS super-linearly with concurrent flows and the collapse survives full process separation, so it is in the kernel path, not in iperf3's structure. macOS 27's `lo0` interface counters do not move during these runs (loopback TCP bypasses interface accounting), so the per-segment mechanism is not observable from userspace. Two corollaries. First, the gateway BEATS this raw curve at every multi-flow point (81--83 vs 64 at two flows; 61--63 vs 40--45 at four): the splice's 1 MiB `writev` batching outperforms iperf3's own sender under concurrency, and E14's "adding flows is not free" surprise is this curve. Second, the client-side per-flow cost is nonlinear the same way: a P1 flow at ~50 Gbit/s costs its receiving thread 40% while a P2 flow at ~40 costs 46%. - **Iterate.** At P2 the gateway is already above the raw two-flow regime; the only direction with large measured headroom is FEWER flows: the sink takes 96.8 on one flow and the gateway delivers 54. Decompose P1: E20. ### E20. "P1 is pacing-limited" (falsified: the per-flow chain pins a core) - **Assumption.** A single flow underuses four shards, but 54 against a 96.8 sink ceiling with an idle client could still be pacing or wakeup latency rather than CPU. - **Experiment.** Two P1 legs with 1 Hz per-thread polling of the VZ process alongside the client poll and the per-pair counters. - **Result.** 54.00 and 53.99 Gbit/s -- rock stable once placement is stable (both legs kept the flow on pair 0; an earlier boot's P1 legs read 39.4 with a 90/10 mid-run pair migration against 52.7 pinned, so P1's run-to-run spread is largely placement churn). One VZ thread ran at 88--93% for the whole leg while the next busiest sat at 8%: the owning shard's domain -- ring drain with its payload copy, the TCP core, and the splice's `writev` with its kernel copy -- is a serial chain pinned on one core. The client thread was at 39--43%; the sink can take 96.8. - **Iterate.** The chain does two payload-sized copies per byte (guest memory to drain string, batch to socket buffer) plus core work, all on one domain. Ranked levers: (1) remove the drain copy and its allocation (92.9% of TX allocation in E17) by letting the socket write gather directly from guest element buffers and returning elements only after the write completes -- Apple documents holding and out-of-order return, and the ring depth flow-controls the guest. This applies P4's ownership design to the VZ TX ring, now that the core carries ropes end to end. (2) Split the chain across two domains (drain+core versus socket delivery) so the remaining copies overlap. The loan removes work; a second domain only moves it, so test the loan first and re-measure P1 and P2 before considering 2. ### E21. The modern gateway re-baselines the macOS 26 transport - **Assumption.** The file-handle numbers (21.7 receive / 32.8 transmit) reflect the transport's ceiling; macOS 26 hosts, which have no `VZCustomVirtioDevice`, are stuck there. - **Experiment.** Boot the same image and gateway with `--network=file-handle`, four shards, jumbo MTU, and sweep P1/P2/P8 both directions by hand in one busy-host window. - **Result.** Receive 27.7 / 48.9 / 68.5 at P1/P2/P8; transmit 48.4 / 44.0 / 45.8. Receive is 3.2x the old baseline and transmit 1.5x, on the same transport those old numbers were measured on: the campaign's gateway-side work -- shard lanes, per-shard host sockets, write batching, checksum offload -- was transport-independent, and most of what this study bought carries to macOS 26 unchanged. Transmit's flat stream response (48/44/46) shows the E20 single-flow bottleneck again, arriving through a serial transport. - **Iterate.** A quiet-window pass for the record; beyond that, the E20 chain levers are the same levers here. ### E22. The loan API feasibility probe (zero-copy TX is possible; holds must be brief) - **Assumption.** E20's guest-buffer loan needs four properties from the undocumented element API: a no-copy buffer view, guest-memory pointer stability while an element is held, late out-of-order return, and a header/payload split it can work with. - **Experiment.** A temporary `VZ_TX_LOAN_PROBE` block in the shard drain (reverted after measurement; 97 lines): dump `readBuffers` on live bulk elements, then hold exactly one mid-stream element, checksum its raw payload pointer at hold and at release, and return it late and out of order. Three probe iterations were themselves instructive failures: the first re-queried `readBuffers` at release (empty -- wrong instrument), the second gated the hold on buffer 0's length (always the 78-byte header, so every bulk element was consumed and dropped: sustained ~1.5% large-segment loss collapsed P1 to 0.65-0.80 Gbit/s), and the third held one element indefinitely. - **Result.** All four properties answered on live traffic: 1. `readBuffers` returns `NSArray` whose `.bytes` are direct guest-memory pointers -- and it CONSUMES the element's readable state (readable 523830 -> 0; a second call returns an empty array). Capture once at drain; never mix with `readBytesIntoBuffer:`. 2. A full-GSO element is a scatter chain of ~17 buffers: buffer 0 is exactly the 78-byte header stack (virtio 12 + Ethernet 14 + IPv4 20 + TCP-with-timestamps 32), then 32 KiB payload fragments. The header/payload split the loan design wants is the hardware layout. 3. Late, out-of-order `returnToQueue` works while traffic keeps kicking the queue, but E23 found the stronger tail condition: a delayed return from an arbitrary OCaml domain can fail to produce a timely used-ring notification once traffic stops. Returns must be handed to the custom-device provider's serial dispatch queue. Payload policy remains in OCaml; the native boundary only schedules the framework call on its owning queue. 4. The raw payload pointer's content is byte-stable across a hold (checksum d857a3e7 unchanged), and a 4-frame (sub-millisecond) hold costs nothing: 53.28 Gbit/s against the 55.8 same-window baseline. The indefinite hold is the sharpest finding: the guest STOPPED SENDING entirely (tx_packets frozen) with one 512 KiB element unreturned -- Linux's TSQ/BQL accounting treats the un-completed skb as in-flight and blocks the flow's next large send. Release latency is therefore a hard design constraint, not a nicety: elements must return within one socket write cycle, and a flow whose socket stalls must either fall back to copy-and-return or accept that holding IS the backpressure (which is the correct semantic -- but only with releases wired to write completion, so a recovering socket recovers the flow). - **Iterate.** Write the loan contract tests with these facts pinned: single-shot capture, per-buffer leaves, release-on-write-completion, a per-pair hold budget protecting the ring, and a copying fallback. ### E23. Guest-transmit loans: ownership proven; rare tail completion still open - **Assumption.** E20's guest-buffer loan -- lend guest TX buffers through the whole gateway instead of copying them -- removes the drain copy and its allocation without corrupting payloads, leaking ring elements, or stalling flows through the guest's in-flight accounting. - **Experiment.** The full loan path landed as an eight-commit series (rope external leaves with owner tokens; ethernet borrowed frames; TCP rope-tail decode and vectored splice writes; slirp Frame_rope with eager Release effects on every reject path; the vz loan drain with per-pair return queues). `--virtio-tx-loan-budget` (bench: `VIRTIO_TX_LOAN_BUDGET`) bounds ring occupancy; zero restores the copying drain as the control arm. Smoke plus interleaved A/B legs, all on a polluted host (mediaanalysisd at 212--219%, a concurrent qemu at 114%, and two neighbouring sessions building -- every absolute below is suspect and labelled). - **Result: the ownership mechanism is correct end to end.** Over a 23.75 GB P1 leg: 45,362 loans and exactly 45,362 releases, zero outstanding at teardown, zero malformed, zero budget fallbacks, 27 small-frame layout fallbacks, and every `tx_frame` accounted. The splice reported 17,705 of 17,705 batches as DIRECT vectored writes -- the flattening copy left the transmit path entirely, so payload bytes now cross exactly once, from guest memory into the socket buffer inside `writev`. iperf3 byte counts matched throughout. Throughput is indicative only: P1 with loans read 63.28 / 61.63 in the two mildest windows against 55.77 for the same-window copy control and the 54.0--55.8 historical baseline (roughly +13%), then 39.10 when mediaanalysisd and the qemu overlapped; P2 readings (71.50 loan under the worst noise, 80.02 control under moderate noise, 67.32 loan under the worst again) contradict each other and are not usable. No verdict is claimed from this window. - **Result: zero application-owned loans was not initially enough.** The first P2 writer experiments ended with `loans == releases` and `tx_loans_outstanding == 0`, yet Linux raised `NETDEV WATCHDOG` five seconds later and incremented `tx_errors`. Two independent bugs were present. First, cancelling a splice released its active batch but stranded `Data` still in the writer stream; a deterministic blocked-writer test now proves active, queued, and post-close external owners are all released. Second, calling delayed `returnToQueue` on the pair's OCaml drain domain cleared our counter but did not reliably deliver the tail completion to the guest. Batched returns now hop to the VZ custom-device provider's serial dispatch queue. The validating polluted-host P2 leg completed normally at 72.69 Gbit/s: 68,184 loans / 68,184 releases / zero outstanding, zero malformed, zero guest drops and errors, no watchdog, and byte-exact iperf completion. - **Result: application release and framework return are now distinguished.** `vz.tx_loan_releases` increments when the OCaml owner relinquishes a loan; `vz.tx_loan_returns_completed` increments only after `returnToQueue` was actually invoked on the VZ provider queue. A direct P1 leg completed at 58.88 Gbit/s with 56,264 / 56,264, and two target-attached profiled P1 legs completed at 50.45 and 48.92 with 12,068 / 12,068 and 46,760 / 46,760; all had zero outstanding and zero guest errors. This proves those legs, not the intermittent case: one earlier profiled P1 leg still reached the guest sender summary and then emitted repeated `NETDEV WATCHDOG`, without a usable end snapshot. Until a failing leg shows whether releases exceed completed returns, tail-completion correctness remains an open race rather than a solved invariant. - **Result: batching several loaned frames is not the P2 blocker.** A 524,280 byte target still gathered two 523,752-byte TCP payloads because the gather test is applied before adding the next rope; it reached 72.61 Gbit/s with 24,970 partial writes. A target below one normal payload made almost every batch one rope and reached 73.0 Gbit/s, still with 22,880 partial writes. After correct VZ-queue returns, the normal 1 MiB target reached 72.69 with 10,836 partial writes. These are noisy absolute numbers, but they falsify multi-frame gather size as the missing 25--30 Gbit/s improvement. Keep the 1 MiB default; do not turn the experimental 500,000-byte test value into a self-imposed frame limit. - **Result: the first trustworthy post-loan native profile names `writev`.** Obs now attaches xctrace to the target PID and reaps exporter processes through `EINTR`; the former all-process capture and interrupted-`waitpid` path produced valid raw traces but routinely lost automatic exports. In a four-second P1 leg (48.92 Gbit/s under profiler perturbation), 46,760 loaned 523,818-byte frames became 18,726 socket calls, 101 partial; 24.48 GB / call is about 1.31 MB and the largest observed call was 1.57 MB. Native self time was `writev` 45.1% (2,567 of 5,696 weighted samples), then `poll` 7.4%; the largest named OCaml function was `Bytes.loop` at 2.4%. The removed VZ drain allocation is no longer the CPU leader. The next test is the P2 path: determine whether two owning shards provide enough parallel socket-copy capacity or whether the macOS loopback write itself is the remaining aggregate ceiling. - **Iterate.** Profile P2 at the established four-pair configuration, then the quiet-window A/B decides the number: interleaved budget 16 / 0 / 16, `STREAMS="1 2"`, plus a TSQ comparison (tx_loans_outstanding against throughput; raise tcp_limit_output_bytes via the guest agent) if P1 regresses. The integrity gates are automated by the counters and held on the first live run. Profile the now-correct loan path before further writer changes: Preserve releases/completed/guest-error gates on every leg; a failing leg is now more valuable than another clean repeat because it localises the tail race. ### E24. "The loan un-pins the P1 chain" (half true: faster, still pinned) - **Assumption.** Removing the drain copy and allocation frees enough of the owning shard that the single-flow chain stops being CPU-bound. - **Experiment.** Back-to-back thread-polled P1 legs, loan against copy, one busy window (mediaanalysisd at 217% plus a concurrent qemu; the relative claim is same-window, the absolutes are depressed). - **Result.** Loan 60.87 / 57.92 against copy 50.93 / 46.20 -- a same-window +21%, the loan's first controlled win. But the owning shard still pins: top thread 84.5% mean loaned against 88.5% copying. The freed cycles were spent, not banked. The second-busiest thread rose 7.4% to 18.5% with loans -- the batched return dispatch riding the device queue -- work that no longer runs on the pinned shard. - **Iterate.** Splitting socket delivery from drain and core across two domains) survives E24: the chain is still the wall, now spending its budget on the TCP core plus the kernel-side writev copy. The quiet A/B prices the loan's win properly first. ### E25. "The element's write side mirrors readBuffers" (falsified) - **Assumption.** Receive placement could lose one of its two copies the way transmit did, by writing socket bytes straight into the guest's posted receive buffers through a write-side view API. - **Experiment.** Re-read the E22 runtime dump for writable element methods. - **Result.** Closed negative. The write side offers only copying entry points -- `writeBuffer:exactLength:`, `write:`, `writeData:`, `writeData:length:` -- plus byte counts. No call exposes a destination pointer into the posted buffer, so direct receive placement is impossible on this API and receive keeps exactly one mandatory framework copy. E28 later resolved the smaller selector question: the three-argument `writeData:length:error:` entry point takes the same raw pointer and length as the old selector and is materially faster here; it does not add a guest-memory view or remove the caller's Rope flattening. - **Iterate.** Receive levers are now bounded: the staging-blit probe, and fewer per-frame costs -- not zero-copy placement. ### E26. "Release latency has a measurable knee" (it is a cliff, at one batch) - **Assumption.** Sweeping an artificial release lag of K frames maps a throughput-versus-hold-latency curve that calibrates budgets and return batching. - **Experiment.** A reverted probe deferred each loan's release until K more loans had passed (P1 only; the lag queue was source-local). K swept 0/2/8/32/64. - **Result.** K=0 ran 62.68; EVERY K >= 2 wedged the flow outright -- no throughput, permanent stall. The mechanism is E22's, now bounded: at 512 KiB frames the guest's default in-flight allowance (tcp_limit_output_bytes, 1 MiB) is TWO frames, and a lag expressed in arrivals strands its last K elements the moment the pipeline pauses, which stops arrivals, which stops releases. Not a knee: a cliff whose edge is one write batch. The shipped design lives because its release trigger is write completion -- it fires whether or not new frames arrive -- and E23's teardown paths flush the rest. - **Iterate.** Two rules graduate to design law: a loan's release must be driven by CONSUMPTION, never by subsequent arrivals; and every quiescence path (close, cancel, teardown) must flush. Both hold in the shipped code, both are now test-pinned, and any future return batching must keep the completion trigger. ### E27. Partial-write ownership and a separate writer pool - **Assumption.** The P2 loan regression comes from holding guest-backed ropes across repeated partial `writev` calls. Copying only after the first partial write should return the VZ element promptly without giving up the direct path on an uncongested socket. If the remaining socket copy still serialises the owning shard, a separate ordered writer executor should let the shard process credits and new frames in parallel. - **Experiment.** First compare loans against the copying control on the same four-shard/four-pair boot. Then make splice ownership adaptive: try external leaves directly once; after a partial write, copy only the unwritten suffix into the splice's reusable buffer and release its owners before retrying. Keep a bounded 16-batch recovery window that copies before the first syscall, then probes direct delivery again. Finally, run each ordered batch on a distinct four-domain executor. All variants kept exact splice, VZ return, and guest counters. - **Result: direct loans are a P1 win but a P2 loss.** In the adjacent control window, loans reached 62.43 at P1 and 62.19 at P2; copying reached 52.10 and 72.35 Gbit/s. The P2 loan leg made about 43,948 writes with 21,179 partials, versus 24,499 writes and 1,378 partials for copying. Eio's `IOV_MAX` is 1024, so this is socket backpressure, not userspace iovec truncation. - **Result: adaptive ownership preserves the useful sides of both paths.** An unprofiled same-boot sweep reached 57.47 at P1 and 73.35 Gbit/s at P2. P1 kept 22,161 of 22,245 batches direct; P2 entered recovery mode and copied 23,902 of its next 24,166 batches. A target-attached P2 profile still reached 67.84 under profiling: `writev` was 49.3% of 10,137 weighted native samples, `_platform_memmove` 6.1%, and `poll` 6.7%. Its 24,295 logical batches became 32,434 writes with 8,139 partials. The ownership fix removes descriptor pinning, but user copy and kernel socket copy remain sequential on the shard. - **Result: the writer executor is rejected.** P1 fell to 50.14 and P2 to 63.17 Gbit/s. In P1 alone, delayed credits exhausted the per-pair loan budget 10,992 times; only 8,758 of 15,984 batches stayed direct. P2 added 18,018 budget fallbacks. Loans, OCaml releases, and provider-queue completions still matched exactly and guest errors stayed zero, so this is not an ownership failure: the extra scheduling boundary delays completion enough to starve the guest/VZ pipeline. The writer-pool API and wiring were removed instead of being left as a dormant option. The immediate reverted control recovered to 62.57 P1 / 78.92 P2 in the same noisy host window. - **Result: pre-copy-before-schedule also loses.** A second implementation copied each payload into a lazily allocated 16-slot OCaml pool, released its VZ owner synchronously, and only then scheduled the ordered write. Its first run exposed an independent `nox-rope` bug: external leaves passed through `Cstruct.to_bytes` before `blit_to_cstruct`, adding a payload-sized allocation and an extra copy. Native chunk-to-chunk blits plus a regression test removed that bug and improved P1 from 15.58 to 37.24 Gbit/s. Replacing unbounded pool misses (18,772 fresh P1 buffers) with the existing direct-loan fallback made memory genuinely bounded and reached 58.17 P1, but P2 still collapsed to 53.02 versus the 78.92 control. Exact releases/completions and zero guest errors held. The pool/executor API was removed; the independently useful `Rope.blit_to_{bytes,cstruct}` fix remains. - **Iterate.** Keep adaptive partial-copy ownership. Do not move completion behind an executor or yield between batches. The next TX change must overlap work without delaying the current batch's credit: either make socket writes cheaper/increase their accepted extent, or pipeline independently owned flows while each owning shard performs its own write synchronously. Reprofile the current adaptive path in a quiet window before changing its 16-batch recovery length. ### E28. The alternate raw-pointer RX selector wins - **Assumption.** `writeData:length:error:` and `writeBuffer:exactLength:error:` expose the same `B40@0:8^v16Q24^@32` raw-pointer ABI, but may take different framework implementations. Changing only that selector could reduce the mandatory host-to-guest placement cost without adding a copy or another framework call. - **Experiment.** Build one binary with a construction-time selector switch and run fresh-boot P16/Q4 legs in A/B/A/B/A order, where A is `writeBuffer:exactLength:error:` and B is `writeData:length:error:`. Every arm writes the virtio header and payload from the same buffers with identical calls and arguments. Six-second legs ran under severe, changing host load: the first three were 30.38 / 40.24 / 26.39 Gbit/s and the second bracket was 26.39 / 32.19 / 25.11. Judge each B against the mean of its adjacent controls, not against a historical absolute. - **Result.** B wins both brackets: +42% and +25%. All five legs negotiated the 524280-byte BIG TCP limits and recorded zero guest drops/errors, VZ refusals, and VZ write failures. The first B and its trailing control had the same 327411-byte maximum RX frame; B delivered 40.24 versus 26.39 Gbit/s while sampled payload placement fell from 4.82 to 3.64 microseconds per frame. The second bracket also wins despite a downward host trend. The collapsed production path then completed a 31.60 Gbit/s smoke leg under two busy processes: 124280/124280 frames written, zero VZ/guest errors, and 524280-byte GSO/GRO/TSO limits. - **Iterate.** Keep `writeData:length:error:` as the only data-frame method and remove the switch; keep `writeBuffer:exactLength:error:` only for the low-rate control virtqueue. This is a backend-specific selector result, not zero-copy placement. Re-run the current path in a quiet window before replacing the 85.0 Gbit/s record. ### E29. A full-ceiling host read does not make full-sized frames - **Assumption.** The host splice still reads at most 256 KiB while the pure TCP/GSO and device paths accept 524280 bytes. Raising the per-flow read buffer to the full ceiling might halve input callbacks and produce fewer, larger guest-bound GSO frames. - **Experiment.** Select the splice read size once at process startup and run fresh-boot P16/Q4 A/B/A legs with 262144 / 524280 / 262144-byte buffers. The production `writeData:length:error:` RX path is common to all arms. The six-second results were 30.01 / 28.20 / 37.22 Gbit/s under one to three unrelated busy processes. - **Result.** Reject the larger buffer: B is 16% below the mean of its adjacent controls. It did not increase actual aggregation: B averaged 137879 bytes per frame with 46.0% GSO frames, versus 190355/74.1% and 132185/52.7% in the controls. All arms had zero VZ refusals/write failures and zero guest drops/errors. The likely mechanism is socket readiness: `single_read` returns the bytes currently available rather than waiting to fill its capacity, so a larger destination cannot force two arrivals into one core input. - **Iterate.** Keep the 256 KiB read size and remove the switch. Reducing framework calls now requires code that can drain several ready inputs in one batch, not a larger buffer for one readiness event. ### E30. RX distributions expose the frame and placement tails - **Assumption.** Cumulative byte and time totals hide whether receive is limited by many small frames or a long tail in device placement. - **Experiment.** Add per-domain HDR recorders to the RX lanes. Record every successfully written frame size and one out of every 64 device-emission durations. The control API snapshots the recorders and publishes count, minimum, p50, p90, p99, p99.9, maximum, mean, and standard deviation. A P16/Q4 six-second leg ran while three unrelated processes were busy. - **Result.** The leg reached 40.64 Gbit/s with zero VZ refusals/write failures and zero guest drops/errors. The frame histogram counted all 237857 device frames. Frame size was p50 131007, p90/p99 262015, p99.9 327423, raw maximum 327168, mean 128313, and standard deviation 95147 bytes. The 3717 sampled emissions measured p50 11375, p90 22671, p99 65599, p99.9 286975, maximum 392704, mean 14239, and standard deviation 20090 ns. Most frames therefore remain at or below 256 KiB, while p99 placement time is 5.8 times p50. - **Iterate.** Keep these distributions in `/transport/stats`. Use them as integrity and tail-latency gates for the next RX batching experiment; do not infer a new quiet-host throughput record from this loaded run. ### E31. Ready host reads can fill the GSO ceiling - **Assumption.** E29 showed that a larger read buffer does not make one socket readiness event return more data. Draining ready events first and combining reads of the same flow should let the TCP core build 524280-byte GSO frames without increasing the 256 KiB splice buffer. - **Experiment.** Each core drains at most 64 queued inputs. Within a run of TCP receive inputs it preserves per-flow order and combines at most two reads per flow; guest frames, EOF, and control inputs end the run. Three P16/Q4 one-chunk/two-chunk pairs measured 71.95/61.39, 68.65/77.99, and 61.36/77.20 Gbit/s under changing host load. The last pair used eight-second legs. A profiled two-chunk leg captured 480 batch events despite 48 million dropped probe events. - **Result.** The median moved from 68.65 to 77.20 Gbit/s (+12.5%); the mean moved from 67.32 to 72.19 (+7.2%). One pair regressed, so this is a promising loaded-host result, not a new absolute record. Of the captured batches, 208 (43.3%) combined two reads. Batch bytes were p50 263 KiB, p99 525 KiB, and mean 359 KiB. The device frame histogram moved from p90/p99 262015 bytes in the last one-chunk leg to 524031 in the paired two-chunk leg. Every leg had zero VZ refusals/write failures and zero guest drops/errors. - **Follow-up.** A later eight-second A/B/A/B/A bracket ran with four or five background processes above 60% CPU. One-read/two-read throughput was 30.46/28.26/34.44/33.96/33.84 Gbit/s. The two B legs were 12.9% and 0.5% below the means of their adjacent controls, so neither passed the promotion gate. Batching still moved frame p90/p99 from 262015 to 524031 bytes; all five legs negotiated the full offload limits and had zero VZ failures or guest drops/errors. This loaded run contradicts the earlier median win and cannot close the quiet-host decision. - **Iterate.** Keep two-read batching opt-in until a quiet bracket confirms the win. The abstract `Tcp_config.gso` constructor owns the executor pool, shard count, emitters, GSO size, and batch count; receive batching is a GSO property. The VZ CLI rejects incompatible flags before starting Eio or the VM. A paired native sample can determine whether the larger-frame arm spends the saved placement calls in the core or whether host contention explains the contradictory brackets. ### E32. Independent VZ providers can form one logical receive port - **Assumption.** E4 varied queue pairs inside one custom device, so all pairs still shared its serial VZ provider queue. Four single-queue devices should expose four independent provider queues while Slirp keeps one published port, one IP address, and one iperf3 server. - **Experiment.** Add a typed virtio topology with either one multi-queue device or several single-queue devices. The multi-device layout gives every device the same logical-port MAC, maps Slirp shards to devices, and attaches every device to the VM. The Linux guest agent discovers all `eth*` links and applies the primary link's MTU and BIG TCP limits to each. P16 RX compared one device/four pairs with four devices/one pair, then checked eight devices/eight shards. iperf3 was unchanged and still connected to the one host forward. - **Result.** The loaded pair measured 70.68 versus 73.63 Gbit/s. In the next pair, where the harness reported no process above 60% CPU, one device with four pairs measured 73.25 and four single-queue devices measured 76.65 Gbit/s (+4.6%). The four devices received 16.15, 20.27, 20.18, and 20.13 GB. All had zero VZ refusals/write failures and the guest had zero drops/errors. Eight devices regressed to 54.88 Gbit/s despite the same clean integrity counters. A shared MAC is part of the logical-port contract: distinct MACs made the one neighbour entry move between devices and prevented all P16 flows from connecting. - **Iterate.** Keep four-by-one as an opt-in RX topology. It removes the single-device provider funnel but the measured gain is about 3.4 Gbit/s, not the missing 23--27 Gbit/s. Eight devices repeat E4's core-oversubscription shape, so adding provider queues cannot by itself reach 100 Gbit/s. The remaining RX work stays with E31's frame economy and per-frame placement cost. ### E33. Preserve host-read buffers through VZ placement - **Assumption.** The splice already reserves header space beside each host read, but a GSO frame often spans several reads. Flattening that rope into a 256 KiB staging buffer adds a full payload copy before VZ performs its mandatory guest-memory copy. - **Experiment.** Keep the short Ethernet/IP/TCP prefix contiguous and expose the remaining headroom-backed rope leaves as validated byte slices. The VZ adapter writes the prefix and slices directly into the guest receive element and copies them into owned storage only if the bounded pending queue is used. Add separate headroom, split, and staged emission histograms. - **Result.** A loaded P16/Q4 leg reached 73.14 Gbit/s with exact host/guest byte accounting, zero guest errors, zero VZ refusals or write failures, and no pending frames. Of 152472 frames, 127467 (83.6%) used the split path, 12466 used one headroom buffer, and 12539 were staged. Split placement was 9.8 us at p50 and 22.0 us mean, versus roughly 16.2 us and 28.1 us for the earlier staged path in the same loaded regime. - **Iterate.** Keep the slice path. It removes an OCaml payload copy but cannot remove VZ's guest-memory copy. Re-run P16 on a quiet host before replacing the 85.0 Gbit/s absolute. ### E34. Direct socket reads expose a registration race and smaller reads - **Assumption.** Reading a host socket directly into a headroom-backed OCaml buffer removes the remaining Cstruct-to-bytes copy at the splice boundary. - **Experiment.** An opt-in Unix-fd reader first exposed an ordering bug: it could publish `Tcp_recv` before the accepted flow's `Tcp_incoming`. Registering the flow before starting its splice fixed that race. The first valid P16/Q4 leg delivered 38.43 Gbit/s under two host-load warnings. After ordinary frames gained the same checksum-valid slice placement as GSO, a clean leg reached 71.44 Gbit/s: 824481 reads carried 53.66 GB, but frame p50/p99 stayed 65535 bytes and 809443 frames reached the device. - **Result.** The bypass traded one payload copy for a much higher call rate. A follow-up drained immediately ready bytes into the same 256 KiB buffer: 157997 reads carried 37.73 GB (239 KiB mean), and frame p50/p99 moved to 262015 bytes with zero device or guest errors. Throughput nevertheless fell to 50.24 Gbit/s versus 66.85 for the adjacent Eio control in the same noisy window. The extra nonblocking reads cost more than the larger frames saved. - **Iterate.** Remove the Unix-fd path, its flag, and its counters. Host socket reads remain `Eio.Flow.single_read`. The zero-copy follow-up is an Eio-owned Cstruct ring whose external rope leaves stay valid through acknowledgement and reach VZ as slices; E31 remains the independent frame-economy knob. ### E35. "Keep Eio and carry its Cstruct to VZ" - **Assumption.** The E34 copy can be removed without escaping Eio: read into a recyclable Cstruct slot, retain it as an external rope leaf until the guest acknowledges it, and let VZ copy that view directly into guest memory. - **Experiment.** Change `Payload_ring` slots from `bytes` to `Cstruct.t` and pass each slot to `Eio.Flow.single_read`. Add a native VZ external-slice packet boundary plus a distinct external-path counter and emission histogram. The benchmark image now pins `init=/init`; without it the rebuilt kernel reached DHCP but never launched the embedded benchmark init. On the loaded host, three P16/Q4 four-second controls delivered 75.08, 73.73, and 73.69 Gbit/s; three external-ring legs delivered 76.34, 74.36, and 75.96 Gbit/s. - **Result.** Median RX rose 73.73 -> 75.96 Gbit/s (3.0%); the mean rose 74.17 -> 75.55 Gbit/s (1.9%). Frame p50/p99 stayed 262015 bytes and every guest/VZ integrity counter stayed zero. The downstream VZ copy remained the cost: sampled payload placement rose 6.33 -> 6.99 us and mean emission rose 18.9 -> 23.4 us, but removing the preceding staging work still won overall. A killed first B leg was not an external-memory failure: a later ordinary A leg was also killed by signal 9 while other sessions shared the machine. - **Iterate.** Keep the Cstruct ring opt-in. Run a quiet A/B/A with the new `vz.rx.external_*` distribution, and make it the default only if both B brackets beat their adjacent controls. ### E36. The guest TSQ allowance becomes a knob; its first read-back corrects E26 - **Assumption.** E23 and E26 name raising `tcp_limit_output_bytes` as the precondition for any intra-flow TX pipelining, and E26's arithmetic assumed the kernel default is 1 MiB, two BIG TCP frames of in-flight allowance. - **Experiment.** The guest agent now applies the sysctl at boot (`VZ_AGENT_TCP_LIMIT_OUTPUT`) or on request (`POST /tcp-limit-output-bytes`, so one boot can A/B values), verifies the write by read-back, and reports the live value in every `/stats` snapshot; the bench sets it per boot via `TCP_LIMIT_OUTPUT_BYTES` and fails the leg if it does not apply. Fresh-boot P1 pairs, control against raised, in two busy windows: a dev-profile window (mediaanalysisd ~200%) and a release-profile window that also carried a second, unrelated VM at ~99% and LTO links. - **Result.** The first live read-back corrected the record: the pinned 6.18.39 guest defaults to 4194304 bytes, EIGHT 524280-byte frames, not the 1 MiB / two frames E26's explanation used. E26's K>=2 cliff was real, but its allowance arithmetic was 4x off and its mechanism deserves re-derivation. The throughput probes disagree across windows: the dev window read 66.59 Gbit/s at 8 MiB against 57.76/53.41 adjacent controls (+20%), while the release window read 31.72/34.65 at 16 MiB against 35.76/33.43 controls (neutral, second VM present). Both boots verified the applied value (8388608 and 16777216) in `/stats`. - **Iterate.** Quiet P1 bracket control/raised/control before any verdict; the E20 two-domain split experiment now has its enabling knob either way. ### E37. The first-TX-leg watchdog reproduces with exact loan accounting - **Assumption.** E23 left tail-completion correctness open and asked for a failing leg: one where releases and provider-queue completions could be compared across a guest-visible stall. - **Experiment.** The E36 dev-profile TX boots supplied it, unasked: two of three boots raised `NETDEV WATCHDOG: transmit queue 1 timed out` (~5.0 s timeout reported at ~14.8 s uptime, i.e. during the boot's first leg), incremented guest `tx_errors` to 1, and failed that iperf leg; the next leg on the same boot completed normally (66.59 and 51.71 Gbit/s). - **Result.** The loan ledger is exact in the failing boots at every snapshot: loans = OCaml releases = provider-queue completions (106,719 at the failing leg's end), zero outstanding, zero malformed. `vz.stalled` reads 1 even in clean boots (a boot-time event), so it does not discriminate. The five release-profile TX legs in the busier window never watchdogged. The race is therefore below the OCaml counters -- either the scheduled `returnToQueue` block does not always produce the guest's used-ring completion/interrupt, or the completion counter increments before the framework actually delivers -- and it is timing-dependent (slower dev builds widen it; 2 of 3 dev boots against 0 of 3 release boots). Artifacts: `/tmp/vz-tsq-2026-07-22/`. The release-build immunity did not hold: E39's brackets reproduced it on release binaries in 2 of 5 TX boots, on both P1 and P2 and with both default and raised TSQ, so this is a production-path bug, not a dev-profile artifact. E40's retroactive per-pair analysis then resolved the either/or above: neither -- the stranded frame was never consumed at all; the resumption kick on a quiesced pair is lost before the loan path begins. - **Iterate.** Dev-profile TX boots are the cheap reproducer. The next discriminator must count on the other side of the boundary: guest `/proc/interrupts` deltas (or a framework-side notification counter) across a failing first leg, compared with `tx_loan_returns_completed`. ### E38. The composed RX opt-ins do not compose (falsified in two windows) - **Assumption.** E31 (+7-12%), E32 (+4.6%), and E35 (+2-3%) each beat their own control, so enabling all three should multiply from the 85.0 quiet base toward the target. - **Experiment.** Fresh-boot interleaved A/B/A/B/A at P16/four shards, six seconds, worktree at the E36 commit: A the shipped default (one device, four pairs, one-read, no ring), B all three opt-ins (`VIRTIO_DEVICES=4 QPAIRS=1 TCP_RX_BATCH_CHUNKS=2 RECYCLE_PAYLOAD=1`). Run once on dev-profile binaries and once on release, then each opt-in alone and each pair, every arm bracketed by controls, all in the release window. Both windows carried heavy load (the second included an unrelated VM), so every claim is relative to adjacent controls. - **Result.** The trio lost in BOTH windows: dev 49.57/40.80 against 56.98-59.77 controls (-16% and -30% versus adjacent-control means); release 30.32/27.58 against 34.75-39.24 controls (-18% and -26%). Yet every single opt-in was neutral-to-positive in the same release window (devices 38.21, +8.8% versus its adjacent-control mean; two-read 37.08, +5.1%; ring 37.45, +4.2%). The pairs localize the interaction: both pairs containing the ring lost (devices+ring 31.85, -8.4%; two-read+ring 33.35, -5.3%) while devices+two-read was within noise (34.05, -4.2% against a window whose controls spread 13%). The ring's win does not survive composition -- consistent with E7's history: recycled-slot lifetimes stretch under two-read ropes and four concurrent placement paths, and the recycled working set turns cold exactly when more consumers hold it. - **Iterate.** The quiet-window brackets in section 6 are rescoped: promote candidates are devices-only (the best single here and in E32) and devices+two-read; the Cstruct ring is only ever a LONE opt-in and its E35 quiet A/B stands alone. Do not carry a "multiply the opt-ins" estimate forward; composition must be measured per pair. ### E39. The promotion brackets reject every opt-in; the defaults stand - **Assumption.** With `mediaanalysisd` suspended (stuck near 200% since the macOS 27 upgrade; SIP forbids `launchctl bootout`, but a SIGSTOP holds because launchd keeps the stopped pid as the live instance), the E38 survivors could be decided: devices-only, devices+two-read, the ring alone, the IPERF_BLOCK client-write arm, the P2 protocol legs, and the E36 TSQ bracket. - **Experiment.** Twenty-one fresh boots, release binaries, worktree at the E38 commit, one window. Not a true quiet window -- one unrelated VM held ~100% of a core throughout and root-owned `mds_stores` flapped to 200% -- but the tightest control spread of the campaign: eight interleaved P16 controls read 41.19-47.82 Gbit/s. Every arm is judged against the mean of its adjacent controls; the gate is both legs above. - **Result.** No candidate passes. | Arm | Legs (Gbit/s) | vs adjacent controls | Verdict | |---|---|---|---| | devices-only (E32) | 42.98 / 41.33 | +0.0% / -5.9% | fail | | devices + two-read | 43.94 / 47.33 | -1.8% / +0.5% | fail (neutral) | | ring alone (E35) | 41.65 / 37.44 | -8.5% / -12.4% | fail decisively | | IPERF_BLOCK=1M | 43.35 | +2.4% | one leg, noise | | TSQ 16 MiB, P1 (E36) | 58.73 / X | +2% vs 58.05/57.00 | neutral | The four-device gain is not robust across windows (+4.6% in E32, +8.8% in E38's release window, -5.9%/flat here). The ring has now lost in two consecutive fresh windows even alone; E35's +3.0% does not reproduce. The TSQ raise is neutral in its second consecutive bracket, so the dev-window +20% does not survive: the P1 chain is CPU-bound as E20 said, and the knob's value is as the E20 enabler, not a standalone lever. The P2 protocol legs read 66.50/67.27/X/71.34/77.11 across two boots, rising as host load fell. Every RX leg was byte-clean (zero guest drops/errors, zero VZ write failures) and every loan ledger exact. Two release TX boots raised the E37 watchdog (P2/default-TSQ third leg, queue 0 at 29.7 s; P1/raised-TSQ second leg, queue 1 at 22.0 s); the failing P2 boot had accumulated 11,442 loan-budget fallbacks during its two CLEAN legs and its ledger stayed exact through the failure. Artifacts: `/tmp/vz-tsq-2026-07-22/quiet-legs/`. - **Iterate.** The configuration search is CLOSED: the shipped defaults -- four shards, four pairs, one device, full 524280-byte GSO, P16 receive, P2 transmit, loan budget 16, adaptive partial-copy, 1 MiB write target -- are the operating point, and no measured knob or combination beats them robustly across windows. The remaining route to 100 is structural: fix the E37 completion race (now the top correctness item), test the E20 two-domain split behind the TSQ knob, and re-baseline absolutes on a host with no second VM and settled indexing. ### E40. The watchdog is a lost resumption kick on a quiesced pair - **Assumption.** E37 framed the race as completion-side: a returned element whose used-ring notification fails to reach the guest. - **Experiment.** No new boots: cross-read the four watchdog boots' per-pair `vz.pair.N.tx_frames` snapshots against the guest's per-queue `/proc/interrupts` tables and watchdog timestamps (dev txT1/txP2, release qTX1/qTT1; artifacts under `/tmp/vz-tsq-2026-07-22/`). - **Result.** All four boots show one signature. The watchdogged queue's pair is QUIESCENT when its leg starts -- either near-zero frames since boot (txT1 pair 1: 3 frames; txP2 pair 1: 5; qTX1 pair 0: 164 across two whole legs) or drained since its previous burst across the inter-leg gap (qTT1 pair 1: busy in leg 1, idle through the gap, watchdogged at leg-2 start). The stall begins exactly at leg start, the guest holds its one in-flight TSQ frame with no completion for 5 s, the watchdog resets the queue, and the SAME pair then flows perfectly for the leg's remainder (5 -> 52,887 frames; 3 -> 85,951; 164 -> 43,855; 84,225 -> 171,441). The loan ledger is exact throughout because the host never consumed the stranded frame: the first avail notification after the pair quiesces is lost in the notification re-arm window, and under TSQ the guest never kicks again. Completion delivery was never the fault. Interrupt coalescing corroborates the busy pairs (2,848 interrupts for 53,777 frames on the healthy queue in the same leg). - **Iterate.** The fix belongs in the vz custom-device drain (ocaml-vz): the classic device-side double-check -- after a drain empties a pair and notifications re-arm, poll the ring once more so a frame that landed in the window is caught; if the framework's arming is opaque, a bounded idle-pair poll (~10 ms) turns a 5 s watchdog into a 10 ms hiccup and a counter (`stranded frames caught`) makes the race observable. Regression shape: many short legs separated by idle gaps (the resumption pattern) at P1, asserting zero guest `tx_errors` across N resumptions. Filed in the ocaml-vz TODO for the session that owns that file today; dev-profile TX boots remain the amplifier. ### E41. Tx_pull closes the lost kick: stranded frames caught by the walk - **Assumption.** E40's lost resumption kick can be absorbed in the lane pull loop by bounding the wake-pipe wait -- expiries walking 2/10/50 ms and holding at 250 ms, each expiry draining again -- so a stranded frame is recovered within the walk instead of by the guest's watchdog. - **Experiment.** `Vz_slirp.Tx_pull` landed with its contract pinned test-first: five spec cases (immediate frames, poke, stranded catch, the walk holding at 250 ms, a poke restarting the walk) failed on a stub and passed on the implementation. The regression is E40's resumption-pattern soak on dev-profile binaries, the documented amplifier: three boots of six 4-second P1 legs, eighteen quiesce/resume cycles. - **Result.** Zero watchdogs and zero guest `tx_errors` across all eighteen legs, against the historical dev-profile rate of two in three boots failing within two legs. The mechanism fired rather than the race disappearing: `vz.tx_stranded_catches` read 3/2/2 per boot -- seven lost kicks recovered by the walk, each of which was previously a five-second stall, a queue reset, and a failed iperf leg. Idle polls ran at the designed steady cost (~16/s across four lanes, 583-631 per boot), and every leg completed at normal dev-profile P1 rates (35-61 Gbit/s under load). - **Iterate.** `vz.tx_stranded_catches` is now the lost-kick rate made visible: read it in every TX leg, and treat a jump after a macOS update as the early warning. The dev-profile soak stands as the regression. ### E42. The E20 overlap lands as a systhread write offload and wins P1 - **Assumption.** E20's split does not need a second domain: offloading the socket write to a systhread whose stub releases the runtime lock lets the owning shard run its drain and TCP core while the kernel copies the batch -- the same overlap with none of E27's scheduling latency, and a discriminator for whether a full writer-domain split could add more. - **Experiment.** `Eio_net_unix.Stream.offload_single_write` (vectored writev, `Eio.Flow.single_write` semantics) behind `Splice.v ?write_offload`, `--tcp-write-offload`, and `TCP_WRITE_OFFLOAD=1`; vectored contract test pinned. Interleaved P1 and P2 brackets in the campaign's cleanest window (loads 3.7-5.3, no process above 60%). - **Result.** P1 offload beat every adjacent control in two independent brackets: 71.55/74.55 against 62.06-70.81 controls (+7.1%/+11.6% versus adjacent-control means), then after the teardown fix 65.84/67.71 against 62.49-64.30 (+3.0%/+6.8%) -- the promotion gate passed both times. P2 is neutral (61.70 against 61.09-61.40), as expected: two flows already overlap each other across shards. The first bracket also exposed a real bug: the raw stub surfaced teardown EPIPE as `Unix.Unix_error` where the inline path reports `Eio.Io`, killing the gateway when a peer closed with a batch in flight; the writer's reset handling now covers both shapes, and a post-fix boot ran both legs with 61,390 offload writes and zero uncaught exceptions. The same window's defaults re-baseline read 41.95/42.87/41.98 at P16 receive and 61.09-67.73 at P2 transmit. - **Iterate.** Keep `--tcp-write-offload` opt-in until one quiet P2 leg confirms the neutrality, then flip the default: it wins the single-flow regime and costs nothing at the operating point. A full writer-domain split is now bounded: it can only improve on the offload by overlapping the OCaml-side batch assembly too, and E27 prices its scheduling risk. ### Summary figure ``` 0 10 20 30 40 50 60 70 80 90 100 Gbit/s |----|----|----|----|----|----|----|----|----|----| RECEIVE file-handle ########### 21.7 E1+E2 custom virtio ####### 13.2 E3 lanes+sockets ################################# 65.9 (b) E4 qpairs=shards ##################################### 74.1 (b) E5 shard ingest ####################################### 78.5 (b) quiet verified ########################################### 85.0 E12 128K GSO ###################################### 76.3 (busy) E13 full GSO current ##################################### 74.2 (busy) E31 two-read batch ####################################### 77.2 (busy) E32 four providers ###################################### 76.7 (no load warning) E33 sliced payloads ##################################### 73.1 (busy) E34 Unix read (removed) ######################### 50.2 (busy) E35 Eio Cstruct ring ###################################### 76.0 (busy) goal .................................................. 100 TRANSMIT file-handle ################ 32.8 E1 custom virtio ################### 38.3 checksum offload ########################## 51.4 one-copy delivery ################################# 66.6 (b) E5 shard ingest (P4) ################################### 70.7 (b) quiet verified (P4) ################################## 68.9 E13 full GSO P4 ################################ 62-65 (noisy) current adaptive P2 ######################################### 80.4-83.6 (busy) goal .................................................. 100 ``` E15 is the first byte-exact TX aggregate above 80. E18 records three P2 legs on one boot at 79.11 / 81.36 / 80.36; Appendix B records five more clean legs across two boots on the current adaptive path. P2 is therefore repeatable; a quiet-window absolute is still pending and the quiet 68.9 record remains the formal record. The older noisy configuration only peaked at 77.7 for one second (74.2 aggregate). ## 4. Negative results Each is measured and recorded where it would be retried. Temporary implementations and probes are removed unless the table names a preserved gate or adaptive form. Publishing them is the cheapest profiling tool this project owns. | Experiment | Measured effect | Where recorded | |---|---|---| | MRG_RXBUF + large GSO, either kernel (E2/E12) | guest drops, TCP collapse | E12 table; `--virtio-gso-max-size` doc | | GC `space_overhead=400` (E6) | TX 63.1 -> 48.2 (35.8 at P4) | `--gc-space-overhead` doc | | Minor arena `s=4M` (E6) | RX 84.8 -> 64.9 | E6 | | Per-input lane emission (E9) | TX 63.1 -> 42.4 (16.1 at P4) | comment in `start_shard` | | Stretch acknowledgements (E10) | neutral | E10; RFC 5681 pin test | | SO_RCVBUF/SO_SNDBUF 4 MB (E11) | neutral | E11 | | Splice write target 2/4 MiB (E16) | fewer writes but 4--6% slower than 1 MiB | E16; `--tcp-write-batch-bytes` doc | | Unconditional direct loans at P2 (E27) | 62.19 versus 72.35 copying; partial writes 21,179 versus 1,378 | E27; adaptive partial-copy kept | | Arrival-driven loan release lag K >= 2 (E26) | permanent flow wedge | E26; probe removed, completion-driven release test-pinned | | Separate writer executor / bounded pre-copy offload (E27) | 50.14/63.17 direct; 58.17/53.02 bounded pre-copy, versus 62.57/78.92 control | E27; implementations removed | | Direct receive placement through a write-side buffer view (E25) | API does not exist | E25 runtime dump | | Full 524280-byte host read buffer (E29) | 28.20 versus adjacent 30.01/37.22 controls; no larger actual frames | E29; 256 KiB kept | | Payload ring end to end (E7) | RX -22% despite the GC win | gate comment in `slirp_eio` | | Scatter-gather device send | worse: per-call framework cost exceeds one memcpy | stub comment | | dispatch_apply parallel TX copy | worse: thread-wakeup churn | stub comment | | Pool-slot copy on the serial thread | 6.02 Gbit/s; bounded queue dropped 1.3% | pre-E7 record | | vsock frame transport | 3.2 / 2.8 Gbit/s: the in-guest tap hop | pre-E1 benchmark record | | HOST_TSO toward a terminating gateway | link death at MTU 1500; slower at jumbo | device feature doc | | Eight shards/pairs/streams with full GSO | 62.25 -> 41.66; core oversubscription, zero lock pressure | E13 counters | | Eight single-queue devices (E32) | RX 54.88 versus 76.65 at four devices; core oversubscription | E32 counters | | Unix host-socket reads (E34, removed) | RX 71.44 with 64 KiB frames; eager drain 50.24 versus adjacent Eio 66.85 | E34 counters | ## 5. Threats to validity - **Host noise dominates absolute numbers.** One unchanged configuration measured 62.5 and 85.1 in different windows of the same night. `mediaanalysisd` (up to 219% CPU), `corespotlightd`, `system_installd`, and unrelated agents' builds appear without warning. Every comparative claim above is same-window or interleaved; every absolute is labelled. - **Profilers perturb this workload.** macOS `sample(1)` at its 1 ms default cut an aligned transmit run from 52.2 to 23.6 Gbit/s and saturated the callback queue; `obs` full-span capture warns above 25k events/s. In E13 the six-second native sample held the first five TX seconds to 38--43 Gbit/s; after it detached, the same run rose to 55.7--57.8. Counters carry the numbers; profilers carry stack identity, read as deltas between configurations. - **Profiled-versus-profiled can invert.** Under the observer the payload ring measured *faster* (75.6 vs 67.7) while unprofiled runs showed the opposite by 22%. No throughput decision is made from profiled runs. - **Attribution had a blind spot** until E8; allocation conclusions drawn before 2026-07-21 saw domain 0 only. - **Counter inference has limits.** The buffer-return stack reaches Mach IPC, but used-ring completion and guest interrupt coalescing are distinct: do not infer one guest interrupt per API call, and treat the buffer-economy counters as observations to correlate with guest deltas and kernel configuration, never as verdicts on their own. ## 6. Open questions and next measurements Answered questions stay in their experiment entries and the current-findings summary; this list contains only work that can still change a decision. 1. Establish the quiet P2 transmit number for the current adaptive-loan path with full 524280-byte GSO/TSO and every integrity gate. The busy-host reproduction establishes repeatability but does not replace the 68.9 Gbit/s quiet record. A true quiet window needs: no other VM on the host, `mediaanalysisd` SIGSTOPped (SIP blocks bootout; the stopped pid holds the launchd slot), and `mds_stores` settled. E39's best legs (77.11 at P2, 47.8 at P16 with a competing VM present) bound what such a window should exceed. The E31/E32/E35 opt-in brackets are DECIDED (E39): none promotes; re-open them only after a structural RX change. 2. Re-sweep P1/P2/P4 and loan budgets 4--64 under the current adaptive partial-copy policy. E27 established the direct-loan P2 regression and the adaptive recovery, but did not test every stream-count/budget combination or price its 16-batch recovery length. 3. E7's residual: an emission-side A/B differing only in buffer provenance (recycled versus fresh, identical content) to catch the interaction the isolation bench does not model. 4. The E19 mechanism: what in the macOS loopback path makes independent flows collapse super-linearly (97 to 64 to 45), and why our splice's large writes partially escape it. Not directly actionable -- the sink is the benchmark's fixed endpoint -- but it prices every multi-stream number this study quotes, and a macOS point release could move it. 5. Head-of-line protection: stall one of two flows deliberately and assert that its peer keeps its rate while budget fallbacks count up. The per-pair budget exists for this case, but the end-to-end stall test is not yet in the log. 6. The lifetime detector: run checksum verification over external leaves for millions of loaned frames, then soak while continuously asserting loans = releases = provider returns and flat RSS. This is the remaining proof that no path reads recycled guest memory. 7. The serial queue's second act: measure its share with loans on and off. E23 correctly moved batched returns onto the device provider queue; the assumption that the return block is too cheap to matter still needs a number. 8. Re-baseline the current adaptive path over the macOS 26 file-handle transport. E21 predates loans and adaptive partial-copy ownership. ## Appendix A: the instrument ladder Work down this list; each step is cheaper than the one after it and rules out more. Skipping ahead to a fancy tool before the cheap ones have spoken is how sessions get lost optimising the wrong layer. **A.1 `iperf3 -J` as harness.** Alternate directions on one boot (a number that only reproduces on a fresh boot is a state leak: the device-queue bufferbloat showed as tx 49.6 fresh but 8.4 after an rx phase -- 67 MB of stale frames). Sweep `-P` before comparing anything. Read `Retr` first: zero retransmits with bad throughput is a stall; nonzero is loss -- disjoint causes. **A.2 Guest counters (free, and nobody else looks).** The bench guest prints `/proc/interrupts` per vCPU (settled the MSI-versus-INTx question: per-queue `MSI Edge` vectors appear only under EFI boot), `/sys/class/net/*/statistics` (`rx_dropped=0` over millions of frames is what *proves* an offload safe), the negotiated feature words (the only ground truth for what the driver accepted), and softnet/ARP/MTU (found a tap at 1500 silently eating jumbo frames). Read host and guest deltas together: a run whose host counters are all clean but whose guest `rx_dropped` moved by 160 is a localisation result (E12), not a dead end. **A.3 macOS `sample(1)`.** Samples all threads including Objective-C dispatch. Three finds: the buffer-return stack reaching `dispatch_mach_send` (host cost scales with buffers, not just frames -- with the counter-inference caveat in section 5), `crc_internet_checksum_bytes` at 17% (a checksum computed for a guest that never reads it -- gone via offload), and the wait-versus-work top-of-stack ratio (10,500 wait vs 2,000 work = coordination-bound, which is why adding shards made it slower). Read deltas, not absolutes: idle workers inflate wait buckets, and the sampler's suspensions perturb. **A.4 `obs run / report / hotspots / query`.** Budget verdict first (waiting vs GC vs CPU), then drill only the named regime. Multi-domain allocation attribution works as of 2026-07-21 (E8). For VM launchers, align the native-sample window with the workload (`--sample-trigger`; the bench does this under `OBS_OUT`). Native sampling is on by default for six seconds; no extra `--sample 6` is required. Use `--sqlite` plus ad-hoc SQL over the slice table for named waits. Full span capture is not free (warns above 25k events/s); confirm throughput without it. The always-on probes publish the callback-queue counters and `vz.virtio` every 500 ms; read them together -- a shallow callback queue with rising native pending depth is guest-ring backpressure, the reverse is OCaml-side backpressure. **A.5 Purpose-built C counters where the layer is opaque.** Virtualization.framework's data path is invisible to every profiler. `VZ_VIRTIO_TRACE=1` prints bring-up, notifications, spans, and stalls; the `vz.virtio` probe publishes cumulative queued/written/refused/stalled, per-pair depths, 1-in-64 phase timings, and the receive buffer economy (`rx_buffers`, `rx_multi_frames`, `rx_max_buffers`, `rx_write_failures`) that decided E2 and E12. The active shard-ingest pull also publishes header/payload read, allocation, descriptor, return, pair-lock, drain/empty-drain, and maximum-batch counters; callback-queue counters remain visible even when that path is bypassed -- measured at the native boundary because only VZ exposes posted element capacities. The guest agent has the same switch (`SLIRP_VSOCK_TRACE`); its ethertype trace found an IPv6-only guest in one look. When a black box sits in the pipeline, spend the twenty lines. **A.6 Runtime API dumps instead of documentation.** With headers lagging the OS, `objc_copyClassList` / `class_copyMethodList` / `method_getTypeEncoding` against the live framework found the real setter signatures, the serial-queue contract, and -- by absence -- that no batched `returnToQueue` exists. A negative from an exhaustive dump is evidence; a negative from missing docs is not. **A.7 Crash forensics.** `~/Library/Logs/DiagnosticReports/*.ips` parsed for the faulting thread identified both the concurrent-dispatch-queue trap (`EXC_BREAKPOINT` in `handle_notification`) and the `Domain.DLS`-in-memprof segfault. First stop for any exit 133 or 139. **Microbenchmarks bound; they do not predict.** `bench/bench.ml` (pure core), `bench/edge_bench.ml` (socketpair edge), and `bench/splice_bench.exe` (splice payload path) each bound one stage. edge_bench's 34.5 was once misread as a gateway ceiling; the same gateway later did 85 through a better transport. Quote them only with the harness named. ## Appendix B: reproduction ```sh # both directions at their defaults: P16 receive, then P2 transmit SECS=6 QPAIRS=4 ocaml-slirp/bench/vz_bench.sh 4 # isolate either direction while keeping its operating point explicit SECS=6 STREAMS=16 QPAIRS=4 ocaml-slirp/bench/vz_bench.sh 4 rx SECS=6 STREAMS=2 QPAIRS=4 VIRTIO_TX_LOAN_BUDGET=16 \ ocaml-slirp/bench/vz_bench.sh 4 tx # five P2 legs across two boots, retaining both integrity-counter logs SECS=6 STREAMS="2 2 2" QPAIRS=4 VIRTIO_TX_LOAN_BUDGET=16 \ COUNTERS_LOG=/tmp/p2-boot1.counters ocaml-slirp/bench/vz_bench.sh 4 tx BUILD_IMAGE=0 SECS=6 STREAMS="2 2" QPAIRS=4 \ VIRTIO_TX_LOAN_BUDGET=16 COUNTERS_LOG=/tmp/p2-boot2.counters \ ocaml-slirp/bench/vz_bench.sh 4 tx # profiled run: obs bundle + sqlite sidecar, sampler aligned to iperf OBS_OUT=/tmp/prof STREAMS=16 QPAIRS=4 ocaml-slirp/bench/vz_bench.sh 4 rx obs report /tmp/prof.sh4 obs hotspots /tmp/prof.sh4.memtrace/0.ctf # the splice payload path in isolation (no VM) dune exec ocaml-tcp/bench/splice_bench.exe -- --recycle --domains=4 ``` The 2026-07-21 reproduction on macOS 27.0 (26A5388g) measured P2 TX at 81.90 / 83.59 / 81.07 Gbit/s on boot 1 and 80.44 / 82.30 on boot 2 (median 81.90). P16 RX measured 64.07. Every leg had host-load warnings; the RX leg overlapped `mediaanalysisd`, so these are repeatability and integrity results, not quiet absolutes. All legs negotiated 524280-byte GSO/GRO/TSO and had zero guest drops/errors, no VZ refusal or write-failure deltas, and exact loan/release/provider-return equality with zero outstanding or malformed loans. A post-refactor P16/Q4 RX leg reached 31.50 Gbit/s while one unrelated process used more than 60% CPU. The measurement added 114394 VZ RX frames and exactly 114394 frame-histogram records, carrying 23.666 GB with zero VZ refusals/write failures and zero guest drops/errors. The guest again reported 524280-byte GSO/GRO/TSO limits. Frame size was p50/p99 262015 bytes and p99.9 524031; the sampled device-emission duration was p50 20383 ns and p99 490751 ns. This checks the refactored module wiring and counter coverage, not the quiet-host absolute. The matrix maps each experiment to its reproduction path. A knob row means the flag exists and both settings can be tested. A recipe row is an invocation or a measured, reverted probe that survives as this log plus a code-site comment; a fixed row describes the shipped configuration. Variants the spec itself sanctions keep their knob even when slower here: RFC 5681 s.4.2 makes every-second-segment acknowledgement a SHOULD and RFC 3449 s.5.2.1 describes when stretching it is right, so the winning setting depends on the deployment and the flag costs nothing on the main path. Configurations that measured worse and have no spec standing keep only their log entry, with one refinement: when the loss is priced by this platform rather than by the design -- E9's guest-interrupt coalescing, E7's placement cost, both properties of how VZ delivers interrupts and returns buffers -- the verdict is per backend, and a KVM or Xen device edge re-runs the experiment rather than inheriting the VZ answer. E7's flag exists for exactly that reason. | Exp | Reproduce with | Kind | |---|---|---| | E1 | `vz run --network=file-handle` vs `custom-virtio` (bench pins the latter) | knob | | E2 | `VIRTIO_MERGEABLE=1` (default `false`) | knob | | E3 | shipped unconditionally; the pre-lane configuration exists only in history | fixed | | E4 | `QPAIRS`/`SHARDS` (operating point 4/4) | knob | | E5 | `VIRTIO_SHARD_INGEST=0` restores the serial-thread baseline; `STREAMS` for Figure 2 | knob | | E6 | `GC_SPACE_OVERHEAD` (default 120); `OCAMLRUNPARAM=s=4M` for the arena arm | knob | | E7 | `RECYCLE_PAYLOAD=1`; isolation via `splice_bench.exe --recycle`; library: `Slirp_eio.run ?recycle_payload`, `Splice.v ?recycle_payload` | knob | | E8 | any `OBS_OUT` run; multi-domain attribution is now observe's default | fixed | | E9 | `EMIT_PER_INPUT=1` (`--emit-per-input`, `Slirp_eio.run ?emit_per_input`) | knob | | E10 | `TCP_ACK_EVERY=N` (`--tcp-ack-every`, `Slirp.tcp ~ack_every`, `Stack.config.ack_every`, `Connection ?ack_every`; default 2 per RFC 5681 s.4.2) | knob | | E11 | reverted (neutral); 4 MB socket buffers described in the log | recipe | | E12 | `VIRTIO_GSO_MAX_SIZE` (bench default 524280) with `VIRTIO_MERGEABLE=0` and the pinned 6.18.39 guest kernel | knob | | E13 | default full-GSO TX; allocation diff with `obs hotspots --baseline OLD.ctf NEW.ctf`; `QPAIRS=8 STREAMS=8 ... 8 tx` reproduces the oversubscription arm | fixed + recipe | | E14 | `iperf3 -c localhost -t 6 -O 1 -l SIZE -P STREAMS -J`; sizes 128K/512K/1M and streams 1/2/4/8 | recipe | | E15 | `STREAMS="1 2 4" QPAIRS=4 ... 4 tx` with a guest reporting 524280 for GSO/GRO/TSO | recipe | | E16 | `STREAMS=2 TCP_WRITE_BATCH_BYTES=N QPAIRS=4 ... 4 tx`; N=1048576/2097152/4194304 | knob | | E17 | `OBS_OUT=/tmp/e17 STREAMS=2 QPAIRS=4 ... 4 tx`; `obs report /tmp/e17.sh4` plus its target-attached native sidecar | fixed + recipe | | E18 | per-pair counters are always on (`vz.pair.*` in the counters log); `STREAMS="2 2 2 4" QPAIRS=4 ... 4 tx`, plus 1 Hz `ps -M` on the iperf3 client | fixed + recipe | | E19 | N independent `iperf3 -s -p PORT` / `iperf3 -c 127.0.0.1 -p PORT -P 1 -l 1M` pairs on distinct ports, same window, N=1/2/3 | recipe | | E20 | `STREAMS="1 1" QPAIRS=4 ... 4 tx` with 1 Hz `ps -M` on the vz-signed process | recipe | | E21 | `vz --network=file-handle --pool-domains=4` with the bench image and ports; sweep `iperf3 -P` by hand | recipe | | E22 | runtime method dump plus the temporary `VZ_TX_LOAN_PROBE` described in E22; the unsafe hold probe is removed | recipe | | E23 | `VIRTIO_TX_LOAN_BUDGET=16` versus `0`, `STREAMS="1 2" QPAIRS=4 ... 4 tx`; require loan/release/return and guest-error gates | knob | | E24 | back-to-back `STREAMS=1` loan/copy legs as in E23, with 1 Hz `ps -M` on vz-signed | recipe | | E25 | repeat the exhaustive runtime method dump from A.6; the absence of a write-side view is the result | recipe | | E26 | temporary arrival-driven release lag K=0/2/8/32/64 described in E26; probe removed, completion-driven release test-pinned | recipe | | E27 | current `VIRTIO_TX_LOAN_BUDGET=16` adaptive path; read `tcp.splice.{partial_copy_fallbacks,backpressure_copy_batches}` and loan-return gates. Writer-executor and pre-copy variants are removed | fixed + recipe | | E28 | data frames use `writeData:length:error:` unconditionally; to repeat the A/B, temporarily point `vz_element_write_data` at `vz_sel.write_buffer` and bracket P16/Q4 fresh boots, requiring zero VZ/guest errors | fixed + recipe | | E29 | splice host-read size is fixed at 256 KiB; temporarily select 524280 at process startup and bracket P16/Q4 fresh boots to repeat the rejected larger-buffer arm | fixed + recipe | | E30 | `GET /transport/stats`; inspect `vz.rx.frame_bytes.*` and `vz.rx.emit_ns.*` after a P16/Q4 RX leg | fixed | | E31 | `TCP_RX_BATCH_CHUNKS=2 STREAMS=16 QPAIRS=4 ... 4 rx`; compare with `TCP_RX_BATCH_CHUNKS=1`; use `OBS_SAMPLE=0 OBS_OUT=/tmp/e31` and `catapult summary /tmp/e31.sh4.pb` for the batch distribution | knob | | E32 | `VIRTIO_DEVICES=4 QPAIRS=1 STREAMS=16 ... 4 rx`; compare with `VIRTIO_DEVICES=1 QPAIRS=4`; `... 8 rx` with eight devices reproduces the oversubscription arm | knob | | E33 | inspect `vz.rx.{headroom,split,staged}_frames` and matching emission histograms after a P16/Q4 RX leg | fixed | | E34 | removed; host sockets use `Eio.Flow.single_read` | rejected | | E35 | `RECYCLE_PAYLOAD=1 STREAMS="16 16 16" QPAIRS=4 ... 4 rx`; compare with the flag unset and inspect `vz.rx.external_{frames,emit_ns.*}` | knob | | E36 | `TCP_LIMIT_OUTPUT_BYTES=N STREAMS="1 1" QPAIRS=4 ... 4 tx` (agent: `VZ_AGENT_TCP_LIMIT_OUTPUT` at boot, `POST /tcp-limit-output-bytes` at runtime); `/stats.TcpLimitOutputBytes` is the ground truth | knob | | E37 | TX boots on dev-profile binaries (build without `--profile release`) reproduce the first-leg watchdog; require the loan-ledger and guest `tx_errors` gates; 2026-07-22 artifacts in `/tmp/vz-tsq-2026-07-22/` | recipe | | E38 | the E31/E32/E35 knobs combined, then singly, then in pairs, each arm bracketed by default-path controls in one window | recipe | | E39 | the E38 survivors plus `IPERF_BLOCK=1M` and the E36 bracket, 21 interleaved fresh boots against defaults, `mediaanalysisd` SIGSTOPped; 2026-07-22 artifacts in `/tmp/vz-tsq-2026-07-22/quiet-legs/` | recipe | | E40 | cross-read `vz.pair.N.tx_frames` snapshots against the guest interrupt tables in any watchdog boot's log; the resumption-pattern P1 soak (short legs, idle gaps) is the regression once the fix lands | recipe | | E41 | dev-profile `STREAMS="1 1 1 1 1 1" ... 4 tx` boots; require zero guest `tx_errors` and read `vz.tx_stranded_catches`/`vz.tx_idle_polls` from the counters log; the Tx_pull contract is pinned in `ocaml-vz/test/test_vz_slirp.ml` | fixed + recipe | | E42 | `TCP_WRITE_OFFLOAD=1` against the default in interleaved P1 brackets; read `tcp.splice.offload_writes` from the counters log | knob | The complete environment-variable list, including `VIRTIO_TX_LOAN_BUDGET`, is documented at the top of `bench/vz_bench.sh`. ## Appendix C: arm64 LTO linker regression The release build of the Linux guest agent exposed two AArch64 LTO bugs. The old compiler failed with this signature: ```console $ ./docker-test.sh --profile release ocaml-slirp/guest/vz_net_agent.exe -use-lto: kept 4438 of 9519 functions, eliminated 5081 as dead code (53.4% of functions, -7001.6% of code size) .../caml_link....s:22238907: Error: pc-relative address offset out of range ``` Both the negative eliminated-code percentage and the PC-relative offset error matter. The fix is `samoht/ocaml` commit `d775aa2d3999d9fa5dfc33e3e3733b0ac4639b32`; the fixed link reports: ```text -use-lto: cleanup round grew the program (341341 -> 1063572 size units); discarding it -use-lto: kept 4374 of 9519 functions, eliminated 5145 as dead code (54.0% of functions, 44.5% of code size) ``` An error that `-use-lto` requires the flambda compiler is a separate configuration regression; `docker-test.sh` installs both the pinned compiler and `ocaml-option-flambda`, so one command checks both.