name: golden-read-until-timeout-flake description: FIXED 2026-06-17 — leveva boot-golden read_until now waits to an overall deadline instead of bailing on the first per-read timeout; the load-induced partial-buffer flake is gone metadata: node_type: memory type: project originSessionId: 29eaf964-8bde-4b89-952c-0208be9c6a10#

FIXED 2026-06-17 (commit 7718cdc1). The shared read_until (now a single read_until_impl in leveva/tests/harness/mod.rs, used by both Client and Peer) no longer breaks on the first per-read timeout. A WouldBlock/TimedOut is treated as a transient stall — keep reading until a generous 30s overall deadline; break early only on real EOF (Ok(0)) or a genuine socket error. This is exactly the "wall-clock deadline that distinguishes timed-out-mid-reply (retry) from stream-closed (stop)" the old note below called for. Common path (sentinel arrives) is unchanged; negative tests never depend on the timeout path (they use a second PING/PONG observable) so the higher ceiling costs them nothing. Verified: golden_restricted_op green 6/6 under concurrent golden-suite load; full cargo test -p leveva exit 0. So: a load-induced read_until partial-buffer "no NNN" panic should no longer happen — if you see one now, suspect a real logic bug, not this flake. (The deterministic fakelag-parking failures below were separate and already fixed in slice 238 via per-test max-penalty headroom.)

Historical context (pre-fix):

The leveva/tests/harness boot-golden Client::read_until(sentinel) loops stream.read() and breaks on Err(_) (a socket read-timeout) as well as on finding the sentinel. Under load (full cargo test -p leveva parallel run, or a slow/busy machine) the server can be slow enough that the read times out before the expected reply arrives, so read_until returns a partial buffer and the test panics with a "no NNN" assertion — and the panic point shifts between runs (e.g. golden_restricted_op::a_third_party_op_cannot_op_a_restricted_member failed at the +v NAMES on one run and at carol's +o NAMES on the next).

This is an environmental flake, not a regression. Confirm the standard way: git stash the working changes and re-run the single test binary — if it still fails on clean HEAD, it is pre-existing (observed 3/3 failing on this machine 2026-06-16, independent of P11 slice 226's connect/exit notices; the affected client was not +s, so the new notices added zero bytes to its stream).

UPDATE 2026-06-17 (slice 238) — golden_restricted_op was NOT this flake; FIXED. On re-investigation it failed deterministically 3/3 at the same point (carol's +o NAMES) on clean HEAD — so not load-dependent. Real root cause: the slice-208 fakelag delay half. The test drives several expensive commands (three NAMES whole-table scans cost +4 each, plus JOIN/MODE) at the harness's 50ms pacing → op's command-penalty clock runs ~21s ahead, past the default 10s MAX_PENALTY, so the serve loop parks op's read arm (flood_until / sleep_until in main.rs) and never processes its final NAMES within the client's 5s read timeout — read_until then breaks on the Err timeout with only a buffered JOIN. Fix: the test's options block now sets max-penalty 600 (disables input throttling for this non-flood characterization test; well under the 60s kill ceiling). Lesson: a deterministic read_until "no NNN" panic on a command-heavy boot golden is likely fakelag parking (slice 208), not the load flake below — give that test fakelag headroom rather than blaming load.

A second boot-golden failure (2026-06-17, slice 234) — ALSO fakelag, FIXED in slice 238: golden_s2s_keepalive::silent_peer_times_out_and_its_users_are_reaped fails with a BrokenPipe (harness/mod.rs Client::send write_all().unwrap()), not a read_until partial. Same deterministic root cause as golden_restricted_op: the observing client alice must poll WHOIS (cost +4 each → ~5s penalty/command) every 300ms for ~12s to stay active against the deliberately-tiny 2s ping-freq; that polling trips the slice-201 excess-flood kill (>60s ahead → server closes alice → the next send writes to a dead socket → BrokenPipe). (It is not a timing flake; it reproduces 3/3 on clean HEAD because the WHOIS-spam → kill is deterministic.) Fix: s2s_fastping.kdl (shared by golden_s2s_keepalive + golden_keepalive) now sets options { max-penalty 6000; flood-kill-ahead 6000 } to disable fakelag for these keepalive tests (both the parking and the kill must be lifted — parking would starve the activity that keeps alice alive → ping-reap; the kill is the proximate BrokenPipe).

Why: keeps a shifting golden panic from being mistaken for a logic bug a new slice introduced. How to apply: before blaming your change for a golden read_until panic, stash-and-rerun; if it reproduces clean, it's this flake. A real fix would give read_until a wall-clock deadline that distinguishes "timed out mid-reply" (retry) from "stream closed" (stop) instead of bailing on any Err. Related flakes: [[s-conf-oline-snap-flake]], [[p5-s2s-stats-flake]]. Recording discipline: [[leveva-slice-recording]].