Commits
Let the collector accept a queued short-lived producer before checking whether all connections have drained. This preserves complete runtime and allocation captures when native sampler startup temporarily occupies the Eio domain.
Also stop immediately, with a useful notice, when xctrace cannot attach before the target exits.
Replace the polymorphic (=) on cp_kind in the critical-path cost fold with
Report.equal_cp_kind, a match-based typed equality on the three-tag enum.
Replace sample(1)'s blocked-thread snapshots with scheduled Time Profiler samples on macOS. Capture all processes once, retain process identities, and report the target alongside external CPU consumers.
Save the native sidecar in every default bundle, load it from obs report, and handshake benchmark triggers only after recording is active.
Use the reconstructed critical path to classify blocking and CPU hotspots. Keep aggregate multidomain waiting in the capacity budget, but do not recommend work on idle listener fibers or count suspended span time as CPU.
The domain_writer fields lose their dw_ prefix (type disambiguation
covers the call sites), the span-sampler guard reports the exception
through the tracer's own error channel instead of swallowing it
blind, and start's promote/dealloc/tracker closures become named
functions. cmd_run gains install_stop_signals for the same
length-threshold reason. No behavior change.
The vendored multicore trace format was already domain-complete --
per-domain writers over one mutex-serialised fd, atomic id chunks, a
domain id on every event -- but the tracer's callback gates dropped
every non-main-domain sample, so an Executor_pool workload's
allocation profile read as empty (a 100 GB run yielded a 2 KB
boot-only trace, which cost the vz campaign its ring diagnosis).
Non-main domains now write through their own per-domain writer under
a plain mutex, which is sound where the vendored Lock is not: they
have no synchronous users, and memprof never re-enters a callback on
the domain running one. Tracked objects carry their writer so
promotion and collection land in the right stream, orphaned events
are counted and dropped rather than crossing domains, and stop
flushes every writer before the fd closes. The per-domain lookup is
an atomic slot array over a mutexed registry: Domain.DLS.get can
grow its backing array, which segfaults inside a memprof callback.
High-rate multidomain captures previously distorted wall and GC budgets, hid application counters, and required an opt-in native sampler. Make the default capture self-diagnosing while preserving explicit opt-outs for overhead measurements.
The ring producer drained the runtime_events ring through a cursor,
which under wasm is a no-op -- wasm_of_ocaml ships runtime_events as an
empty stub, so nothing is ever read. Probe has a sink escape hatch for
exactly this: when the ring is inactive, install a sink that decodes
each written packet and pushes it as a probe frame, stamped from the
producer's clock. Application spans then reach the collector from a wasm
guest as they do from a solo5 one. GC and Eio events are
runtime_events-native and have no sink, so they stay absent under wasm.
The auto-calibrating burn/count loops lived as byte-identical clones in
the crypto and tls bench executables and had started to drift. Home
them with the perf tooling as a dependency-free sublibrary: input
generation and result printing stay with each bench, so the shared
surface is just the probe-then-scale timing core.
root moved the bytesrw pin to the samoht/bytesrw fork (upstream of_slice fix plus the gc-sections stub-elimination patch); propagate the same pin into every direct dependent's pin-depends so the whole closure resolves the one fork instead of the stale dbuenzli commit.
The nox- prefix existed only because the bare name tty is taken on
central opam. console is free there and says what the library is:
inline styled terminal output rather than a full-screen TUI. Rename
the opam packages (nox-tty -> console, nox-tty-eio -> console-eio),
the modules (Tty -> Console, Tty_eio -> Console_eio), the C stub
namespace, the repository directory (ocaml-tty -> ocaml-console) and
the bundled Claude skill, and update every in-tree consumer. The
tangled repo must be renamed to ocaml-console to match before the
next monopam push.
One opam package shipped both, so a program that wanted Cmdliner logging terms declared a dependency on sqlite, hdr, uuidm, cstruct, eio_main and monitor. Those belong to the profiler. obs now carries the CLI, its collector and its trace adapters; observe carries the library. observe also drops nox-tty, since Tty.Display now installs the log-suspending reporter wrapper itself.
Dune ties a public_name to its package, so every observe.* sublibrary stays in observe, and the CLI's four libraries take public names under obs: obs.model, obs.fxt, obs.memtrace, obs.offline. They had to become public because dune forbids a public library depending on a private one, and being public brought their modules under merlint's untested-module rule. So they are tested now: 18 cases over the Fuchsia trace codec and 24 over the collector, its adapters and the CTF reader, each with a hostile case the public API must reject.
observe.opam still declares nox-protobuf, nox-varint, bytesrw and vsock through observe.producer, and nox-json and nox-catapult through observe.capture. Neither is the logging library's business; both are addressable by moving those sublibraries into packages of their own.
ctf_input.mli promises Sys_error when the trace cannot be opened, but feed let Unix.Unix_error escape from the reader. No caller catches either, so a missing .ctf surfaced as an unhandled Unix_error rather than the documented failure. The test added alongside this asserts the documented contract; reverting this line turns it red.
Observe_eio.setup was Observe.setup composed with Probe_eio.install. Bolting a scheduler's span provider onto a logging library gave observe a dependency on eio for one function call, and gave every Eio program a second setup entry point to choose between.
A library exposing an Eio interface installs the provider for its users, which is what quic_eio, h2_eio, slirp_eio and iperf3_eio already do. The two programs that drove Eio themselves now do the same. Observe.setup is the single call for Eio and non-Eio programs alike.
2cc174d44 extended the small-object-churn advice but left run.t stale, so dune test failed on a diff that was never a regression.
Forwarder's stop joins the session domain, so it blocks. test_server_end_to_end called it from inside Eio_main.run, parking that scheduler: the collector's handler fiber could then never drain the socket the session domain was blocked writing to, and neither side ever observed the stop flag. dune test hung there indefinitely -- the suite was killed, never passed.
The tests stop through Eio_unix.run_in_systhread instead, and the blocking contract is stated on the stop type. Observe.setup is unaffected: it defers stop to at_exit, which runs after Eio_main.run has returned.
lib/memtrace is a fork of janestreet/memtrace, MIT licensed, copyright Jane Street. None of its files carried a copyright header, while the package declared itself ISC with a single author -- which the MIT terms do not permit. Every derived file now carries the same style of attribution header the repo already uses for the eio-trace fork in lib/obs/fxt, LICENSE.md carries upstream's text, and the package license is the SPDX expression ISC AND MIT.
stdlib_shim is not upstream -- it stubs the OxCaml stdlib features the multicore branch uses -- so it takes the first-party ISC header instead. README.md records why observe cannot just depend on memtrace.0.2.3: that release predates multicore, its Promote and Collect carry no domain, its Alloc has no span_name, and it has no currently_tracing, all of which Obs.Hotspots and Forwarder_core rely on. It also records the Location_codec clamp bug this fork fixes, which belongs upstream.
observe depends on observe.producer, which depended on observe.producer.core, which depended on eio. Every CLI that calls Observe.setup therefore linked eio, and with it bigstringaf, cstruct, eio.core, lwt-dllist and optint -- to print log lines. The forwarder never touches Eio: the only users inside producer.core were Ring_eio, an Eio driver the forwarder path never calls, and Clock.of_eio, two clock reads.
Both move to a new observe.producer.core.eio, linked explicitly by the two Eio unikernel examples and the ring tests. producer.core now needs neither eio nor mtime, and obs_guest (observe.producer + probe) no longer drags the Eio core into a guest image. Clock_eio gets its own test module, split out of test_clock.
ocamlformat wraps by width, so dropping the Memtrace_eio level let these lines fit. The pre-commit hook promoted the reflow after staging, which is why b60ed01 shipped without it.
bytesrw 0.3.0 Reader.of_slice ignores a slice's first offset and reads from byte 0 (dbuenzli/bytesrw#13), so a Reader cannot address a sub-range of a larger buffer without a copy. The fix is on the bytesrw main branch but unreleased; pin to it in the root and in every package that depends on bytesrw directly, so the whole closure resolves one fixed version.
dune-project sets (implicit_transitive_deps false), so a declared library is a real compile edge, not documentation. lib/dune listed nox-json and unix; observe.ml and observe.mli name neither.
memprof_tracer_410.ml_ is a 4.10-era memprof shim parked behind a suffix dune ignores. Nothing has referenced it since it landed in 103850ead, and it would ship in the release tarball.
memtrace_eio dates from `eiotrace', the tool that merged a memtrace .ctf with Eio runtime_events. The module no longer mentions Memtrace at all -- the .ctf reader is Ctf_input -- and it reports fine on traces carrying no Eio scheduler events. Its library was already observer_model, so naming the module after it drops a redundant level: callers write Observer_model.Model.t, not Observer_model.Memtrace_eio.Model.t.
The four file renames reached HEAD in 31a64e002, which swept a staged `git mv' from a concurrent session; this commit carries the call sites that go with them, so the tree compiles again.
The DHCP client and server bind a datagram socket to 0.0.0.0 with
SO_REUSEADDR and SO_BROADCAST. That used to mean creating a Unix fd,
setting the options, binding, then importing into Eio, which pulled
eio.unix and unix into an otherwise portable library.
Eio 1.4 carries SO_BROADCAST as a portable Net.Sockopt and reuse_addr
as a datagram_socket argument, so the socket is built directly with
Eio.Net.datagram_socket and Eio.Net.setsockopt. The library now links
no unix; lease/renew/serve take the net capability.
eio moves to a dev commit: it adds portable Net socket options, so a
consumer can set TCP_NODELAY and friends without linking eio.unix. Every package
that depends on eio carries the pin, since opam resolves pin-depends per package
and not from the root.
The solo5 and unikraft cross-toolchains move to wip-macos+runtime_events+lto,
one branch carrying macOS cross-build support, an in-process runtime_events ring
and a flambda compiler that defaults to link-time DCE. The unikernel arms link
with -use-lto, so the compiler has to carry those patches.
merlint now measures functions inside functor bodies, and Model.report
lives in one. Its cyclomatic complexity (12) was driven by the boolean
operators of two derived flags, no_runtime_timing and no_evidence, computed
inline. Lift each to a named predicate above report. They are pure boolean
tests, which merlint does not count as control-flow complexity, and the
report body drops well under the threshold. No behaviour change.
The Rate class (a small object allocated too often, low promotion) is where
avoidable amplification hides -- a value boxed or tupled in a hot loop that did
no lasting work. The fix text just said 'cut the number of allocating calls',
which reads the same for an inherent per-row cost as for a boxed int64 that
should never have allocated.
When the object is tiny (<= 24 bytes, ~one to three words) refine the FIX: name
the usual culprit -- a boxed int64/float, a tuple, or an option/ref -- and the
SOTA move (hoist it out of the loop, or use an unboxed/native path). This is the
signature behind this session's biggest wins (the cat_cursor 12-byte value
round-trip, varint/solo5 int64 boxes), so obs now flags them as avoidable rather
than as the cost of the work. The base guidance is preserved, so the hotspots
suite and cram are unchanged.
The Eio forwarder's hot path encoded each event with Frame.to_framed -- which
allocates a fresh buffer and a string per frame -- then copied that string into
the batch buffer with add_string. Frame.add_framed (already used by the blocking
forwarder, documented as the producer hot path) writes the same length-delimited
bytes straight into the batch buffer, dropping a buffer and a string per streamed
event. Switch the Eio forwarder and the bench to it; add_framed emits identical
bytes, so the 52 protocol and 28 obs tests are unchanged.
The trace held on-CPU span durations but not the time fibers spent suspended, so blocked time never showed up as critical-path cost. The offline adapter now pairs each suspend-fiber with the next resume into a wait slice, and a critical_path canonical query ranks on-CPU spans and waits together.
The cross-process critical path needs each producer's parent span, which
reached the collector but was never written to the bundle. obs run records
a "process <id> parent_span <pid>" line per producer in the .meta (and sets
it on the live model from the collector's Hellos); obs report reads it back
and feeds the model, so the distributed path survives to disk. Also extract
the summary's row builders to keep render_summary_tty under the length cap.
Stitch each producer's critical path into one across process boundaries
using the W3C trace context the Hello already carries (set_process_context
records each process's parent span; 0 = a root). The backward walk crosses
on two edges, like Dapper distributed critical-path tracing: a host's
external wait that a child process ran inside jumps into that child (join,
not double-counted), and the child's first segment is preceded by the
launcher's segment ending at its start (launch). Test: host build_plan 5ms
-> launch -> guest boot 15ms = 20ms, crossing the process boundary.
obs run's triage was plain == BUDGET ==/== FINDINGS == text. Render it with
the house tty library in the green Matrix theme: the verdict in a panel,
the wall-clock budget and the ranked findings as bordered tables, and the
Amdahl ceiling (at most the critical path is reclaimable) as a footer.
Colour follows the formatter's style renderer, so a pipe or --json stays
plain. obs report keeps the full text tables. Empty captures render an
"unattributed" budget rather than a misleading cpu=100%. cram greps the
table header instead of the old text markers.
The waiting-fiber finding charged a parked fiber's whole idle life as
reclaimable (-100%), topping the list, even when the fiber was off the
critical path and optimizing it would return nothing. Re-express it as
critical-path zeroing: fire only for a fiber whose wait is on the path
(its on-path external-block time), with that on-path wait as the payoff;
an off-path idle fiber has slack and is not flagged (the non-bottleneck
mirage). Drop the span-explained heuristic (and the now-unused
fiber_block_in_span) it replaces. Add an Amdahl ceiling to the findings:
no fix returns more wall than the critical path. Tests: exact on-path
payoff (90ms), off-path fiber not flagged, and the waker edge survives the
fxt round-trip (2ms+8ms=10ms).
Compute the critical path -- the backward chain of "what unblocked me",
the bottleneck -- the way Go's tracer / Perfetto do. Ingest the wakeup
edges obs dropped (`Put` records who resolved each object; a resumed
fiber's waker is the latest Put on the object it awaited, else external),
record each fiber's Run/Block segments, and backward-walk the segment DAG
from the last-ending segment: a run is preceded by the block it resumed
from; a fiber-woken block jumps to the waker's run (its wait overlaps that
run, not double-counted); an external block is a path leaf. Report gains
critical_path (cost buckets, the reclaimable-wall ceiling) and per-fiber
on_path/slack, with a == CRITICAL PATH == section. Crosses domains within a
process. Tests: single-domain (cpu+wait=wall) and cross-domain (waker hop).
The analysis model keyed scheduler state by ring and fiber/object/cc state
by bare id, and the fxt reader folded every record onto ring 0 -- so two
domains merged and two producers (each a separate process: host child,
spawned sub-process, VM guest) with the same fiber/object id collided.
Re-key the per-process tables by (process, id) / (process, ring), thread an
optional process (default 0) through eio_event/gc_*, and have the reader
route each record to the (process, ring) of its own fxt thread (pid/tid),
pending creates included. Prerequisite for per-process / critical-path
analysis. New test: two processes sharing a fiber id stay two fibers with
distinct run-times.
The GC section showed CPU time = wall x domains (capacity), which read
34.229s right under the budget cpu=2.498s and made GC overhead (% of CPU)
meaningless for an idle guest (0.02% of wall vs 0.22% of on-CPU). Use the
budget cpu (wall - blocked - gc) so the two numbers match and the overhead
is a fraction of time actually spent computing.
The alloc rate divided total_alloc_words, which only the memtrace .ctf
fills -- a guest streams the exact minor/major allocation counters over
the ring but writes no .ctf (no filesystem), so the rate read 0 w/s
despite millions of words. Fall back to minor+major words when there are
no memtrace samples.
The verdict lead already says the category and the blocked/GC share (e.g.
"I/O-bound. 93% of wall-clock is blocked;"), so the appended finding
diagnosis should name the cause, not restate them. The syscall diagnosis
is now "blocks on <op> in <span>" and the gc one "allocates too much",
so the verdict reads "I/O-bound. 93% ...; blocks on Stream.take in
ssh.eio.connection" instead of repeating itself.
The ring-free producer is gone, so the no_runtime_timing verdict and
budget line should describe the trace condition, not a producer that no
longer exists: a trace with spans but no GC-phase or scheduler events. It
stays a dormant safety net -- a real ring capture carries fibers, GC
slices, or (now) the idle suspend_domain, so it never fires for one.
With the ring-free Direct producer gone, nothing streams the
Gc.quick_stat gauges (minor-collections, major-collections, heap-words):
the ring drives collection counts from the GC-phase events and has no
heap-size source. Remove the gauge cases from gc_counter, the heap_words
field, and its render line; the model keeps only the three runtime_events
allocation counters it accumulates.
The producer framed every event by allocating a temp Buffer plus a body
and a contents string in to_framed, then copied that string into the
output buffer -- three allocations per event on the ring drain path. Add
Frame.add_framed, which writes the length-delimited encoding straight
into the caller buffer; Out.push (the freestanding ring producer) and the
host forwarder emit use it. to_framed stays as a thin wrapper.
The in-process runtime_events ring works on solo5 now (verified end to
end under the tender), so the ring path is the single producer everywhere
and the ring-free Direct sampler -- which approximated GC from
Gc.quick_stat because no ring was available -- is dead. Delete Direct,
Direct_eio, and their tests; the unikernel already streams over Ring. The
shared output (Out) and ring framing (Ring) stay.
A sync object created before the collector attached has no captured
Create/Name, so the contention table showed it as a bare scheduler id
(26, 37) with no hint of what it is, and runtime_events cannot replay the
missing create. Record the span each Get/Try_get fires under and surface
its dominant span as a Where column, so a nameless object still shows where
it is contended -- the same span attribution the fiber/suspend tables use.
An object seen only via Get/Try_get (created before the collector attached,
so its Create/Name was never captured) renders as a bare id in the
contention table with no hint of what it is. runtime_events cannot replay
the missing create, so the row should fall back to the span it was
contended under. The test contends such an object inside a span and asserts
the row names that span; it fails today.
A bundle whose meta recorded wall time but whose runtime_events stream was
empty and that streamed no memtrace (mono_first never set) has no event of
any kind to attribute, yet the budget booked the whole wall as cpu and the
verdict claimed CPU-bound -- the misread a target without Observe.setup
produces. Add a no_evidence flag (no runtime_events timestamp, no memtrace,
positive wall): the verdict states the capture is empty, the budget marks
the wall unattributed, and the findings drop the "run looks balanced" line.
obs report emits "CPU-bound" for a capture with wall time from the run
meta but no events at all -- no scheduler events, GC, spans, suspends or
memtrace samples. The leftover wall is unknown, not on-CPU. The test sets
only the wall clock and asserts the verdict is not CPU-bound, names the
empty case, and the report drops the "run looks balanced" line. Fails today.
obs run printed the full report -- budget, verdict, findings, then every
detail table -- the same output obs report gives offline. Print only the
triage and point at `obs report <id>` for the rest; obs report stays full.
Attribute fibers and suspends to the span active when they ran or blocked,
so the report names where work is stuck instead of bare scheduler ids:
- fibers and the suspend hotspots gain a Where column (the dominant span);
- the sleep wait finding names the polling loop's span and drops the
"batch the syscalls" advice that never fit a timer wait;
- the waiting-fiber finding names where the fiber runs and what it blocks
on, and is suppressed when a named span already explains the wait, so it
stops topping the list with a -100% non-fix.
A fiber's run time is charged only to spans it opened, so one fiber's time
never leaks to a span another left open across a suspend.
Let the collector accept a queued short-lived producer before checking whether all connections have drained. This preserves complete runtime and allocation captures when native sampler startup temporarily occupies the Eio domain.
Also stop immediately, with a useful notice, when xctrace cannot attach before the target exits.
Replace sample(1)'s blocked-thread snapshots with scheduled Time Profiler samples on macOS. Capture all processes once, retain process identities, and report the target alongside external CPU consumers.
Save the native sidecar in every default bundle, load it from obs report, and handshake benchmark triggers only after recording is active.
The domain_writer fields lose their dw_ prefix (type disambiguation
covers the call sites), the span-sampler guard reports the exception
through the tracer's own error channel instead of swallowing it
blind, and start's promote/dealloc/tracker closures become named
functions. cmd_run gains install_stop_signals for the same
length-threshold reason. No behavior change.
The vendored multicore trace format was already domain-complete --
per-domain writers over one mutex-serialised fd, atomic id chunks, a
domain id on every event -- but the tracer's callback gates dropped
every non-main-domain sample, so an Executor_pool workload's
allocation profile read as empty (a 100 GB run yielded a 2 KB
boot-only trace, which cost the vz campaign its ring diagnosis).
Non-main domains now write through their own per-domain writer under
a plain mutex, which is sound where the vendored Lock is not: they
have no synchronous users, and memprof never re-enters a callback on
the domain running one. Tracked objects carry their writer so
promotion and collection land in the right stream, orphaned events
are counted and dropped rather than crossing domains, and stop
flushes every writer before the fd closes. The per-domain lookup is
an atomic slot array over a mutexed registry: Domain.DLS.get can
grow its backing array, which segfaults inside a memprof callback.
The ring producer drained the runtime_events ring through a cursor,
which under wasm is a no-op -- wasm_of_ocaml ships runtime_events as an
empty stub, so nothing is ever read. Probe has a sink escape hatch for
exactly this: when the ring is inactive, install a sink that decodes
each written packet and pushes it as a probe frame, stamped from the
producer's clock. Application spans then reach the collector from a wasm
guest as they do from a solo5 one. GC and Eio events are
runtime_events-native and have no sink, so they stay absent under wasm.
The auto-calibrating burn/count loops lived as byte-identical clones in
the crypto and tls bench executables and had started to drift. Home
them with the perf tooling as a dependency-free sublibrary: input
generation and result printing stay with each bench, so the shared
surface is just the probe-then-scale timing core.
The nox- prefix existed only because the bare name tty is taken on
central opam. console is free there and says what the library is:
inline styled terminal output rather than a full-screen TUI. Rename
the opam packages (nox-tty -> console, nox-tty-eio -> console-eio),
the modules (Tty -> Console, Tty_eio -> Console_eio), the C stub
namespace, the repository directory (ocaml-tty -> ocaml-console) and
the bundled Claude skill, and update every in-tree consumer. The
tangled repo must be renamed to ocaml-console to match before the
next monopam push.
One opam package shipped both, so a program that wanted Cmdliner logging terms declared a dependency on sqlite, hdr, uuidm, cstruct, eio_main and monitor. Those belong to the profiler. obs now carries the CLI, its collector and its trace adapters; observe carries the library. observe also drops nox-tty, since Tty.Display now installs the log-suspending reporter wrapper itself.
Dune ties a public_name to its package, so every observe.* sublibrary stays in observe, and the CLI's four libraries take public names under obs: obs.model, obs.fxt, obs.memtrace, obs.offline. They had to become public because dune forbids a public library depending on a private one, and being public brought their modules under merlint's untested-module rule. So they are tested now: 18 cases over the Fuchsia trace codec and 24 over the collector, its adapters and the CTF reader, each with a hostile case the public API must reject.
observe.opam still declares nox-protobuf, nox-varint, bytesrw and vsock through observe.producer, and nox-json and nox-catapult through observe.capture. Neither is the logging library's business; both are addressable by moving those sublibraries into packages of their own.
ctf_input.mli promises Sys_error when the trace cannot be opened, but feed let Unix.Unix_error escape from the reader. No caller catches either, so a missing .ctf surfaced as an unhandled Unix_error rather than the documented failure. The test added alongside this asserts the documented contract; reverting this line turns it red.
Observe_eio.setup was Observe.setup composed with Probe_eio.install. Bolting a scheduler's span provider onto a logging library gave observe a dependency on eio for one function call, and gave every Eio program a second setup entry point to choose between.
A library exposing an Eio interface installs the provider for its users, which is what quic_eio, h2_eio, slirp_eio and iperf3_eio already do. The two programs that drove Eio themselves now do the same. Observe.setup is the single call for Eio and non-Eio programs alike.
Forwarder's stop joins the session domain, so it blocks. test_server_end_to_end called it from inside Eio_main.run, parking that scheduler: the collector's handler fiber could then never drain the socket the session domain was blocked writing to, and neither side ever observed the stop flag. dune test hung there indefinitely -- the suite was killed, never passed.
The tests stop through Eio_unix.run_in_systhread instead, and the blocking contract is stated on the stop type. Observe.setup is unaffected: it defers stop to at_exit, which runs after Eio_main.run has returned.
lib/memtrace is a fork of janestreet/memtrace, MIT licensed, copyright Jane Street. None of its files carried a copyright header, while the package declared itself ISC with a single author -- which the MIT terms do not permit. Every derived file now carries the same style of attribution header the repo already uses for the eio-trace fork in lib/obs/fxt, LICENSE.md carries upstream's text, and the package license is the SPDX expression ISC AND MIT.
stdlib_shim is not upstream -- it stubs the OxCaml stdlib features the multicore branch uses -- so it takes the first-party ISC header instead. README.md records why observe cannot just depend on memtrace.0.2.3: that release predates multicore, its Promote and Collect carry no domain, its Alloc has no span_name, and it has no currently_tracing, all of which Obs.Hotspots and Forwarder_core rely on. It also records the Location_codec clamp bug this fork fixes, which belongs upstream.
observe depends on observe.producer, which depended on observe.producer.core, which depended on eio. Every CLI that calls Observe.setup therefore linked eio, and with it bigstringaf, cstruct, eio.core, lwt-dllist and optint -- to print log lines. The forwarder never touches Eio: the only users inside producer.core were Ring_eio, an Eio driver the forwarder path never calls, and Clock.of_eio, two clock reads.
Both move to a new observe.producer.core.eio, linked explicitly by the two Eio unikernel examples and the ring tests. producer.core now needs neither eio nor mtime, and obs_guest (observe.producer + probe) no longer drags the Eio core into a guest image. Clock_eio gets its own test module, split out of test_clock.
bytesrw 0.3.0 Reader.of_slice ignores a slice's first offset and reads from byte 0 (dbuenzli/bytesrw#13), so a Reader cannot address a sub-range of a larger buffer without a copy. The fix is on the bytesrw main branch but unreleased; pin to it in the root and in every package that depends on bytesrw directly, so the whole closure resolves one fixed version.
dune-project sets (implicit_transitive_deps false), so a declared library is a real compile edge, not documentation. lib/dune listed nox-json and unix; observe.ml and observe.mli name neither.
memprof_tracer_410.ml_ is a 4.10-era memprof shim parked behind a suffix dune ignores. Nothing has referenced it since it landed in 103850ead, and it would ship in the release tarball.
memtrace_eio dates from `eiotrace', the tool that merged a memtrace .ctf with Eio runtime_events. The module no longer mentions Memtrace at all -- the .ctf reader is Ctf_input -- and it reports fine on traces carrying no Eio scheduler events. Its library was already observer_model, so naming the module after it drops a redundant level: callers write Observer_model.Model.t, not Observer_model.Memtrace_eio.Model.t.
The four file renames reached HEAD in 31a64e002, which swept a staged `git mv' from a concurrent session; this commit carries the call sites that go with them, so the tree compiles again.
The DHCP client and server bind a datagram socket to 0.0.0.0 with
SO_REUSEADDR and SO_BROADCAST. That used to mean creating a Unix fd,
setting the options, binding, then importing into Eio, which pulled
eio.unix and unix into an otherwise portable library.
Eio 1.4 carries SO_BROADCAST as a portable Net.Sockopt and reuse_addr
as a datagram_socket argument, so the socket is built directly with
Eio.Net.datagram_socket and Eio.Net.setsockopt. The library now links
no unix; lease/renew/serve take the net capability.
eio moves to a dev commit: it adds portable Net socket options, so a
consumer can set TCP_NODELAY and friends without linking eio.unix. Every package
that depends on eio carries the pin, since opam resolves pin-depends per package
and not from the root.
The solo5 and unikraft cross-toolchains move to wip-macos+runtime_events+lto,
one branch carrying macOS cross-build support, an in-process runtime_events ring
and a flambda compiler that defaults to link-time DCE. The unikernel arms link
with -use-lto, so the compiler has to carry those patches.
merlint now measures functions inside functor bodies, and Model.report
lives in one. Its cyclomatic complexity (12) was driven by the boolean
operators of two derived flags, no_runtime_timing and no_evidence, computed
inline. Lift each to a named predicate above report. They are pure boolean
tests, which merlint does not count as control-flow complexity, and the
report body drops well under the threshold. No behaviour change.
The Rate class (a small object allocated too often, low promotion) is where
avoidable amplification hides -- a value boxed or tupled in a hot loop that did
no lasting work. The fix text just said 'cut the number of allocating calls',
which reads the same for an inherent per-row cost as for a boxed int64 that
should never have allocated.
When the object is tiny (<= 24 bytes, ~one to three words) refine the FIX: name
the usual culprit -- a boxed int64/float, a tuple, or an option/ref -- and the
SOTA move (hoist it out of the loop, or use an unboxed/native path). This is the
signature behind this session's biggest wins (the cat_cursor 12-byte value
round-trip, varint/solo5 int64 boxes), so obs now flags them as avoidable rather
than as the cost of the work. The base guidance is preserved, so the hotspots
suite and cram are unchanged.
The Eio forwarder's hot path encoded each event with Frame.to_framed -- which
allocates a fresh buffer and a string per frame -- then copied that string into
the batch buffer with add_string. Frame.add_framed (already used by the blocking
forwarder, documented as the producer hot path) writes the same length-delimited
bytes straight into the batch buffer, dropping a buffer and a string per streamed
event. Switch the Eio forwarder and the bench to it; add_framed emits identical
bytes, so the 52 protocol and 28 obs tests are unchanged.
The cross-process critical path needs each producer's parent span, which
reached the collector but was never written to the bundle. obs run records
a "process <id> parent_span <pid>" line per producer in the .meta (and sets
it on the live model from the collector's Hellos); obs report reads it back
and feeds the model, so the distributed path survives to disk. Also extract
the summary's row builders to keep render_summary_tty under the length cap.
Stitch each producer's critical path into one across process boundaries
using the W3C trace context the Hello already carries (set_process_context
records each process's parent span; 0 = a root). The backward walk crosses
on two edges, like Dapper distributed critical-path tracing: a host's
external wait that a child process ran inside jumps into that child (join,
not double-counted), and the child's first segment is preceded by the
launcher's segment ending at its start (launch). Test: host build_plan 5ms
-> launch -> guest boot 15ms = 20ms, crossing the process boundary.
obs run's triage was plain == BUDGET ==/== FINDINGS == text. Render it with
the house tty library in the green Matrix theme: the verdict in a panel,
the wall-clock budget and the ranked findings as bordered tables, and the
Amdahl ceiling (at most the critical path is reclaimable) as a footer.
Colour follows the formatter's style renderer, so a pipe or --json stays
plain. obs report keeps the full text tables. Empty captures render an
"unattributed" budget rather than a misleading cpu=100%. cram greps the
table header instead of the old text markers.
The waiting-fiber finding charged a parked fiber's whole idle life as
reclaimable (-100%), topping the list, even when the fiber was off the
critical path and optimizing it would return nothing. Re-express it as
critical-path zeroing: fire only for a fiber whose wait is on the path
(its on-path external-block time), with that on-path wait as the payoff;
an off-path idle fiber has slack and is not flagged (the non-bottleneck
mirage). Drop the span-explained heuristic (and the now-unused
fiber_block_in_span) it replaces. Add an Amdahl ceiling to the findings:
no fix returns more wall than the critical path. Tests: exact on-path
payoff (90ms), off-path fiber not flagged, and the waker edge survives the
fxt round-trip (2ms+8ms=10ms).
Compute the critical path -- the backward chain of "what unblocked me",
the bottleneck -- the way Go's tracer / Perfetto do. Ingest the wakeup
edges obs dropped (`Put` records who resolved each object; a resumed
fiber's waker is the latest Put on the object it awaited, else external),
record each fiber's Run/Block segments, and backward-walk the segment DAG
from the last-ending segment: a run is preceded by the block it resumed
from; a fiber-woken block jumps to the waker's run (its wait overlaps that
run, not double-counted); an external block is a path leaf. Report gains
critical_path (cost buckets, the reclaimable-wall ceiling) and per-fiber
on_path/slack, with a == CRITICAL PATH == section. Crosses domains within a
process. Tests: single-domain (cpu+wait=wall) and cross-domain (waker hop).
The analysis model keyed scheduler state by ring and fiber/object/cc state
by bare id, and the fxt reader folded every record onto ring 0 -- so two
domains merged and two producers (each a separate process: host child,
spawned sub-process, VM guest) with the same fiber/object id collided.
Re-key the per-process tables by (process, id) / (process, ring), thread an
optional process (default 0) through eio_event/gc_*, and have the reader
route each record to the (process, ring) of its own fxt thread (pid/tid),
pending creates included. Prerequisite for per-process / critical-path
analysis. New test: two processes sharing a fiber id stay two fibers with
distinct run-times.
The GC section showed CPU time = wall x domains (capacity), which read
34.229s right under the budget cpu=2.498s and made GC overhead (% of CPU)
meaningless for an idle guest (0.02% of wall vs 0.22% of on-CPU). Use the
budget cpu (wall - blocked - gc) so the two numbers match and the overhead
is a fraction of time actually spent computing.
The alloc rate divided total_alloc_words, which only the memtrace .ctf
fills -- a guest streams the exact minor/major allocation counters over
the ring but writes no .ctf (no filesystem), so the rate read 0 w/s
despite millions of words. Fall back to minor+major words when there are
no memtrace samples.
The verdict lead already says the category and the blocked/GC share (e.g.
"I/O-bound. 93% of wall-clock is blocked;"), so the appended finding
diagnosis should name the cause, not restate them. The syscall diagnosis
is now "blocks on <op> in <span>" and the gc one "allocates too much",
so the verdict reads "I/O-bound. 93% ...; blocks on Stream.take in
ssh.eio.connection" instead of repeating itself.
The ring-free producer is gone, so the no_runtime_timing verdict and
budget line should describe the trace condition, not a producer that no
longer exists: a trace with spans but no GC-phase or scheduler events. It
stays a dormant safety net -- a real ring capture carries fibers, GC
slices, or (now) the idle suspend_domain, so it never fires for one.
With the ring-free Direct producer gone, nothing streams the
Gc.quick_stat gauges (minor-collections, major-collections, heap-words):
the ring drives collection counts from the GC-phase events and has no
heap-size source. Remove the gauge cases from gc_counter, the heap_words
field, and its render line; the model keeps only the three runtime_events
allocation counters it accumulates.
The producer framed every event by allocating a temp Buffer plus a body
and a contents string in to_framed, then copied that string into the
output buffer -- three allocations per event on the ring drain path. Add
Frame.add_framed, which writes the length-delimited encoding straight
into the caller buffer; Out.push (the freestanding ring producer) and the
host forwarder emit use it. to_framed stays as a thin wrapper.
The in-process runtime_events ring works on solo5 now (verified end to
end under the tender), so the ring path is the single producer everywhere
and the ring-free Direct sampler -- which approximated GC from
Gc.quick_stat because no ring was available -- is dead. Delete Direct,
Direct_eio, and their tests; the unikernel already streams over Ring. The
shared output (Out) and ring framing (Ring) stay.
A sync object created before the collector attached has no captured
Create/Name, so the contention table showed it as a bare scheduler id
(26, 37) with no hint of what it is, and runtime_events cannot replay the
missing create. Record the span each Get/Try_get fires under and surface
its dominant span as a Where column, so a nameless object still shows where
it is contended -- the same span attribution the fiber/suspend tables use.
An object seen only via Get/Try_get (created before the collector attached,
so its Create/Name was never captured) renders as a bare id in the
contention table with no hint of what it is. runtime_events cannot replay
the missing create, so the row should fall back to the span it was
contended under. The test contends such an object inside a span and asserts
the row names that span; it fails today.
A bundle whose meta recorded wall time but whose runtime_events stream was
empty and that streamed no memtrace (mono_first never set) has no event of
any kind to attribute, yet the budget booked the whole wall as cpu and the
verdict claimed CPU-bound -- the misread a target without Observe.setup
produces. Add a no_evidence flag (no runtime_events timestamp, no memtrace,
positive wall): the verdict states the capture is empty, the budget marks
the wall unattributed, and the findings drop the "run looks balanced" line.
obs report emits "CPU-bound" for a capture with wall time from the run
meta but no events at all -- no scheduler events, GC, spans, suspends or
memtrace samples. The leftover wall is unknown, not on-CPU. The test sets
only the wall clock and asserts the verdict is not CPU-bound, names the
empty case, and the report drops the "run looks balanced" line. Fails today.
obs run printed the full report -- budget, verdict, findings, then every
detail table -- the same output obs report gives offline. Print only the
triage and point at `obs report <id>` for the rest; obs report stays full.
Attribute fibers and suspends to the span active when they ran or blocked,
so the report names where work is stuck instead of bare scheduler ids:
- fibers and the suspend hotspots gain a Where column (the dominant span);
- the sleep wait finding names the polling loop's span and drops the
"batch the syscalls" advice that never fit a timer wait;
- the waiting-fiber finding names where the fiber runs and what it blocks
on, and is suppressed when a named span already explains the wait, so it
stops topping the list with a -100% non-fix.
A fiber's run time is charged only to spans it opened, so one fiber's time
never leaks to a span another left open across a suspend.