Commits
is_build_dir matched _build and _build_* but let the hyphenated
_build-linux (docker-test.sh's build dir) through, so merlint's
project rules analysed stale build copies of whole packages and
reported ghost findings against paths under the build tree. Any
_build prefix now reads as a build directory; the skip test covers
the hyphen variant.
Library.re_exports surfaces the (re_export ...) subset of a library's
dependencies, so dead-dependency analysis can skip entries an umbrella
library hands to its dependents without importing.
The parallel-scan merge also learns about the four per-library tables
added for the link-level queries (wrapped, links_all, has_foreign_stubs
and now re_exports): partial indexes dropped them on merge, so pooled
runs saw every source library as wrapped, never -linkall and never
re-exporting, which both broke the -linkall exemption and misread
(wrapped false) libraries' unit names.
Library.modules reports source basenames, which are not a wrapped
library's link-level identity: an internal wire.ml is only addressable
as Lib__Wire, yet it leaked out under the bare name Wire and collided
with the wire codec's entry module, masking a dead dependency in
cfdp-eio. Library.unit_names reports what the linker sees instead: the
entry unit for a wrapped source library, module names for an unwrapped
one, and dune-package object names for installed libraries.
Stanzas also now carry links_all and has_foreign_stubs (on Library.t
and source_stanza), the two shapes where a dependency is live without
any module reference.
eio moves to a dev commit: it adds portable Net socket options, so a
consumer can set TCP_NODELAY and friends without linking eio.unix. Every package
that depends on eio carries the pin, since opam resolves pin-depends per package
and not from the root.
The solo5 and unikraft cross-toolchains move to wip-macos+runtime_events+lto,
one branch carrying macOS cross-build support, an in-process runtime_events ring
and a flambda compiler that defaults to link-time DCE. The unikernel arms link
with -use-lto, so the compiler has to carry those patches.
Several synopses had drifted from the code (mirage-eio is backend-agnostic,
not Solo5-only; nox-quic ships HTTP/3; squashfs writes as well as reads) and
varied in style. Describe each package by what it does rather than through
its eio, mirage or solo5 consumer, and apply the house synopsis conventions:
no leading article, spec ids in trailing parens. Also corrects a few factual
description and tag errors.
These packages are published only to the opam overlay, not the public
opam-repository. Without the repo-add line a reader who runs opam install
gets nothing and has to hunt down the overlay URL by hand.
Dune accepts (vendored_dirs ...) in every dune file, marking that
file's own subdirectories; the index only read the monorepo root's
dune, so a package vendoring an upstream library under lib/ was not
seen as vendored and linters kept asking its verbatim sources to
change. Each scanned dune file now contributes its declarations,
resolved against its own directory.
Dune honors (vendored_dirs ...) in any dune file, marking that file's
own subdirectories; the index only reads the monorepo root's dune, so
a package vendoring an upstream library under lib/ is not seen as
vendored and linters keep asking its verbatim sources to change.
nox-memtrace was a maintained fork of Jane Street's memtrace; it now lives in
ocaml-observe as observe.memtrace, so there is no external fork to track and the
capture format can evolve with the obs tooling. This drops the nox-memtrace and
nox-memtrace-eio packages and repoints every bench and opam dependency in the
tree at observe, and adds the obs SQLite sink so traces can be queried with SQL.
Across the packages monopam lint flagged: declare the libraries each
package actually uses (test-only ones scoped with-test, dev tools and
benches scoped with-dev-setup), drop the opam deps and dune (libraries)
entries nothing references, declare pin-depends for the wire and zarith
git pins so a standalone opam install can resolve them, and point the
five stale dune-project source URLs at the names sources.toml already
uses. The generated .opam files are regenerated to match.
Where monopam lint and merlint's project rules disagree, merlint wins:
[protocol] packages keep probe linked (E932), CSV-trace interop tests
keep csv (E825), and dev-scope deps carry the right filter (E943).
The few dead-lib reports that remain are monopam lint blind spots, not
real dead deps: usage reached only through an mdx README block (slirp),
through copy_files# (vz), or inside a library's own private modules
(builder), plus a tar gz dep kept live for a stub test still to write.
A platform- or config-gated stanza (an enabled_if library, a build_if
test) may legitimately have no .cmt on the analysis host. Add gated_source_files
so consumers can tell such an off-host artefact from one that was simply never
built. Record each library's gated flag and -- crucially -- carry it through the
parallel index merge (merge_library_tables dropped it, so libraries scanned in a
worker lost their gating). Cover both the enabled_if library and build_if test
paths, built through a pool, in a regression test.
Follow the ocaml-probe API rename from Probe.run to Probe.with_span.
A private executable (no public_name) shipped by an (install (package P))
stanza -- directly or through (copy ...) rules -- is a runtime artifact of P:
the package builds and installs it, so its libraries belong in P's runtime
depends. Follow the install -> copy -> NAME.exe chain to credit those
libraries as runtime uses, and treat such an install as attaching the dune
file to P. Fixes a false E943 on packages that codesign a copy of their
tender before installing it.
record_libraries_from_dune_package called both add_dune_package and
Package.libraries on the same content, parsing every installed
dune-package twice (and up to four times across the two install roots) --
the bulk of the project_index.scan allocation. Parse once with
Package.libraries and feed the stanzas to Lib_index.add_packages.
A dependency installed with only .cmi files records just its bare
package name, so using one of its sublibraries (pkg.sub) resolved to no
package and the dependency looked unreached -- masking real missing
deps and tripping false E943s the other way. Fall back to the
public-name prefix, which dune guarantees is the providing package.
Add an ISC LICENSE.md (E913), link the type in the (/) doc with
{!type-t} (E420), and lift the per-directory readdir cache out of
resolve_module_files into a dir_entries helper so the function is back
under the length threshold (E005).
Introduce Project_index.Path: an abstract, canonical (absolute, normalised)
filesystem path built once at construction, with structural equality, ordering
and ancestor tests. Unify the index's internal path helpers (path_key,
descendant checks) onto it. This is the shared home for the path type that
merlint's Context and the index query API will adopt, replacing the scattered
per-module path normalisation helpers.
Hoist the path-key helpers (cwd, normalise_dir, absolute_path, path_key) to the
top so every cache and comparison shares one canonical absolute string per
path, and add str_is_descendant, a plain prefix test over those strings.
is_vendored_path now canonicalises the queried path once and compares against
vendored_dirs held as canonical strings, instead of re-normalising both sides
on every ancestor via path_is_descendant.
resolve_module_files joined each candidate module into an Fpath, rendered it to
a string for dedup, then re-parsed it with Fpath.v/parent to rebuild the
directory. Carry the (directory, base name) pair through instead, so the final
existence check reuses the directory it already had.
resolve_module_files ran Sys.readdir for a directory during its traversal and
then again in the per-module existence check -- once per candidate module and
suffix -- reallocating the entry array each time. Memoise the directory listing
within the call. On a merlint self-run this drops the module-resolution
allocation from the top of the profile to zero (about a sixth of total).
The installed-package roots feed one thing: resolving a referenced
(libraries ...) use to its providing package. Eager scanning parsed every
installed dune-package up front, but a build only resolves the libraries its
source tree actually names. Replace the installed:bool flag with a three-way
installed_scan: Skip, Eager, and Referenced, which scans the source tree
first, then visits only the install dirs whose name is the first segment of a
referenced library and that no source package already provides. Dune installs
a public library under the directory named by its first segment, so this
resolves every referenced use while parsing a fraction of the switch;
excluding source packages keeps them authoritative without the install scan
clobbering them.
build scanned _opam/lib and _build/install/default/lib on every call, but
their only product is the installed library-to-package mapping that the
dependency-resolution queries consume. A caller running none of those checks
paid a serial scan of every installed dune-package -- tens of MB of parsing --
for nothing. Add ?installed (default true) to skip both roots, and parse each
source dune file once, shared between the walk's skip filter and the stanza
scan, rather than reading it twice.
Remove libraries that no module in the stanza directory
references (monopam lint dead-lib).
Opam.field_of_path allocated the default IO buffer per call; the
project index scans hundreds of opam files of a few KB each, so the
transient buffers added up to 226 MB (11% of merlint's allocation).
Cap the reader slice at the file length.
project_index's opam_field was a private duplicate of field_of_path
(modulo catching End_of_file); delegate to the canonical one.
Libraries with a public name were registered under that name only, so
a sibling package's (libraries local_name) reference -- which dune
resolves workspace-wide -- looked unreached and E941/E943 misfired
(git-x not reaching nox-git, nox-tcp-eio not reaching nox-tcp). A
lib_alias table maps each local (name ...) to its canonical
(pkg, public_name) id and Library.in_pkg / resolve_all consult it.
doc_files returns every .md/.mld next to an indexed dune file,
annotated with whether an mdx stanza covers it; this feeds the E923
skipped-example rule.
Comment reflow and header reindent produced by the new ocamlformat;
no code changes.
Add merlint rule E334 (Redundant Variant/Field Prefixes): a variant
whose every constructor, or a record whose every field, shares a leading
underscore-delimited word has that prefix dropped, relying on OCaml's
type-directed disambiguation at call sites. A bare field that would be a
keyword takes a trailing underscore (entity_type -> type_).
Apply it across ~45 packages, annotating call sites where the bare name
is ambiguous between sibling record types.
By default [project-index files] lists only non-vendored sources. Add an
opt-in flag that passes ~include_vendored:true through to
Project_index.source_files so vendored (vendored_dirs ...) subtrees can be
listed too.
The recent quadratics (source_presence rebuilding the file list per
file, resolve_library rescanning every library per call) were invisible
until profiled. Add invocation counters to the whole-repo builders
(source_packages, source_package_list, source_files, package_libraries,
public_library_source_files, private_module_names, test_module_names)
and surface them via Project_index.scan_stats, logged by merlint at -v.
A count orders of magnitude above the rule count flags a builder being
recomputed inside a per-file or per-stanza loop. Counters are
pre-registered and atomic so the cross-domain rule fibres only read the
table. After the earlier fixes the worst offender is source_files=3
(was per-file); package_libraries is O(rules x packages) and cheap.
resolve_public_library replaces the per-call rescan that backed
Project.Query.resolve_library: every lookup rebuilt the full source
library list and linearly searched it by public name. E606 and E615
call it per declared library per test stanza, so the cost was quadratic
in the number of libraries. Build the public-to-local map once (first
declaration wins, matching the previous List.find_opt) and cache it on
the index, mirroring the other lazy caches. Any rule resolving a public
library reference now pays O(1). E615 drops from ~890ms to ~450ms and
E606 leaves the hot list; results are unchanged.
mem_source_file rebuilt and sorted the entire monorepo source-file list
on every call, and source_presence calls it once per file. Rules that
moved from a filesystem stat to the index (E505, E705, E600, E606) then
paid O(n^2 log n): on this tree E505 alone spent 2730 ms. Cache the
membership set per include_vendored flag, mirroring the existing
file_libraries_cache, so the query is O(1) after the first build. E505
drops to ~240 ms and total project-rule time roughly halves; results are
unchanged.
ocamllex (.mll) and ocamlyacc/menhir (.mly) generate a .ml module of the
same name at build time, but the project index only scanned for .ml/.mli, so
a lexer.mll or parser.mly contributed no module to Library.files.
E610 (test without library) pairs every test_X.ml with a library module X;
relying on Library.files, it reported test_lexer.ml as having no library
module when the source tree held lexer.mll + lexer.mli but no checked-in
lexer.ml -- a false positive, since the lexer module really exists (generated
by ocamllex). Adding a stub lexer.ml would be the wrong fix.
- project-index: resolve_module_files now treats .mll and .mly as module
sources, the way dune does, so they appear in Library.files.
- merlint E610: normalise a .mll/.mly library file to the .ml module it
generates before matching it against a test file.
Add a good-generated E610 fixture (lexer.mll + parser.mly with test_lexer.ml
and test_parser.ml, no .ml sources) that stays silent. The e523 ocamllex
fixture now counts lexer.mll among the analysed files (16 -> 17).
source_status reported Missing for any file not in the project index, but a
roots-scoped Project_index.build (as merlint uses for --no-build file-list
runs) drops sibling files: a fuzz_*.mli that exists on disk read as Missing, so
E705 falsely flagged a fuzz module as lacking its interface (and E600 likewise
for test .mli) on commit.
Add Project_index.source_presence : Indexed | Unindexed | Absent -- a file not
indexed but present on disk is Unindexed, only a genuinely absent file is
Absent. merlint's Build.source_status gains a matching Skipped status
(on-disk-but-unindexed) distinct from Missing; E600/E705 treat Skipped as
present, so they only fire when the interface is truly absent.
The filesystem stat lives in the index layer, not the rules. Test covers all
three presences.
E600/E705 keyed "does this .mli exist" off the run's selected/analyse
set. When merlint analyses a subset of the project (a pre-commit hook
passing only the changed files), an existing-but-unchanged .mli is not
in that set, so the rule wrongly reported the interface as missing.
Add Project_index.mem_source_file as the source of truth for source
existence, and Build.source_status returning Compiled | Not_compiled |
Missing -- existence from the index, compilation freshness from the
.cmt/.cmti artefact. E600 and E705 now report a missing interface only
on Missing, so a present-but-unchanged or merely-uncompiled .mli no
longer trips them.
E941 now also checks libraries linked by public-executable stanzas, since opam
install builds those through @install -- a runtime dep, not a test-only one.
project-index records each source stanza's (public_name ...) so the rule can
tell public executables from private ones and tests.
The extended check surfaced real gaps, now fixed: gdocs/gsheets/gslides linked
nox-crypto-rng from a public executable but declared it :with-test;
project-index's CLI links cmdliner/eio_main/nox-json at runtime; prune links
nox-dune. New bad-exe/good-exe cram fixtures cover the executable path.
Pure dune fmt output: doc-comment placement and line wrapping. No behaviour
change.
- dune-workspace: remove the second [afl] build context. The default
build no longer duplicates every target into an afl-instrumented
context (the afl env flags stay for when the profile is selected
explicitly), cutting build work.
- (mdx ...) stanzas across the tree gain an explicit [(package ...)]
so the README doc-tests are owned by the right opam package.
- merlint: new rule E922 (with cram + unit tests); E605/E727/data and
ocaml-dune / project-index follow-on.
merlint: E410/E523 and Docs/Issue/Rule follow-on, regenerated
index.html / STYLE_GUIDE.md, cram promotes. Continues the
monorepo-wide doc-comment pass (value docs lead with the value name;
undocumented public values documented) across monopam, ocaml-atp,
ocaml-atproto-oauth, ocaml-auth and others.
E420 drops a useless [{ target with ... }] record-with that tripped
warning 23. Continues the monorepo-wide doc-comment pass: value docs
lead with the value name, undocumented public values documented.
- merlint: new rule E420 (with cram + unit tests); E415 fixtures and
docs updated, Docs module follow-on.
- ocaml-crypto/test: the per-backend test_native.ml files are replaced
by a thin test.ml that runs the shared crypto_backend_tests suite
against nox-crypto.c / nox-crypto.ocaml.
- continued doc-comment pass across dupfind, irmin, ocaml-crypto,
ocaml-crow, ocaml-cookie/cpio/csv/dsp/dune/fdir/git/gpt and others:
value docs lead with the value name, undocumented public values get
a one-line summary.
Wide doc-comment pass: value docs lead with the value name / [fn args]
form instead of an imperative sentence, and previously undocumented
public values (mostly test [suite] values and a few accessors) get a
one-line description -- satisfying the public-value-documentation
rule across the tree.
Also includes the ocaml-bloom package (lib + opam + dune-project) and
small follow-on edits to merlint rules (E305/E320/E405), monopam, and
ocaml-atp / ocaml-claude / ocaml-btree / ocaml-cfdp libraries that the
same pass touched, plus assorted eio test/dune tweaks.
Lands the in-flight work across the tree:
- project-index: package-context resolution and stanza ownership
reach the merlint rules that consume them (E5xx/E6xx/E7xx/E9xx),
plus the JSON dump binary under ocaml-project-index/bin.
- merlint: new rules E622 and E727 with cram + unit tests; the
existing rule batch updated for the new Project_index / Context /
Build surface.
- monopam: lint and opam_sync follow the project-index API changes.
- assorted eio test/dune tweaks in ocaml-atp / ocaml-cop1 / ocaml-ltp.
The usage block named eio / monopam / fmt and threw the results away
with [ignore] -- arbitrary inside this package and demonstrating
nothing. Replace with realistic, parameterised helper functions a
consuming tool would actually write: load the index, pair every
package with its libraries, and resolve a package's [(libraries ...)]
uses within its own scope (noting the package-scoping that keeps
same-named private helpers from leaking).
MDX stanza drops [eio_main] (the example no longer runs an event loop;
it parameterises over [fs] like real library code) and keeps
[eio eio.core eio.unix] for the [Eio.Path.t] type.
The usage block was a [let queries fs = ...] wrapper that existed only
to type-check without a real fs. Rewrite it as natural top-level code
([open Project_index] + [let () = Eio_main.run ...]) matching the
house style (ocaml-merlin, ocaml-block READMEs), so the example reads
like real usage and still compiles under MDX.
The MDX stanza gains [eio eio.core eio.unix] (alongside the existing
[eio_main]) so [Eio.Stdenv.fs] resolves -- the same library list other
eio-using READMEs in the repo declare.
A package's [(libraries name)] use was resolved with a global name
lookup (Library.resolve_all), so a private test helper named [helper]
in one package pulled in every same-named helper across the repo --
test_module_names then leaked unrelated packages' modules.
Adds [library_used_by pkg name]: resolve within [pkg] first
(Library.in_pkg), then fall back only to a globally-unique
public/installed library. test_module_names and the dep-resolution
paths route package-scoped uses through it; the bare global lookup
stays available as [libraries_of_name] for inventory questions.
Test test_package_context_library_use_resolution sets up two packages
with a private [helper] library each (one a test helper, one a runtime
helper) and asserts the package-context resolver picks the right one
and the other package's modules don't leak into the aggregate.
Two installed-package fixes plus tests:
- Local precedence: a locally built package now resets its installed
library state before recording (reset_installed_package_libraries),
so a library removed or renamed in the local build no longer lingers
from the _opam copy. origin reports Local. Test:
test_local_install_shadows_opam_copy.
- Plural stanzas: (tests (names a b)) / (executables (names a b)) now
record one source_stanza per name instead of dropping all but the
first, so Package.test_stanzas / executable_stanzas see every name
(E615 walks these). Tests: test_plural_tests_stanza_keeps_all_names,
test_plural_executables_stanza_keeps_all_names.
The lib now logs a warning on the precedence reset path, so it gains a
[logs] dependency.
In a multi-package dune-project, ownership was decided per dune file
from its public libraries / executables, so a [(test ... (package
host_eio))] sitting in a file alongside a [host] library was either
mis-attributed to host or dropped entirely (the ocaml-tls /
nox-tls-eio shape: eio/tests/dune named nox-tls-eio but the test never
landed on it).
ocaml-dune gains a typed accessor for a test/tests stanza's [(package
...)] field; project-index records each stanza under its own declared
owner instead of treating the whole dune file as one owner.
Adds 18 ownership / project-shape tests to test_project_index:
explicit [(package X)] on test and tests stanzas, the package-less
ambiguous case, mixed ownership within one dune file, adapter packages
(both test_modules and test_stanzas), unknown-package drop, deep
nesting, include_subdirs (recursive lib files + test exclusion),
data_only_dirs, cram .t fixtures, vendored subtree flagging, and
nested-workspace isolation.
is_build_dir matched _build and _build_* but let the hyphenated
_build-linux (docker-test.sh's build dir) through, so merlint's
project rules analysed stale build copies of whole packages and
reported ghost findings against paths under the build tree. Any
_build prefix now reads as a build directory; the skip test covers
the hyphen variant.
Library.re_exports surfaces the (re_export ...) subset of a library's
dependencies, so dead-dependency analysis can skip entries an umbrella
library hands to its dependents without importing.
The parallel-scan merge also learns about the four per-library tables
added for the link-level queries (wrapped, links_all, has_foreign_stubs
and now re_exports): partial indexes dropped them on merge, so pooled
runs saw every source library as wrapped, never -linkall and never
re-exporting, which both broke the -linkall exemption and misread
(wrapped false) libraries' unit names.
Library.modules reports source basenames, which are not a wrapped
library's link-level identity: an internal wire.ml is only addressable
as Lib__Wire, yet it leaked out under the bare name Wire and collided
with the wire codec's entry module, masking a dead dependency in
cfdp-eio. Library.unit_names reports what the linker sees instead: the
entry unit for a wrapped source library, module names for an unwrapped
one, and dune-package object names for installed libraries.
Stanzas also now carry links_all and has_foreign_stubs (on Library.t
and source_stanza), the two shapes where a dependency is live without
any module reference.
eio moves to a dev commit: it adds portable Net socket options, so a
consumer can set TCP_NODELAY and friends without linking eio.unix. Every package
that depends on eio carries the pin, since opam resolves pin-depends per package
and not from the root.
The solo5 and unikraft cross-toolchains move to wip-macos+runtime_events+lto,
one branch carrying macOS cross-build support, an in-process runtime_events ring
and a flambda compiler that defaults to link-time DCE. The unikernel arms link
with -use-lto, so the compiler has to carry those patches.
Several synopses had drifted from the code (mirage-eio is backend-agnostic,
not Solo5-only; nox-quic ships HTTP/3; squashfs writes as well as reads) and
varied in style. Describe each package by what it does rather than through
its eio, mirage or solo5 consumer, and apply the house synopsis conventions:
no leading article, spec ids in trailing parens. Also corrects a few factual
description and tag errors.
Dune accepts (vendored_dirs ...) in every dune file, marking that
file's own subdirectories; the index only read the monorepo root's
dune, so a package vendoring an upstream library under lib/ was not
seen as vendored and linters kept asking its verbatim sources to
change. Each scanned dune file now contributes its declarations,
resolved against its own directory.
nox-memtrace was a maintained fork of Jane Street's memtrace; it now lives in
ocaml-observe as observe.memtrace, so there is no external fork to track and the
capture format can evolve with the obs tooling. This drops the nox-memtrace and
nox-memtrace-eio packages and repoints every bench and opam dependency in the
tree at observe, and adds the obs SQLite sink so traces can be queried with SQL.
Across the packages monopam lint flagged: declare the libraries each
package actually uses (test-only ones scoped with-test, dev tools and
benches scoped with-dev-setup), drop the opam deps and dune (libraries)
entries nothing references, declare pin-depends for the wire and zarith
git pins so a standalone opam install can resolve them, and point the
five stale dune-project source URLs at the names sources.toml already
uses. The generated .opam files are regenerated to match.
Where monopam lint and merlint's project rules disagree, merlint wins:
[protocol] packages keep probe linked (E932), CSV-trace interop tests
keep csv (E825), and dev-scope deps carry the right filter (E943).
The few dead-lib reports that remain are monopam lint blind spots, not
real dead deps: usage reached only through an mdx README block (slirp),
through copy_files# (vz), or inside a library's own private modules
(builder), plus a tar gz dep kept live for a stub test still to write.
A platform- or config-gated stanza (an enabled_if library, a build_if
test) may legitimately have no .cmt on the analysis host. Add gated_source_files
so consumers can tell such an off-host artefact from one that was simply never
built. Record each library's gated flag and -- crucially -- carry it through the
parallel index merge (merge_library_tables dropped it, so libraries scanned in a
worker lost their gating). Cover both the enabled_if library and build_if test
paths, built through a pool, in a regression test.
A private executable (no public_name) shipped by an (install (package P))
stanza -- directly or through (copy ...) rules -- is a runtime artifact of P:
the package builds and installs it, so its libraries belong in P's runtime
depends. Follow the install -> copy -> NAME.exe chain to credit those
libraries as runtime uses, and treat such an install as attaching the dune
file to P. Fixes a false E943 on packages that codesign a copy of their
tender before installing it.
record_libraries_from_dune_package called both add_dune_package and
Package.libraries on the same content, parsing every installed
dune-package twice (and up to four times across the two install roots) --
the bulk of the project_index.scan allocation. Parse once with
Package.libraries and feed the stanzas to Lib_index.add_packages.
A dependency installed with only .cmi files records just its bare
package name, so using one of its sublibraries (pkg.sub) resolved to no
package and the dependency looked unreached -- masking real missing
deps and tripping false E943s the other way. Fall back to the
public-name prefix, which dune guarantees is the providing package.
Introduce Project_index.Path: an abstract, canonical (absolute, normalised)
filesystem path built once at construction, with structural equality, ordering
and ancestor tests. Unify the index's internal path helpers (path_key,
descendant checks) onto it. This is the shared home for the path type that
merlint's Context and the index query API will adopt, replacing the scattered
per-module path normalisation helpers.
Hoist the path-key helpers (cwd, normalise_dir, absolute_path, path_key) to the
top so every cache and comparison shares one canonical absolute string per
path, and add str_is_descendant, a plain prefix test over those strings.
is_vendored_path now canonicalises the queried path once and compares against
vendored_dirs held as canonical strings, instead of re-normalising both sides
on every ancestor via path_is_descendant.
resolve_module_files ran Sys.readdir for a directory during its traversal and
then again in the per-module existence check -- once per candidate module and
suffix -- reallocating the entry array each time. Memoise the directory listing
within the call. On a merlint self-run this drops the module-resolution
allocation from the top of the profile to zero (about a sixth of total).
The installed-package roots feed one thing: resolving a referenced
(libraries ...) use to its providing package. Eager scanning parsed every
installed dune-package up front, but a build only resolves the libraries its
source tree actually names. Replace the installed:bool flag with a three-way
installed_scan: Skip, Eager, and Referenced, which scans the source tree
first, then visits only the install dirs whose name is the first segment of a
referenced library and that no source package already provides. Dune installs
a public library under the directory named by its first segment, so this
resolves every referenced use while parsing a fraction of the switch;
excluding source packages keeps them authoritative without the install scan
clobbering them.
build scanned _opam/lib and _build/install/default/lib on every call, but
their only product is the installed library-to-package mapping that the
dependency-resolution queries consume. A caller running none of those checks
paid a serial scan of every installed dune-package -- tens of MB of parsing --
for nothing. Add ?installed (default true) to skip both roots, and parse each
source dune file once, shared between the walk's skip filter and the stanza
scan, rather than reading it twice.
Opam.field_of_path allocated the default IO buffer per call; the
project index scans hundreds of opam files of a few KB each, so the
transient buffers added up to 226 MB (11% of merlint's allocation).
Cap the reader slice at the file length.
project_index's opam_field was a private duplicate of field_of_path
(modulo catching End_of_file); delegate to the canonical one.
Libraries with a public name were registered under that name only, so
a sibling package's (libraries local_name) reference -- which dune
resolves workspace-wide -- looked unreached and E941/E943 misfired
(git-x not reaching nox-git, nox-tcp-eio not reaching nox-tcp). A
lib_alias table maps each local (name ...) to its canonical
(pkg, public_name) id and Library.in_pkg / resolve_all consult it.
Add merlint rule E334 (Redundant Variant/Field Prefixes): a variant
whose every constructor, or a record whose every field, shares a leading
underscore-delimited word has that prefix dropped, relying on OCaml's
type-directed disambiguation at call sites. A bare field that would be a
keyword takes a trailing underscore (entity_type -> type_).
Apply it across ~45 packages, annotating call sites where the bare name
is ambiguous between sibling record types.
The recent quadratics (source_presence rebuilding the file list per
file, resolve_library rescanning every library per call) were invisible
until profiled. Add invocation counters to the whole-repo builders
(source_packages, source_package_list, source_files, package_libraries,
public_library_source_files, private_module_names, test_module_names)
and surface them via Project_index.scan_stats, logged by merlint at -v.
A count orders of magnitude above the rule count flags a builder being
recomputed inside a per-file or per-stanza loop. Counters are
pre-registered and atomic so the cross-domain rule fibres only read the
table. After the earlier fixes the worst offender is source_files=3
(was per-file); package_libraries is O(rules x packages) and cheap.
resolve_public_library replaces the per-call rescan that backed
Project.Query.resolve_library: every lookup rebuilt the full source
library list and linearly searched it by public name. E606 and E615
call it per declared library per test stanza, so the cost was quadratic
in the number of libraries. Build the public-to-local map once (first
declaration wins, matching the previous List.find_opt) and cache it on
the index, mirroring the other lazy caches. Any rule resolving a public
library reference now pays O(1). E615 drops from ~890ms to ~450ms and
E606 leaves the hot list; results are unchanged.
mem_source_file rebuilt and sorted the entire monorepo source-file list
on every call, and source_presence calls it once per file. Rules that
moved from a filesystem stat to the index (E505, E705, E600, E606) then
paid O(n^2 log n): on this tree E505 alone spent 2730 ms. Cache the
membership set per include_vendored flag, mirroring the existing
file_libraries_cache, so the query is O(1) after the first build. E505
drops to ~240 ms and total project-rule time roughly halves; results are
unchanged.
ocamllex (.mll) and ocamlyacc/menhir (.mly) generate a .ml module of the
same name at build time, but the project index only scanned for .ml/.mli, so
a lexer.mll or parser.mly contributed no module to Library.files.
E610 (test without library) pairs every test_X.ml with a library module X;
relying on Library.files, it reported test_lexer.ml as having no library
module when the source tree held lexer.mll + lexer.mli but no checked-in
lexer.ml -- a false positive, since the lexer module really exists (generated
by ocamllex). Adding a stub lexer.ml would be the wrong fix.
- project-index: resolve_module_files now treats .mll and .mly as module
sources, the way dune does, so they appear in Library.files.
- merlint E610: normalise a .mll/.mly library file to the .ml module it
generates before matching it against a test file.
Add a good-generated E610 fixture (lexer.mll + parser.mly with test_lexer.ml
and test_parser.ml, no .ml sources) that stays silent. The e523 ocamllex
fixture now counts lexer.mll among the analysed files (16 -> 17).
source_status reported Missing for any file not in the project index, but a
roots-scoped Project_index.build (as merlint uses for --no-build file-list
runs) drops sibling files: a fuzz_*.mli that exists on disk read as Missing, so
E705 falsely flagged a fuzz module as lacking its interface (and E600 likewise
for test .mli) on commit.
Add Project_index.source_presence : Indexed | Unindexed | Absent -- a file not
indexed but present on disk is Unindexed, only a genuinely absent file is
Absent. merlint's Build.source_status gains a matching Skipped status
(on-disk-but-unindexed) distinct from Missing; E600/E705 treat Skipped as
present, so they only fire when the interface is truly absent.
The filesystem stat lives in the index layer, not the rules. Test covers all
three presences.
E600/E705 keyed "does this .mli exist" off the run's selected/analyse
set. When merlint analyses a subset of the project (a pre-commit hook
passing only the changed files), an existing-but-unchanged .mli is not
in that set, so the rule wrongly reported the interface as missing.
Add Project_index.mem_source_file as the source of truth for source
existence, and Build.source_status returning Compiled | Not_compiled |
Missing -- existence from the index, compilation freshness from the
.cmt/.cmti artefact. E600 and E705 now report a missing interface only
on Missing, so a present-but-unchanged or merely-uncompiled .mli no
longer trips them.
E941 now also checks libraries linked by public-executable stanzas, since opam
install builds those through @install -- a runtime dep, not a test-only one.
project-index records each source stanza's (public_name ...) so the rule can
tell public executables from private ones and tests.
The extended check surfaced real gaps, now fixed: gdocs/gsheets/gslides linked
nox-crypto-rng from a public executable but declared it :with-test;
project-index's CLI links cmdliner/eio_main/nox-json at runtime; prune links
nox-dune. New bad-exe/good-exe cram fixtures cover the executable path.
- dune-workspace: remove the second [afl] build context. The default
build no longer duplicates every target into an afl-instrumented
context (the afl env flags stay for when the profile is selected
explicitly), cutting build work.
- (mdx ...) stanzas across the tree gain an explicit [(package ...)]
so the README doc-tests are owned by the right opam package.
- merlint: new rule E922 (with cram + unit tests); E605/E727/data and
ocaml-dune / project-index follow-on.
- merlint: new rule E420 (with cram + unit tests); E415 fixtures and
docs updated, Docs module follow-on.
- ocaml-crypto/test: the per-backend test_native.ml files are replaced
by a thin test.ml that runs the shared crypto_backend_tests suite
against nox-crypto.c / nox-crypto.ocaml.
- continued doc-comment pass across dupfind, irmin, ocaml-crypto,
ocaml-crow, ocaml-cookie/cpio/csv/dsp/dune/fdir/git/gpt and others:
value docs lead with the value name, undocumented public values get
a one-line summary.
Wide doc-comment pass: value docs lead with the value name / [fn args]
form instead of an imperative sentence, and previously undocumented
public values (mostly test [suite] values and a few accessors) get a
one-line description -- satisfying the public-value-documentation
rule across the tree.
Also includes the ocaml-bloom package (lib + opam + dune-project) and
small follow-on edits to merlint rules (E305/E320/E405), monopam, and
ocaml-atp / ocaml-claude / ocaml-btree / ocaml-cfdp libraries that the
same pass touched, plus assorted eio test/dune tweaks.
Lands the in-flight work across the tree:
- project-index: package-context resolution and stanza ownership
reach the merlint rules that consume them (E5xx/E6xx/E7xx/E9xx),
plus the JSON dump binary under ocaml-project-index/bin.
- merlint: new rules E622 and E727 with cram + unit tests; the
existing rule batch updated for the new Project_index / Context /
Build surface.
- monopam: lint and opam_sync follow the project-index API changes.
- assorted eio test/dune tweaks in ocaml-atp / ocaml-cop1 / ocaml-ltp.
The usage block named eio / monopam / fmt and threw the results away
with [ignore] -- arbitrary inside this package and demonstrating
nothing. Replace with realistic, parameterised helper functions a
consuming tool would actually write: load the index, pair every
package with its libraries, and resolve a package's [(libraries ...)]
uses within its own scope (noting the package-scoping that keeps
same-named private helpers from leaking).
MDX stanza drops [eio_main] (the example no longer runs an event loop;
it parameterises over [fs] like real library code) and keeps
[eio eio.core eio.unix] for the [Eio.Path.t] type.
The usage block was a [let queries fs = ...] wrapper that existed only
to type-check without a real fs. Rewrite it as natural top-level code
([open Project_index] + [let () = Eio_main.run ...]) matching the
house style (ocaml-merlin, ocaml-block READMEs), so the example reads
like real usage and still compiles under MDX.
The MDX stanza gains [eio eio.core eio.unix] (alongside the existing
[eio_main]) so [Eio.Stdenv.fs] resolves -- the same library list other
eio-using READMEs in the repo declare.
A package's [(libraries name)] use was resolved with a global name
lookup (Library.resolve_all), so a private test helper named [helper]
in one package pulled in every same-named helper across the repo --
test_module_names then leaked unrelated packages' modules.
Adds [library_used_by pkg name]: resolve within [pkg] first
(Library.in_pkg), then fall back only to a globally-unique
public/installed library. test_module_names and the dep-resolution
paths route package-scoped uses through it; the bare global lookup
stays available as [libraries_of_name] for inventory questions.
Test test_package_context_library_use_resolution sets up two packages
with a private [helper] library each (one a test helper, one a runtime
helper) and asserts the package-context resolver picks the right one
and the other package's modules don't leak into the aggregate.
Two installed-package fixes plus tests:
- Local precedence: a locally built package now resets its installed
library state before recording (reset_installed_package_libraries),
so a library removed or renamed in the local build no longer lingers
from the _opam copy. origin reports Local. Test:
test_local_install_shadows_opam_copy.
- Plural stanzas: (tests (names a b)) / (executables (names a b)) now
record one source_stanza per name instead of dropping all but the
first, so Package.test_stanzas / executable_stanzas see every name
(E615 walks these). Tests: test_plural_tests_stanza_keeps_all_names,
test_plural_executables_stanza_keeps_all_names.
The lib now logs a warning on the precedence reset path, so it gains a
[logs] dependency.
In a multi-package dune-project, ownership was decided per dune file
from its public libraries / executables, so a [(test ... (package
host_eio))] sitting in a file alongside a [host] library was either
mis-attributed to host or dropped entirely (the ocaml-tls /
nox-tls-eio shape: eio/tests/dune named nox-tls-eio but the test never
landed on it).
ocaml-dune gains a typed accessor for a test/tests stanza's [(package
...)] field; project-index records each stanza under its own declared
owner instead of treating the whole dune file as one owner.
Adds 18 ownership / project-shape tests to test_project_index:
explicit [(package X)] on test and tests stanzas, the package-less
ambiguous case, mixed ownership within one dune file, adapter packages
(both test_modules and test_stanzas), unknown-package drop, deep
nesting, include_subdirs (recursive lib files + test exclusion),
data_only_dirs, cram .t fixtures, vendored subtree flagging, and
nested-workspace isolation.