builder, oci: split network-free packages out of the engine
opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface.
Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.
builder, oci: split network-free packages out of the engine
opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface.
Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.
builder: add a content-addressed derivation graph + plan view
`builder.graph` is a new sublibrary giving the engine a layer above its
per-step layer cache: `Derivation` keys a whole built artifact (an image, an
initramfs, a disk) by its inputs — the artifact-level companion to
`Cache_key`, which keys an individual step/layer within an image — and
`Build_graph` is the dependency DAG of those derivations annotated with each
node's key and cache status (cached / to-build / deferred), rendered as a
plain-text tree or `encode_json`.
Together they let any consumer compute a build plan — what a build would
reuse, fetch, or rebuild, with the cached artifacts' digests and sizes —
before running it, and draw it as a `--plan`/dry-run view. The code was
written against the SpaceOS build but carries no product specifics, so it
belongs in the open engine rather than the closed layer; tests cover the key
algebra and invalidation propagation, and the graph model (status, stats,
tree and JSON rendering).
builder: key engine steps as derivations, collapsing Cache_key into Derivation
The engine computed each step's cache id inline as digest(parent ^ op_key)
-- the same formula Cache_key.step_id already provided -- so the engine's
intra-image step graph was keyed by a scheme parallel to, but separate
from, the Derivation keying the artifact graph uses. Make them one model:
- Cache_key.step_id is now the key of a step Derivation over [parent; op],
so every engine cache id is a Derivation.key and an image is a chain of
step derivations -- the same content-addressed keying as Composition
artifacts. (parent_id is a step id, op_key an op identity like "FROM" or
a run/copy key; the two are disjoint string spaces, so the derivation's
sorted inputs never swap-collide.)
- The engine's FROM, RUN and COPY id sites route through Cache_key.step_id
instead of re-inlining the digest, so there is one keying, single-sourced.
This re-keys the step/snapshot cache once (a derived cache, always safe to
drop): the new ids differ from the old digest-based ones, so a warm cache
rebuilds each step a single time. Caching behaviour is unchanged -- same
inputs still yield the same id and reuse -- which the build, export and
cache tests confirm.
builder: a multi-artifact composition engine for builder.graph
builder.graph shipped the IR only (Derivation + Build_graph): building a
graph of artifacts - images -> an init -> disks, each keyed on a prior
artifact's built output - was hand-wired by each consumer, which built the
Derivation nodes and called realise in dependency order itself, then walked
the same graph a second time to render the plan.
Add the two generic pieces so a consumer only names its nodes:
- Composition: a typed, JSON-serialisable multi-artifact spec. Each artifact
names its rule, its source leaves, and its dependency edges (Key on a
dependency's derivation key, or Output on a dependency's realised output
digest). It validates (unique names, known deps, acyclic), topologically
orders, and lowers to a graph of Derivations.
- Driver.Make (S): realise a whole composition in one call - topological
order, dependencies first, threading each realised output into its
dependents' build - over the existing Derivation.store/realise primitive,
and render the Build_graph plan view from the same composition so the plan
cannot drift from the build. An Output dependent is reported deferred until
its dependency is cached.
Event adds ocaml-probe instrumentation: spans around the whole walk and each
artifact, build/hit point events, a realise-totals counter, and a plan
summary event, all decoded and locked down by a runtime_events test.
The tests assert exact rebuild sets, including across an Output edge where a
dependent re-keys on a dependency's realised output - the case the bare
Derivation tests never exercised.
builder: add a content-addressed derivation graph + plan view
`builder.graph` is a new sublibrary giving the engine a layer above its
per-step layer cache: `Derivation` keys a whole built artifact (an image, an
initramfs, a disk) by its inputs — the artifact-level companion to
`Cache_key`, which keys an individual step/layer within an image — and
`Build_graph` is the dependency DAG of those derivations annotated with each
node's key and cache status (cached / to-build / deferred), rendered as a
plain-text tree or `encode_json`.
Together they let any consumer compute a build plan — what a build would
reuse, fetch, or rebuild, with the cached artifacts' digests and sizes —
before running it, and draw it as a `--plan`/dry-run view. The code was
written against the SpaceOS build but carries no product specifics, so it
belongs in the open engine rather than the closed layer; tests cover the key
algebra and invalidation propagation, and the graph model (status, stats,
tree and JSON rendering).
builder: a multi-artifact composition engine for builder.graph
builder.graph shipped the IR only (Derivation + Build_graph): building a
graph of artifacts - images -> an init -> disks, each keyed on a prior
artifact's built output - was hand-wired by each consumer, which built the
Derivation nodes and called realise in dependency order itself, then walked
the same graph a second time to render the plan.
Add the two generic pieces so a consumer only names its nodes:
- Composition: a typed, JSON-serialisable multi-artifact spec. Each artifact
names its rule, its source leaves, and its dependency edges (Key on a
dependency's derivation key, or Output on a dependency's realised output
digest). It validates (unique names, known deps, acyclic), topologically
orders, and lowers to a graph of Derivations.
- Driver.Make (S): realise a whole composition in one call - topological
order, dependencies first, threading each realised output into its
dependents' build - over the existing Derivation.store/realise primitive,
and render the Build_graph plan view from the same composition so the plan
cannot drift from the build. An Output dependent is reported deferred until
its dependency is cached.
Event adds ocaml-probe instrumentation: spans around the whole walk and each
artifact, build/hit point events, a realise-totals counter, and a plan
summary event, all decoded and locked down by a runtime_events test.
The tests assert exact rebuild sets, including across an Output edge where a
dependent re-keys on a dependency's realised output - the case the bare
Derivation tests never exercised.
builder: a multi-artifact composition engine for builder.graph
builder.graph shipped the IR only (Derivation + Build_graph): building a
graph of artifacts - images -> an init -> disks, each keyed on a prior
artifact's built output - was hand-wired by each consumer, which built the
Derivation nodes and called realise in dependency order itself, then walked
the same graph a second time to render the plan.
Add the two generic pieces so a consumer only names its nodes:
- Composition: a typed, JSON-serialisable multi-artifact spec. Each artifact
names its rule, its source leaves, and its dependency edges (Key on a
dependency's derivation key, or Output on a dependency's realised output
digest). It validates (unique names, known deps, acyclic), topologically
orders, and lowers to a graph of Derivations.
- Driver.Make (S): realise a whole composition in one call - topological
order, dependencies first, threading each realised output into its
dependents' build - over the existing Derivation.store/realise primitive,
and render the Build_graph plan view from the same composition so the plan
cannot drift from the build. An Output dependent is reported deferred until
its dependency is cached.
Event adds ocaml-probe instrumentation: spans around the whole walk and each
artifact, build/hit point events, a realise-totals counter, and a plan
summary event, all decoded and locked down by a runtime_events test.
The tests assert exact rebuild sets, including across an Output edge where a
dependent re-keys on a dependency's realised output - the case the bare
Derivation tests never exercised.
builder: idmapped-mount rootfs for multi-tenant runc
Present the build snapshot to a user-namespaced RUN step through an
idmapped mount instead of recursively chowning it. open_tree plus
mount_setattr(MOUNT_ATTR_IDMAP) against a fresh user namespace maps the
on-disk ids into the container range at the VFS layer: ownership shifts
with no tree walk and no on-disk change, and per-file ownership survives
where the chown round-trip flattened it onto the mapped root. The OCI
runtime spec cannot idmap the rootfs itself, so the builder creates the
mount (new Builder.Idmap) and points runc root.path at it, the way
containerd's overlay snapshotter does. A kernel without mount_setattr,
or a filesystem that cannot idmap, falls back to the chown.
Multi-tenant RUN secrets are chowned to the mapped container-root id so
the namespaced step can read a mode-0600 secret.
Add container-escape hardening regressions: container root never maps to
host root, noNewPrivileges, a seccomp profile, dropped CAP_MKNOD and
CAP_AUDIT_WRITE, and a read-only secret bind. Verified end to end on
Linux via docker-test.sh (Alpine/musl, privileged).
Rides along with an in-progress network-spec extraction (Builder_spec
.Network) interleaved in runc_runtime.ml and the test runner; the two
cannot be separated without breaking the build.
builder: export one OCI layer per build step, not a squashed layer
OCI export tarred the final merged snapshot as a single layer, so images
shared no base layer, every push re-sent everything, and there was no
per-step dedup. Now each build step contributes its own layer.
New Builder.Layers turns the snapshot chain into per-step layer
directories, ready for Oci.Build.Make.layer (an OCI whiteout is just a
.wh. file, so the tar path is unchanged):
- copy backend (macOS, local/tests): diff consecutive merged snapshots,
bringing changed files in by hardlink (no data copied) and recording
deletions as .wh. markers;
- overlay backend (Linux VM, production): re-export each step's upper
directory directly -- the upper already is the diff -- translating
overlayfs char-device(0,0) whiteouts to OCI .wh., as BuildKit does
(util/overlay). Opaque dirs and redirect_dir are not yet translated.
The store walks its Db parent links to recover the step chain on both
backends; Export.of_chain builds a layer per directory and chains them,
and the image record now carries `layers` (bottom-first) instead of a
single `layer`. run_oci_layout wires it together.
Tested on the copy backend end to end (fake runtime): a two-RUN build
yields three layers -- base, then one per step -- each carrying only its
own change. The overlay upper translation is verified in CI (no
overlayfs/char-devices on macOS).
runc: parse the full OCI memory block, swappiness as uint64
The resources codec decoded only memory.limit and skip_unknown'd the
rest of linux.resources.memory, so every other field was silently
dropped on a decode/encode round-trip. The lost field that matters in
practice is swappiness: older oci-runtime-tool wrote its "unset"
sentinel as the OCI uint64 2^64-1, and hundreds of real config.json
bundles (e.g. jessfraz/containers) carry {"swappiness":
18446744073709551615} verbatim. nox-json parses that exactly (it never
overflowed our decoder, unlike the rlimit case), but skip_unknown threw
it away.
Model the block as Resources.Memory.t. The byte controls (limit,
reservation, swap, kernel, kernelTCP) decode with Json.Codec.int64 so a
full-width count cannot overflow OCaml's 63-bit int; swappiness decodes
with Json.Codec.uint64, held as the int64 two's-complement bit pattern
(2^64-1 is -1L), mirroring the rlimit and seccomp-arg codecs. The three
flags are tri-state bool option. Resources.t now carries
memory : Memory.t option; multi_tenant and the ocaml-builder call sites
build a Memory.t. The two Seccomp [kernel] argument references become
[~kernel] so the prose does not collide with the new field name.
*: migrate every digestif consumer to ocaml-hash
Tree-wide mechanical migration (Digestif.* -> Hash.*, dune and opam
deps digestif -> ocaml-hash) across 43 packages, dropping digestif
from the dependency graph entirely (root.opam no longer lists it).
Blob digests, git object ids, KEX exchange hashes, JWT/COSE/X.509
signatures and every other SHA-256 user now ride the hardware path on
CPUs that have it.
Five libraries had an internal hash.ml that would shadow the Hash
library inside their own compilation units; their internal module is
renamed by what it actually is, with the public alias kept so
consumers (Irmin.Hash, Git.Hash, Tpm.Hash, ...) are unchanged:
- irmin, ocaml-git: Object_hash (content-addressed object ids)
- ocaml-scitt, ocaml-sigstore: Hash_alg (algorithm registries)
- ocaml-tpm: Hashing (digest/HMAC/KDFa over TPM algorithms)
Semantic adjustments the rename surfaced:
- x509: the OID<->hash tables lose their arms for algorithms the
hash library does not carry (SHA-3, BLAKE2, RIPEMD, WHIRLPOOL);
Hash.kind is closed over the six supported algorithms so the
catch-all arms were dead.
- sigstore: bundles declaring SHA-3 digests now fail verification
loudly (Invalid_argument) instead of mapping to an unimplemented
kind; the protobuf enum still parses.
- x509/jwt: local [let module Hash = (val module_of ...)] bindings
renamed to H -- they would shadow the library inside their scope.
- irmin re-exports the raw SHA-1/SHA-256 implementations through
Object_hash so its schema instantiations need no second name for
the library under the public Hash alias.
SSH throughput is unchanged (170/537/460 MB/s socketpair / 2-domain /
2-process, same as the digestif baseline): the negotiated transport
cipher is chacha20-poly1305, whose MAC is Poly1305, so SHA-256 only
runs in the handshake. SSH's data-path ceiling remains nox-crypto's
scalar ChaCha20/Poly1305.
All migrated package suites pass (ssh, tls, x509, paseto, jwt, kdf,
git, tpm, scitt, sigstore, oci, srp, spake2, cose, bpsec, hash).
gazagnaire.org