ocaml-builder/lib/ at main · gazagnaire.org/ocaml-builder · Tangled

gazagnaire.org / ocaml-builder

0

OCI-oriented build engine

0

ocaml-builder / lib

5 folders 29 files

at main

Thomas Gazagnaire linkedin: stop accepting session cookies on the command line 12hrs ago

builder: support Docker context ignore rules Expose Docker-compatible .dockerignore matching from builder.dockerfile so callers can apply the same context boundary before hashing or transfer. Cover anchoring, double-star patterns, last-match negation, excluded-directory traversal, and malformed patterns.

3 days ago

builder: digest derivation key parts directly, no concat Derivation.key hashed String.concat "\x00" (rule_version :: sorted inputs), allocating the joined string per key (obs: 127K allocs, ~17 MB on the graph bench, proportional to the input count). Feed the header and the NUL-separated inputs straight into digesti_string instead; the digested byte stream is identical to the concatenation, so the key is byte-identical (the cache_key tests, which pin key stability, pass). Bench allocation 168 MB -> 160 MB.

4 weeks ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

builder: preserve Docker runtime metadata Lower EXPOSE and VOLUME into resolved plan state, merge declarations inherited from base images, and export both fields in the OCI image configuration.

4 days ago

linkedin: stop accepting session cookies on the command line li_at and JSESSIONID are a live LinkedIn login in one string. As options they were visible in ps(1) to every process on the machine and were written to shell history, and neither can be taken back afterwards. The environment variables stay and are now the only way to pass one by hand. Not private either, but not world-readable on Linux, not persisted to disk, and already the documented path -- the help text only ever told people to export them. cmdliner reads and documents a variable given Cmd.Env.info with no flag names, so nothing else changes. Same class as the --password options removed from ocaml-atp.

12 hours ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

builder_clone_stubs.c

builder: fix spawn leak-on-raise and reject NUL in path stubs builder_spawn_session leaked the copied executable/cwd/argv when a later copy hit ENOMEM and raised; the copy helpers now return NULL so the caller frees every copy and raises once. The clone, statvfs and xattr stubs passed String_val paths to syscalls with no embedded-NUL check, which silently truncates to a different path; add a shared reject_nul guard mirroring pid1.

2 weeks ago

builder_idmap_stubs.c

builder: release the runtime lock across the idmap userns handshake make_userns_fd blocks in pipe reads, /proc map writes and waitpid while the child unshares its user namespace; doing that with the runtime lock held stalls every OCaml domain for the window. Fork still happens under the lock so the child inherits a consistent runtime image, then the parent releases it for the blocking region, mirroring the realpath pattern in self_exe_stubs.c. errno is restored after reacquisition. Compile-checked on macOS only: the docker-test lane is currently broken by an unrelated opam-overlay build failure (tw vs new cascade/css).

1 month ago

builder_reject_nul.h

builder: fix spawn leak-on-raise and reject NUL in path stubs builder_spawn_session leaked the copied executable/cwd/argv when a later copy hit ENOMEM and raised; the copy helpers now return NULL so the caller frees every copy and raises once. The clone, statvfs and xattr stubs passed String_val paths to syscalls with no embedded-NUL check, which silently truncates to a different path; add a shared reject_nul guard mirroring pid1.

2 weeks ago

builder_statvfs_stubs.c

linkedin: stop accepting session cookies on the command line li_at and JSESSIONID are a live LinkedIn login in one string. As options they were visible in ps(1) to every process on the machine and were written to shell history, and neither can be taken back afterwards. The environment variables stay and are now the only way to pass one by hand. Not private either, but not world-readable on Linux, not persisted to disk, and already the documented path -- the help text only ever told people to export them. cmdliner reads and documents a variable given Cmd.Env.info with no flag names, so nothing else changes. Same class as the --password options removed from ocaml-atp.

12 hours ago

builder_xattr_stubs.c

builder: fix spawn leak-on-raise and reject NUL in path stubs builder_spawn_session leaked the copied executable/cwd/argv when a later copy hit ENOMEM and raised; the copy helpers now return NULL so the caller frees every copy and raises once. The clone, statvfs and xattr stubs passed String_val paths to syscalls with no embedded-NUL check, which silently truncates to a different path; add a shared reject_nul guard mirroring pid1.

2 weeks ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

repo: remove the redundant lchown wrapper Eio already provides non-following path ownership changes. Keeping a package around a single direct call adds metadata and API surface without adding portability or behavior.

1 week ago

linkedin: stop accepting session cookies on the command line li_at and JSESSIONID are a live LinkedIn login in one string. As options they were visible in ps(1) to every process on the machine and were written to shell history, and neither can be taken back afterwards. The environment variables stay and are now the only way to pass one by hand. Not private either, but not world-readable on Linux, not persisted to disk, and already the documented path -- the help text only ever told people to export them. cmdliner reads and documents a variable given Cmd.Env.info with no flag names, so nothing else changes. Same class as the --password options removed from ocaml-atp.

12 hours ago

builder: reclaim store space with prune and free-space GC The snapshot store grew without bound, so a build eventually died on ENOSPC with no way to reclaim space short of deleting the whole cache disk. Add a `buildr prune` subcommand over the existing Runner.prune (--keep/--max-age), plus gc_until_free, which sheds least-recently-used leaf snapshots until a target percent of the store filesystem is free (--target-free) and is a no-op when the target is already met, so a caller can run it cheaply before every build. It only removes childless leaves and does not touch the OCI blob cache, so it never drops a base image a build is about to use.

2 weeks ago

builder: preserve Docker runtime metadata Lower EXPOSE and VOLUME into resolved plan state, merge declarations inherited from base images, and export both fields in the OCI image configuration.

4 days ago

builder: preserve Docker runtime metadata Lower EXPOSE and VOLUME into resolved plan state, merge declarations inherited from base images, and export both fields in the OCI image configuration.

4 days ago

builder: idmapped-mount rootfs for multi-tenant runc Present the build snapshot to a user-namespaced RUN step through an idmapped mount instead of recursively chowning it. open_tree plus mount_setattr(MOUNT_ATTR_IDMAP) against a fresh user namespace maps the on-disk ids into the container range at the VFS layer: ownership shifts with no tree walk and no on-disk change, and per-file ownership survives where the chown round-trip flattened it onto the mapped root. The OCI runtime spec cannot idmap the rootfs itself, so the builder creates the mount (new Builder.Idmap) and points runc root.path at it, the way containerd's overlay snapshotter does. A kernel without mount_setattr, or a filesystem that cannot idmap, falls back to the chown. Multi-tenant RUN secrets are chowned to the mapped container-root id so the namespaced step can read a mode-0600 secret. Add container-escape hardening regressions: container root never maps to host root, noNewPrivileges, a seccomp profile, dropped CAP_MKNOD and CAP_AUDIT_WRITE, and a read-only secret bind. Verified end to end on Linux via docker-test.sh (Alpine/musl, privileged). Rides along with an in-progress network-spec extraction (Builder_spec .Network) interleaved in runc_runtime.ml and the test runner; the two cannot be separated without breaking the build.

1 month ago

builder: idmapped-mount rootfs for multi-tenant runc Present the build snapshot to a user-namespaced RUN step through an idmapped mount instead of recursively chowning it. open_tree plus mount_setattr(MOUNT_ATTR_IDMAP) against a fresh user namespace maps the on-disk ids into the container range at the VFS layer: ownership shifts with no tree walk and no on-disk change, and per-file ownership survives where the chown round-trip flattened it onto the mapped root. The OCI runtime spec cannot idmap the rootfs itself, so the builder creates the mount (new Builder.Idmap) and points runc root.path at it, the way containerd's overlay snapshotter does. A kernel without mount_setattr, or a filesystem that cannot idmap, falls back to the chown. Multi-tenant RUN secrets are chowned to the mapped container-root id so the namespaced step can read a mode-0600 secret. Add container-escape hardening regressions: container root never maps to host root, noNewPrivileges, a seccomp profile, dropped CAP_MKNOD and CAP_AUDIT_WRITE, and a read-only secret bind. Verified end to end on Linux via docker-test.sh (Alpine/musl, privileged). Rides along with an in-progress network-spec extraction (Builder_spec .Network) interleaved in runc_runtime.ml and the test runner; the two cannot be separated without breaking the build.

1 month ago

builder: compare filesystem kinds explicitly

5 days ago

docs: update MDX examples to the v constructor after create/make rename The repo-wide create/make -> v rename (E332) updated the libraries but left doc-comment and README MDX examples calling the old .create/.make names, so dune test failed across ~17 packages (dsp, dune, gpt, jwt, loc, mal, mbr, monitor, prometheus, qemu, requests h2, runc, scc, sqlite, srp, sse, stripe). Point each example at the renamed constructor (.v). Also fixes the stale doc references the Store de-functorization left: {!Oci.Build.Make.*} -> {!Oci.Build.*} and {!Oci_server.Store.Memory} -> {!Oci_server.Store.in_memory}. Changelog and skill prose mentioning the historical names are left as-is.

1 month ago

ocaml-builder: code-quality cleanup (value-name doc comments, extracted helpers)

1 month ago

ocaml-builder: code-quality cleanup (value-name doc comments, extracted helpers)

1 month ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

macos_spawn_stubs.c

builder: fix spawn leak-on-raise and reject NUL in path stubs builder_spawn_session leaked the copied executable/cwd/argv when a later copy hit ENOMEM and raised; the copy helpers now return NULL so the caller frees every copy and raises once. The clone, statvfs and xattr stubs passed String_val paths to syscalls with no embedded-NUL check, which silently truncates to a different path; add a shared reject_nul guard mirroring pid1.

2 weeks ago

runc_runtime.ml

builder: disable apt's redundant sandbox in the userns RUN jail apt drops privileges to the unprivileged _apt user to fetch and verify a repository's package indexes, but that privilege-drop does not survive the runc backend's user namespace: apt rejects every repository signature as invalid and no Debian/Ubuntu image can apt-get install. The RUN step already runs in a user-namespace + seccomp + capability jail, so apt's own sandbox is redundant here. Bind an apt.conf.d drop-in (APT::Sandbox::User root) into a networked RUN, ephemeral like the resolv.conf bind so it is not committed to the built layer, and only when the image ships an apt.conf.d.

2 weeks ago

runc_runtime.mli

builder, oci: split network-free packages out of the engine opam resolves packages whole, so depending on any sub-library of builder or oci pulled the package's entire dependency set: runc, seatbelt and sqlite from the builder engine, and the requests/TLS/HTTP registry client from oci. A consumer that needs only the build model or the local OCI layer cache inherited the whole execution and network surface. Extract the network-free cores into their own packages so they can be consumed without that surface: oci-layer (spec, local store, layer build), builder-spec (spec, plan, runtime, backend), builder-graph (the build DAG) and builder-opam (the opam binary-layer cache and solver). The engine and registry client now depend on these.

1 month ago

builder: reclaim store space with prune and free-space GC The snapshot store grew without bound, so a build eventually died on ENOSPC with no way to reclaim space short of deleting the whole cache disk. Add a `buildr prune` subcommand over the existing Runner.prune (--keep/--max-age), plus gc_until_free, which sheds least-recently-used leaf snapshots until a target percent of the store filesystem is free (--target-free) and is a no-op when the target is already met, so a caller can run it cheaply before every build. It only removes childless leaves and does not touch the OCI blob cache, so it never drops a base image a build is about to use.

2 weeks ago

builder: reclaim store space with prune and free-space GC The snapshot store grew without bound, so a build eventually died on ENOSPC with no way to reclaim space short of deleting the whole cache disk. Add a `buildr prune` subcommand over the existing Runner.prune (--keep/--max-age), plus gc_until_free, which sheds least-recently-used leaf snapshots until a target percent of the store filesystem is free (--target-free) and is a no-op when the target is already met, so a caller can run it cheaply before every build. It only removes childless leaves and does not touch the OCI blob cache, so it never drops a base image a build is about to use.

2 weeks ago

builder: emit opaque whiteouts (.wh..wh..opq) from overlay export of_overlay_upper translated overlayfs deletions (char device 0,0) into .wh.<name> markers but left opaque directories untranslated -- a documented gap that meant an overlay step which made a directory opaque exported a layer the import side would merge wrong (lower-layer entries surviving under a directory the step meant to replace). A directory now carrying the overlay opaque attribute emits a .wh..wh..opq marker inside the corresponding layer directory, the OCI equivalent the checkout side already consumes. The attribute is read from both the trusted namespace (privileged overlay mount) and the user namespace (userxattr / rootless mount), matching how overlayfs records it. Adds a small Xattr primitive (portable get/set C stub: l{get,set}xattr on Linux, {get,set}xattr + XATTR_NOFOLLOW on macOS) since the repo had no extended-attribute access, and the first test covering of_overlay_upper: regular/dir/symlink translation plus the opaque marker, planting user.overlay.opaque so it runs unprivileged. redirect_dir stays untranslated, as in BuildKit.

1 month ago

builder: emit opaque whiteouts (.wh..wh..opq) from overlay export of_overlay_upper translated overlayfs deletions (char device 0,0) into .wh.<name> markers but left opaque directories untranslated -- a documented gap that meant an overlay step which made a directory opaque exported a layer the import side would merge wrong (lower-layer entries surviving under a directory the step meant to replace). A directory now carrying the overlay opaque attribute emits a .wh..wh..opq marker inside the corresponding layer directory, the OCI equivalent the checkout side already consumes. The attribute is read from both the trusted namespace (privileged overlay mount) and the user namespace (userxattr / rootless mount), matching how overlayfs records it. Adds a small Xattr primitive (portable get/set C stub: l{get,set}xattr on Linux, {get,set}xattr + XATTR_NOFOLLOW on macOS) since the repo had no extended-attribute access, and the first test covering of_overlay_upper: regular/dir/symlink translation plus the opaque marker, planting user.overlay.opaque so it runs unprivileged. redirect_dir stays untranslated, as in BuildKit.

1 month ago