xeiaso.net / objgit

0

Git backed by object storage because you can't stop me

git object-storage kefka

0

objgit / internal / s3fs

1 folder 24 files

at main

Xe Iaso fix: further optimize pack caching 1mo ago

feat(s3fs): add optional Unix-metadata storage as S3 user metadata Store POSIX file attributes (uid, gid, mode, mtime) as x-amz-meta-* headers on S3 objects so POSIX filesystems layered on Tigris can round-trip ownership and permissions instead of always reporting mode 0666 with no owner. The feature is opt-in and off by default: NewS3FS takes functional options, and without WithUnixMetadata the filesystem behaves exactly as before. Callers opt in by passing WithUnixMetadata(uid, gid, umask) and resolve any name strings to numeric IDs themselves. New unixmeta package implements Encode/Decode and PosixMode/GoFileMode conversion; the wire format is documented in docs/reference. Assisted-by: Claude Opus 4.7 via Claude Code Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

fix: further optimize pack caching Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

chore: import s3fs, adapt to billy v6 Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

chore: import s3fs, adapt to billy v6 Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

feat(s3fs): cache immutable pack files on local disk to fix clone hangs Serving a clone via upload-pack's delta-compression re-reads pack objects thousands of times. Commit 76bb55d made pack reads lazy to avoid RAM buffering; this causes each access to issue a fresh S3 GetObject, making real-repo clones never finish. Implement PackCache: immutable pack-dir files (.pack/.idx/.rev) download once to a local temp file (LRU-bounded by -pack-cache-bytes, default 2GB) and all reads serve from local disk. Results on real Tigris bucket: - objgit (318 objects, 200KB pack): 8500+ GetObjects → 10, 7s - kefka (197 commits, 19.6MB pack): hang → 14s, full clone The cache is wired into OpenFile and shared across Chroot children like the listing cache. Add flags: -pack-cache-bytes, -pack-cache-dir. See docs/plans/pack-temp-file-cache.md for design. Assisted-by: Claude Opus 4.8 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

perf(s3fs): register chroot roots as recursive subtree prefixes Make each repository's Chroot root register itself as a recursive subtree prefix with the listing cache. This revives the subtree-scan optimization (previously dead in chrooted deployments) so the entire repo is answered from one delimiter-less S3 ListObjectsV2 instead of a list per folder. Eliminates ~256 loose-object negative-lookup lists per clone and collapses the background warmer from O(folders) lists/tick to O(1). Changes: - listingcache.go: roots field is now atomic.Pointer[[]string] with registerRoot() method for lock-free runtime registration - chroot.go: calls registerRoot() when creating a chroot - listingcache_test.go: updated TestListingCacheChrootShares and added TestListingCacheChrootSubtreeCollapsesLooseLookups regression guard Verification: all tests pass, including race detector and cmd/objgitd protocol tests. Plan: docs/plans/why-does-objgit-do-zazzy-bentley.md Assisted-by: Claude Opus 4.8 via Claude Code

1 month ago

feat(protocol): add git upload-pack support via git protocol Implement native git protocol support in objgitd daemon. Enables clients to use standard git commands (git clone, git fetch) over the git protocol instead of requiring HTTP. Includes proper request handling and response formatting for upload-pack operations. Assisted-by: Claude Haiku 4.5 via Claude Code Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

feat: keep pushed packs whole Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

feat(s3fs): implement lazy, range-based reads with read-ahead window Replace eager GetObject+ReadAll buffering with lazy on-demand fetching. Files now stream their bodies only when read, and ReadAt uses S3 Range requests backed by a 1 MiB read-ahead window. This prevents whole-object buffering in RAM (critical for large pack files since feat: keep pushed packs whole) and reduces redundant HeadObject calls by deriving metadata from the GetObject response. Sequential Read opens a body at Open or on first read (depending on cache hit); body streams on demand and reopens after Seek. ReadAt issues Range requests; nearby offsets are served from the window without new S3 calls. The redundant HeadObject is dropped from the main path; only Stat on a never-read handle falls back to it (effectively unreachable in normal flow). The public billy.File API is unchanged; this is a performance and memory optimization. All three transport protocols (HTTP, git://, SSH) exercise real pack/idx reads through go-git's FSObject and Scanner; the full test suite passes. See docs/plans/s3fs-lazy-read.md for the design. Assisted-by: Claude Haiku 4.5 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

file_lazy_test.go

feat(s3fs): implement lazy, range-based reads with read-ahead window Replace eager GetObject+ReadAll buffering with lazy on-demand fetching. Files now stream their bodies only when read, and ReadAt uses S3 Range requests backed by a 1 MiB read-ahead window. This prevents whole-object buffering in RAM (critical for large pack files since feat: keep pushed packs whole) and reduces redundant HeadObject calls by deriving metadata from the GetObject response. Sequential Read opens a body at Open or on first read (depending on cache hit); body streams on demand and reopens after Seek. ReadAt issues Range requests; nearby offsets are served from the window without new S3 calls. The redundant HeadObject is dropped from the main path; only Stat on a never-read handle falls back to it (effectively unreachable in normal flow). The public billy.File API is unchanged; this is a performance and memory optimization. All three transport protocols (HTTP, git://, SSH) exercise real pack/idx reads through go-git's FSObject and Scanner; the full test suite passes. See docs/plans/s3fs-lazy-read.md for the design. Assisted-by: Claude Haiku 4.5 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

feat(s3fs): implement lazy, range-based reads with read-ahead window Replace eager GetObject+ReadAll buffering with lazy on-demand fetching. Files now stream their bodies only when read, and ReadAt uses S3 Range requests backed by a 1 MiB read-ahead window. This prevents whole-object buffering in RAM (critical for large pack files since feat: keep pushed packs whole) and reduces redundant HeadObject calls by deriving metadata from the GetObject response. Sequential Read opens a body at Open or on first read (depending on cache hit); body streams on demand and reopens after Seek. ReadAt issues Range requests; nearby offsets are served from the window without new S3 calls. The redundant HeadObject is dropped from the main path; only Stat on a never-read handle falls back to it (effectively unreachable in normal flow). The public billy.File API is unchanged; this is a performance and memory optimization. All three transport protocols (HTTP, git://, SSH) exercise real pack/idx reads through go-git's FSObject and Scanner; the full test suite passes. See docs/plans/s3fs-lazy-read.md for the design. Assisted-by: Claude Haiku 4.5 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

feat(s3fs): add optional Unix-metadata storage as S3 user metadata Store POSIX file attributes (uid, gid, mode, mtime) as x-amz-meta-* headers on S3 objects so POSIX filesystems layered on Tigris can round-trip ownership and permissions instead of always reporting mode 0666 with no owner. The feature is opt-in and off by default: NewS3FS takes functional options, and without WithUnixMetadata the filesystem behaves exactly as before. Callers opt in by passing WithUnixMetadata(uid, gid, umask) and resolve any name strings to numeric IDs themselves. New unixmeta package implements Encode/Decode and PosixMode/GoFileMode conversion; the wire format is documented in docs/reference. Assisted-by: Claude Opus 4.7 via Claude Code Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

feat(s3fs): cache immutable pack files on local disk to fix clone hangs Serving a clone via upload-pack's delta-compression re-reads pack objects thousands of times. Commit 76bb55d made pack reads lazy to avoid RAM buffering; this causes each access to issue a fresh S3 GetObject, making real-repo clones never finish. Implement PackCache: immutable pack-dir files (.pack/.idx/.rev) download once to a local temp file (LRU-bounded by -pack-cache-bytes, default 2GB) and all reads serve from local disk. Results on real Tigris bucket: - objgit (318 objects, 200KB pack): 8500+ GetObjects → 10, 7s - kefka (197 commits, 19.6MB pack): hang → 14s, full clone The cache is wired into OpenFile and shared across Chroot children like the listing cache. Add flags: -pack-cache-bytes, -pack-cache-dir. See docs/plans/pack-temp-file-cache.md for design. Assisted-by: Claude Opus 4.8 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

feat(protocol): add git upload-pack support via git protocol Implement native git protocol support in objgitd daemon. Enables clients to use standard git commands (git clone, git fetch) over the git protocol instead of requiring HTTP. Includes proper request handling and response formatting for upload-pack operations. Assisted-by: Claude Haiku 4.5 via Claude Code Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

listingcache.go

perf(s3fs): register chroot roots as recursive subtree prefixes Make each repository's Chroot root register itself as a recursive subtree prefix with the listing cache. This revives the subtree-scan optimization (previously dead in chrooted deployments) so the entire repo is answered from one delimiter-less S3 ListObjectsV2 instead of a list per folder. Eliminates ~256 loose-object negative-lookup lists per clone and collapses the background warmer from O(folders) lists/tick to O(1). Changes: - listingcache.go: roots field is now atomic.Pointer[[]string] with registerRoot() method for lock-free runtime registration - chroot.go: calls registerRoot() when creating a chroot - listingcache_test.go: updated TestListingCacheChrootShares and added TestListingCacheChrootSubtreeCollapsesLooseLookups regression guard Verification: all tests pass, including race detector and cmd/objgitd protocol tests. Plan: docs/plans/why-does-objgit-do-zazzy-bentley.md Assisted-by: Claude Opus 4.8 via Claude Code

1 month ago

listingcache_test.go

perf(s3fs): register chroot roots as recursive subtree prefixes Make each repository's Chroot root register itself as a recursive subtree prefix with the listing cache. This revives the subtree-scan optimization (previously dead in chrooted deployments) so the entire repo is answered from one delimiter-less S3 ListObjectsV2 instead of a list per folder. Eliminates ~256 loose-object negative-lookup lists per clone and collapses the background warmer from O(folders) lists/tick to O(1). Changes: - listingcache.go: roots field is now atomic.Pointer[[]string] with registerRoot() method for lock-free runtime registration - chroot.go: calls registerRoot() when creating a chroot - listingcache_test.go: updated TestListingCacheChrootShares and added TestListingCacheChrootSubtreeCollapsesLooseLookups regression guard Verification: all tests pass, including race detector and cmd/objgitd protocol tests. Plan: docs/plans/why-does-objgit-do-zazzy-bentley.md Assisted-by: Claude Opus 4.8 via Claude Code

1 month ago

feat(metrics): add Prometheus instrumentation and a /metrics endpoint Instrument the three things an operator needs to see and serve them on a dedicated listener (-metrics-bind, default :9090; empty disables): - s3fs: count + latency per S3 API operation (GetObject, PutObject, …) via a process-level observer set with s3fs.SetMetricsObserver, keeping s3fs free of any prometheus import. - git ops: count + latency + in-flight gauge per protocol+service+status, wrapped once per transport handler (http/git/ssh). - auth: count + latency per transport+operation+decision, routed through a new (*daemon).authorize chokepoint that all three transports now call. Extras: push-hook runs (ok/error/timeout) with latency, repos auto-created counter, and the default registry's Go-runtime + process collectors. Per the operator's call there is no repo label (unbounded cardinality); git ops are keyed by protocol+service+status only. All vectors live in internal/metrics via promauto on the default registry, exposed by promhttp.Handler under the existing errgroup Serve/Shutdown idiom.

1 month ago

fix: further optimize pack caching Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

packcache_test.go

fix: further optimize pack caching Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

fix(s3fs): harden S3 client transport to fail fast on stale connections The AWS SDK's default transport (IdleConnTimeout: 90s, no ResponseHeaderTimeout) allows reused keep-alive connections to Tigris to sit stale and never respond. With context.TODO() (no timeout), ReadObject hangs forever instead of failing and retrying. Inject a hardened HTTP client (ResponseHeaderTimeout: 30s, IdleConnTimeout: 30s) into every S3 call via per-call options, so stale connections error quickly and the retryer recovers. Relates to: lazy-read clone hangs (independently useful). Assisted-by: Claude Opus 4.8 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

resilient_test.go

fix(s3fs): harden S3 client transport to fail fast on stale connections The AWS SDK's default transport (IdleConnTimeout: 90s, no ResponseHeaderTimeout) allows reused keep-alive connections to Tigris to sit stale and never respond. With context.TODO() (no timeout), ReadObject hangs forever instead of failing and retrying. Inject a hardened HTTP client (ResponseHeaderTimeout: 30s, IdleConnTimeout: 30s) into every S3 call via per-call options, so stale connections error quickly and the retryer recovers. Relates to: lazy-read clone hangs (independently useful). Assisted-by: Claude Opus 4.8 via claude.ai Signed-off-by: Xe Iaso <me@xeiaso.net>

1 month ago

chore: import s3fs, adapt to billy v6 Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

refactor(protocol): rename daemon files to git_protocol Move daemon.go and daemon_test.go to git_protocol.go and git_protocol_test.go to better reflect their purpose. No logic changes. Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

refactor(protocol): rename daemon files to git_protocol Move daemon.go and daemon_test.go to git_protocol.go and git_protocol_test.go to better reflect their purpose. No logic changes. Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

refactor(protocol): rename daemon files to git_protocol Move daemon.go and daemon_test.go to git_protocol.go and git_protocol_test.go to better reflect their purpose. No logic changes. Signed-off-by: Xe Iaso <me@xeiaso.net>

2 months ago

README.md

s3fs#

created by Austin Poor

An implementation of the go-billy Filesystem for treating an S3 bucket like a filesystem.