core/knotmirror/knotstream/ at master · mihaizaurus.at/core · Tangled

mihaizaurus.at / core

forked from tangled.org/core

0

Monorepo for Tangled

0

core / knotmirror / knotstream

5 files

at master

Seongmin Lee knotmirror: performant language indexer 2mo ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

knotmirror: introduce knotmirror KnotMirror is an external service that is intended to be used by appview. It will ingest all known git repos and provide xrpc methods to inspect them, so appview won't need to fetch individual knots on every page render. Using postgres exclusively instead of sqlite to support A LOT of concurrent writes. Signed-off-by: Seongmin Lee <git@boltless.me>

5 months ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

subscription.go

knotmirror: introduce knotmirror KnotMirror is an external service that is intended to be used by appview. It will ingest all known git repos and provide xrpc methods to inspect them, so appview won't need to fetch individual knots on every page render. Using postgres exclusively instead of sqlite to support A LOT of concurrent writes. Signed-off-by: Seongmin Lee <git@boltless.me>

5 months ago