Self-hosted app to archive YouTube videos and playlists by link, store them locally, and play them back
  • Python 74.8%
  • HTML 13.3%
  • CSS 8.2%
  • Nix 2%
  • Dockerfile 1.7%
Find a file
benmandrew e66d9902e9
feat(downloader): show per-phase progress for merge and finishing steps
The progress bar tracked only yt-dlp's byte download, so a video job
sat at 100% for as long as ffmpeg took to mux separate video and
audio streams — the slowest and least visible part of a large
download.

Adds a `JobPhase` enum and nullable `Job.phase` column (nullable so
`db.py::_add_missing_columns` can migrate existing SQLite databases).
The bar now restarts at 0% per phase: downloading_video,
downloading_audio, merging, finishing. yt-dlp's postprocessor_hooks
report the Merger starting/finishing but carry no percentage, so
`downloader.py::MergeProgress` estimates it: since the merge is a
stream copy, a daemon thread samples the download directory's total
size every 0.5s and measures growth past the pre-merge baseline,
capped at 99%, with no dependency on yt-dlp's temp-file naming.
`finishing` covers the remaining postprocessors and _save_video; it
has no measurable size, so its bar sweeps (CSS .bar--sweeping)
instead of showing a percentage.

The worker's progress throttle now writes through immediately on a
phase change, and speed/eta clear once downloading ends. Phase is
reset in _claim_jobs, _finish_job, and the retry route.

Claude-Session: https://claude.ai/code/session_015AntLJDoCzkDRAkpTYoaDN
2026-08-31 00:57:23 +01:00
.github/workflows feat(ci/ytdlp): auto-rebuild on new yt-dlp release + runtime overlay 2026-08-31 00:20:43 +01:00
tests feat(downloader): show per-phase progress for merge and finishing steps 2026-08-31 00:57:23 +01:00
yam feat(downloader): show per-phase progress for merge and finishing steps 2026-08-31 00:57:23 +01:00
.dockerignore build(docker): split the image build and run it unprivileged 2026-07-26 22:25:13 +01:00
.envrc chore: add .envrc for direnv, quiet flake shellHook 2026-07-07 23:20:12 +01:00
.gitignore chore: add .envrc for direnv, quiet flake shellHook 2026-07-07 23:20:12 +01:00
CLAUDE.md feat(ci/ytdlp): auto-rebuild on new yt-dlp release + runtime overlay 2026-08-31 00:20:43 +01:00
docker-compose.yml build: drop Tailscale sidecar, build image from Dockerfile, add Docker Hub CI 2026-07-08 00:14:42 +01:00
Dockerfile feat(ci/ytdlp): auto-rebuild on new yt-dlp release + runtime overlay 2026-08-31 00:20:43 +01:00
flake.lock feat: scaffold project skeleton (Nix flake, FastAPI app, Tailscale deploy) 2026-07-07 22:36:23 +01:00
flake.nix test: M8 — pytest suite, CI, and structured logging 2026-07-08 23:34:46 +01:00
LICENCE Add MIT licence 2026-07-26 22:52:46 +01:00
PLAN.md feat(downloader): show per-phase progress for merge and finishing steps 2026-08-31 00:57:23 +01:00
pytest.ini test: M8 — pytest suite, CI, and structured logging 2026-07-08 23:34:46 +01:00
README.md feat(ci/ytdlp): auto-rebuild on new yt-dlp release + runtime overlay 2026-08-31 00:20:43 +01:00
requirements.txt feat(ci/ytdlp): auto-rebuild on new yt-dlp release + runtime overlay 2026-08-31 00:20:43 +01:00

Yam

Self-hosted app to archive YouTube videos and playlists by link, store them locally, and play them back through one web interface. See PLAN.md for the full design and roadmap.

Status: Milestones 16 done — downloads, playback, playlists, library/job management, and playlist sync. M7 underway (cookies guidance + subtitles). See PLAN.md for the remaining roadmap.

Features

  • Archive by link — paste a YouTube video or playlist URL; a background worker (MAX_CONCURRENT_DOWNLOADS in parallel) fetches it with yt-dlp, preferring H.264/AAC mp4 (~1080p cap) for universal playback.
  • Live downloads page (/downloads) — job status with retry, clear, and playlist jobs nesting their per-entry child jobs.
  • Library grid (/) — thumbnail cards with search and sort (by title/channel, date/duration/size).
  • Playback — native <video> player at /watch/{id} backed by Range streaming from /media/{id}.
  • Playlists (/playlist/{id}) — ordered entries, cover thumbnail, manual "Next ▸", one stored file per video shared across playlists, and sync to re-enumerate (adding new entries, pruning removed links without deleting files).
  • Management — delete a video (guarded while any playlist references it) or a full playlist (full-deletes orphaned videos), and retry pending playlist entries.

Requirements

Dependencies are managed entirely by the Nix flake (Python, FastAPI, yt-dlp, ffmpeg, …). You only need Nix with flakes enabled.

Develop

nix develop            # enters devShell; sets MEDIA_DIR/DATA_DIR to ./.local
uvicorn yam.main:app --reload --port 8080

Then open http://localhost:8080 (and http://localhost:8080/healthz).

You can also run the packaged app directly, without a shell:

MEDIA_DIR=./.local/media DATA_DIR=./.local/data nix run .#yam

Build & publish the container image

The image is built from the Dockerfile (works on any Docker host):

docker build -t yam:latest .

CI (.github/workflows/docker-publish.yml) builds and pushes to Docker Hub on every push to main and on v* tags. Set two repo secrets:

  • DOCKERHUB_USERNAME — your Docker Hub username (also the image namespace)
  • DOCKERHUB_TOKEN — a Docker Hub access token

The published image is docker.io/<DOCKERHUB_USERNAME>/yam (tags: latest, the git tag, and the short commit SHA). Reference it as the image: in compose.

Deploy

docker-compose.yml runs a single container and publishes port 8080; Yam speaks plain HTTP, so put TLS / access control in front of it. If the host is already a tailnet node, front it with host-level tailscale serve (any reverse proxy works too):

tailscale serve --bg --https=8449 http://127.0.0.1:8080
#   -> https://<host>.<tailnet>.ts.net:8449
docker compose up -d

Configuration

Env var Default Purpose
MEDIA_DIR /media Where video files/thumbnails are stored
DATA_DIR /data SQLite DB location
MAX_CONCURRENT_DOWNLOADS 2 Parallel download workers
DOWNLOAD_SUBTITLES false Fetch subtitles alongside videos
COOKIES_FILE Path to cookies.txt for restricted content
MIN_FREE_SPACE_MB 500 Refuse to start a download below this free space (0 disables)
BASIC_AUTH_USER Username for optional HTTP Basic auth
BASIC_AUTH_PASS Password for optional HTTP Basic auth

The effective settings and storage stats are visible read-only at /config.

The app listens on port 8080 inside the container/package. To use a different port in local dev, pass it to uvicorn: uvicorn yam.main:app --port 9000.

Subtitles

Set DOWNLOAD_SUBTITLES=true to fetch the English subtitle track (manual, else auto-generated) as WebVTT alongside each new video; the player then shows a selectable subtitle track. Already-archived videos are unaffected — re-download them to pick up subs.

Cookies (restricted / age-gated content)

Some videos, and increasingly ordinary ones, return "Sign in to confirm you're not a bot". To get past it, export a cookies.txt from a browser where you're signed in to YouTube (e.g. the "Get cookies.txt" extension, Netscape format), mount it into the container, and point COOKIES_FILE at it:

# docker-compose.yml
    environment:
      COOKIES_FILE: /data/cookies.txt   # mounted alongside the DB volume

When a download or playlist enumeration hits the bot check, the failed job on /downloads shows this guidance instead of the raw yt-dlp error.

Keeping yt-dlp up to date

YouTube changes the interfaces yt-dlp speaks to on its own schedule, and a yt-dlp that has fallen behind starts failing downloads with HTTP 403 on videos that worked the week before. Two things keep Yam's copy current, and neither needs a change to Yam itself.

CI rebuilds and republishes the image every day that PyPI has a newer stable yt-dlp than the published latest (the pinned release is recorded in the image's io.yam.ytdlp.version label), so docker compose pull && docker compose up -d gets you a current image.

To fix it the same day instead of waiting for the next pull, open /config and press "Check for update". Yam installs the current stable yt-dlp into the data volume ($DATA_DIR/ytdlp), where it survives container restarts. Restart the container to apply it — the old version is already loaded by the time you press the button. /config shows the running version and where it came from, the image or that install. Once a pulled image ships a yt-dlp at least as new, the installed copy is ignored automatically and there is nothing to clean up.

Access control (optional Basic auth)

Yam speaks plain HTTP and expects TLS/access control in front of it (see Deploy). As defense-in-depth, set both BASIC_AUTH_USER and BASIC_AUTH_PASS to require HTTP Basic auth on every route except /healthz (kept open for container health probes). Leave either unset to disable it.

Disk guard

Downloads are refused when free space on MEDIA_DIR would drop below MIN_FREE_SPACE_MB (default 500 MB; set 0 to disable); the job fails with a clear message so you can free space and retry. Current archived size and free space are shown at /config.