AlexWortega/moe-100m-volta-week
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
100M MoE pretrain spec build + V100 4-GPU smoke + optimizations
8 commits 1 branch 0 tags 814 KiB
  • Python 91.6%
  • Shell 5.4%
  • Rebol 3%
Find a file
Exact
AlexWortega 53100d354d
 round 4: lm-eval-harness zero-shot eval (boolq 56.5, arc_easy 46.2 beat gpt2-124M)
2026-05-15 12:24:32 +02:00
bench round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
eval round 4: lm-eval-harness zero-shot eval (boolq 56.5, arc_easy 46.2 beat gpt2-124M) 2026-05-15 12:24:32 +02:00
model Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
moe round 2: Ultra-FineWeb streaming + 7 optimizations measured + MFU honesty 2026-05-15 00:12:16 +02:00
optim round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
prof/baseline round 2: Ultra-FineWeb streaming + 7 optimizations measured + MFU honesty 2026-05-15 00:12:16 +02:00
results round 4: lm-eval-harness zero-shot eval (boolq 56.5, arc_easy 46.2 beat gpt2-124M) 2026-05-15 12:24:32 +02:00
scripts round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
tests Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
train round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
.gitignore Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
.resume_pid.txt round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
.run_start_t.txt round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
eval.log round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
EVAL.md round 4: lm-eval-harness zero-shot eval (boolq 56.5, arc_easy 46.2 beat gpt2-124M) 2026-05-15 12:24:32 +02:00
eval_pre_resume.log round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
gen_samples.log round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
model.py round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
model_standalone.py round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
OPTIMIZATIONS.md round 2: Ultra-FineWeb streaming + 7 optimizations measured + MFU honesty 2026-05-15 00:12:16 +02:00
PLAN.md Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
PLAN_1B.md round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
prompt.txt Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
PUBLISHED.md round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
reproduce.sh Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
RESEARCH.md Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
RESULTS.md round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
RESULTS.md.r2 round 2: Ultra-FineWeb streaming + 7 optimizations measured + MFU honesty 2026-05-15 00:12:16 +02:00
RESULTS_1B.md round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
RESULTS_R2.md round 2: Ultra-FineWeb streaming + 7 optimizations measured + MFU honesty 2026-05-15 00:12:16 +02:00
RESUME_CHECK.md round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
run.sh Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
run_1b.sh round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
run_resume_1b.sh round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
TASK.md Initial 100M-MoE-on-4xV100 reproducer (smoke + bench matrix) 2026-05-14 20:47:45 +00:00
train.log round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
train.stdout round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
train_pre_resume.log round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
train_pre_resume.stdout round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00
VERIFY.md round 2: Ultra-FineWeb streaming + 7 optimizations measured + MFU honesty 2026-05-15 00:12:16 +02:00
VERIFY_1B.md round 3: 1B-token full-pipeline shakedown (WSD + EMA + resume + HF push) 2026-05-15 10:27:10 +02:00