Searchable Anthropic interview questions, solutions, and follow-ups
- HTML 100%
Aleksandr Nikolich
172d9aad4etheory/ultrascale.md — a condensed digest of the 246-page HuggingFace book: memory budget and the 16/20 bytes-per-parameter tables, activation recomputation, ZeRO stages with their communication cost, tensor + sequence parallelism (conjugate f/f* and g/g* transitions), context parallelism and zig-zag balancing, pipeline schedules through zero-bubble and DualPipe (B/W backward decomposition), expert parallelism, the three-step configuration search with benchmark takeaways, kernel fusion, FlashAttention and mixed precision (format table, epsilon, the three fp16 tricks, why naive fp8 diverges), plus profiling and overlap math. Each section cross-links to the gym tasks where the corresponding mistake is reproduced in code. Readable via `python gym.py book [N]` and published as a separate section of the site (13 sections, 23 cross-links). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| transformer-gym | ||
| .nojekyll | ||
| 404.html | ||
| index.html | ||
| robots.txt | ||