// the find
stas00/ml-engineering
Machine Learning Engineering Open Book
A living reference book from Stas Bekman — one of the engineers who trained BLOOM-176B — covering the full stack of running large-scale ML training: GPUs, networking, storage, SLURM, debugging hangs, and inference. Not a framework or a library, just documented hard-won operational knowledge. Aimed at engineers who already have access to a GPU cluster and are dealing with the chaos of actually using it.
The debugging section is the real value here — particularly the distributed hanging solutions and underflow/overflow detection, which are the exact problems that eat days when you're running multi-node PyTorch jobs. The network benchmarking scripts (all_reduce_bench.py, torch-distributed-gpu-test.py) are immediately usable, not just explained. The SLURM content is unusually complete: launchers for torchrun, accelerate, and lightning, plus admin and user guides. The training instabilities section includes actual loss curves from real BLOOM and IDEFICS runs, which is rare — most documentation pretends training always converges.
It's a Markdown book in a git repo, not a structured reference — navigation relies on the README table of contents and your ability to grep. Content density is uneven: some sections are deep and specific, others are thin stubs. Nothing here helps you if you're on a cloud provider that isn't bare-metal (Modal, Lambda, RunPod) — the SLURM-centric worldview assumes you control the cluster. The inference section is noticeably thinner than training, so if you're primarily deploying rather than training you'll hit a ceiling quickly.