// the take
A companion repo to Eugene Yan's blog post on testing ML code, showing pre-train, post-train, and evaluation tests against a from-scratch numpy DecisionTree/RandomForest trained on the Titanic dataset. It's for ML engineers who want a concrete pattern for testing model code beyond unit tests on utility functions.
The pre-train/post-train/evaluation split is a genuinely useful mental model: implementation correctness, learned behavior, and performance are different failure modes and deserve different tests. The invariance and directional expectation tests are the most valuable part - checking that changing a passenger's ticket number shouldn't move the prediction, but changing sex or class should, is a pattern most ML codebases skip entirely. Standard engineering hygiene (pytest fixtures, coverage, mypy, pylint, CI) is wired up and actually exercised, not just configured and ignored.
Last commit is from September 2022 - this is a frozen teaching artifact, not a maintained project, so don't expect fixes or updates. There's no installable library here; it's a worked example on a toy dataset with a hand-rolled decision tree, so applying it means re-deriving the pattern for your own stack (gradient boosting, neural nets, LLMs) rather than importing anything. The evaluation tests hardcode specific thresholds like 'AUC ROC on test should be > 0.84' against a fixed random split - that's exactly the kind of brittle, seed-dependent assertion that causes flaky CI if copied directly into a real pipeline with retraining.
We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →