// the take
A free online book documenting the internals of Apache Spark Core — scheduler, memory management, shuffle, broadcast, dynamic allocation, and dozens of other subsystems, explained class-by-class with diagrams. It's for engineers who need to actually understand how Spark works under the hood (debugging weird executor behavior, tuning memory, writing a custom scheduler listener) rather than just use the DataFrame API.
The coverage is unusually deep and specific — individual pages for internal classes like TaskSchedulerImpl, BlockManager, ExecutorAllocationManager, and UnsafeExternalSorter that you won't find explained anywhere in Spark's own docs. Built with MkDocs and deployed via GitHub Actions, so it's a proper searchable site, not a wall of markdown. The diagrams (architecture, RDD lineage, scheduler flow) are the kind of thing people usually have to draw themselves after reading source code for a week.
It documents a specific era of Spark internals and the source is the only real indicator of freshness — if you're on a very recent Spark version, some class internals (memory manager, shuffle writers) may have drifted from what's described. It's Spark Core only; nothing here on Spark SQL/Catalyst, Structured Streaming, or MLlib internals, so for most modern Spark work (which is DataFrame/SQL-centric) this covers a shrinking slice of what you'll actually touch. It's a one-person-maintained reference work (Jacek Laskowski), so depth is high but there's no community review process catching stale or wrong explanations the way a widely-contributed project would.
We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →