finds.dev

japila-books/apache-spark-internals

The Internals of Apache Spark

View on GitHubHomepage ↗

★ 1,552Apache-2.0updated Oct 2026

// the take

A free online book documenting the internals of Apache Spark Core — scheduler, memory management, shuffle, broadcast, dynamic allocation, and dozens of other subsystems, explained class-by-class with diagrams. It's for engineers who need to actually understand how Spark works under the hood (debugging weird executor behavior, tuning memory, writing a custom scheduler listener) rather than just use the DataFrame API.

The coverage is unusually deep and specific — individual pages for internal classes like TaskSchedulerImpl, BlockManager, ExecutorAllocationManager, and UnsafeExternalSorter that you won't find explained anywhere in Spark's own docs. Built with MkDocs and deployed via GitHub Actions, so it's a proper searchable site, not a wall of markdown. The diagrams (architecture, RDD lineage, scheduler flow) are the kind of thing people usually have to draw themselves after reading source code for a week.

It documents a specific era of Spark internals and the source is the only real indicator of freshness — if you're on a very recent Spark version, some class internals (memory manager, shuffle writers) may have drifted from what's described. It's Spark Core only; nothing here on Spark SQL/Catalyst, Structured Streaming, or MLlib internals, so for most modern Spark work (which is DataFrame/SQL-centric) this covers a shrinking slice of what you'll actually touch. It's a one-person-maintained reference work (Jacek Laskowski), so depth is high but there's no community review process catching stale or wrong explanations the way a widely-contributed project would.

We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →

search the same ground:

// at a glance

Stars
1,552
Forks
457
License
Apache-2.0
Last commit
10 Oct 2026

// nearby

If apache-spark-internals isn't quite it.

All repos →