finds.dev

mahmoudparsian/data-algorithms-book

MapReduce, Spark, Java, and Scala for Data Algorithms Book

View on GitHubHomepage ↗

★ 1,082JavaNOASSERTIONupdated Oct 2024

// the take

This is the companion code repository for Mahmoud Parsian's 2015 O'Reilly book 'Data Algorithms: Recipes for Scaling up with Hadoop and Spark.' It's a teaching resource for classic big-data patterns — secondary sort, top-N, joins, partitioning — implemented twice over: once in raw Hadoop MapReduce and once in Spark (Java, Scala, and PySpark). Useful for engineers who want to see the same algorithm solved at both abstraction levels, not for pulling in as a dependency.

Side-by-side implementations of the same problem in MapReduce and Spark make the tradeoffs concrete instead of abstract — you can actually diff a custom Partitioner/GroupingComparator against the equivalent repartitionAndSortWithinPartitions call. It covers the unglamorous but genuinely hard parts of MapReduce (composite keys, secondary sort, custom partitioners) that most tutorials skip. Build support for Ant, Maven, and Gradle plus runnable shell scripts for YARN/cluster submission means the examples are meant to actually be run, not just read.

The core examples are Hadoop 2.6 / Spark 3.1.2 era (RDD-centric), and the promised 2nd edition has been 'coming end of 2021' with no sign of landing — treat this as a historical reference, not current Spark idiom (no Structured Streaming, no DataFrame-first patterns). The repo vendors roughly 150 prebuilt JARs directly into lib/ instead of leaning on Maven/Gradle dependency resolution, which bloats the clone and locks in old, unpatched library versions. There are no tests and no CI, so nothing guarantees the examples still compile or run correctly against the Spark/Hadoop versions pinned in the build files. Last meaningful code activity is years old despite the October 2024 push, which looks like a doc/README touch-up rather than engagement with the codebase.

We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
1,082
Forks
651
Language
Java
License
NOASSERTION
Last commit
14 Oct 2024

// nearby

If data-algorithms-book isn't quite it.

All repos →