finds.dev

nchammas/flintrock

A command-line tool for launching Apache Spark clusters.

View on GitHub

★ 651PythonApache-2.0updated Dec 2024

// the take

Flintrock is a CLI tool for spinning up Apache Spark clusters on EC2, built as a faster replacement for the now-dead spark-ec2 script. It handles cluster launch, teardown, scaling, and basic file/command operations against the cluster. The target user is someone who wants throwaway Spark clusters for experimentation or batch jobs without paying EMR prices.

Parallel node provisioning is the core win — 100 nodes in ~3 minutes versus spark-ec2's linear hour-plus. The YAML config file with per-run CLI overrides is well-designed; you keep a baseline and tweak at launch time without rewriting everything. Ephemeral storage is configured automatically and wired into HDFS/Spark, which is easy to get wrong manually. The exit codes and YAML output mode make it usable in automated pipelines without fragile stdout parsing.

EC2-only, full stop — there's no GCP, Azure, or on-prem path, and the architecture makes adding providers non-trivial. It's a personal project with low commit frequency; the last push was late 2024 but activity has been sparse for years, so you're adopting something that may not track new Spark or AWS API changes. No state management beyond what AWS holds — if you lose your local config or the cluster gets into a weird state, recovery is manual. The standalone package was dropped in 2.1.0, so Python is now a hard dependency with no good explanation for why that tradeoff was made.

We read the README, the directory tree and a key source file in July 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
651
Forks
120
Language
Python
License
Apache-2.0
Last commit
13 Dec 2024

// nearby

If flintrock isn't quite it.

All repos →