// the find
nchammas/flintrock
A command-line tool for launching Apache Spark clusters.
Flintrock is a CLI tool for spinning up Apache Spark clusters on EC2, built as a faster replacement for the now-dead spark-ec2 script. It handles cluster launch, teardown, scaling, and basic file/command operations against the cluster. The target user is someone who wants throwaway Spark clusters for experimentation or batch jobs without paying EMR prices.
Parallel node provisioning is the core win — 100 nodes in ~3 minutes versus spark-ec2's linear hour-plus. The YAML config file with per-run CLI overrides is well-designed; you keep a baseline and tweak at launch time without rewriting everything. Ephemeral storage is configured automatically and wired into HDFS/Spark, which is easy to get wrong manually. The exit codes and YAML output mode make it usable in automated pipelines without fragile stdout parsing.
EC2-only, full stop — there's no GCP, Azure, or on-prem path, and the architecture makes adding providers non-trivial. It's a personal project with low commit frequency; the last push was late 2024 but activity has been sparse for years, so you're adopting something that may not track new Spark or AWS API changes. No state management beyond what AWS holds — if you lose your local config or the cluster gets into a weird state, recovery is manual. The standalone package was dropped in 2.1.0, so Python is now a hard dependency with no good explanation for why that tradeoff was made.