// the find
big-data-europe/docker-spark
Apache Spark docker image
Docker images for running Apache Spark in standalone cluster mode, covering versions from Spark 1.5 through 3.3. Aimed at developers who want a local or small-team Spark environment without wrestling with bare-metal setup. Works with both Docker Compose and Kubernetes.
The multi-image split (base, master, worker, history-server, submit) is clean and lets you swap individual pieces without rebuilding everything. Maven, Python, and sbt example templates are genuinely useful for getting a job running in under an hour. Kubernetes deployment is included and not an afterthought — the headless service setup for spark-submit driver reachability is a real problem this actually solves. The version matrix is wide enough that you can match whatever Spark version your existing jobs were built against.
Newest supported version is Spark 3.3.0, released in 2022 — Spark 3.5 and 4.0 are out and this repo hasn't kept pace. If you need Delta Lake, Iceberg, or modern connector support, you'll be patching these images yourself. The `INIT_DAEMON_STEP` environment variable is part of a larger BDE pipeline framework that isn't well-documented here, so copy-pasting the docker-compose example without that context leaves you with a dangling env var. No TLS or authentication between master and workers out of the box — fine for a laptop, a liability if you accidentally expose port 7077.