// the find
jupyter-incubator/sparkmagic
Jupyter magics and kernels for working with remote Spark clusters
Sparkmagic lets you run Spark code on a remote cluster from a Jupyter notebook by proxying everything through a Livy REST server. You get PySpark, Scala, and R kernels plus SQL magic, automatic Pandas DataFrame capture, and basic auto-visualization — all without installing Spark locally. The target user is a data engineer or analyst on a team that already has a Livy-fronted Spark cluster (YARN or Kubernetes).
The Livy proxy architecture is genuinely useful: zero local Spark dependencies, and you can point at multiple clusters from one notebook. The `%%sql` → Pandas capture pipeline works well in practice and removes a lot of boilerplate. Kerberos auth support is real and configurable, not an afterthought — mutual authentication, hostname override, preemptive mode are all exposed. The Docker Compose stack with a local-mode Spark + Livy is a fast path to actually trying the thing without standing up infrastructure.
Livy itself is effectively abandoned by the Apache project (last release 2020), which puts the entire foundation of this tool on shaky ground — Lighter and Ilum are third-party alternatives but add their own operational surface area. The auto-visualization layer (autovizwidget + Plotly) is thin and tends to break on anything but small result sets; for real analysis work you'll drop to `%%local` and matplotlib anyway. CI still shows a Travis badge pointing at a build that almost certainly doesn't pass anymore — the GitHub Actions workflows exist but the README was never updated, suggesting maintenance is inconsistent. There's no streaming output support: long-running Spark jobs show nothing until they complete, which makes interactive debugging painful.