// the find
intel/BigDL
BigDL: Distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray
BigDL was Intel's attempt at a unified platform for running deep learning workloads on Spark and Ray clusters — covering everything from distributed training (Orca) to time series (Chronos) to recommendation systems (Friesian) to SGX-protected computation (PPML). It served teams that already had Spark infrastructure and wanted to run TF/PyTorch at scale without standing up separate ML clusters. The project is archived as of June 30, 2026, with the LLM work having already migrated to ipex-llm.
The Orca abstraction for scaling single-node PyTorch/TF code to Spark clusters with minimal code changes was genuinely useful — three lines to switch from local to YARN/K8s. The Nano inference optimizer's benchmark table approach (run all quantization strategies, show latency vs accuracy tradeoff, pick the winner) is a good pattern that most teams do manually. PPML's use of Intel SGX for confidential computing on big data pipelines filled a real gap for regulated industries that needed to run analytics on sensitive data without trusting the cloud operator.
It's archived — full stop. Any production dependency on this is a liability you're accepting. The scope was too wide: six distinct sub-libraries (Orca, Nano, DLlib, Chronos, Friesian, PPML) meant none got the depth it needed, and the DLlib Keras-style Scala API in particular was always playing catch-up with actual Keras. Intel SGX dependency in PPML creates a hard hardware constraint that rules out most cloud deployments. The migration story for existing users is fragmented — LLM users go to ipex-llm, but there's no clear successor for Orca or Chronos.