// the find
apache/datafusion-comet
Apache DataFusion Comet Spark Accelerator
Comet is a drop-in Spark accelerator that replaces Spark's row-based operators with Arrow-native columnar execution backed by Rust via Apache DataFusion. You add a jar and some config flags; your existing Spark SQL and PySpark code runs unchanged but faster. The claimed 2x speedup on TPC-DS at 1TB is real and benchmarked, not a marketing estimate.
The end-to-end Arrow-native design is the right call — keeping data in columnar format across scan, shuffle, and join avoids the serialization tax that hurts other accelerators. The fallback model is honest: when Comet can't handle an operator it falls back to Spark rather than silently producing wrong results, and `spark.comet.explainFallback.enabled=true` tells you exactly where. Off-heap memory via Arrow means GC pressure drops sharply for large aggregations. The `.claude/skills/` directory shows they're using Claude to help contributors implement new expressions consistently, which is a practical answer to the long tail of missing functions.
Coverage is still incomplete — Python UDFs are not in the Comet pipeline (only Scala/Java UDFs via whole-stage codegen), so any shop with heavy pandas_udf usage sees zero benefit on those stages. The off-heap requirement (`spark.memory.offHeap.enabled=true` with a fixed `offHeap.size`) means you need to tune a new memory knob per job; set it too low and you fall back, too high and you starve the JVM heap. Spark 4.2 support is marked experimental, which matters if you're on a recent EMR or Databricks runtime. Build complexity is significant — Rust + Scala + JNI means contributors need a full Rust toolchain and the CI matrix is large, slowing the expression coverage gap closure.