finds.dev

apache/carbondata

High performance data store solution

View on GitHub

★ 1,453ScalaApache-2.0updated Oct 2026

// the take

Apache CarbonData is a columnar file format and storage layer for Hadoop/Spark that bakes multi-level indexing (min-max, bloom filter, Lucene, secondary indexes) directly into the file format to speed up filtered OLAP-style scans. It's for data engineers already committed to the Spark/Hive/Presto ecosystem who want faster query performance on large HDFS datasets without standing up a separate index service.

The indexing is genuinely built into the storage layer, not bolted on - block/blocklet-level min-max stats let a query engine skip I/O before it even reaches columnar decode, and optional bloom filter or Lucene indexes handle point lookups the base index can't. Integration surface is wide and real: Spark datasource, Hive SerDe, Presto connector, Flink sink, plus an SDK (including a C++ SDK) for writing files outside a Spark job. It's a graduated Apache TLP with actual CI (GitHub Actions build-test) and long-running Coverity/Coveralls tracking, not a one-person side project. CDC/SCD support and streaming ingestion run on the same storage format as batch, so you're not maintaining two pipelines.

Completely coupled to the Hadoop/Spark/Hive stack - there's no story for anyone not already running that infrastructure, and the status badges still reference Spark 2.4, which is a bad sign for anyone on Spark 3.x+ wondering how current the integration actually is. Operationally heavier than it looks: segment management, table status locks, and an optional distributed index server are all extra infrastructure compared to just using Parquet/ORC with Spark's native stats. A chunk of the feature surface is explicitly experimental - hybrid-format segments, MV on parquet/orc, non-transactional Hive writes - meaning syntax and behavior can shift under a production user. The README's sudden, unrelated 'Agent module' addition (a Python package for multi-agent metadata storage grafted onto a Java/Scala columnar engine) is a scope-creep red flag worth checking before trusting recent commit activity as a signal of where the project is actually headed.

We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
1,453
Forks
694
Language
Scala
License
Apache-2.0
Last commit
2 Oct 2026

// nearby

If carbondata isn't quite it.

All repos →