// the find
OrionExplorer/dcpam
Data Construct-Populate-Access-Manage - Open source data warehouse solution.
DCPAM is a C-based ETL/ELT engine and data warehouse toolkit. It pulls from PostgreSQL, MySQL/MariaDB, Oracle, SQLite, MongoDB, ODBC sources, flat files and HTTP APIs into a target database, configured through JSON and SQL files. It suits teams who want scriptable on-prem warehouse loads and are comfortable hand-writing config and building from source.
- Each ETL process runs in its own thread, and the design supports multiple instances on one host or across machines, so parallel loading is part of the architecture rather than bolted on.
- Native drivers cover the common databases (Postgres, MySQL/MariaDB, Oracle, SQLite, MongoDB), and ODBC is the fallback for everything else, which is a sensible split for a C codebase.
- CSV/TSV/PSV and JSON parsing live in-process under src/file, with cJSON vendored, so the core ingest path does not depend on a runtime like Python.
- The repo has a GitHub Actions C/C++ workflow and a CodeQL analysis workflow, which is more CI than most projects this small bother with.
- The last push was March 2021 and the roadmap still shows LDAP and WDS cache TTL as in progress, so read the README as a snapshot of an unfinished plan, not current status.
- Admin (DCPAM Construct) and monitoring (DCPAM Monitoring) are unbuilt, so operating this means editing JSON by hand and reading per-component log files. The README's claim that anyone with SQL knowledge can run it skips over that.
- XLS, XLSX, ODS and XML support is handled by Python scripts in data-processor/, which adds a Python dependency to an otherwise C deployment.
- Zero stars, zero forks and no commits in years mean no community is maintaining it. Any bug you hit is yours to fix, and the vendored sqlite3 and cJSON copies will not update themselves.