// the find
databrickslabs/dlt-meta
Metadata driven Spark Declarative Pipelines framework for bronze/silver pipelines
DLT-META is a configuration-driven layer on top of Databricks Lakeflow Declarative Pipelines (formerly DLT) that lets you define bronze/silver pipelines via JSON onboarding files instead of writing pipeline code. One generic pipeline reads the DataflowSpec and wires up readers, CDC flows, DQ expectations, and sink writers dynamically. It's for Databricks-heavy data engineering teams who want to onboard 50 tables without writing 50 pipeline scripts.
The DataflowSpec abstraction is genuinely useful — source/target metadata, DQ rules, and CDC config all live in JSON, so adding a new table is a config change not a code change. CDC support via apply_changes and apply_changes_from_snapshot is non-trivial to get right and it's covered. The CLI (databricks labs dlt-meta onboard/deploy) wraps the full workspace deployment flow in two interactive commands, which is a real time save. The Databricks Asset Bundles integration means you can slot this into an existing DAB-based CI/CD workflow without hacks.
Hard Databricks lock-in is complete — this is useless outside Databricks, and if Lakeflow Declarative Pipelines pricing or behavior changes, you have no exit. Silver transformations are expressed as SQL strings in a JSON file, which means no IDE support, no type checking, and debugging a bad SQL transform means hunting through JSON configs. The project is officially 'Databricks Labs' which means no SLA, no support tickets, and maintainer bandwidth is whatever spare time Databricks engineers have. At v0.0.9 with 263 stars, you'd be an early adopter betting on a lab project that could be abandoned or superseded by native Databricks features at any time.