// the take
dbldatagen is a PySpark library for generating synthetic data at scale inside Databricks, driven by a declarative column spec rather than row-by-row code. It's aimed at data engineers who need large, repeatable test or demo datasets with realistic referential integrity across multiple tables, not at people wanting a lightweight general-purpose faker.
Generation runs as a Spark job, so it scales to billions of rows across partitions instead of generating row-by-row in a single process like Faker does. It has real support for consistent primary/foreign keys across tables, which makes it usable for CDC, merge, and join test scenarios rather than just flat tables. Column values can be seeded deterministically via hashing, so you get reproducible datasets instead of random ones that differ on every run. It plugs into Faker for realistic text generation rather than reimplementing that, and it hooks directly into Delta Live Tables as a source.
It's hard-locked to the Databricks/Spark runtime - there's no path to use this as a standalone Python library outside that ecosystem, which rules it out for most non-Databricks shops. Compatibility is tied to specific runtime versions (13.3 LTS+) and Unity Catalog access modes, with real feature gaps (3rd-party libs, Python UDFs) on older or shared-cluster configurations. It's a Databricks Labs project, meaning explicitly no SLA and best-effort support - issues get triaged as time permits, not on a maintenance schedule. The 'generate spec from existing schema/data' feature, which is the part most people actually want when bootstrapping test data, is still marked experimental.
We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →