// the find
awslabs/deequ
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Deequ is an AWS-built Spark library for expressing data quality rules as executable checks — completeness, uniqueness, value ranges, pattern matches — and running them as ordinary Spark jobs over arbitrarily large datasets. It also ships anomaly detection over time-series metrics, automatic constraint suggestion from data profiling, and a DQDL declarative rule language that mirrors what AWS Glue uses. The target audience is data engineers running Spark pipelines who want something more structured than ad-hoc assertions.
The algebraic state model is the real engineering here: analyzers compute mergeable summaries so multiple checks scan the data once, not once-per-check — this matters a lot at scale. The MetricsRepository for storing historical metrics and running anomaly detection over them (including Holt-Winters seasonal detection) is a feature most competitors skip entirely. DQDL support means rules can be written by analysts in a readable DSL rather than Scala lambdas, and it's compatible with AWS Glue's rule format. The constraint suggestion engine that profiles a dataset and auto-generates a starter ruleset is genuinely useful for bootstrapping quality checks on a new table.
The Spark version coupling is painful: Deequ 2.x locks you to Spark 3.1 specifically, not 3.x generally — if your cluster is on 3.2+ you're on your own with community forks. Row-level evaluation is a second-class citizen: composite rules don't support it at all, and several rule types just skip it silently rather than failing loudly, which makes debugging failures harder than it should be. The DQDL parser depends on AWS Glue's ANTLR grammar (a transitive dependency that isn't obvious from the README), which creates an awkward coupling to a specific AWS service for what presents itself as a generic library. Test coverage is good for core analyzers but the DQDL executor layer has noticeably thinner tests, which is exactly the path most new adopters will use.