// the find
zinggAI/zingg
Scalable master data management, identity resolution, entity resolution, and deduplication using ML
Zingg is a Spark-based entity resolution and deduplication system that trains ML models on labeled pairs to match records across messy real-world data. It uses a two-stage approach — a blocking model to cut down comparisons, then a similarity model on those candidates — which makes it practical at the millions-of-records scale where naive O(n²) approaches fall apart. It's aimed at data engineers dealing with MDM, KYC/AML, or merging datasets from disparate sources.
The active-learning labeler is the standout feature: instead of asking you to label thousands of pairs, it picks the most informative uncertain pairs and builds a usable model from a small sample. The blocking model is legitimately clever — it learns which fields to hash for candidate generation rather than requiring you to define blocking rules manually, which is usually the hardest part of entity resolution. Spark as the execution engine means you can throw this at genuinely large datasets without rewriting anything; the connector abstraction covers S3, Snowflake, JDBC, and most file formats without custom code. CI includes actual performance tests against 120K and 5M-record Febrl datasets, so regressions in throughput don't silently ship.
The Spark dependency is a hard prerequisite — you need a Spark cluster or Databricks to run anything, which rules out smaller teams or single-machine workloads where something like splink would work fine without the infrastructure overhead. The interactive labeling UI is a terminal CLI gif, which means no web UI for non-engineers to label data — in practice the people who know what 'these two records are the same person' looks like are rarely comfortable in a bash shell. AGPL v3 license will stop most commercial teams cold unless they buy a separate license, and the README buries that fact. Documentation exists but the step-by-step guides appear to live in a separate docs site that isn't in the repo, so you're dependent on an external URL not going stale.