finds.dev

GokuMohandas/data-engineering

Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.

View on GitHub

★ 250Jupyter Notebookupdated Sep 2022

// the take

A companion repo to the Made With ML MLOps course, walking through a full ELT stack: Airbyte extracts CSVs into BigQuery, dbt transforms them, Great Expectations validates each stage, and Airflow orchestrates the whole thing. It's a learning resource, not a library — the 'data' is two toy CSV files with ML project titles and tags.

The Airflow DAG is well-structured: extract → validate → transform → validate again is the right order and most tutorials skip the post-load validation step. The dbt schema.yml tests are minimal but correct. Great Expectations integration via the Airflow provider is shown end-to-end, which is genuinely hard to find documented in one place. The dependency chain between tasks is explicit and readable.

Last pushed September 2022 — Airflow is pinned to 2.3.3, Great Expectations to 0.15.19, both of which have had breaking API changes since. The dataset is two CSV files pulled from a public URL; nothing here is reusable for a real pipeline without gutting everything. No environment management beyond a flat requirements.txt, so getting this running without version conflicts is its own project. The BigQuery project ID is hardcoded throughout the SQL models, meaning every learner has to find-and-replace across multiple files just to run the examples.

We read the README, the directory tree and a key source file in July 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
250
Forks
40
Language
Jupyter Notebook
Last commit
12 Sep 2022

// nearby

If data-engineering isn't quite it.

All repos →