// the find
GokuMohandas/data-engineering
Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.
A companion repo to the Made With ML MLOps course, walking through a full ELT stack: Airbyte extracts CSVs into BigQuery, dbt transforms them, Great Expectations validates each stage, and Airflow orchestrates the whole thing. It's a learning resource, not a library — the 'data' is two toy CSV files with ML project titles and tags.
The Airflow DAG is well-structured: extract → validate → transform → validate again is the right order and most tutorials skip the post-load validation step. The dbt schema.yml tests are minimal but correct. Great Expectations integration via the Airflow provider is shown end-to-end, which is genuinely hard to find documented in one place. The dependency chain between tasks is explicit and readable.
Last pushed September 2022 — Airflow is pinned to 2.3.3, Great Expectations to 0.15.19, both of which have had breaking API changes since. The dataset is two CSV files pulled from a public URL; nothing here is reusable for a real pipeline without gutting everything. No environment management beyond a flat requirements.txt, so getting this running without version conflicts is its own project. The BigQuery project ID is hardcoded throughout the SQL models, meaning every learner has to find-and-replace across multiple files just to run the examples.