// the find
josephmachado/beginner_de_project
Beginner data engineering project - batch edition
A tutorial project that wires together Airflow, DuckDB, Spark, MinIO, and Quarto into a single Docker Compose setup to demonstrate a batch ETL pipeline. It's aimed at people who want a working local environment to learn data engineering concepts without setting up cloud infrastructure first. Companion to a blog post series on startdataengineering.com.
Codespaces support is a genuine win — someone can be running the pipeline in under 10 minutes without touching their local machine. The stack choices are sensible for learning: MinIO as a local S3 stand-in avoids AWS account friction, DuckDB as the warehouse is far more approachable than standing up Redshift. CI/CD workflows are included and the Makefile keeps the command surface small. The test suite, even if minimal, means there's at least a validity check on the DAGs.
Everything runs in one container, which is convenient for setup but teaches the wrong mental model — in production, Airflow, Spark, and your warehouse are not cohabitants of the same process. The topics mention Redshift and EMR but the actual code uses DuckDB and local Spark; anyone expecting to transfer this directly to AWS will hit a gap. The .env file is committed to the repo with credentials in plaintext, which is a bad habit to teach beginners. Test coverage is essentially a DAG import check — no tests for the actual transformation logic.