finds.dev

damklis/DataEngineeringProject

Example end to end data engineering project.

View on GitHub

★ 1,411PythonMITupdated Dec 2022

// the take

A complete data pipeline demo that ingests news from RSS feeds through Kafka → MongoDB → Elasticsearch, with a Django REST API for querying. It's a portfolio/learning project that wires together about a dozen open-source tools to show how they fit together. Aimed at data engineers who want to see a realistic (if simplified) streaming architecture running locally.

The component selection is realistic — Kafka Connect with Debezium for CDC, MinIO as a local S3 stand-in, and a proper CQRS split between write (MongoDB) and read (Elasticsearch) models. The proxy pool with rotating user agents is a practical touch most tutorial projects skip. Test coverage exists for the scraping and API layers, not just a single smoke test. The `manage.sh` wrapper keeps the Docker Compose lifecycle simple enough to actually run.

Abandoned since December 2022, pinned to Python 3.8, and the CI badge points to a dead Travis CI URL — you're on your own if anything breaks. Running this locally requires 8GB of Docker memory for what amounts to a news aggregator, which tells you the infrastructure is the demo, not the product. The Airflow DAG uses public proxy scraping as its data source, so it will fail silently whenever those proxy lists go stale. No schema registry for Kafka, meaning the Avro-less JSON messages will quietly drift if you extend the pipeline.

We read the README, the directory tree and a key source file in June 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
1,411
Forks
278
Language
Python
License
MIT
Last commit
8 Dec 2022

// nearby

If DataEngineeringProject isn't quite it.

All repos →