// the find
dagster-io/dagster
An orchestration platform for the development, production, and observation of data assets.
Dagster is a data pipeline orchestrator built around the concept of 'assets' — declarative Python functions that produce named data artifacts. It handles scheduling, dependency resolution, lineage tracking, and observability for the entire data engineering lifecycle. Aimed at data engineers and ML teams who've outgrown cron jobs and need something between Airflow's complexity and a simple task queue.
Asset-centric model is genuinely better than task-centric (Airflow) for data work — you think in terms of what data you're producing, not what code runs when. The web UI's asset lineage graph is excellent; tracing why downstream assets are stale is actually useful rather than decorative. First-class testability is real — assets are plain Python functions with typed inputs/outputs, so unit tests don't require a running orchestrator. The integration ecosystem (dbt, Spark, Snowflake, etc.) is extensive and maintained by the core team, not community stragglers.
The learning curve is steep and the abstraction surface is large — resources, IO managers, partitions, sensors, schedules, and asset checks all have their own mental models that take time to internalize before you're productive. Local development still requires running the daemon and webserver alongside your code, which is heavier than it should be for simple pipelines. The open-source version lacks some observability features (alerting, SLAs, asset catalog search) that live behind the Dagster+ paywall, which will frustrate teams expecting a complete self-hosted solution. Monorepo structure with 100+ packages means `pip install dagster` pulls in a lot, and dependency conflicts with existing data stacks (Spark, older Pandas) surface regularly.