// the find
apache/airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Apache Airflow is the standard Python-based workflow orchestrator — you define DAGs as code, the scheduler runs them, and a web UI shows you what's running and what failed. It's aimed at data engineers who need to coordinate pipelines across heterogeneous systems (databases, cloud storage, ML platforms) with retry logic, backfill, and dependency tracking built in. If you're already using it, you know it; if you're evaluating orchestrators, it's the incumbent you're comparing against.
The provider ecosystem is genuinely vast — 80+ first-party provider packages covering everything from Snowflake to GCP to dbt, meaning you rarely write a custom operator from scratch. Dynamic task mapping (added in 2.3, matured since) lets you fan out tasks at runtime based on upstream output without restructuring the DAG. The Airflow 3.x architecture properly separates the scheduler, API server, and DAG processor, fixing the long-standing problem where a misbehaving DAG file could starve the scheduler. The constraint-file approach to installation is annoying to explain but genuinely solves the real problem of Airflow's sprawling transitive dependencies breaking on fresh installs.
XCom is still the wrong abstraction for passing data between tasks — it was designed for small metadata and people keep stuffing DataFrames through it anyway, which silently bloats your metadata database. Local development is painful: you need Postgres or MySQL, a working Python environment with constrained deps, and ideally the full Breeze Docker setup just to iterate on a DAG. The 2.x to 3.x migration is non-trivial and the 2.x branch hits EOL April 2026, so anyone still on 2.x is looking at a forced upgrade soon. Dynamic DAG generation via Python import-time side effects still confuses beginners and can cause scheduler thrash when DAG files take too long to parse.