// the find
PrefectHQ/prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Prefect is a Python workflow orchestration framework that turns decorated functions into observable, retriable, scheduled pipelines. The decorator model (@flow, @task) means you can drop it onto an existing script without restructuring around a DAG builder. It targets data engineers and ML teams who outgrew cron but don't want to run Airflow.
1. Dynamic workflows at runtime — unlike Airflow's static DAGs, you can loop, branch, and create tasks based on values computed mid-run. This matters a lot when your pipeline shape depends on upstream data. 2. Minimal ceremony to get started: two decorators and a local server that's a single process, not a five-service stack. The quickstart example in the README actually works. 3. The prefect-client package is a smart separation — ephemeral workers in Lambda or containers can report state without dragging in the full server dependency. 4. First-party integrations with AWS, GCP, Azure, dbt, Databricks, and Kubernetes are maintained in the same org and versioned alongside the core, so they don't quietly fall behind.
1. The OSS/Cloud split is aggressive — RBAC, SSO, managed work pools, and incident management are all Cloud-only. The self-hosted server is functional but deliberately limited, making OSS feel like a trial tier rather than a complete product. 2. The API has broken significantly across v1 → v2 → v3; if you're on anything older than 2.x, you're rewriting, not upgrading. The Blocks system in particular has been redesigned multiple times and the docs still show conflicting patterns depending on which version they were written for. 3. The Server → Work Pool → Worker → Deployment mental model has more moving parts than the decorator API implies. Simple use cases end up requiring infra decisions (which work pool type? local process vs. Docker vs. Kubernetes?) before you can deploy anything. 4. Self-hosted observability is thin — the UI shows run state and logs but lacks the alerting, SLAs, and audit trails that make the Cloud tier worth paying for, so teams doing serious production ops on OSS will need to bolt on their own monitoring.