// the find
apache/nifi
Apache NiFi
Apache NiFi is a mature, long-running dataflow automation platform with a visual editor for building and monitoring pipelines, plus MiNiFi and Registry subprojects for edge collection and flow versioning. It's aimed at data engineering and infra teams who need drag-and-drop ETL/streaming pipelines with provenance tracking rather than hand-written code.
Provenance tracking with full lineage graphs is genuinely useful for debugging and audit requirements that most streaming tools skip entirely. The extensible processor/controller-service model plus native Python processor support means you're not locked into Java for extensions. Clustering and backpressure/prioritization for throughput vs latency tradeoffs are built in rather than bolted on, and the CI setup (system tests, integration tests, code coverage, docker tests) reflects an actual engineering discipline behind a 15+ year old ASF project.
This is a heavyweight JVM monolith - Java 21 requirement, huge multi-module Maven build, and a UI-driven flow model that doesn't version well in git without the separate Registry component. The visual flow paradigm makes code review and diffing pipeline changes painful compared to code-first tools like Airflow or dbt. Operationally it's not lightweight: clustering, ZooKeeper dependencies historically, and JVM tuning are real costs for what's often just data movement. Default single-user auth with generated random credentials is a rough starting point for anyone expecting sane defaults for team use.