// the find
insitro/redun
Yet another redundant workflow engine
redun is a Python workflow engine from Insitro where pipelines are just plain Python functions decorated with @task, not a YAML/DSL — the scheduler builds a lazy expression graph and runs it with automatic caching, parallelism, and data lineage tracking. It's aimed at data science and bioinformatics teams running multi-step DAGs across local processes, AWS Batch, Spark, or k8s who are tired of Airflow/Luigi-style boilerplate.
Workflows are regular Python with recursion, closures, and higher-order functions intact, so you don't fight a restrictive DAG-definition DSL to express conditional or dynamic pipelines. Cache invalidation is driven by hashing both input data (file content) and task code (function bytecode), so a code change correctly busts the cache without manual versioning. The call graph recording gives real data provenance — you can query 'what produced this file and from what inputs' after the fact, which most lightweight pipeline tools don't bother with. Per-task executor annotation (thread/process/Batch/Spark/k8s) lets you mix compute backends in one workflow without rewriting it.
The Postgres backend (what you'd want for anything beyond a single laptop) pulls in libpq-dev and gcc as native build deps — not a pip install away for teams on locked-down environments. Caching by hashing function bytecode is clever but fragile: closures over mutable module-level state or non-deterministic code paths can produce stale cache hits that silently return wrong results. The project is heavily AWS-shaped in its examples and defaults (Batch, S3); GCP Batch and k8s executors read as newer and less battle-tested by comparison. Observability is a terminal TUI console, not a web dashboard — fine for one engineer debugging a run, less fine for a team that wants shared visibility into what's running.