// the find
kubeflow/pipelines
Machine Learning Pipelines for Kubeflow
Kubeflow Pipelines is a platform for building and running ML workflows on Kubernetes, using Argo Workflows under the hood for DAG execution. It gives you a Python SDK to define pipeline components as containerized steps, a UI for tracking runs and experiments, and versioned pipeline storage. It's aimed at ML engineers who need repeatable, auditable training and inference workflows at scale.
The separation between pipeline definition (Python SDK compiling to protobuf IR) and execution (Argo on Kubernetes) is clean — you write Python, it runs as containers, no hidden magic. The SDK v2 IR format is portable enough that other backends (Vertex AI Pipelines) can run the same pipeline spec without changes. The caching layer is genuinely useful: it fingerprints task inputs and skips re-execution on reruns, which matters a lot for expensive data preprocessing steps. Active development cadence with a push as recent as yesterday and solid CI coverage (20+ workflow files) gives reasonable confidence it won't stall on you.
Getting it running locally for development is painful — you realistically need a Kubernetes cluster, and the setup docs assume you have one ready. The legacy v1 API surface is still very much alive alongside v2, which means the documentation is inconsistently split and you'll waste time on examples that use the old SDK patterns. Multi-user isolation relies on Istio for namespace-level auth, which adds substantial ops complexity that catches teams off guard if they're not already running a service mesh. Error messages from failed pipeline runs surface as Argo workflow conditions and pod logs scattered across namespaces — there's no aggregated failure view in the KFP UI, so debugging a broken step means hunting through kubectl.