Kubeflow Pipelines

Kubeflow Pipelines (KFP) is a platform for building and deploying portable, scalable machine learning workflows on Kubernetes, using a Python SDK to define pipelines as containerized components.

PythonKubernetesMLMLOpsOpen SourceContainers

Visit website GitHub Docs

Stars: 4,237
Forks: 2,145
Contributors: 586
Latest: 2.17.2

About Kubeflow Pipelines

Kubeflow Pipelines is the workflow component of the Kubeflow project. Pipelines are written in Python with the KFP SDK, compiled to an intermediate YAML representation, and executed on Kubernetes, where each step runs in its own container. KFP tracks runs, artifacts and metadata, provides a UI for comparing experiments, and supports caching of step outputs. The same pipeline definition can also run on managed backends such as Google Cloud Vertex AI Pipelines.

Key Features

Python SDK (@dsl.component, @dsl.pipeline) for defining pipelines

Each step runs as a container on Kubernetes

Artifact and metadata tracking (ML Metadata)

Experiments, runs and recurring runs in a web UI

Step-level caching to skip unchanged work

Compiled pipelines are portable YAML (IR) and run on Vertex AI Pipelines

Control flow: conditions, loops (ParallelFor) and exit handlers

Pros

Purpose-built for ML workflows and experiment tracking

Portable pipeline definitions across Kubernetes clusters and Vertex AI

Strong isolation and reproducibility through containers

Part of the broader Kubeflow MLOps ecosystem

Cons

Requires a Kubernetes cluster and non-trivial operations

Heavier than needed for simple or non-ML pipelines

Container-per-step adds latency and build overhead

Breaking changes between KFP v1 and v2 SDKs

Tool Information

Language: Python SDK; Go backend
License: Apache-2.0
First release: 2018
Latest version: 2.17.2
Maintainer: Kubeflow community (CNCF)

Compare Kubeflow Pipelines with other tools

See how Kubeflow Pipelines stacks up against other workflow orchestration engines, feature by feature.