Metaflow

Metaflow is an open-source Python framework, originally developed at Netflix, for building and operating ML and data science workflows from prototype to production.

PythonMLData ScienceOpen SourceAWSKubernetes

Visit website GitHub Docs

Stars: 10,299
Forks: 1,377
Contributors: 152
Latest: 2.19.39

About Metaflow

Metaflow lets data scientists write workflows as plain Python classes where each @step is a node in a graph. It automatically versions code, data and artifacts for every run, makes it easy to scale individual steps to the cloud (AWS Batch, Kubernetes) with decorators like @resources, and deploys flows to production orchestrators such as AWS Step Functions, Argo Workflows or Airflow without code changes. Netflix open-sourced Metaflow in 2019; it is now developed together with Outerbounds.

Key Features

Flows defined as Python classes with @step methods

Automatic versioning of code, data and artifacts for every run

Scale steps to AWS Batch or Kubernetes with @batch, @kubernetes and @resources

Deploy to AWS Step Functions, Argo Workflows or Airflow for production scheduling

Foreach fan-out and join for parallel workloads

Resume failed runs from the failing step

Client API and notebook cards for inspecting results

Pros

Very approachable for data scientists: plain Python, minimal boilerplate

Seamless path from laptop to cloud compute

Excellent experiment reproducibility through built-in versioning

Proven at Netflix scale

Cons

Relies on another orchestrator (Step Functions, Argo, Airflow) for production scheduling

Cloud deployment requires setting up infrastructure (metadata service, datastore)

Focused on ML/data science; less suited for general-purpose automation

UI is more limited than dedicated orchestration platforms

Tool Information

Language: Python
License: Apache-2.0
First release: 2019 (open-sourced by Netflix)
Latest version: 2.19.39
Maintainer: Netflix / Outerbounds / community

Used By

NetflixCNN23andMeRealtor.com

Compare Metaflow with other tools

See how Metaflow stacks up against other workflow orchestration engines, feature by feature.