Apache DolphinScheduler

Apache DolphinScheduler is a distributed, extensible data orchestration platform with a visual DAG editor for building, scheduling and monitoring batch and data workflows.

JavaBatch ProcessingData PipelineBig DataLow-CodeOpen Source

Visit website GitHub Docs

Stars: 14,516
Forks: 5,108
Contributors: 697
Latest: 3.4.3

About Apache DolphinScheduler

DolphinScheduler was open-sourced in 2019 (as EasyScheduler) and became an Apache top-level project in 2021. Workflows are designed as DAGs in a drag-and-drop web UI, or as code with the Python SDK, and support a wide range of task types including Shell, SQL, Python, Spark, Flink, Hive, DataX, Kubernetes and sub-processes. Its decentralized master/worker architecture provides high availability and horizontal scaling, and it adds multi-tenancy, resource management, alerting and complement (backfill) runs, making it popular for big-data batch scheduling.

Key Features

Visual drag-and-drop DAG editor

Workflows as code with the Python SDK (PyDolphinScheduler)

Task types for Shell, SQL, Python, Spark, Flink, Hive, DataX, Kubernetes and more

Decentralized master/worker architecture with high availability

Multi-tenancy, projects and resource center

Backfill (complement data), pause/resume and failure recovery

Alerting via email, Slack, DingTalk, WeChat and others

Pros

Easy for teams that prefer a visual workflow designer

Built-in support for big-data engines

Scales horizontally with high availability

Strong multi-tenant features for shared platforms

Cons

Heavier deployment (registry such as ZooKeeper, database, master and worker servers)

Community and documentation are strongest in Chinese

UI-defined workflows are harder to version and review than pure code

Smaller ecosystem outside the big-data space

Tool Information

Language: Java
License: Apache-2.0
First release: 2019 (Apache TLP 2021)
Latest version: 3.4.3
Maintainer: Apache Software Foundation

Compare Apache DolphinScheduler with other tools

See how Apache DolphinScheduler stacks up against other workflow orchestration engines, feature by feature.