Dagster

Dagster is a cloud-native orchestration and observability tool for data engineering, built with a focus on data assets, type safety, testing, lineage, and developer productivity. You author pipelines (or “jobs/assets”) in Python using constructs like ops, graphs, and software-defined assets (SDAs), and Dagster provides execution backends, scheduling, observability via its UI (Dagit), and integration with many data systems. It unifies development, testing, and production in a consistent model, exposing metadata about data assets (freshness, lineage, partitions) and enabling workflows that are data-aware and introspective.