Luigi

Luigi was built at Spotify and open-sourced in 2012 to stitch together long-running batch processes such as Hadoop jobs, database dumps and machine learning steps. Pipelines are plain Python: each Task declares the tasks it requires, the output Target it produces and how to run. Luigi resolves the dependency graph, skips tasks whose outputs already exist, and ships a lightweight central scheduler with a web UI for visualizing progress. It has no built-in cron-style scheduler, so it is typically triggered by cron or another orchestrator.