luigi

Luigi is a Python package (tested on Python 3.10 through 3.14) for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, failure handling, command-line integration and more. Installation is via pip: `pip install luigi` for the latest stable release from PyPI, or `pip install luigi[toml]` to add TOML-based configuration support. Bleeding-edge code can be installed directly from the GitHub repository. Background: Luigi addresses the plumbing typically associated with long-running batch processes. Tasks can be anything, but are commonly long-running jobs such as Hadoop jobs, dumping data to or from databases, or running machine learning algorithms. Luigi is not intended to replace lower-level data processing tools like Hive, Pig or Cascading; instead it stitches many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python snippet, a database table dump, or anything else. It supports building long-running pipelines comprising thousands of tasks that take days or weeks to complete. Luigi ships with a toolbox of common task templates, including support for Python mapreduce jobs in Hadoop as well as Hive and Pig jobs. It also provides file system abstractions for HDFS and local files that ensure file system operations are atomic, so a data pipeline will not crash in a state containing partial data. The Luigi server includes a web interface for searching and filtering tasks, and the visualiser provides a graphical overview of a workflow's dependency graph, where each node represents a task to be run and colors indicate completed versus pending tasks. Philosophy: conceptually Luigi is similar to GNU Make, with tasks that may depend on other tasks, and shares some similarities with Oozie and Azkaban. A major difference is that Luigi is not built specifically for Hadoop and can be extended with other kinds of tasks. Everything is in Python: instead of XML configuration or similar external data files, the dependency graph is specified within Python, making it easy to build complex dependency graphs where dependencies can involve date algebra or recursive references to other versions of the same task. Workflows can still trigger things outside Python, such as running Pig scripts or scp'ing files. Adoption: Luigi is used internally at Spotify to run thousands of tasks every day organized in complex dependency graphs, mostly Hadoop jobs, powering recommendations, toplists, A/B test analysis, external reports and internal dashboards. The README lists many other organizations that have written about or presented on Luigi, including Foursquare, Stripe, Buffer, SeatGeek, Treasure Data, AdRoll, Groupon/OrderUp, Red Hat Marketing Operations, GetNinjas, voyages-sncf.com, Open Targets, Leipzig University Library, Glossier, Data Revenue, Uppsala University, GIPHY, xtream and CIAN, plus additional companies that have not published about it. The exact number of users is unknown since Luigi is open source without registration walls. External links include a Google Groups mailing list for discussions and questions, PyPI releases, the GitHub source code, and a Hubot integration plugin for Slack, Hipchat and similar services. Authors: Luigi was built at Spotify, mainly by Erik Bernhardsson and Elias Freider, with many other contributors since it was open sourced in late 2012. Arash Rouhani was chief maintainer from 2015 to 2019, and Spotify's Data Team now maintains Luigi.