dagster — Dagster is a cloud-native data pipeline orchestrator designed for the development, production, and observation of data assets throughout their entire lifecycle.

Dagster is a cloud-native data pipeline orchestrator designed for the development, production, and observation of data assets throughout their entire lifecycle.

About this project

Dagster is an orchestration platform focused on the development and maintenance of data assets such as tables, datasets, machine learning models, and reports. It utilizes a declarative programming model where users define data assets as Python functions, allowing the system to manage execution timing and asset updates. Key capabilities include: - Integrated lineage and observability to track data flow and diagnose performance issues. - A declarative approach to asset creation and task management. - Support for the full development lifecycle, including local development, unit testing, integration testing, and production deployment. - A unified control plane for centralized metadata, cataloging, and diagnostics. - A library of integrations for various popular data tools and infrastructure. Dagster is available on PyPI and supports Python versions 3.9 through 3.14.