Discover quality open-source projects, submit projects anonymously, and claim and edit your own project.
A little curiosity. A world of open source.
THE FIRST COLLECTIONLightweight Python execution pipeline library that uses @action decorators to define tasks, supports retry, parallel execution, observability, and one-click deployment to 34+ cloud platforms including AWS Lambda, GCP Cloud Run, and Kubernetes.
Dagster is a cloud-native data pipeline orchestrator designed for the development, production, and observation of data assets throughout their entire lifecycle.
Apache Airflow is a platform to programmatically author, schedule, and monitor workflows as code, commonly used for data pipelines and AI/ML orchestration.
Pathway is a Python ETL framework for stream processing, real-time analytics, and AI pipelines including LLM and RAG applications. It runs Python code on a Rust engine for incremental computation, supports batch and streaming data with the same code, and deploys via Docker and Kubernetes.
A fully serverless, event-driven clickstream data pipeline on AWS that ingests, transforms, and stores clickstream events in Parquet format using API Gateway, Kinesis, Firehose, Lambda, and Athena, with production-ready IaC via AWS SAM.
Flow is a single-binary, high-performance ETL engine and workflow orchestrator written in Go. It features an embedded visual web builder, declarative XML pipelines, cross-database support, and built-in audit telemetry.
Kestra is an open-source, event-driven orchestration platform designed for data, AI, and infrastructure workflows, utilizing a declarative YAML interface.
CocoIndex is an open-source incremental indexing framework designed to provide AI agents and LLM applications with continuously fresh context by reprocessing only changed data.
Vector is an open-source observability data pipeline written in Rust. It collects, transforms and routes logs and metrics as either an agent or an aggregator, letting teams control where telemetry goes and reduce vendor costs.
