Curated open source apps, developer tools and self-hosted projects.
Ibis is a portable Python dataframe library that compiles a single dataframe API into SQL or native code for over 20 backends, including DuckDB, Polars, BigQuery, Snowflake and PostgreSQL. It supports lazy expressions, interactive exploration, and mixing Python with SQL.
Spark NLP is an Apache Spark-based NLP library offering scalable annotations, 100,000+ pretrained pipelines and models in 200+ languages, and support for transformers, LLMs, speech and vision tasks across Python, R, Java, Scala and Kotlin.
SynapseML is an open-source library from Microsoft that simplifies building massively scalable machine learning pipelines on Apache Spark, offering composable distributed APIs for text analytics, vision, anomaly detection and more, usable from Python, R, Scala, Java and .NET.
Sail is a Rust-native, drop-in Apache Spark replacement supporting batch, streaming, and AI workloads via Spark Connect. It offers high performance, low cost, and compatibility with existing PySpark code.
