sail

Sail is an open-source, Rust-native distributed compute engine designed as a drop-in replacement for Apache Spark. It unifies batch processing, stream processing, and compute-intensive AI workloads on a single multimodal engine. Sail is compatible with the Spark Connect protocol, allowing existing PySpark applications to connect without code changes. The engine is built on Apache Arrow and Apache DataFusion, leveraging columnar execution and SIMD instructions for high performance. It supports Python UDFs, UDAFs, UDWFs, and UDTFs with zero-copy data sharing. Sail provides native support for Delta Lake and Apache Iceberg table formats, and integrates with catalog providers such as AWS Glue, Unity Catalog, Hive Metastore, and Microsoft OneLake. Storage backends include AWS S3, Azure, Google Cloud Storage, HDFS, and more. The project claims significant performance improvements over Spark, with derived TPC-H benchmarks showing about 10x faster query execution and 98% lower infrastructure costs. Sail can be deployed locally via CLI or Python API, or on Kubernetes for distributed processing. The documentation includes installation guides, migration guides, and detailed feature references.