datafusion — Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, providing SQL and DataFrame APIs for building fast analytic systems.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, providing SQL and DataFrame APIs for building fast analytic systems.

About this project

Apache DataFusion is an extensible query engine designed for developers building high-performance database and analytic systems. Written in Rust and utilizing Apache Arrow for its in-memory format, it features a full query planner and a columnar, streaming, multi-threaded, vectorized execution engine. Key capabilities include: - APIs: Provides both SQL and DataFrame interfaces for querying. - Data Support: Built-in support for CSV, Parquet, JSON, and Avro formats. - Extensibility: Allows customization of data sources, query languages, functions, and operators. - Ecosystem: Integrated with subprojects including DataFusion Python, DataFusion Java, DataFusion Comet (Spark accelerator), and DataFusion Ballista (distributed execution). The engine is suitable for creating domain-specific query engines, new database platforms, and data pipelines. It includes various optional features for compression, cryptographic expressions, and Parquet modular encryption.