arrow — Apache Arrow is a universal columnar memory format and multi-language toolbox designed for efficient data interchange and in-memory analytics.

Apache Arrow is a universal columnar memory format and multi-language toolbox designed for efficient data interchange and in-memory analytics.

About this project

Apache Arrow provides a standardized columnar format for in-memory representation of various data types, enabling data systems to store, process, and move data efficiently. It includes a set of technologies for fast data interchange and analytics across heterogeneous environments. Key components include: - Arrow Columnar Format: A standard for in-memory representation of plain or nested datatypes. - Arrow IPC Format: Efficient serialization for communication between processes. - ADBC (Arrow Database Connectivity): API and drivers for accessing databases and query engines. - Arrow Flight RPC: A protocol for remote services to exchange Arrow data. - Gandiva: An LLVM-based expression compiler for Arrow. The project provides libraries across multiple languages, including C++, Python, Java, JavaScript, Rust, Go, R, Ruby, .NET, Julia, and Swift. Technical capabilities include: - Columnar vector and table-like containers supporting flat or nested types. - Reference-counted off-heap buffer memory management for zero-copy memory sharing. - IO interfaces for local and remote filesystems. - Readers and writers for common file formats such as Parquet and CSV. - A metadata messaging layer based on Google's FlatBuffers.