hudi

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lakehouse platform built around a high-performance open table format. It enables organizations to ingest, index, store, serve, transform, and manage data across multiple cloud data environments while maintaining data in open formats on cloud storage. Key capabilities include: **Ingestion**: Built-in tools for Apache Spark and Apache Flink users, supporting half-a-dozen file formats, database change logs, and streaming data systems. Includes a Kafka Connect sink for bringing external data sources. **Storage**: Optimized storage format supporting both row and columnar data. Features timeline metadata to track change history, automatic file size and layout management using statistics, savepoints for data versioning and recovery, and schema tracking with evolution. **Indexing**: Scalable indexing subsystem that accelerates snapshot queries, maintained automatically by writes. Tracks file listings, column-level and partition-level statistics for efficient query planning. Provides record-level indexing via row-oriented file formats and bloom filters, plus logical partitioning using expression indexes decoupled from physical storage partitioning. **Writing**: Atomic commits with rollback/restore support. Fast upsert/delete operations leveraging record-level indexes. Snapshot isolation between writers and queries. Offers both optimistic concurrency control (Read-Modify-Write for relational data model) and non-blocking concurrency control (streaming data model with out-of-order/late data handling). **Queries**: Multiple query types on a single table: Snapshot Query (latest committed state), Incremental Query (changes since a point in time), Change-Data-Capture Query (change stream with before/after images), Time-Travel Query (table as of a given time), and Read Optimized Query (columnar storage performance via compaction). **Table Management**: Automatic, hands-free table services integrated into Spark/Flink writers or operated independently. Configurable scheduling with failure handling. Cleaning and TTL management for storage reclamation. Clustering and space-filling curve algorithms for data layout optimization. Asynchronous compaction of row-oriented data into columnar formats. Consistent index building during ongoing operations. Catalog synchronization with Hive Metastore, AWS Glue, Google BigQuery, Apache XTable, and more. Hudi supports building with multiple Spark versions (3.3 through 4.2) and Flink versions (1.18 through 2.2), with Scala 2.12/2.13 options. The project uses Maven for builds and provides comprehensive test profiles (unit, functional, integration).