fluss

Apache Fluss is an open-source streaming storage system designed for real-time analytics and AI workloads. It serves as a real-time data layer for Lakehouse architectures, bridging the gap between data streaming and data Lakehouse. Fluss enables low-latency, high-throughput data ingestion and processing while integrating with popular compute engines like Apache Flink and Apache Spark, with StarRocks support planned. Key features include: - **Sub-Second Data Freshness**: Continuous ingestion and immediate data availability for low-latency analytics and real-time decision-making. - **Streaming & Lakehouse Unification**: Uses tables as a single abstraction to unify real-time and historical data across engines, enabling streaming-native storage with low-latency access on top of lakehouse. - **Columnar Streaming**: Based on Apache Arrow, it supports database primitives on data streams, column pruning, and predicate pushdown to minimize I/O and network costs. - **Compute-Storage Separation**: Stream processors focus on computation while Fluss manages state and storage, offering features like deduplication, partial updates, delta joins, and aggregation merge engines. - **ML & AI-Ready Storage**: A unified storage layer supporting row-based, columnar, vector, and multi-modal data for real-time feature stores and centralized ML/AI data repositories. - **Changelogs & Decision Tracking**: Built-in changelog generation provides an append-only history of state and decision evolution for auditing, reproducibility, and observability. ## Building Prerequisites: Unix-like environment, Git, Maven (>= 3.8.6), and Java 11. ```bash git clone https://github.com/apache/fluss.git cd fluss ./mvnw clean package -DskipTests ``` The build uses Maven Wrapper to ensure the correct Maven version. The project is installed in `build-target`. ## Contributing Apache Fluss is open-source and welcomes contributions. You can join discussions, open issues for bugs or feature requests, and contribute code or documentation. ## License Apache Fluss is licensed under the Apache License 2.0.