materialize

Materialize is a real-time data integration platform that creates and continually updates consistent views of transactional data from across an organization. Its SQL interface is meant to make serving and accessing live data broadly accessible, and it can be deployed wherever your infrastructure runs. Common adoption patterns described in the README: - Query Offload (CQRS): scale complex read queries more efficiently than a read replica, without cache invalidation headaches. - Integration Hub (ODS): extract, load and incrementally transform data from multiple sources, creating live views that can be queried directly or pushed downstream. - Operational Data Mesh (ODM): use SQL to create and deliver real-time, strongly consistent data products across services and domains. Consistency and execution model: Materialize aims to give correct, consistent answers with minimal latency rather than approximate or eventually consistent results, including when joining data from multiple upstream systems. Queries are recast as dataflows that react efficiently to data changes. The managed service is described as cloud-native with high availability via multi-active replication, horizontal scalability across machines, and storage backed by cloud object storage such as Amazon S3. Enterprise and Community editions can be self-managed. Data ingestion: direct reads from PostgreSQL or MySQL replication streams, Kafka and Kafka API-compatible systems such as Redpanda, and SaaS applications via webhooks. Transformation and querying: views are defined and read through the PostgreSQL protocol, so existing SQL clients including psql can be used; dbt Core is mentioned as common in production use. Supported SQL features include multi-column and multi-way joins, self-joins, cross-joins, inner and outer joins, subqueries with automatic decorrelation, incremental view maintenance under inserts, updates and deletes, aggregations (min, max, count, sum, stddev and others), HAVING, ORDER BY, LIMIT, DISTINCT, JSON operators and functions in the PostgreSQL dialect, nested views, and shared indices across overlapping subplans. Recursion is also supported for incrementally updating tree and graph structures. The README includes a TPC-H query 15 example showing CREATE SOURCE, CREATE VIEW, CREATE MATERIALIZED VIEW and CREATE INDEX. Data egress: pull-based access through any PostgreSQL-compatible driver, and push-based access via SUBSCRIBE or streaming results to a Kafka topic, with the option to copy updates to object storage. Licensing and development: the standalone database engine source is in this repository under BSL 1.1, converting to Apache 2.0 after four years; single-node use of the standalone engine is free forever. The managed cloud service includes proprietary features such as horizontal scalability, high availability and a web management console. Materialize is primarily written in Rust, with developer docs in doc/developer and Rust API docs hosted separately. Contributions are welcome, with guidance in CONTRIBUTING.md.