peerdb

PeerDB is an ETL/ELT tool focused on moving data in and out of PostgreSQL. It supports several streaming modes: log-based CDC (recommended and actively maintained), cursor-based replication using timestamps or integers, and XMIN-based replication. The Query Replication (QRep) and XMIN mirror types are marked deprecated but remain functional. Sources listed as actively maintained include Postgres, MySQL, MongoDB, CockroachDB and BigQuery. Actively maintained destinations are ClickHouse, ClickHouse Cloud and Postgres. Several destination connectors (S3, Snowflake, BigQuery, ElasticSearch, Kafka, Redpanda, Confluent, Azure Event Hubs, Google Pub/Sub) are deprecated but still functional; BigQuery remains supported as a source. A distinguishing feature is a Postgres-compatible SQL interface for ETL, so users can manage replication with familiar client tools (psql, pgAdmin), BI tools, migration/versioning tools, and existing schedulers such as Airflow. The project emphasizes Postgres-native optimizations: parallel initial loads, handling of TOAST columns, rich data-type mapping including jsonb, arrays and geospatial types, and support for schema changes. It also describes fault-tolerance mechanisms such as state management, automatic retries and idempotency handling, plus configurable batching and parallelism to avoid out-of-memory failures. Getting started is via a shell script that brings up Docker containers for Postgres as catalog, Temporal, the PeerDB server, flow API and workers, and the PeerDB UI. A local development path builds images locally and requires the buf compiler for protobuf generation. The README notes that when ClickHouse runs outside Docker, MinIO staging access must be configured via AWS_ENDPOINT_URL_S3 and network/security-group rules. End-to-end testing is supported locally through Tilt-orchestrated Docker Compose, with test launchers for Postgres, MySQL (GTID and file-position variants), MariaDB and MongoDB to ClickHouse. Data stores use manual initialization to conserve resources, and named volumes provide persistence with a cleanup option in the Tilt UI. PeerDB is also offered natively in ClickHouse Cloud. The project is licensed under AGPLv3 (the README badge references Elv2, while the license section states AGPLv3).