canal

## Introduction Canal [kə'næl], meaning waterway/pipe/channel, is primarily used for incremental data subscription and consumption based on MySQL database binary log parsing. In its early days, Alibaba transitioned from business triggers to database log parsing due to cross-datacenter synchronization needs, spawning numerous incremental subscription and consumption businesses. Businesses based on log incremental subscription and consumption include: database mirroring, real-time database backup, index building and real-time maintenance (splitting heterogeneous indexes, inverted indexes, etc.), business cache refresh, and incremental data processing with business logic. Currently, canal supports source MySQL versions including 5.1.x, 5.5.x, 5.6.x, 5.7.x, and 8.0.x. ## How It Works ### MySQL Master-Slave Replication Principle - MySQL master writes data changes to the binary log (recorded as binary log events) - MySQL slave copies master's binary log events to the relay log - MySQL slave replays events from the relay log, applying data changes to its own data ### Canal Working Principle - Canal simulates the MySQL slave interaction protocol, masquerading as a MySQL slave, and sends the dump protocol to the MySQL master - Upon receiving the dump request, the MySQL master starts pushing binary logs to canal - Canal parses the binary log objects (originally byte streams) ## Major Version Update Notes ### Canal 1.1.x Versions - Overall performance testing and optimization, improved by 150% (#726) - Native support for Prometheus monitoring (#765) - Native support for Kafka message delivery (#695) - Native support for Alibaba Cloud RDS binlog subscription (solving automatic master-slave switch/oss binlog offline parsing) - Native support for Docker images (#801) ### Canal 1.1.4 Version - Introduced WebUI capability, added the canal-admin project, supporting dynamic management of canal via WebUI, including online white-screen operations for configuration, tasks, logs, etc. ## Documentation Comprehensive documentation is provided, including Home, Introduction, QuickStart (Docker, Kafka/RocketMQ, Alibaba Cloud RDS, Prometheus), Canal Admin (QuickStart, Guide, ServerGuide, Docker), AdminGuide, ClientExample, ClientAPI, Performance, DevGuide, BinlogChange (MySQL 5.6/MariaDB), TableMetaTSDB, ReleaseNotes, Download, and FAQ. ## Multiple Languages Canal adopts a client-server model, with the interaction protocol using protobuf 3.0. Clients can implement consumption logic in different languages: - Java client: official examples - C# client: CanalSharp - Go client: canal-go - PHP client: canal-php - Python client: canal-python - Rust client: canal-rs - Nodejs client: canal-nodejs As a tool for MySQL binlog incremental retrieval and parsing, canal can deliver change records to MQ systems (such as Kafka/RocketMQ), leveraging MQ's multi-language capabilities. ## Tools Developed Based on Canal - canal2sql (generates SQL based on binlog) ## Related Open Source Projects and Products - Canal consumer-side open source project: Otter - Alibaba's Oracle data migration and synchronization tool: yugong - Alibaba's offline synchronization open source project: DataX - Alibaba's database connection pool open source project: Druid - Alibaba's real-time data synchronization tool: DTS ## Issue Feedback - Report issues: GitHub Issues - Project issues are also synchronized to the Alibaba Cloud Developer Community