Debezium MySQL Source Connector

The Debezium’s MySQL Connector is a source connector that can obtain a snapshot of the existing data and record all of the row-level changes in the databases on a MySQL server/cluster. The first time it connects to a MySQL server, it reads a consistent snapshot of all of the databases. When that snapshot is complete, the connector continuously reads the changes that were committed to MySQL and generates corresponding insert, update, and delete events. All of the events for each table are recorded in a separate Kafka topic, where they can be easily consumed by applications and services.

  • Confluent supports MySQL connector version 0.9.3 and later.
  • Confluent supports using this connector with MySQL 5.6 or later.

Install the MySQL Connector

You can install this connector by using the Confluent Hub client (recommended) or you can manually download the ZIP file.

confluent-hub install debezium/debezium-connector-mysql:latest

You can install a specific version by replacing latest with a version number. For example:

confluent-hub install debezium/debezium-connector-mysql:0.9.4

Enable the Binary Log on MySQL Server

The MySQL server must be configured to use a row-level binary log, which is described in more detail in the MySQL documentation. MySQL’s binary log, or binlog, records all operations in the same order they are committed by the database, including changes to the schemas of tables or changes to data stored within the tables. MySQL uses its binlog for replication and recovery.

Debezium’s MySQL connector reads MySQL’s binary log to understand what and in what order data has changed. It then produces a change event for every row-level insert, update, and delete operation in the binlog, recording all the change events for each table in a separate Kafka topic.

This is most often done in the MySQL server configuration file, and will look similar to the following fragment:

server-id         = 223344
log_bin           = mysql-bin
binlog_format     = row
binlog_row_image  = full
expire_logs_days  = 10

where:

  • the value for server-id must be unique for each server and replication client within the MySQL cluster. When you set up the connector, you also assign the connector a unique server ID.
  • the value for log_bin is the base name for the sequence of binlog files.
  • the value for binlog_format must be set to row or ROW.
  • the value for binlog_row_image must be set to full or FULL.
  • the value for expire_log_days is the number of days for automatic binary log file removal. The default is 0, which means "no automatic removal," so be sure to set a value that is appropriate for your environment.

License

The Debezium MySQL connector is an open source connector and does not require a Confluent Enterprise License.

Documentation

Note

Portions of the information provided derives from documentation originally produced by the Debezium Community. Work produced by Debezium is licensed under Creative Commons 3.0.