Backward Incompatible Changes in Debezium CDC V2 Connectors

This document highlights backward incompatible changes between version 2 of the Debezium CDC Source connectors and the latest version 1 connectors available on Confluent Cloud.

Version 2 introduces new features and breaking changes that are not backward compatible with version 1. As such, careful consideration and planning is required before moving to version 2.

The breaking changes can be classified into the categories listed.

Configuration property changes

Note the following incompatible changes to configuration properties.

  • Renamed the following configuration properties (while retaining the same semantics):
    • database.server.name renamed to topic.prefix (MySQL, PostgreSQL, and SQL Server plugins)
    • JDBC pass-through properties were earlier passed with the prefix database. but now with the prefix driver. (MySQL, PostgreSQL, and SQL Server plugins)
    • Database schema history configuration properties were earlier prefixed with database.history. but now prefixed with schema.history.internal. (MySQL, PostgreSQL, and SQL Server)
    • event.deserialization.failure.handling.mode renamed to event.processing.failure.handling.mode (MySQL)
  • Replaced the following property with a newer one that has different semantics.
    • database.names replaced database.dbname (SQL Server plugin)
  • Changed the default value for the following properties:
    • schema.name.adjustment.mode default changed from avro to none [Ref] (MySQL, PostgreSQL, and SQL Server)
    • skipped.operations default changed from none to t [truncate] (MySQL and SQL Server)
    • database.ssl.mode (MySQL) and database.sslmode (PostgreSQL) connectors now use a secure (encrypted) connection if available by default. They fall back to an unencrypted connection if a secure connection cannot be established. [Ref]
    • after.state.only default changed from true to false (MySQL, PostgreSQL, and SQL Server)

Record schema changes

Note the following changes to change event topics and transaction metadata topics.

  • All schemas used by Debezium are now defined in a central point, properly named and versioned [Ref 1 and 2]. This can lead to schema compatibility issues if a schema registry is used and an existing connector is upgraded from version 1 to 2. (MySQL, PostgreSQL, and SQL Server)

    For example, in version 2.x, consider the following schema for the transaction field which is present in the record for change event topic:

    {
  "default": null,
  "name": "transaction",
  "type": [
    "null",
    {
      "connect.name": "event.block",
      "connect.version": 1,
      "fields": [ ... ],
      "name": "block",
      "namespace": "event",
      "type": "record"
    }
  ]
}

    The name and namespace fields cause the schema to be backward incompatible. In the older version, they were unnamed and defaulted to:

    "name": "ConnectDefault"
"namespace": "io.confluent.connect.avro"

  • The ts_ms field is added to the transaction metadata records. This can break schema compatibility if a schema based format is used and the connector is upgraded from version 1 to 2. (MySQL, PostgreSQL, and SQL Server)

    {
  "name": "ts_ms",
  "type": "long"
}

  • Debezium MySQL connector represented FLOAT datatype as FLOAT64. Now the correct type FLOAT32 is used. [Ref] (MySQL)

  • Debezium MySQL connector did not set precision for BIGINT UNSIGNED type in precise mode. Now the decimal precision is set to 20. [Ref] (MySQL)

  • The record schema in version 2 contains the before field encapsulating the state of the row before the event occurred. To revert to the version 1 behaviour of excluding the before field, set the after.state.only configuration property to true. (MySQL, PostgreSQL, and SQL Server)

Record content changes

This includes changes to the change event and transaction metadata event content.

  • In version 1, Debezium truncated the trailing zeroes from the millisecond/microsecond part for timezone types regardless of the schema width setting. In version 2, this is no longer the case and Debezium provides the correct number of trailing zeroes. [Ref 1 and 2] (MySQL, PostgreSQL, and SQL Server)
  • In version 1, the transaction ID in the transaction metadata topic consisted of the PostgreSQL transaction ID. In version 2, this is updated to include the PostgreSQL transaction ID and LSN of a given operation separated by colon, that is, the format is txID:LSN. [Ref 1] (PostgreSQL)
  • The record content in version 2 contains the state of the row before the event occurred. To revert to the version 1 behaviour of excluding the state of the row before the event occurred, set the after.state.only configuration property to true. (MySQL, PostgreSQL, and SQL Server)

Removed wal2json plugin support for PostgreSQL

In the older version of the connector, the plugin.name property supported the following values: decoderbufs, pgoutput, wal2json, wal2json_rds, wal2json_streaming, wal2json_rds_streaming. However, the newer version only supports decoderbufs and pgoutput [Ref]. As a result, there is no migration path for existing connectors that are configured with the wal2json plugin.

Given these backward incompatible changes between version 1 and 2 of the CDC connectors, version 2 is being provided as a new set of connectors on Confluent Cloud, alongside the existing version 1 connectors. You have the choice of provisioning either the version 1 or 2 connector. At a later stage, the version 1 connectors will be deprecated and then unsupported.

More information on the set of backward incompatible changes can be found in the Debezium release notes for each version.