Kafka Connect TimescaleDB (Debezium) SMT Usage Reference for Confluent Platform

This topic provides usage information for the open-source Debezium SMT io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb.

Description

The TimescaleDb SMT processes raw change event records captured from TimescaleDB databases. Use this SMT to perform logical routing, add relevant metadata, and streamline the handling of time-series data captured by the Debezium PostgreSQL connector.

For complete details, see the official Debezium TimescaleDB SMT documentation.

Limitations

  • The TimescaleDb SMT is only available for Debezium PostgreSQL CDC V2 Source connectors.

  • The SMT requires a dedicated database connection to the TimescaleDB database for metadata lookups. It cannot reuse the connector-level database configuration.

Predicates

Transformations can be configured with predicates so that the transformation is applied only to records which satisfy a condition. You can use predicates in a transformation chain and, when combined with the Kafka Connect Filter (Kafka) SMT Usage Reference for Confluent Cloud or Confluent Platform, predicates can conditionally filter out specific records. For details and examples, see Predicates.

Example

The following example shows how to configure and use the TimescaleDb SMT with a Debezium PostgreSQL connector.

"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "192.168.99.100",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname": "postgres",
"topic.prefix": "dbserver1",
"plugin.name": "pgoutput",
"schema.include.list": "_timescaledb_internal",
"transforms": "timescaledb",
"transforms.timescaledb.type": "io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb",
"transforms.timescaledb.database.hostname": "timescaledb",
"transforms.timescaledb.database.port": "5432",
"transforms.timescaledb.database.user": "postgres",
"transforms.timescaledb.database.password": "postgres",
"transforms.timescaledb.database.dbname": "postgres"

Properties

Name

Description

Type

Default

Importance

database.hostname

The hostname or IP address of the TimescaleDB metadata server used by the SMT for metadata lookups.

string

high

database.port

The port number of the TimescaleDB metadata server.

int

5432

high

database.user

The database user for the TimescaleDB metadata connection.

string

high

database.password

The password for the TimescaleDB metadata connection.

string

high

database.dbname

The name of the TimescaleDB database containing the change data.

string

high

topic.prefix

The namespace prefix for the routed target topics.

string

timescaledb

medium

schema.include.list

Comma-separated list of schema names that contain raw internal TimescaleDB data tables.

string

_timescaledb_internal

medium