Kafka Connect TimescaleDB (Debezium) SMT Usage Reference for Confluent Platform

The TimescaleDb SMT (io.debezium.connector.v2.postgresql.transforms.timescaledb.TimescaleDb) processes raw change event records captured from TimescaleDB databases.

Description

Use the TimescaleDb SMT to perform logical routing, add metadata, and streamline the handling of time-series data captured by the Debezium PostgreSQL connector.

For complete details, see the official Debezium TimescaleDB SMT documentation. For other topic-routing SMTs, see Kafka Connect Single Message Transformations Reference for Confluent Platform.

Limitations

  • The TimescaleDb SMT is only available for Debezium PostgreSQL change data capture (CDC) V2 Source connectors.

  • The SMT requires a dedicated database connection to the TimescaleDB database for metadata lookups. It cannot reuse the connector-level database configuration.

Properties

The following table lists the configuration options that you can set for the TimescaleDB integration SMT.

Name

Description

Type

Default

Importance

database.hostname

The hostname or IP address of the TimescaleDB metadata server used by the SMT for metadata lookups.

string

high

database.port

The port number of the TimescaleDB metadata server.

int

5432

high

database.user

The database user for the TimescaleDB metadata connection.

string

high

database.password

The password for the TimescaleDB metadata connection.

string

high

database.dbname

The name of the TimescaleDB database containing the change data.

string

high

target.topic.prefix

The namespace prefix for the routed target topics.

string

timescaledb

medium

schema.list

Comma-separated list of schema names that contain raw internal TimescaleDB data tables.

list

_timescaledb_internal

high

Example: Route TimescaleDB change events

The following example shows how to configure and use the TimescaleDb SMT with a Debezium PostgreSQL connector.

"connector.class": "io.debezium.connector.v2.postgresql.PostgresConnector",
"database.hostname": "192.168.99.100",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname": "postgres",
"topic.prefix": "dbserver1",
"plugin.name": "pgoutput",
"schema.include.list": "_timescaledb_internal",
"transforms": "timescaledb",
"transforms.timescaledb.type": "io.debezium.connector.v2.postgresql.transforms.timescaledb.TimescaleDb",
"transforms.timescaledb.database.hostname": "timescaledb",
"transforms.timescaledb.database.port": "5432",
"transforms.timescaledb.database.user": "postgres",
"transforms.timescaledb.database.password": "postgres",
"transforms.timescaledb.database.dbname": "postgres",
"transforms.timescaledb.target.topic.prefix": "timescaledb",
"transforms.timescaledb.schema.list": "_timescaledb_internal"

Predicates

Configure transformations with predicates to ensure they process only records that satisfy a particular condition. You can also use predicates in a transformation chain with the Kafka Connect Filter (Kafka) SMT Usage Reference for Confluent Platform to conditionally filter specific records. For more information, see Predicates.