Kafka Connect TimescaleDB (Debezium) SMT Usage Reference for Confluent Platform

This topic provides usage information for the open-source Debezium SMT io.debezium.connector.v2.postgresql.transforms.timescaledb.TimescaleDb.

Description

The TimescaleDb SMT processes raw change event records captured from TimescaleDB databases. Use this SMT to perform logical routing, add relevant metadata, and streamline the handling of time-series data captured by the Debezium PostgreSQL connector.

For complete details, see the official Debezium TimescaleDB SMT documentation.

Limitations

  • The TimescaleDb SMT is only available for Debezium PostgreSQL CDC V2 Source connectors.

  • The SMT requires a dedicated database connection to the TimescaleDB database for metadata lookups. It cannot reuse the connector-level database configuration.

Predicates

Configure transformations with predicates to ensure they only process records satisfying a particular condition. You can also use predicates in a transformation chain along with the Kafka Connect Filter (Kafka) SMT Usage Reference for Confluent Platform to conditionally filter specific records. For more information, refer to Predicates.

Example

The following example shows how to configure and use the TimescaleDb SMT with a Debezium PostgreSQL connector.

"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "192.168.99.100",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname": "postgres",
"topic.prefix": "dbserver1",
"plugin.name": "pgoutput",
"schema.include.list": "_timescaledb_internal",
"transforms": "timescaledb",
"transforms.timescaledb.type": "io.debezium.connector.v2.postgresql.transforms.timescaledb.TimescaleDb",
"transforms.timescaledb.database.hostname": "timescaledb",
"transforms.timescaledb.database.port": "5432",
"transforms.timescaledb.database.user": "postgres",
"transforms.timescaledb.database.password": "postgres",
"transforms.timescaledb.database.dbname": "postgres",
"transforms.timescaledb.target.topic.prefix": "timescaledb",
"transforms.timescaledb.schema.list": "_timescaledb_internal"

Properties

The following table lists the configuration options that you can set for the TimescaleDB integration SMT.

Name

Description

Type

Default

Importance

database.hostname

The hostname or IP address of the TimescaleDB metadata server used by the SMT for metadata lookups.

string

high

database.port

The port number of the TimescaleDB metadata server.

int

5432

high

database.user

The database user for the TimescaleDB metadata connection.

string

high

database.password

The password for the TimescaleDB metadata connection.

string

high

database.dbname

The name of the TimescaleDB database containing the change data.

string

high

target.topic.prefix

The namespace prefix for the routed target topics.

string

timescaledb

medium

schema.list

Comma-separated list of schema names that contain raw internal TimescaleDB data tables.

list

_timescaledb_internal

high