Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

MongoDB Source Connector (Debezium) Configuration Properties

The MongoDB Source Connector can be configured using a variety of configuration properties.

mongodb.hosts

The comma-separated list of hostname and port pairs (in the form host or host:port) of the MongoDB servers in the replica set. The list can contain a single hostname and port pair. If mongodb.members.auto.discover is set to false, then the host and port pair should be prefixed with the replica set name (e.g., rs0/localhost:27017)

  • Type: List of Strings
  • Importance: High
mongodb.name

A unique name that identifies the connector and/or MongoDB replica set or sharded cluster that this connector monitors. Each server should be monitored by at most one Debezium connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster.

  • Type: Integer
  • Importance: High
mongodb.user

Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.

  • Type: String
  • Importance: Medium
mongodb.password

Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.

  • Type: Password
  • Importance: Medium
database.history.kafka.topic

The full name of the Kafka topic where the connector will store the database schema history.

  • Type: String
  • Importance: High
database.history.kafka.bootstrap.servers

A list of host/port pairs that the connector will use for establishing an initial connection to the Kafka cluster. This connection will be used for retrieving database schema history previously stored by the connector and for writing each DDL statement read from the source database. This should point to the same Kafka cluster used by the Connect process.

  • Type: List of Strings
  • Importance: High

Note

If the Kafka cluster is secured, you must add the security properties prefixed with database.history.consumer.* and database.history.producer.* to the connector configuration, as shown below:

"database.history.consumer.security.protocol": "SASL_SSL",
"database.history.consumer.sasl.mechanism": "PLAIN",
"database.history.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"key\" password=\"secret\";",

"database.history.producer.security.protocol": "SASL_SSL",
"database.history.producer.sasl.mechanism": "PLAIN",
"database.history.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"key\" password=\"secret\";",
mongodb.ssl.enabled

Connector uses SSL to connect to MongoDB instances.

  • Type: String
  • Importance: Low
  • Default: false
mongodb.ssl.invalid.hostname.allowed

When SSL is enabled this setting controls whether strict hostname checking is disabled during connection phase. If true the connection does not prevent man-in-the-middle attacks.

  • Type: String
  • Importance: Low
  • Default: false
database.whitelist

An optional comma-separated list of regular expressions that match database names to be monitored; any database name not included in the whitelist will be excluded from monitoring. By default all databases will be monitored. May not be used with database.blacklist.

  • Type: List of Strings
  • Importance: Medium
  • Default: empty string
database.blacklist

An optional comma-separated list of regular expressions that match database names to be excluded from monitoring; any database name not included in the blacklist will be monitored. May not be used with database.whitelist.

  • Type: List of Strings
  • Importance: Medium
  • Default: empty string
collection.whitelist

An optional comma-separated list of regular expressions that match fully-qualified namespaces for MongoDB collections to be monitored. Any collection not included in the whitelist will be excluded from monitoring. Each identifier is in the form databaseName.collectionName. By default the connector will monitor all collections except those in the local and admin databases. May not be used with collection.blacklist.

  • Type: List of Strings
  • Importance: Low
  • Default: empty string
collection.blacklist

An optional comma-separated list of regular expressions that match fully-qualified namespaces for MongoDB collections to be excluded from monitoring. Any collection not included in the blacklist will be monitored. Each identifier is in the form databaseName.collectionName. May not be used with collection.whitelist.

  • Type: List of Strings
  • Importance: Low
snapshot.mode

Specifies the criteria for running a snapshot (for example, the initial sync) upon startup of the connector. The default is initial, and specifies the connector reads a snapshot when either no offset is found or if the oplog no longer contains the previous offset. The never option specifies that the connector should never use snapshots, instead the connector should tail the log.

  • Type: String
  • Importance: Low
  • Default: intial
field.blacklist

An optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are in the form databaseName.collectionName.fieldName.nestedFieldName, where databaseName and collectionName may contain the wildcard (*) which matches any characters.

  • Type: List of Strings
  • Importance: Low
  • Default: empty string
field.renames

An optional comma-separated list of the fully-qualified replacements of fields that should be used to rename fields in change event message values. Fully-qualified replacements for fields are in the form databaseName.collectionName.fieldName.nestedFieldName:newNestedFieldName, where databaseName and collectionName may contain the wildcard (*) which matches any characters. The colon character (:) is used to determine rename mapping of field. Note that the next field replacement is applied to the result of the previous field replacement in the list. Keep this in mind when renaming multiple fields that are in the same path.

  • Type: String
  • Importance: High
  • Default: empty string
tasks.max

The maximum number of tasks that should be created for this connector. The MongoDB connector attempts to use a separate task for each replica set, so the default is acceptable when using the connector with a single MongoDB replica set. When using the connector with a MongoDB sharded cluster, Confluent recommends specifying a value that is equal to or more than the number of shards in the cluster, so that the work for each replica set can be distributed by Kafka Connect.

  • Type: Integer
  • Importance: Medium
  • Default: 1
initial.sync.max.threads

Positive integer value that specifies the maximum number of threads used to perform an initial sync of the collections in a replica set. Defaults to 1.

  • Type: Integer
  • Importance: Medium
  • Default: 1
tombstones.on.delete

Controls whether a tombstone event should be generated after a delete event. When set to true, the delete operations are represented by a delete event and a subsequent tombstone event. When set to false, only a delete event is sent. Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record is deleted.

  • Type: String
  • Importance: Low
  • Default: true
snapshot.delay.ms

An interval in milliseconds that the connector should wait before taking a snapshot after starting up. Can be used to avoid snapshot interruptions (and potential rebalancing) when starting multiple connectors in a cluster. of connectors.

  • Type: List of Strings
  • Importance: Low
  • Default: empty string

Advanced Configuration Properties

max.queue.size

Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the oplog reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the max.batch.size property.

  • Type: Integer
  • Importance: Low
  • Default: 8192
max.batch.size

Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.

  • Type: Integer
  • Importance: Low
  • Default: 2048
poll.interval.ms

Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 500 milliseconds (1 second).

  • Type: Integer
  • Importance: Low
  • Default: 500
connect.backoff.initial.delay.ms

Positive integer value that specifies the initial delay when trying to reconnect to a primary after the first failed connection attempt or when no primary is available. Defaults to 1000 milliseconds (1 second).

  • Type: Integer
  • Importance: Low
  • Default: 1000
connect.backoff.max.delay.ms

Positive integer value that specifies the maximum delay when trying to reconnect to a primary after repeated failed connection attempts or when no primary is available. Defaults to 120000 milliseconds (120 seconds).

  • Type: Integer
  • Importance: Low
  • Default: 120000
connect.max.attempts

Positive integer value that specifies the maximum number of failed connection attempts to a replica set primary before an exception occurs and task is aborted. Defaults to 16` which results in a little over 20 minutes of attempts before failing, if using the defaults for ``connect.backoff.initial.delay.ms and connect.backoff.max.delay.ms.

  • Type: Integer
  • Importance: Low
  • Default: 16
mongodb.members.auto.discover

Boolean value that specifies whether the addresses in mongodb.hosts are seeds that should be used to discover all members of the cluster or replica set (true), or whether the address(es) in mongodb.hosts should be used as is (false). The default is true and should be used in all cases except where MongoDB is fronted by a proxy.

  • Type: String
  • Importance: Low
  • Default: true

More details can be found in the Debezium connector properties documentation.

Note

Portions of the information provided here derives from documentation originally produced by the Debezium Community. Work produced by Debezium is licensed under Creative Commons 3.0.