Configuration Reference for Databricks Delta Lake Sink for Confluent Platform

The following connector configuration properties are used with the Databricks Delta Lake Sink connector for Confluent Platform. For common sink connector configuration properties, see the Kafka Sink Connector Configuration Reference page.

Note

These are properties for the self-managed connector. If you are using Confluent Cloud, see Databricks Delta Lake Sink Connector for Confluent Cloud.

delta.lake.table2partition.map

A map of tables to partition fields in the format of comma-separated tuples (for example, <table-1>:<partition-1>,<table-2>:<partition-2>,...)

  • Type: String

  • Importance: Low

  • Default value: “”

delta.lake.host.name

The Databricks Delta Lake server hostname.

  • Type: String

  • Importance: High

delta.lake.http.path

The HTTP path used to connect to the Databricks Delta Lake instance.

  • Type: String

  • Importance: High

delta.lake.token

The personal access token authenticates the user when connecting to the Databricks Delta Lake instance using JDBC.

  • Type: Password

  • Importance: High

staging.s3.access.key.id

The AWS access key is used to connect to the S3 staging bucket.

  • Type: Password

  • Importance: High

staging.s3.secret.access.key

The AWS secret access key is used to connect to the S3 staging bucket.

  • Type: Password

  • Importance: High

staging.bucket.name

The S3 staging bucket where files get written to from Kafka, and that subsequently get copied into the Databricks Delta Lake table.

  • Type: String

  • Importance: High

delta.lake.table.format

A format string for the destination table name, which may contain ${topic} as a placeholder for the originating topic name. For example, to create a table named kafka-orders based on a Kafka topic named orders, you would enter kafka-${topic} in this field. Note that you must use the ${topic} placeholder if you have multiple originating topics.

  • Type: String

  • Default value: ${topic}

  • Importance: Medium

delta.lake.topic2table.map

Map of topics to tables in the format of comma-separated tuples. For example, <topic-1>:<table-1>,<topic-2>:<table-2>,....

  • Type: String

  • Default value: ${topic}

  • Importance: Low

delta.lake.table.auto.create

Whether to automatically create the destination table (based on the record schema) if it does not exist. Defaults to false.

  • Type: String

  • Default value: false

  • Importance: Medium

delta.lake.tables.location

The underlying location where the data in the Delta Lake table or tables are stored.

  • Type: String

  • Default value: “”

  • Importance: Medium

delta.lake.table2partition.map

Map of tables to partition fields in the format of comma-separated tuples. For example, <table-1>:<partition-1>,<table-1>:<partition-2>,<table-2>:<partition-3>....

  • Type: String

  • Default value: “”

  • Importance: Low

flush.interval.ms

The time interval in milliseconds (ms) to periodically invoke file commits. This property ensures the connector invokes file commits at every configured interval. The commit time is adjusted to 00:00 UTC. The commit is performed at the scheduled time, regardless of the last commit time or number of messages. This configuration is useful when you have to commit your data based on current server time, like at the beginning of each hour. The default value used is 300,000 ms (5 minutes).

  • Type: Long

  • Default value: 300000

  • Importance: Medium

delta.lake.topic2table.map

Map of topics to tables (optional). Create mapping as comma-separated tuples. For example: <topic-1>:<table-1>,<topic-2>:<table-2>,.... If you use this property, the connector ignores any string entered for delta.lake.table.format.

  • Type: String

  • Default value: ${topic}

  • Importance: Low

CSFLE configuration

csfle.enabled

Accepts a boolean value. CSFLE is enabled for the connector if csfle.enabled is set to True.

  • Type: boolean

  • Default: False

auto.register.schemas

Specifies if the Serializer should attempt to register the Schema with Schema Registry.

  • Type: boolean

  • Default: true

  • Importance: medium

use.latest.version

Only applies when auto.register.schemas is set to false. If auto.register.schemas is set to false and use.latest.version is set to true, then instead of deriving a schema for the object passed to the client for serialization, Schema Registry uses the latest version of the schema in the subject for serialization.

  • Type: boolean

  • Default: true

  • Importance: medium