Configuration Reference for Databricks Delta Lake Sink for Confluent Platform

The following connector configuration properties are used with the Databricks Delta Lake Sink connector for Confluent Platform. For common sink connector configuration properties, see the Kafka Sink Connector Configuration Reference page.

Note

These are properties for the self-managed connector. If you are using Confluent Cloud, see Databricks Delta Lake Sink Connector for Confluent Cloud.

delta.lake.table2partition.map

A map of tables to partition fields in the format of comma-separated tuples (for example, <table-1>:<partition-1>,<table-2>:<partition-2>,...)

  • Type: String
  • Importance: Low
  • Default value: “”
delta.lake.host.name

The Databricks Delta Lake server hostname.

  • Type: String
  • Importance: High
delta.lake.http.path

The HTTP path used to connect to the Databricks Delta Lake instance.

  • Type: String
  • Importance: High
delta.lake.token

The personal access token authenticates the user when connecting to the Databricks Delta Lake instance using JDBC.

  • Type: Password
  • Importance: High
staging.s3.access.key.id

The AWS access key is used to connect to the S3 staging bucket.

  • Type: Password
  • Importance: High
staging.s3.secret.access.key

The AWS secret access key is used to connect to the S3 staging bucket.

  • Type: Password
  • Importance: High
staging.bucket.name

The S3 staging bucket where files get written to from Kafka, and that subsequently get copied into the Databricks Delta Lake table.

  • Type: String
  • Importance: High
delta.lake.table.format

A format string for the destination table name, which may contain ${topic} as a placeholder for the originating topic name. For example, to create a table named kafka-orders based on a Kafka topic named orders, you would enter kafka-${topic} in this field. Note that you must use the ${topic} placeholder if you have multiple originating topics.

  • Type: String
  • Default value: ${topic}
  • Importance: Medium
delta.lake.topic2table.map

Map of topics to tables in the format of comma-separated tuples. For example, <topic-1>:<table-1>,<topic-2>:<table-2>,....

  • Type: String
  • Default value: ${topic}
  • Importance: Low
delta.lake.table.auto.create

Whether to automatically create the destination table (based on the record schema) if it does not exist. Defaults to false.

  • Type: String
  • Default value: false
  • Importance: Medium
delta.lake.tables.location

The underlying location where the data in the Delta Lake table or tables are stored.

  • Type: String
  • Default value: “”
  • Importance: Medium
delta.lake.table2partition.map

Map of tables to partition fields in the format of comma-separated tuples. For example, <table-1>:<partition-1>,<table-1>:<partition-2>,<table-2>:<partition-3>....

  • Type: String
  • Default value: “”
  • Importance: Low
flush.interval.ms

The time interval in milliseconds (ms) to periodically invoke file commits. This property ensures the connector invokes file commits at every configured interval. The commit time is adjusted to 00:00 UTC. The commit is performed at the scheduled time, regardless of the last commit time or number of messages. This configuration is useful when you have to commit your data based on current server time, like at the beginning of each hour. The default value used is 300,000 ms (5 minutes).

  • Type: Long
  • Default value: 300000
  • Importance: Medium
delta.lake.topic2table.map

Map of topics to tables (optional). Create mapping as comma-separated tuples. For example: <topic-1>:<table-1>,<topic-2>:<table-2>,.... If you use this property, the connector ignores any string entered for delta.lake.table.format.

  • Type: String
  • Default value: ${topic}
  • Importance: Low