# Azure Cosmos DB Sink V2 Connector for Confluent Cloud The fully managed Azure Cosmos DB Sink V2 connector for Confluent Cloud writes data to an Azure Cosmos DB database. The connector polls data from Apache Kafka® and writes to database containers, supporting high-throughput data ingestion with configurable write strategies for enhanced data handling. Confluent Cloud is available through [Azure Marketplace](https://azuremarketplace.microsoft.com/en/marketplace/apps/confluentinc.confluent-cloud-azure-prod?tab=Overview) or [directly from Confluent](https://www.confluent.io/get-started/). #### NOTE If you require private networking for fully managed connectors, make sure to set up the proper networking beforehand. For more information, see [Manage Networking for Confluent Cloud Connectors](networking/internet-resource.md#clusters-connect-cloud). ## V2 Improvements The V2 connector includes the following improvements: * Supports multiple write strategies for enhanced data handling. * Supports service principal authentication using client secrets. * Supports enhanced throughput control for managing data ingestion rates. * Offers improved metadata handling for accurate offset tracking and seamless scalability. ## Features The Azure Cosmos DB Sink V2 connector supports the following features: * **Topic mapping**: Maps the Kafka topic to the Azure Cosmos DB container. * **Multiple key strategies**: - `FullKeyStrategy`: The ID generated is the Kafka record key. This is the default option. - `KafkaMetadataStrategy`: The ID generated is a concatenation of the Kafka topic, partition, and offset. For example: `${topic}-${partition}-${offset}`. - `ProvidedInKeyStrategy`: The ID generated is the `id` field found in the key object. - `ProvidedInValueStrategy`: The ID generated is the `id` field found in the value object. - `TemplateStrategy`: The template string used to populate the document with the `id` field. Every record must have (lower case) `id` field. This is an Azure Cosmos DB requirement. See the [lower case id prerequisite](cc-azure-cosmos-sink.md#cc-azure-cosmos-sink-prereqs). * **Client-side encryption (CSFLE and CSPE) support**: The connector supports CSFLE and CSPE for sensitive data. For more information about CSFLE or CSPE setup, see the [connector configuration](#cc-azure-cosmos-v2-sink-setup-connection). * **Provider integration support**: The connector supports Microsoft’s native identity authorization using Confluent Provider Integration. For more information about provider integration setup, see the [connector authentication](#cc-azure-cosmos-v2-sink-setup-connection). For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ## Limitations Be sure to review the following information. * For connector limitations, see [Azure Cosmos DB Sink Connector](limits.md#azure-cosmos-sink-limits) limitations. * If you plan to use one or more Single Message Transformations (SMTs), see [SMT Limitations](single-message-transforms.md#cc-single-message-transforms-limitations). ## Quick Start Use this quick start to get up and running with the Confluent Cloud Azure Cosmos DB Sink V2 connector. The quick start provides the basics of selecting the connector and configuring it to stream Kafka events to an Azure Cosmos DB container. Prerequisites : - Authorized access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/) cluster on Azure. - The Confluent CLI installed and configured for the cluster. See [Install the Confluent CLI](https://docs.confluent.io/confluent-cli/current/install.html). - [Schema Registry](../get-started/schema-registry.md#cloud-sr-config) must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf). - At least one source Kafka topic must exist in your Confluent Cloud cluster before creating the sink connector. - The Azure Cosmos DB and the Kafka cluster must be in the same region. - The Azure Cosmos DB requires an `id` field in every record. See [ID strategies](cc-azure-cosmos-sink.md#cc-azure-cosmos-sink-id-strategy) for an example of how each of these works. The following strategies are provided to generate the ID: * `FullKeyStrategy`: The ID generated is the Kafka record key. This is the default option. * `KafkaMetadataStrategy`: The ID generated is a concatenation of the Kafka topic, partition, and offset. For example: `${topic}-${partition}-${offset}`. * `ProvidedInKeyStrategy`: The ID generated is the `id` field found in the key object. * `ProvidedInValueStrategy`: The ID generated is the `id` field found in the value object. If you select this ID strategy, you must create a new field named `id`. You can also use the following [ksqlDB statement](/platform/current/ksqldb/developer-guide/ksqldb-reference/create-stream.html). The example below uses a topic named `orders`. * `TemplateStrategy`: The template string used to generate the `id` field. ```sql CREATE STREAM ORDERS_STREAM WITH ( KAFKA_TOPIC = 'orders', VALUE_FORMAT = 'AVRO' ); CREATE STREAM ORDER_AUGMENTED AS SELECT ORDERID AS `id`, ORDERTIME, ITEMID, ORDERUNITS, ADDRESS FROM ORDERS_STREAM; ``` #### NOTE * The connector supports `Upsert` based on `id`. * The connector does not support `Delete` for tombstone records. ### Using the Confluent Cloud Console #### Step 1: Launch your Confluent Cloud cluster To create and launch a Kafka cluster in Confluent Cloud, see [Create a kafka cluster in Confluent Cloud](../get-started/index.md#cloud-create-kafka-cluster). #### Step 2: Add a connector In the left navigation menu, click **Connectors**. If you already have connectors in your cluster, click **+ Add connector**. #### Step 3: Select your connector Click the **Azure Cosmos DB Sink V2** connector card.  #### Step 4: Enter the connector details #### NOTE * Ensure you have all your [prerequisites](cc-azure-cosmos-sink.md#cc-azure-cosmos-sink-prereqs) completed. * An asterisk ( \* ) designates a required entry. At the **Add Azure Cosmos DB Sink V2 Connector** screen, complete the following: ### Topic selection If you’ve already populated your Kafka topics, select the topics you want to connect from the **Topics** list. To create a new topic, click **+Add new topic**. ### Kafka access 1. Select the way you want to provide **Kafka Cluster credentials**. You can choose one of the following options: - **My account**: This setting allows your connector to globally access everything that you have access to. With a user account, the connector uses an API key and secret to access the Kafka cluster. This option is not recommended for production. - **Service account**: This setting limits the access for your connector by using a [service account](service-account.md#s3-cloud-service-account). This option is recommended for production. - **Use an existing API key**: This setting allows you to specify an API key and a secret pair. You can use an existing pair or create a new one. This method is not recommended for production environments. #### NOTE Freight clusters support only service accounts for Kafka authentication. 2. Click **Continue**. ### Authentication 1. Configure the authentication properties: **Authentication method** - **Authentication method**: Under **Azure credentials**, select how you want to authenticate with the database. Your options are: - If you select **Master Key**, enter the following details: * **Cosmos endpoint**: The Cosmos endpoint URL. For example, `https://connect-cosmosdb.documents.azure.com:443/`. * **Cosmos database name**: The name of your Cosmos database. * **Cosmos DB account key**: Enter the account key of the database. - If you select **Microsoft Entra ID application**, enter the following details: * Under the **Provider integration** dropdown, select an existing integration name that has access to your Azure resource required to run this connector. For more information, see [Manage an Microsoft Azure Provider Integration](provider-integration.md#connector-az-pi). * **Cosmos endpoint**: The Cosmos endpoint URL. For example, `https://connect-cosmosdb.documents.azure.com:443/`. * **Cosmos database name**: The name of your Cosmos database. - If you select **Service Principal**, enter the following Cosmos DB connection details: * **Cosmos endpoint**: The Cosmos endpoint URL. For example, `https://connect-cosmosdb.documents.azure.com:443/`. * **Cosmos database name**: The name of your Cosmos database. * **ClientID/ApplicationID**: Enter the clientID/applicationID of the account. * **Client secret/password**: Enter the client secret/password of the account. * **TenantID**: Enter the tenantID of the account. - **Use secret manager**: Fetch sensitive configuration values from a secret manager. **Azure credentials** - **Provider Integration**: Select an existing integration name that has access to your Azure resource required to run this connector. Required for `Microsoft Entra ID application` authentication method. **Secret manager configuration** - **Secret manager**: Select the secret manager to use for retrieving sensitive data. - **Configurations from Secret manager**: Select the configurations whose values Confluent Cloud should fetch from the secret manager. - **Provider Integration**: Select an existing integration name that has access to your Azure resource required to run this connector. Required for `Microsoft Entra ID application` authentication method. **Connect to your Cosmos DB database** - **Cosmos endpoint**: Specify the Cosmos endpoint URL. For example: `https://connect-cosmosdb.documents.azure.com:443/`. **Connect to your Azure Cosmos DB** - **Cosmos Database name**: Specify the Cosmos target database to write records into. **Account details** - **Cosmos DB account key**: Cosmos DB account key. Required for `Master Key` authentication method. - **ClientId/applicationId of service principal**: The clientId/ApplicationId of the service principal. Required for `Service Principal` authentication method. - **Client secret/password of service principal**: The client secret/password of the service principal. Required for `Service Principal` authentication method. - **TenantId of Cosmos DB account**: The tenantId of the Cosmos DB account. Required for `Service Principal` authentication method. 2. Click **Continue**. ### Configuration #### NOTE Configuration properties that are not shown in the Cloud Console use the default values. See [Configuration Properties](#cc-azure-cosmos-v2-sink-config-properties) for all property values and definitions. - **Input Kafka record value format**: Select the input Kafka record value format (data coming from the Kafka topic). A valid schema must be available in [Schema Registry](../get-started/schema-registry.md#cloud-sr-config) to use a schema-based message format (for example, Avro, JSON_SR (JSON Schema), or Protobuf). **Connect to your Azure Cosmos DB** - **Topic-Container map**: In the **Topic-Container Map** field, enter a comma-delimited list of Kafka topics mapped to Cosmos DB containers. For example, `topic#container1,topic2#container2`. **ID Strategy details** - **Id strategy**: The `IdStrategy` class name determines how to generate a unique document ID: - `FullKeyStrategy`: The ID generated is the Kafka record key. - `KafkaMetadataStrategy`: The ID generated is a concatenation of the Kafka topic, partition, and offset. For example: `${topic}-${partition}-${offset}`. - `ProvidedInKeyStrategy`: The ID generated is the `id` field found in the key object. - `ProvidedInValueStrategy`: The ID generated is the `id` field found in the value object. Every record must have (lower case) `id` field. This is an Azure Cosmos DB requirement. See [Lower case id prerequisite](cc-azure-cosmos-sink.md#cc-azure-cosmos-sink-prereqs). - `TemplateStrategy`: The ID generated is based on the template string that you provide. **Write configuration details** - **Cosmos DB item write strategy**: Defaults to **ItemOverwrite**. Select one the following options: * **ItemOverwrite**: Involves using an upsert approach. * **ItemAppend**: Involves using a create approach; ignores pre-existing items, for example, conflicts. * **ItemDelete**: Deletes documents based on id/pk of the data frame. * **ItemDeleteIfNotModified**: Deletes documents based on id/pk of the data frame if the etag hasn’t changed since collecting id/pk. * **ItemOverwriteIfNotModified**: Involves using a create approach if etag is empty; updates/replaces with etag precondition. Otherwise, if document was updated, the precondition failure is ignored. * **ItemPatch**: Partially updates all documents based on the patch config. **Data decryption** - Enable **Client-Side Field Level Encryption** for data decryption. Specify a **Service Account** to access the Schema Registry and associated encryption rules or keys with that schema. Select the connector behavior (`ERROR` or `NONE`) on data decryption failure. If set to `ERROR`, the connector fails and writes the encrypted data in the DLQ. If set to `NONE`, the connector writes the encrypted data in the target system without decryption. For more information on CSFLE or CSPE setup, see [Manage encryption for connectors](csfle.md#connect-csfle). ### **Show advanced configurations** - **Schema context**: Select a schema context to use for this connector, if using a schema-based data format. This property defaults to the **Default** context, which configures the connector to use the default schema set up for Schema Registry in your Confluent Cloud environment. A schema context allows you to use separate schemas (like schema sub-registries) tied to topics in different Kafka clusters that share the same Schema Registry environment. For example, if you select a non-default context, a **Source** connector uses only that schema context to register a schema and a **Sink** connector uses only that schema context to read from. For more information about setting up a schema context, see [What are schema contexts and when should you use them?](../sr/faqs-cc.md#faq-schema-contexts). **Additional Configs** - **Value Converter Replace Null With Default**: Specifies whether to replace fields that have a default value and that are null to the default value. When set to `true`, the connector uses the default value; otherwise, it uses `null`. Applies to the `JSON` converter. - **Value Converter Schema ID Deserializer**: Sets the class name of the schema ID deserializer for values. The deserializer reads schema IDs from message headers. - **Value Converter Reference Subject Name Strategy**: Sets the subject reference name strategy for values. Valid entries are `DefaultReferenceSubjectNameStrategy` or `QualifiedReferenceSubjectNameStrategy`. You can use this strategy only with `PROTOBUF` format; the default strategy is `DefaultReferenceSubjectNameStrategy`. - **Schema ID For Value Converter**: Sets the schema ID to use for deserialization when using `ConfigSchemaIdDeserializer`. This lets you specify a fixed schema ID for deserializing message values. This property is applicable only when `value.converter.value.schema.id.deserializer` is set to `ConfigSchemaIdDeserializer`. - **Value Converter Schemas Enable**: Includes schema within each of the serialized values. Input messages must contain `schema` and `payload` fields and must not contain additional fields. For plain `JSON` data, set this to `false`. Applies to the `JSON` converter. - **Errors Tolerance**: Use this property to configure the connector’s error handling behavior. #### WARNING Use this property with caution for sink connectors, as it can lead to data loss. If you set this property to `all`, the connector does not fail on errant records, but logs them (and sends to DLQ for sink connectors) and continues processing. If you set this property to `none`, the connector task fails on errant records. - **Value Converter Ignore Default For Nullables**: When set to `true`, this property ensures that the corresponding record in Kafka is `null`, instead of showing the default column value. Applies to the `AVRO`, `PROTOBUF`, and `JSON_SR` converters. - **Key Converter Schema ID Deserializer**: Sets the class name of the schema ID deserializer for keys. The deserializer reads schema IDs from message headers. - **Value Converter Decimal Format**: Specifies the `JSON` or `JSON_SR` serialization format for Connect `DECIMAL` logical type values with two allowed literals: `BASE64` to serialize `DECIMAL` logical types as base64 encoded binary data, and `NUMERIC` to serialize `DECIMAL` logical type values in `JSON` or `JSON_SR` as a number representing the decimal value. - **Schema GUID For Key Converter**: Sets the schema GUID to use for deserialization when using `ConfigSchemaIdDeserializer`. This lets you specify a fixed schema GUID for deserializing message keys. This property is applicable only when `key.converter.key.schema.id.deserializer` is set to `ConfigSchemaIdDeserializer`. - **Schema GUID For Value Converter**: Sets the schema GUID to use for deserialization when using `ConfigSchemaIdDeserializer`. This lets you specify a fixed schema GUID for deserializing message values. This property is applicable only when `value.converter.value.schema.id.deserializer` is set to `ConfigSchemaIdDeserializer`. - **Value Converter Connect Meta Data**: Enables the Connect converter to add its metadata to the output schema. Applies to Avro converters. - **Value Converter Value Subject Name Strategy**: Determines how to construct the subject name under which the value schema is registered with Schema Registry. - **Key Converter Key Subject Name Strategy**: Determines how to construct the subject name for key schema registration. - **Schema ID For Key Converter**: Sets the schema ID to use for deserialization when using `ConfigSchemaIdDeserializer`. This lets you specify a fixed schema ID for deserializing message keys. This property is applicable only when `key.converter.key.schema.id.deserializer` is set to `ConfigSchemaIdDeserializer`. **Auto-restart policy** - **Enable Connector Auto-restart**: Enables the auto-restart behavior of the connector and its task in the event of user-actionable errors. Defaults to `true`, enabling the connector to automatically restart in case of user-actionable errors. Set this property to `false` to disable auto-restart for failed connectors. If disabled, you must manually restart the connector. **Consumer configuration** - **Max poll interval(ms)**: Sets the maximum delay between subsequent consume requests to Kafka. Use this property to improve connector performance in cases when the connector cannot send records to the sink system. The default is 300,000 milliseconds (5 minutes). - **Max poll records**: Sets the maximum number of records to consume from Kafka in a single request. Use this property to improve connector performance in cases when the connector cannot send records to the sink system. The default is 500 records. **Account details** - **Azure environment of Cosmos DB account**: The azure environment of the Cosmos DB account: `Azure`, `AzureChina`, `AzureUsGovernment`, `AzureGermany`. - **Gateway mode**: A flag to indicate whether to use gateway mode for connecting to Cosmos DB. Defaults to `false`, meaning the SDK uses direct mode. For more information, see [Azure Cosmos DB connectivity modes](https://learn.microsoft.com/en-gb/azure/cosmos-db/nosql/sdk-connection-modes?session_ref=direct&url_ref=https%3A%2F%2Fdocs.confluent.io%2Fcloud%2Fcurrent%2Fconnectors%2Fcc-azure-cosmos-sink-v2.html). - **Preferred regions list**: Use preferred regions list for a multi-region Cosmos DB account. Specify the regions in a flexible format, either as an array ([East US, West US]) or as a comma-separated list (East US, West US). Confluent recommends to use an Kafka cluster that is collocated in the same Azure region as your Cosmos DB account and use the Azure region hosting your Kafka cluster as preferred region. For more information, see [Azure Cosmos DB supported regions](https://learn.microsoft.com/en-gb/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true&session_ref=direct&url_ref=https%3A%2F%2Fdocs.confluent.io%2Fcloud%2Fcurrent%2Fconnectors%2Fcc-azure-cosmos-sink-v2.html). **Write configuration details** - **Enable bulk mode**: A flag to indicate whether Cosmos DB bulk mode is enabled for Sink connector. By default it is `true`. - **Cosmos DB item write max concurrent cosmos partitions**: Specify this property to optimize bulk processing when the input data in each batch has been repartitioned to balance the number of Cosmos partitions to which each batch needs to write. This is mainly useful for very large containers (with hundreds of physical partitions). If specified, it indicates that each batch contains data from, at most, this number of Cosmos physical partitions. If not specified, it will be determined based on the number of the container’s physical partitions, which suggests that each batch is expected to contain data from all Cosmos physical partitions. Defaults to `-1`. - **Cosmos DB initial bulk micro batch size**: A micro-batch is flushed to the backend when it exceeds the number of documents or the target payload size. This size automatically adjusts based on throttling rate. Defaults to `1`. Reduce it to prevent initial requests from consuming too many RUs. - **Default Cosmos DB patch operation type**: Select the default patch operation that determines how document writes are handled in Cosmos DB. Defaults to `set`. Supported operation types are `none`, `add`, `set`, `replace`, `remove`, `increment`. For more information, see [Supported operations](https://learn.microsoft.com/en-gb/azure/cosmos-db/partial-document-update?session_ref=direct&url_ref=https%3A%2F%2Fdocs.confluent.io%2Fcloud%2Fcurrent%2Fconnectors%2Fcc-azure-cosmos-sink-v2.html#supported-operations). - **Cosmos DB patch json property config**: Explains how to configure patch operations in Cosmos DB using JSON properties. Use multiple definitions matching the following patterns, separated by commas: `property(jsonProperty).op(operationType)` or `property(jsonProperty).path(patchInCosmosDB).op(operationType)`. The difference in the second pattern is that it also allows you to define a different Cosmos DB path. Note that nested JSON property configurations are not supported. - **Conditional patch**: Specify the condition under which a patch operation should be executed when using a specific SDK. For more information, see [Partial document update](https://learn.microsoft.com/en-gb/azure/cosmos-db/partial-document-update-getting-started?tabs=dotnet&session_ref=direct&url_ref=https%3A%2F%2Fdocs.confluent.io%2Fcloud%2Fcurrent%2Fconnectors%2Fcc-azure-cosmos-sink-v2.html#java). - **Cosmos DB max retry attempts on write failures**: The number of retry attempts the connector will make in the event of write failures. By default, the connector retries transient write errors up to `10` times. - **Error tolerance level**: Adjust this setting to specify how errors are handled after the specified number of retries has been exhausted. Setting the level to `All` enables the connector to log the error and then continue with its execution. Defaults to `None`, indicating that the connector will not tolerate any errors after all retries are exhausted. **Throughput control details** - **Enable throughput control**: A flag to indicate whether throughput control is enabled. Defaults to `false`. Set this property to `true` to enable the throughput control and enter the Cosmos DB account details. - **Cosmos auth type**: Select an auth type from the two currently supported: * **MasterKey** (`PrimaryReadWriteKeys`, `SecondReadWriteKeys`, `PrimaryReadOnlyKeys`, `SecondReadWriteKeys`) * **ServicePrincipal** - **Cosmos DB throughput control account key**: Cosmos DB throughput control account key. Required for `MasterKey` authentication. - **ClientId/applicationId of service principal**: The clientId/applicationId of the service principal. Required for `ServicePrincipal` authentication. - **Client secret/password of service principal**: The client secret/password of the service principal. Required for `ServicePrincipal` authentication. - **TenantId of Cosmos DB account**: The tenantId of the Cosmos DB account. Required for `ServicePrincipal` authentication. - **Azure environment of Cosmos DB account**: The azure environment of the Cosmos DB account: `Azure`, `AzureChina`, `AzureUsGovernment`, `AzureGermany`. - **Cosmos DB throughput control account endpoint uri**: Cosmos DB throughput control account endpoint uri. - **Gateway mode for throughput control**: Flag to indicate whether to use gateway mode. By default it is `false`, means SDK uses direct mode. For more information, see [SDK Connection Modes](https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-connection-modes) on Microsoft’s documentation. - **Preferred regions list for throughput control database account**: Preferred regions list to be used for a multi region Cosmos DB account. This is a comma separated value, for example `[East US, West US]` or `East US, West US` provided preferred regions will be used as a hint. You should use a collocated kafka cluster with your Cosmos DB account and pass the kafka cluster region as preferred region. For more information, see [List of azure regions](https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true). - **Throughput control group name**: Throughput control group name. Since customer is allowed to create many groups for a container, the name should be unique. - **Throughput control group target throughput**: Throughput control group target throughput. The value should be larger than 0. - **Throughput control group target throughput threshold**: Throughput control group target throughput threshold. The value should be between (0,1]. - **Throughput control group priority level**: Throughput control group priority level. The value can be None, High or Low. - **Database for throughput global control**: Database which will be used for throughput global control. - **Container for throughput global control**: Container which will be used for throughput global control. - **Throughput control client RU usage update interval**: This controls how often the client is going to update the throughput usage of itself and adjust its own throughput share based on the throughput usage of other clients. Default is 5s, the allowed min value is 5s. - **Throughput control client expire interval**: This controls how quickly we will detect the client has been offline and hence allow its throughput share to be taken by other clients. Default is 11s, the allowed min value is 2 \* renewIntervalInMS + 1. **Transforms** - **Single Message Transformations**: To add a new SMT, see [Add transforms](single-message-transforms.md#cc-single-message-transforms-ui). For more information about unsupported SMTs, see [Unsupported transformations](single-message-transforms.md#cc-single-message-transforms-unsupported-transforms). **Processing position** - **Set offsets**: Click **Set offsets** to define a specific offset for this connector to begin procession data from. For more information on managing offsets, see [Manage offsets](offsets.md#connect-custom-offsets). See [Configuration Properties](#cc-azure-cosmos-v2-sink-config-properties) for all property values and definitions. - Click **Continue**. ### Sizing Based on the number of topic partitions you select, you will be provided with a recommended number of tasks. 1. To change the number of recommended tasks, enter the number of [tasks](/platform/current/connect/concepts.html#tasks) for the connector to use in the **Tasks** field. More tasks may improve performance. 2. Click **Continue**. ### Review and Launch 1. Verify the connection details. 2. Click **Launch**. The status for the connector should go from **Provisioning** to **Running**. #### Step 5: Check for records Verify that records are being produced in your Azure Cosmos database. For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ### Using the Confluent CLI Complete the following steps to set up and run the connector using the Confluent CLI. #### NOTE Make sure you have all your [prerequisites](#cc-azure-cosmos-v2-sink-prereqs) completed. #### Step 1: List the available connectors Enter the following command to list available connectors: ```none confluent connect plugin list ``` #### Step 2: List the connector configuration properties Enter the following command to show the connector configuration properties: ```none confluent connect plugin describe ``` The command output shows the required and optional configuration properties. #### Step 3: Create the connector configuration file Create a JSON file that contains the connector configuration properties. The following example shows the required connector properties. ```json { "name": "CosmosDbSinkV2Connector_0", "config": { "connector.class": "CosmosDbSinkV2", "name": "CosmosDbSinkV2Connector_0", "input.data.format": "AVRO", "kafka.auth.mode": "KAFKA_API_KEY", "kafka.api.key": "****************", "kafka.api.secret": "**********************************************", "topics": "pageviews", "azure.cosmos.account.endpoint": "https://myaccount.documents.azure.com:443/", "azure.cosmos.account.key": "****************************************", "azure.cosmos.sink.database.name": "myDBname", "azure.cosmos.sink.containers.topicMap": "pageviews#Container2", "azure.cosmos.sink.id.strategy": "FullKeyStrategy", "tasks.max": "1" } } ``` Note the following property definitions: * `"connector.class"`: Identifies the connector plugin name. * `"input.data.format"`: Sets the input Kafka record value format (data coming from the Kafka topic). Valid entries are **AVRO**, **JSON_SR**, **PROTOBUF**, or **JSON**. You must have Confluent Cloud Schema Registry configured if using a schema-based message format (for example, Avro, JSON_SR (JSON Schema), or Protobuf). * `"name"`: Sets a name for your new connector. * `"kafka.auth.mode"`: Identifies the connector authentication mode you want to use. There are two options: `SERVICE_ACCOUNT` or `KAFKA_API_KEY` (the default). To use an API key and secret, specify the configuration properties `kafka.api.key` and `kafka.api.secret`, as shown in the example configuration (above). To use a [service account](service-account.md#s3-cloud-service-account), specify the **Resource ID** in the property `kafka.service.account.id=`. To list the available service account resource IDs, use the following command: ```bash confluent iam service-account list ``` For example: ```bash confluent iam service-account list Id | Resource ID | Name | Description +---------+-------------+-------------------+------------------- 123456 | sa-l1r23m | sa-1 | Service account 1 789101 | sa-l4d56p | sa-2 | Service account 2 ``` * `"azure.cosmos.account.endpoint"`: A URI with the form `https://ccloud-cosmos-db-1.documents.azure.com:443/`. * `"azure.cosmos.account.key"`: The Azure Cosmos master key. * `"azure.cosmos.sink.database.name"`: The name of your Cosmos DB. * `"azure.cosmos.sink.containers.topicMap"`: A comma-delimited list of Kafka topics mapped to Cosmos DB containers. Note that this property only supports 1:1 mapping between topic and container name. For example: `topic#container1,topic2#container2`. * (Optional) `"azure.cosmos.sink.id.strategy"`: Defaults to `FullKeyStrategy`. Enter one of the following strategies: - `FullKeyStrategy`: The ID generated is the Kafka record key. - `KafkaMetadataStrategy`: The ID generated is a concatenation of the Kafka topic, partition, and offset. For example: `${topic}-${partition}-${offset}`. - `ProvidedInKeyStrategy`: The ID generated is the `id` field found in the key object. Every record must have (lower case) `id` field. This is an Azure Cosmos DB requirement. See [Lower case id prerequisite](cc-azure-cosmos-sink.md#cc-azure-cosmos-sink-prereqs). - `ProvidedInValueStrategy`: The ID generated is the `id` field found in the value object. - `TemplateStrategy`: The template string used to generate the `id` field. Every record must have (lower case) `id` field. This is an Azure Cosmos DB requirement. See [Lower case id prerequisite](cc-azure-cosmos-sink.md#cc-azure-cosmos-sink-prereqs). * `"tasks"`: The number of [tasks](/platform/current/connect/concepts.html#tasks) to use with the connector. More tasks may improve performance. #### NOTE To enable CSFLE or CSPE for data encryption, specify the following properties: * `csfle.enabled`: Flag to indicate whether the connector honors CSFLE or CSPE rules. * `sr.service.account.id`: A Service Account to access the Schema Registry and associated encryption rules or keys with that schema. * `csfle.onFailure`: Configures the connector behavior (`ERROR` or `NONE`) on data decryption failure. If set to `ERROR`, the connector fails and writes the encrypted data in the DLQ. If set to `NONE`, the connector writes the encrypted data in the target system without decryption. When using CSFLE or CSPE with connectors that route failed messages to a Dead Letter Queue (DLQ), be aware that data sent to the DLQ is written in unencrypted plaintext. This poses a significant security risk as sensitive data that should be encrypted may be exposed in the DLQ. Do not use DLQ with CSFLE or CSPE in the current version. If you need error handling for CSFLE- or CSPE-enabled data, use alternative approaches such as: * Setting the connector behavior to `ERROR` to throw exceptions instead of routing to DLQ * Implementing custom error handling in your applications * Using `NONE` to pass encrypted data through without decryption For more information on CSFLE or CSPE setup, see [Manage encryption for connectors](csfle.md#connect-csfle). **SMTs**: For details about adding SMTs using the Confluent CLI, see the [Single Message Transformations](single-message-transforms.md#cc-single-message-transforms) documentation. See [Configuration Properties](#cc-azure-cosmos-v2-sink-config-properties) for all property values and descriptions. #### Step 4: Load the properties file and create the connector Enter the following command to load the configuration and start the connector: ```none confluent connect cluster create --config-file .json ``` For example: ```none confluent connect cluster create --config-file azure-cosmos-v2-sink-config.json ``` Example output: ```none Created connector CosmosDbSinkV2Connector_0 lcc-do6vzd ``` #### Step 4: Check the connector status. Enter the following command to check the connector status: ```none confluent connect cluster list ``` Example output: ```none ID | Name | Status | Type | Trace +------------+-------------------------------+---------+------+-------+ lcc-do6vzd | CosmosDbSinkV2Connector_0 | RUNNING | sink | | ``` #### Step 5: Check for records Verify that records are populating the endpoint. For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ## V1 to V2 Migration Confluent recommends upgrading from version 1 to version 2 of this connector to take advantage of the latest features, including support for `TemplateStrategy` ID strategy. Use the following steps to migrate to version 2 connector. Implement and validate any connector changes in a pre-production environment before promoting to production. #### IMPORTANT If you plan to migrate from version 1 to version 2, set `azure.cosmos.sink.id.strategy` configuration property as `FullKeyStrategy` to avoid any migration failures. This is also the default ID strategy. 1. Pause the V1 connector. 2. Get the [offset](https://docs.confluent.io/cloud/current/ccloud/delete-connectv-1-connector/) for the V1 connector. 3. Create the V2 connector using the offset from the previous step. ```none confluent connect cluster create [flags] ``` For example: Create a configuration file with connector configs and offsets. ```none { "name": "(connector-name)", "config": { ... // connector specific configuration }, "offsets": [ { "partition": { ... // connector specific configuration }, "offset": { ... // connector specific configuration } } ] } ``` Create a V2 connector in the current or specified Kafka cluster context. ```none confluent connect cluster create --config-file config.json ``` 4. Verify the migration and confirm that the connector is running successfully with the V1 payloads. 5. [Delete](https://docs.confluent.io/cloud/current/ccloud/delete-connectv-1-connector/) the V1 connector. For more information on offsets, see [Sink connectors](offsets.md#custom-offsets-sink-proc). ## Configuration Properties Use the following configuration properties with the fully managed connector. For self-managed connector property definitions and other details, see the connector docs in [Self-managed connectors for Confluent Platform](/platform/current/connect/kafka_connectors.html). ### Authentication method `authentication.method` : How Confluent Cloud authenticates with Azure.
* Type: string * Default: Master Key * Importance: high `secret.manager.enabled` : Fetch sensitive configuration values from a secret manager.
* Type: boolean * Default: false * Importance: high ### Secret manager configuration `secret.manager` : Select the secret manager to use for retrieving sensitive data.
* Type: string * Importance: high `secret.manager.managed.configs` : Select the configurations to fetch their values from the secret manager.
* Type: list * Importance: high `secret.manager.provider.integration.id` : Select an existing provider integration that has access to your secret manager.
* Type: string * Importance: high ### Azure credentials `provider.integration.id` : Azure provider-integration to mint Microsoft Entra ID application tokens.
* Type: string * Importance: high ### How should we connect to your data? `name` : Sets a name for your connector.
* Type: string * Valid Values: A string at most 64 characters long * Importance: high ### Schema Config `schema.context.name` : Add a schema context name. A schema context represents an independent scope in Schema Registry. It is a separate sub-schema tied to topics in different Kafka clusters that share the same Schema Registry instance. If not used, the connector uses the default schema configured for Schema Registry in your Confluent Cloud environment.
* Type: string * Default: default * Importance: medium ### Input messages `input.data.format` : Sets the input Kafka record value format. Valid entries are AVRO, JSON_SR, PROTOBUF, or JSON. Note that you need to have Confluent Cloud Schema Registry configured if using a schema-based message format like AVRO, JSON_SR, and PROTOBUF.
* Type: string * Default: JSON * Importance: high ### Kafka Cluster credentials `kafka.auth.mode` : Kafka Authentication mode. It can be one of KAFKA_API_KEY or SERVICE_ACCOUNT. It defaults to KAFKA_API_KEY mode, whenever possible.
* Type: string * Valid Values: SERVICE_ACCOUNT, KAFKA_API_KEY * Importance: high `kafka.api.key` : Kafka API Key. Required when kafka.auth.mode==KAFKA_API_KEY.
* Type: password * Importance: high `kafka.service.account.id` : The Service Account that will be used to generate the API keys to communicate with Kafka Cluster.
* Type: string * Importance: high `kafka.api.secret` : Secret associated with Kafka API key. Required when kafka.auth.mode==KAFKA_API_KEY.
* Type: password * Importance: high ### Which topics do you want to get data from? `topics.regex` : A regular expression that matches the names of the topics to consume from. This is useful when you want to consume from multiple topics that match a certain pattern without having to list them all individually.
* Type: string * Importance: low `topics` : Identifies the topic name or a comma-separated list of topic names.
* Type: list * Importance: high `errors.deadletterqueue.topic.name` : The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. Defaults to ‘dlq-${connector}’ if not set. The DLQ topic will be created automatically if it does not exist. You can provide `${connector}` in the value to use it as a placeholder for the logical cluster ID.
* Type: string * Default: dlq-${connector} * Importance: low ### Connect to your Azure Cosmos DB `azure.cosmos.account.endpoint` : Cosmos endpoint URL. For example: [https://connect-cosmosdb.documents.azure.com:443/](https://connect-cosmosdb.documents.azure.com:443/).
* Type: string * Importance: high `azure.cosmos.sink.containers.topicMap` : A comma delimited list of Kafka topics mapped to Cosmos containers. For example: topic1#con1,topic2#con2.
* Type: string * Importance: high `azure.cosmos.sink.database.name` : Cosmos target database to write records into.
* Type: string * Importance: high ### Account details `azure.cosmos.account.environment` : The azure environment of the Cosmos DB account: Azure, AzureChina, AzureUsGovernment, AzureGermany.
* Type: string * Default: AZURE * Valid Values: AZURE, AZURE_CHINA, AZURE_CHINA, AZURE_GERMANY, AZURE_US_GOVERNMENT * Importance: medium `azure.cosmos.mode.gateway` : Flag to indicate whether to use gateway mode. By default it is false, means SDK uses direct mode. [https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-connection-modes](https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-connection-modes)
* Type: boolean * Default: false * Importance: low `azure.cosmos.preferredRegionList` : Preferred regions list to be used for a multi region Cosmos DB account. This is a comma separated value (e.g., [East US, West US] or East US, West US) provided preferred regions will be used as hint. You should use a collocated kafka cluster with your Cosmos DB account and pass the kafka cluster region as preferred region. See list of azure regions - [https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true](https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true).
* Type: string * Importance: low `azure.cosmos.account.key` : Cosmos DB account key (only required in case of auth.type as MasterKey).
* Type: password * Importance: medium `azure.cosmos.auth.aad.clientId` : The clientId/ApplicationId of the service principal. Required for ServicePrincipal authentication.
* Type: string * Importance: medium `azure.cosmos.auth.aad.clientSecret` : The client secret/password of the service principal. Required for ServicePrincipal authentication.
* Type: password * Importance: medium `azure.cosmos.account.tenantId` : The tenantId of the Cosmos DB account. Required for ServicePrincipal authentication.
* Type: string * Default: “” * Importance: medium ### Consumer configuration `max.poll.interval.ms` : The maximum delay between subsequent consume requests to Kafka. This configuration property may be used to improve the performance of the connector, if the connector cannot send records to the sink system. Defaults to 300000 milliseconds (5 minutes).
* Type: long * Default: 300000 (5 minutes) * Valid Values: [60000,…,1800000] for non-dedicated clusters and [60000,…] for dedicated clusters * Importance: low `max.poll.records` : The maximum number of records to consume from Kafka in a single request. This configuration property may be used to improve the performance of the connector, if the connector cannot send records to the sink system. Defaults to 500 records.
* Type: long * Default: 500 * Valid Values: [1,…,500] for non-dedicated clusters and [1,…] for dedicated clusters * Importance: low ### Number of tasks for this connector `tasks.max` : Maximum number of tasks for the connector.
* Type: int * Valid Values: [1,…] * Importance: high ### Write configuration details `azure.cosmos.sink.bulk.enabled` : Flag to indicate whether Cosmos DB bulk mode is enabled for Sink connector. By default it is true.
* Type: boolean * Default: true * Importance: medium `azure.cosmos.sink.bulk.maxConcurrentCosmosPartitions` : Cosmos DB item write max concurrent cosmos partitions. If not specified it will be determined based on the number of the container’s physical partitions - which would indicate every batch is expected to have data from all Cosmos physical partitions. If specified it indicates from at most how many Cosmos Physical Partitions each batch contains data. So this config can be used to make bulk processing more efficient when input data in each batch has been repartitioned to balance to how many Cosmos partitions each batch needs to write. This is mainly useful for very large containers (with hundreds of physical partitions).
* Type: int * Default: -1 * Importance: low `azure.cosmos.sink.bulk.initialBatchSize` : Cosmos DB initial bulk micro batch size - a micro batch will be flushed to the backend when the number of documents enqueued exceeds this size - or the target payload size is met. The micro batch size is getting automatically tuned based on the throttling rate. By default the initial micro batch size is 1. Reduce this when you want to avoid that the first few requests consume too many RUs.
* Type: int * Default: 1 * Importance: medium `azure.cosmos.sink.write.strategy` : Cosmos DB item write strategy: ItemOverwrite (using upsert), ItemAppend (using create, ignore pre-existing items i.e., Conflicts), ItemDelete (deletes based on id/pk of data frame), ItemDeleteIfNotModified (deletes based on id/pk of data frame if etag hasn’t changed since collecting id/pk), ItemOverwriteIfNotModified (using create if etag is empty, update/replace with etag pre-condition otherwise, if document was updated the pre-condition failure is ignored), ItemPatch (Partial update all documents based on the patch config)
* Type: string * Default: ItemOverwrite * Valid Values: ItemAppend, ItemDelete, ItemDeleteIfNotModified, ItemOverwrite, ItemOverwriteIfNotModified, ItemPatch * Importance: high `azure.cosmos.sink.write.patch.operationType.default` : Default Cosmos DB patch operation type. Supported ones include none, add, set, replace, remove, increment. Choose none for no-op, for others please reference - [https://docs.microsoft.com/azure/cosmos-db/partial-document-update#supported-operations](https://docs.microsoft.com/azure/cosmos-db/partial-document-update#supported-operations) for full context.
* Type: string * Default: Set * Valid Values: Add, Increment, None, Remove, Replace, Set * Importance: low `azure.cosmos.sink.write.patch.property.configs` : Cosmos DB patch json property configs. It can contain multiple definitions matching the following patterns separated by comma. property(jsonProperty).op(operationType) or property(jsonProperty).path(patchInCosmosdb).op(operationType) - The difference of the second pattern is that it also allows you to define a different cosmosdb path. Note: It does not support nested json property config.
* Type: string * Importance: low `azure.cosmos.sink.write.patch.filter` : Used for Conditional patch. Ref - [https://docs.microsoft.com/azure/cosmos-db/partial-document-update-getting-started#java](https://docs.microsoft.com/azure/cosmos-db/partial-document-update-getting-started#java)
* Type: string * Importance: low `azure.cosmos.sink.maxRetryCount` : Cosmos DB max retry attempts on write failures. By default, the connector will retry on transient write errors for up to 10 times.
* Type: int * Default: 10 * Importance: medium `azure.cosmos.sink.errors.tolerance.level` : Error tolerance level after exhausting all retries. None for fail on error. All for log and continue.
* Type: string * Default: None * Valid Values: All, None * Importance: high ### ID Strategy details `azure.cosmos.sink.id.strategy` : The IdStrategy class name to use for generating a unique document id (id). FullKeyStrategy uses the full record key as ID. KafkaMetadataStrategy uses a concatenation of the kafka topic, partition, and offset as ID, with dashes as separator. i.e. ${topic}-${partition}-${offset}. ProvidedInKeyStrategy and ProvidedInValueStrategy use the id field found in the key and value objects respectively as ID.
* Type: string * Default: FullKeyStrategy * Valid Values: FullKeyStrategy, KafkaMetadataStrategy, ProvidedInKeyStrategy, ProvidedInValueStrategy, TemplateStrategy * Importance: high ### Throughput control details `azure.cosmos.throughputControl.enabled` : A flag to indicate whether throughput control is enabled.
* Type: boolean * Default: false * Importance: medium `azure.cosmos.throughputControl.auth.type` : There are two auth types are supported currently: MasterKey\`(PrimaryReadWriteKeys, SecondReadWriteKeys, PrimaryReadOnlyKeys, SecondReadWriteKeys), \`ServicePrincipal
* Type: string * Default: MasterKey * Valid Values: MasterKey, ServicePrincipal * Importance: low `azure.cosmos.throughputControl.account.key` : Cosmos DB throughput control account key (only required in case of throughputControl.auth.type as MasterKey)
* Type: password * Importance: low `azure.cosmos.throughputControl.auth.aad.clientId` : The clientId/applicationId of the service principal. Required for ServicePrincipal authentication.
* Type: string * Importance: low `azure.cosmos.throughputControl.auth.aad.clientSecret` : The client secret/password of the service principal. Required for ServicePrincipal authentication.
* Type: password * Importance: low `azure.cosmos.throughputControl.account.tenantId` : The tenantId of the Cosmos DB account. Required for ServicePrincipal authentication.
* Type: string * Importance: low `azure.cosmos.throughputControl.account.environment` : The azure environment of the Cosmos DB account: Azure, AzureChina, AzureUsGovernment, AzureGermany.
* Type: string * Default: AZURE * Valid Values: AZURE, AZURE_CHINA, AZURE_GERMANY, AZURE_US_GOVERNMENT * Importance: low `azure.cosmos.throughputControl.account.endpoint` : Cosmos DB throughput control account endpoint uri.
* Type: string * Importance: low `azure.cosmos.throughputControl.mode.gateway` : Flag to indicate whether to use gateway mode. By default it is false, means SDK uses direct mode. [https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-connection-modes](https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-connection-modes)
* Type: boolean * Default: false * Importance: low `azure.cosmos.throughputControl.preferredRegionList` : Preferred regions list to be used for a multi region Cosmos DB account. This is a comma separated value (e.g., [East US, West US] or East US, West US) provided preferred regions will be used as hint. You should use a collocated kafka cluster with your Cosmos DB account and pass the kafka cluster region as preferred region. See list of azure regions - [https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true](https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true)
* Type: string * Importance: low `azure.cosmos.throughputControl.group.name` : Throughput control group name. Since customer is allowed to create many groups for a container, the name should be unique.
* Type: string * Importance: medium `azure.cosmos.throughputControl.targetThroughput` : Throughput control group target throughput. The value should be larger than 0.
* Type: int * Valid Values: [1,…] * Importance: medium `azure.cosmos.throughputControl.targetThroughputThreshold` : Throughput control group target throughput threshold. The value should be between (0,1].
* Type: double * Importance: medium `azure.cosmos.throughputControl.priorityLevel` : Throughput control group priority level. The value can be None, High or Low.
* Type: string * Default: None * Valid Values: High, Low, None * Importance: medium `azure.cosmos.throughputControl.globalControl.database.name` : Database which will be used for throughput global control.
* Type: string * Importance: medium `azure.cosmos.throughputControl.globalControl.container.name` : Container which will be used for throughput global control.
* Type: string * Importance: medium `azure.cosmos.throughputControl.globalControl.renewIntervalInMS` : This controls how often the client is going to update the throughput usage of itself and adjust its own throughput share based on the throughput usage of other clients. Default is 5s, the allowed min value is 5s.
* Type: int * Default: 5000 * Valid Values: [5000,…] * Importance: low `azure.cosmos.throughputControl.globalControl.expireIntervalInMS` : This controls how quickly we will detect the client has been offline and hence allow its throughput share to be taken by other clients. Default is 11s, the allowed min value is 2 \* renewIntervalInMS + 1
* Type: int * Importance: low ### Additional Configs `consumer.override.auto.offset.reset` : Defines the behavior of the consumer when there is no committed position (which occurs when the group is first initialized) or when an offset is out of range. You can choose either to reset the position to the “earliest” offset (the default) or the “latest” offset. You can also select “none” if you would rather set the initial offset yourself and you are willing to handle out of range errors manually. More details: [https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#auto-offset-reset](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#auto-offset-reset)
* Type: string * Importance: low `consumer.override.isolation.level` : Controls how to read messages written transactionally. If set to read_committed, consumer.poll() will only return transactional messages which have been committed. If set to read_uncommitted (the default), consumer.poll() will return all messages, even transactional messages which have been aborted. Non-transactional messages will be returned unconditionally in either mode. More details: [https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#isolation-level](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#isolation-level)
* Type: string * Importance: low `header.converter` : The converter class for the headers. This is used to serialize and deserialize the headers of the messages.
* Type: string * Importance: low `key.converter.use.schema.guid` : The schema GUID to use for deserialization when using ConfigSchemaIdDeserializer. This allows you to specify a fixed schema GUID to be used for deserializing message keys. Only applicable when key.converter.key.schema.id.deserializer is set to ConfigSchemaIdDeserializer.
* Type: string * Importance: low `key.converter.use.schema.id` : The schema ID to use for deserialization when using ConfigSchemaIdDeserializer. This allows you to specify a fixed schema ID to be used for deserializing message keys. Only applicable when key.converter.key.schema.id.deserializer is set to ConfigSchemaIdDeserializer.
* Type: int * Importance: low `value.converter.allow.optional.map.keys` : Allow optional string map key when converting from Connect Schema to Avro Schema. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.auto.register.schemas` : Specify if the Serializer should attempt to register the Schema.
* Type: boolean * Importance: low `value.converter.connect.meta.data` : Allow the Connect converter to add its metadata to the output schema. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.enhanced.avro.schema.support` : Enable enhanced schema support to preserve package information and Enums. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.enhanced.protobuf.schema.support` : Enable enhanced schema support to preserve package information. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.flatten.unions` : Whether to flatten unions (oneofs). Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.generate.index.for.unions` : Whether to generate an index suffix for unions. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.generate.struct.for.nulls` : Whether to generate a struct variable for null values. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.int.for.enums` : Whether to represent enums as integers. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.latest.compatibility.strict` : Verify latest subject version is backward compatible when use.latest.version is true.
* Type: boolean * Importance: low `value.converter.object.additional.properties` : Whether to allow additional properties for object schemas. Applicable for JSON_SR Converters.
* Type: boolean * Importance: low `value.converter.optional.for.nullables` : Whether nullable fields should be specified with an optional label. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.optional.for.proto2` : Whether proto2 optionals are supported. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.scrub.invalid.names` : Whether to scrub invalid names by replacing invalid characters with valid characters. Applicable for Avro and Protobuf Converters.
* Type: boolean * Importance: low `value.converter.use.latest.version` : Use latest version of schema in subject for serialization when auto.register.schemas is false.
* Type: boolean * Importance: low `value.converter.use.optional.for.nonrequired` : Whether to set non-required properties to be optional. Applicable for JSON_SR Converters.
* Type: boolean * Importance: low `value.converter.use.schema.guid` : The schema GUID to use for deserialization when using ConfigSchemaIdDeserializer. This allows you to specify a fixed schema GUID to be used for deserializing message values. Only applicable when value.converter.value.schema.id.deserializer is set to ConfigSchemaIdDeserializer.
* Type: string * Importance: low `value.converter.use.schema.id` : The schema ID to use for deserialization when using ConfigSchemaIdDeserializer. This allows you to specify a fixed schema ID to be used for deserializing message values. Only applicable when value.converter.value.schema.id.deserializer is set to ConfigSchemaIdDeserializer.
* Type: int * Importance: low `value.converter.wrapper.for.nullables` : Whether nullable fields should use primitive wrapper messages. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.wrapper.for.raw.primitives` : Whether a wrapper message should be interpreted as a raw primitive at root level. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `errors.tolerance` : Use this property if you would like to configure the connector’s error handling behavior. WARNING: This property should be used with CAUTION for SOURCE CONNECTORS as it may lead to dataloss. If you set this property to ‘all’, the connector will not fail on errant records, but will instead log them (and send to DLQ for Sink Connectors) and continue processing. If you set this property to ‘none’, the connector task will fail on errant records.
* Type: string * Default: all * Importance: low `key.converter.key.schema.id.deserializer` : The class name of the schema ID deserializer for keys. This is used to deserialize schema IDs from the message headers.
* Type: string * Default: io.confluent.kafka.serializers.schema.id.DualSchemaIdDeserializer * Importance: low `key.converter.key.subject.name.strategy` : How to construct the subject name for key schema registration.
* Type: string * Default: TopicNameStrategy * Importance: low `value.converter.decimal.format` : Specify the JSON/JSON_SR serialization format for Connect DECIMAL logical type values with two allowed literals:
BASE64 to serialize DECIMAL logical types as base64 encoded binary data and
NUMERIC to serialize Connect DECIMAL logical type values in JSON/JSON_SR as a number representing the decimal value.
* Type: string * Default: BASE64 * Importance: low `value.converter.flatten.singleton.unions` : Whether to flatten singleton unions. Applicable for Avro and JSON_SR Converters.
* Type: boolean * Default: false * Importance: low `value.converter.ignore.default.for.nullables` : When set to true, this property ensures that the corresponding record in Kafka is NULL, instead of showing the default column value. Applicable for AVRO,PROTOBUF and JSON_SR Converters.
* Type: boolean * Default: false * Importance: low `value.converter.reference.subject.name.strategy` : Set the subject reference name strategy for value. Valid entries are DefaultReferenceSubjectNameStrategy or QualifiedReferenceSubjectNameStrategy. Note that the subject reference name strategy can be selected only for PROTOBUF format with the default strategy being DefaultReferenceSubjectNameStrategy.
* Type: string * Default: DefaultReferenceSubjectNameStrategy * Importance: low `value.converter.replace.null.with.default` : Whether to replace fields that have a default value and that are null to the default value. When set to true, the default value is used, otherwise null is used. Applicable for JSON Converter.
* Type: boolean * Default: true * Importance: low `value.converter.schemas.enable` : Include schemas within each of the serialized values. Input messages must contain schema and payload fields and may not contain additional fields. For plain JSON data, set this to false. Applicable for JSON Converter.
* Type: boolean * Default: false * Importance: low `value.converter.value.schema.id.deserializer` : The class name of the schema ID deserializer for values. This is used to deserialize schema IDs from the message headers.
* Type: string * Default: io.confluent.kafka.serializers.schema.id.DualSchemaIdDeserializer * Importance: low `value.converter.value.subject.name.strategy` : Determines how to construct the subject name under which the value schema is registered with Schema Registry.
* Type: string * Default: TopicNameStrategy * Importance: low ### Auto-restart policy `auto.restart.on.user.error` : Enable connector to automatically restart on user-actionable errors.
* Type: boolean * Default: true * Importance: medium ## Frequently asked questions Find answers to frequently asked questions about the Azure Cosmos DB Sink V2 connector for Confluent Cloud. ### How do I connect to Azure Cosmos DB using service principal authentication? The connector supports service principal authentication using Confluent Provider Integration for enhanced security. To set up service principal authentication: 1. **Create Azure service principal**: In the Azure portal, create a service principal with appropriate permissions: Navigate to **Azure Active Directory > App registrations > New registration**. 2. **Grant Cosmos DB permissions**: Assign the service principal `Cosmos DB Built-in Data Contributor` role: In your Cosmos DB account, go to **Access control (IAM)** and add role assignments for write access. 3. **Configure provider integration**: In the Cloud Console, set up the provider integration: * Navigate to **Connections > Provider integrations** * Create a new Azure provider integration * Provide the service principal credentials: client ID, tenant ID, and client secret 4. **Configure the connector**: Use provider integration authentication: ```json { "azure.cosmos.auth.type": "SERVICE_PRINCIPAL", "azure.cosmos.account.endpoint": "https://your-account.documents.azure.com:443/", "provider.integration.id": "your-provider-integration-id" } ``` #### IMPORTANT Service principal authentication is more secure than master key authentication and supports role-based access control. It is the recommended authentication method for production environments. ### How do I configure the connector to write to specific containers? The connector writes data from Kafka topics to Cosmos DB containers using topic to container mapping. Configure the connector using the following container options: 1. **One topic to one container**: Map a single topic to a container: ```json { "topics": "orders-topic", "azure.cosmos.sink.database.name": "production-db", "azure.cosmos.sink.containers.topicMap": "orders-topic#orders-container" } ``` Messages from `orders-topic` are written to `orders-container`. 2. **Multiple topics to containers**: Map multiple topics: ```json { "topics": "orders-topic,users-topic", "azure.cosmos.sink.containers.topicMap": "orders-topic#orders-container,users-topic#users-container" } ``` #### IMPORTANT The topic mapping format must follow the pattern `topic#container`. Multiple mappings are comma-separated. Containers must exist before starting the connector. Autocreation of databases and containers is not supported. ### Why is my connector failing with write errors? Write errors can occur due to insufficient permissions, RU throttling, or data validation issues. ```text Request rate is large. More Request Units may be needed ``` Common causes and solutions: 1. **Insufficient Request Units**: Cosmos DB is throttling writes due to RU limits: Increase provisioned throughput or enable autoscale in the Azure portal. Monitor Request Unit consumption under **Metrics** in your Cosmos DB account. 2. **Missing partition key**: Documents are missing the required partition key: Ensure all records include the field that matches the container’s partition key path. For example, if your container’s partition key is `/customerId`, all documents must have the `customerId` field. 3. **Insufficient permissions**: The service principal or master key lacks write access: Grant `Cosmos DB Built-in Data Contributor` role to the service principal, or verify the master key is correct. 4. **Document size exceeds limit**: Documents are larger than the 2 MB limit: Review and reduce document size. Cosmos DB enforces a 2 MB maximum document size. #### NOTE The V2 connector uses bulk operations for improved write performance. Monitor Cosmos DB metrics to optimize throughput allocation. ### How can I optimize connector write performance? Write performance depends on Cosmos DB throughput, bulk operation settings, and network latency. To optimize write performance, consider the following: 1. **Enable bulk operations**: The connector uses bulk mode by default: ```json { "azure.cosmos.sink.bulk.enabled": "true" } ``` Bulk mode provides better throughput than single-document writes. 2. **Increase tasks**: More tasks can improve write throughput: ```json { "tasks.max": "4" } ``` Multiple tasks write to Cosmos DB in parallel. 3. **Tune bulk write efficiency**: For large containers, optimize how the connector handles partition distribution: ```json { "azure.cosmos.sink.bulk.maxConcurrentCosmosPartitions": "-1" } ``` The default `-1` auto-detects based on the container’s physical partitions. For very large containers with hundreds of partitions, set this to limit how many physical partitions each batch targets, which can improve efficiency when input data is pre-partitioned. 4. **Use same region**: Deploy the connector in the same Azure region as Cosmos DB: Cross-region writes have higher latency and cost more. 5. **Optimize Cosmos DB throughput**: Ensure adequate Request Units: * **Use autoscale**: Configure autoscale to handle variable workloads * **Monitor Request Unit consumption**: Check for throttling * **Increase provisioned throughput**: Allocate more Request Units if needed #### NOTE The V2 connector’s bulk operations provide significantly better performance than the V1 connector. Monitor Request Unit consumption to balance performance and cost. ### How do I handle schema evolution for documents? Cosmos DB is schemaless, allowing flexible document structures. The connector writes Kafka records as JSON documents. The connector handles schema as follows: 1. **Schema Registry formats**: Use schema-based formats for structured data: ```json { "input.data.format": "JSON_SR" } ``` Alternatively, use `AVRO` or `PROTOBUF` formats. 2. **Schemaless JSON**: Use JSON format for flexible schemas: ```json { "input.data.format": "JSON" } ``` Documents are written as-is without schema validation. 3. **Adding new fields**: Cosmos DB automatically accepts documents with new fields: No configuration changes needed when adding fields. 4. **Document structure**: The connector converts Kafka records to JSON documents: Record keys and values are preserved in the document structure. #### NOTE Cosmos DB’s schemaless nature provides flexibility for evolving document structures. Use Schema Registry formats for consistency and validation. ### Why is my connector failing with authentication errors? Authentication errors indicate issues with master key or service principal credentials. ```text Unauthorized: The input authorization token can't serve the request ``` Common causes and solutions: 1. **Invalid master key**: The `azure.cosmos.account.key` is incorrect: Verify the master key in the Azure portal under **Keys** in your Cosmos DB account settings. 2. **Service principal permissions**: The service principal lacks required write permissions: Grant the `Cosmos DB Built-in Data Contributor` role for write access. 3. **Expired credentials**: Service principal client secret has expired: Rotate the client secret in Azure Active Directory and update the provider integration. 4. **Wrong account endpoint**: The endpoint URL is incorrect: Verify the endpoint format: ```json { "azure.cosmos.account.endpoint": "https://your-account.documents.azure.com:443/" } ``` #### IMPORTANT Master keys provide full access to the Cosmos DB account. For production environments, use service principal authentication with least-privilege permissions. ### What should I do if my connector keeps failing or restarting? Frequent connector failures indicate configuration issues, network problems, or Cosmos DB capacity constraints. Common causes and solutions: 1. **Check connector logs**: In the Cloud Console, review error messages: * **Authentication failures**: Verify credentials and permissions * **Throttling errors**: Check RU consumption and increase throughput * **Validation errors**: Verify document structure and partition keys * **Network errors**: Check connectivity 2. **Verify Cosmos DB connectivity**: Test that Cosmos DB is accessible: * **Check firewall rules**: Ensure Confluent Cloud can connect to Cosmos DB * **Verify endpoint**: Confirm the account endpoint is correct * **Test credentials**: Use Azure portal to verify account access 3. **Monitor Cosmos DB metrics**: Check for resource constraints: * **Request Unit consumption**: Monitor Request Unit usage and throttling * **Storage capacity**: Ensure adequate storage * **Partition key distribution**: Check for hot partitions 4. **Verify container configuration**: Ensure containers exist and are accessible: Verify containers exist in Cosmos DB. #### IMPORTANT For persistent failures, use the connector diagnostics feature and share logs with Confluent Support. ## Next Steps For an example that shows fully managed Confluent Cloud connectors in action with Confluent Cloud for Apache Flink, see the [Cloud ETL Demo](/platform/current/tutorials/examples/cloud-etl/docs/index.html). This example also shows how to use Confluent CLI to manage your resources in Confluent Cloud. [](https://docs.confluent.io/platform/current/tutorials/examples/cloud-etl/docs/index.html)