Microsoft Azure Key Vault integration

This topic explains how to integrate Azure Key Vault with Confluent Cloud to retrieve authentication-related secrets for fully-managed connectors.

Prerequisites

Before you start integrating Azure Key Vault with Kafka Connect in Confluent Cloud, ensure that you have access to the following resources:

• Azure Portal at https://portal.azure.com

• Azure Key Vault

• Confluent Cloud Console at https://confluent.cloud

Supported regions

For public Dedicated, secret manager integration is supported in all Azure regions except australiaeast, centralus, eastus, eastus2, germanywestcentral, northeurope, southeastasia, westeurope, westus2, qatarcentral, and jioindiawest.

For private link Dedicated, secret manager integration is supported in all Azure regions except Jio regions and qatarcentral.

PrivateLink setup for connectors using Azure Key Vault (Optional)

If you want to create connectors within an Azure PrivateLink Confluent Cloud cluster, see Egress Private Link Endpoints Setup Guide: First-Party Services on Azure for Confluent Cloud.

Ensure that your cluster’s VPC can access the Azure Key Vault endpoint so that the connector has access.

Skip this step if you don’t want to run your connector within an Azure PrivateLink Confluent Cloud cluster.

Step 1: Set up Azure Key Vault

Create an Azure Key Vault and add your secrets such as database credentials, access keys, and similar authentication-related secrets. For more information, see Quickstart: Create a key vault using the Azure portal and Quickstart: Set and retrieve a secret from Azure Key Vault using the Azure portal

Step 2: Configure a Microsoft Azure Provider Integration to access Azure Key Vault

You must configure an Azure Provider Integration in Confluent Cloud to establish secure access to your Azure Key Vault before you create a connector that uses secret manager integration.

For detailed setup instructions, see Create an Azure Provider Integration in Confluent Cloud.

Tip

Note the provider integration’s id. You have to use it as secret.manager.provider.integration.id in your connector configuration.

The Confluent Cloud Console displays the id under the provider integration’s name.

Or, use Read a provider integration API to fetch the provider integration’s id.

Assign the following required permissions to the service principal associated with your Azure provider integration to access your Azure Key Vault:

• Microsoft.KeyVault/vaults/secrets/getSecret/action (Get Secret)

• Microsoft.KeyVault/vaults/secrets/readMetadata/action (List Secrets)

These permissions are included in the built-in Key Vault Secrets User role. You can assign this role to your service principal using the Azure portal or Azure CLI.

Step 3: Use Azure Key Vault integration with a connector

Now that you have successfully integrated Azure Key Vault with Confluent Cloud, you can use it within a connector. For example, you can add a SQL Server CDC Source v2 connector, and use Azure Key Vault to fetch sensitive data such as passwords for authentication.

Authenticate using secrets from Azure Key Vault

Cloud Console

At the Add Source/Sink connector screen, complete the following in the Authentication step:

1. Choose an authentication method that uses static credentials like Password and SAS Key.

2. Toggle the Use secret manager setting on to let Confluent Cloud fetch credentials from Azure Key Vault.

3. Select AZURE_KEY_VAULT from the Secret manager dropdown.

4. From the Configurations from Secret manager multi-select dropdown, select the configurations whose values Confluent Cloud should fetch from the Azure Key Vault. For each selected field, enter a secret identifier for that field instead of the literal credential value. For example:

   • For a plaintext secret, a secret identifier is the vault URI and the secret name delimited using a :: in the format <AZURE_VAULT_URI>::<secret_name>.

     If you select the database.password plaintext secret, then the secret identifier for the database.password field is https://your-key-vault-name.vault.azure.net::sqlserver-db-secret.

   • For a JSON secret, a secret identifier is the vault URI, the secret name, and the configuration key delimited using a :: in the format <AZURE_VAULT_URI>::<secret_name>::<config-key>.

     If you select database.user and database.password keys from a JSON secret:

     • The secret identifier for the database.user field is https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user.

     • The secret identifier for the database.password field is https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password.

   For more information, see Configuration value format.

5. Select your provider integration name from the Provider integration dropdown.

6. Complete your connector’s configuration. For detailed documentation on supported connectors, see Supported connectors.

Confluent CLI

1. Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON:

   • authentication.method

   • secret.manager.enabled

   • secret.manager

   • secret.manager.managed.configs

   • secret.manager.provider.integration.id

   • The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the database.password and the database.user fields. For more information on the Azure Key Vault’s secret identifier format, see Configuration value format.

   {
      "name": "SqlServerCdcSource1",
      "config": {
         "connector.class": "SqlServerCdcSourceV2",
         "name": "SqlServerCdcSource1",
         "kafka.auth.mode": "SERVICE_ACCOUNT",
         "kafka.service.account.id": "sa-xxxxxx",
         "authentication.method": "Password",
         "secret.manager.enabled": "true",
         "secret.manager": "AZURE_KEY_VAULT",
         "secret.manager.managed.configs": "database.password,database.user",
         "secret.manager.provider.integration.id": "cspi-devxxxxxx",
         "database.hostname": "203.0.113.0",
         "database.port": "1433",
         "database.user": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user",
         "database.password": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password",
         "database.names": "db123",
         "database.encrypt": "false",
         "output.data.format": "JSON",
         "output.key.format": "JSON",
         "table.include.list": "table123",
         "topic.prefix": "sqlcdc",
         "tasks.max": "1"
      }
   }
   

2. Enter the following command to load the JSON and start the connector:

   confluent connect cluster create --config-file <file-name>.json
   

   For example, if your filename is sqlcdc_with_key_vault.json, then run:

   confluent connect cluster create --config-file sqlcdc_with_key_vault.json
   

   Example output:

   Created connector SqlServerCdcSource1 lcc-ix4dl
   

Confluent REST API

Include the following fields in the configuration payload of your Create a connector API request.

• authentication.method

• secret.manager.enabled

• secret.manager

• secret.manager.managed.configs

• secret.manager.provider.integration.id

• The secret-name whose value you want Confluent Cloud to fetch. In this example, it is database.password field’s value. For more information on the Azure Key Vault’s secret identifier format, see Configuration value format.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name": "SqlServerCdcSource1",
   "config": {
      "connector.class": "SqlServerCdcSourceV2",
      "name": "SqlServerCdcSource1",
      "kafka.auth.mode": "SERVICE_ACCOUNT",
      "kafka.service.account.id": "sa-xxxxxx",
      "authentication.method": "Password",
      "secret.manager.enabled": "true",
      "secret.manager": "AZURE_KEY_VAULT",
      "secret.manager.managed.configs": "database.password,database.user",
      "secret.manager.provider.integration.id": "cspi-devxxxxxx",
      "database.hostname": "203.0.113.0",
      "database.port": "1433",
      "database.user": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user",
      "database.password": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password",
      "database.names": "db123",
      "database.encrypt": "false",
      "output.data.format": "JSON",
      "output.key.format": "JSON",
      "table.include.list": "table123",
      "topic.prefix": "sqlcdc",
      "tasks.max": "1"
   }
}'

Verify Azure Key Vault authentication

After the connector is provisioned and running, verify the successful authorization through the credentials fetched from Azure Key Vault.

View topic messages

1. Sign in to your Confluent Cloud account.

2. In the left navigation menu, click Connectors.

3. Click the connector you just created.

4. View the messages produced to or received from a topic.

View provider integration resources

1. Go to the Environments page at https://confluent.cloud/environments and select the environment in which you created your provider integration.

2. Click the Provider integrations tab.

3. Under the Resources column, view the resources.

4. Click the resource link to view the associated connector that you created.

Configuration value format

For the connector configurations that are managed through Azure Key Vault, you must specify the secret identifiers as configuration values according to the following specified formats:

1. If your secret is stored as plaintext (non-JSON format), ensure that you specify its configuration value as <AZURE_VAULT_URI>::<secret_name>.

   For example, if you want Confluent Cloud to fetch the database password configuration, and:

   • The secret-name within Azure Key Vault is sqlserver-db-secret

   • The Azure Key Vault’s URI is https://your-key-vault-name.vault.azure.net

   Then, specify the value of the database password as https://your-key-vault-name.vault.azure.net::sqlserver-db-secret.

2. If your secrets are inside a JSON object stored as an Azure Key Vault secret, then ensure that you specify each configuration value as <AZURE_VAULT_URI>::<secret_name>::<config-key>.

   For example, if https://your-key-vault-name.vault.azure.net::sqlserver-db-secret stores a JSON that looks like:

   {
      "user": "dbadmin",
      "password": "$trongPassw0rdExamp1e"
   }
   

   • To fetch the database user, ensure that you specify the connector configuration value as https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user.

   • To fetch the database password, ensure that you specify the connector configuration value as https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password.

Troubleshooting

Connector fails to start with authentication error

• Verify your Azure Key Vault name and secret name are correct.

• Ensure the service principal associated with your provider integration has the required Key Vault Secrets User role on your Azure Key Vault.

• Check that the secret exists in your Azure Key Vault and contains a valid value.

• Validate your provider integration using the Validate option in Confluent Cloud.

• Ensure that secret identifiers of the configurations that Confluent Cloud fetches from the secret manager follow the specified format. For more information, see Configuration value format.