# Microsoft Azure Key Vault integration Integrate Azure Key Vault with Confluent Cloud to let fully managed connectors retrieve authentication credentials such as database passwords and access keys from Azure Key Vault at runtime, instead of storing those values directly in the connector configuration. ## Prerequisites Before you start integrating Azure Key Vault with Kafka Connect in Confluent Cloud, ensure that you have access to the following resources: - Azure Portal at [https://portal.azure.com](https://portal.azure.com) - Azure Key Vault - Confluent Cloud Console at [https://confluent.cloud](https://confluent.cloud) ## Supported regions For Dedicated, secret manager integration is supported in all Azure regions except Jio regions and `qatarcentral`. ## PrivateLink setup for connectors using Azure Key Vault (Optional) Azure Key Vault integration supports VNet Peering and PrivateLink. If your cluster uses VNet Peering, no extra networking setup is needed. If you want to create connectors within an Azure PrivateLink Confluent Cloud cluster, see [Egress Private Link Endpoints Setup Guide: First-Party Services on Azure for Confluent Cloud](../networking/azure-eap-1st-party.md#cc-azure-eap-1st-party). Ensure that your cluster’s VPC can access the Azure Key Vault endpoint so that the connector has access. If you don’t want to run your connector within an Azure PrivateLink Confluent Cloud cluster, skip this step. ## Step 1: Set up Azure Key Vault Create an Azure Key Vault and add your secrets such as database credentials, access keys, and similar authentication-related secrets. For more information, see [Quickstart: Create a key vault using the Azure portal](https://learn.microsoft.com/en-us/azure/key-vault/general/quick-create-portal) and [Quickstart: Set and retrieve a secret from Azure Key Vault using the Azure portal](https://learn.microsoft.com/en-us/azure/key-vault/secrets/quick-create-portal) ## Step 2: Configure a Microsoft Azure Provider Integration to access Azure Key Vault You must configure an Azure Provider Integration in Confluent Cloud to establish secure access to your Azure Key Vault before you create a connector that uses secret manager integration. For detailed setup instructions, see [Create an Azure Provider Integration in Confluent Cloud](../../integrations/provider-integrations/create-provider-integration-azure.md#create-provider-integration-azure). Assign the following required permissions to the service principal associated with your Azure provider integration to access your Azure Key Vault: - `Microsoft.KeyVault/vaults/secrets/getSecret/action` (Get Secret) - `Microsoft.KeyVault/vaults/secrets/readMetadata/action` (List Secrets) These permissions are included in the built-in `Key Vault Secrets User` role. You can assign this role to your service principal using the Azure portal or Azure CLI. (Optional) Use the **Validate** option on the provider integration to verify that trust is established between Confluent Cloud and your Azure account. ## Step 3: Use Azure Key Vault integration with a connector Now that you have successfully integrated Azure Key Vault with Confluent Cloud, you can use it within a connector. For example, you can add a SQL Server CDC Source v2 connector, and use Azure Key Vault to fetch sensitive data such as passwords for authentication. ### Authenticate using secrets from Azure Key Vault ### Cloud Console At the **Add Source/Sink connector** screen, complete the following in the **Authentication** step: 1. Choose an authentication method that uses static credentials like `Password` and `SAS Key`. 2. Toggle the **Use secret manager** setting on to let Confluent Cloud fetch credentials from Azure Key Vault. 3. Select **AZURE_KEY_VAULT** from the **Secret manager** dropdown. 4. From the **Configurations from Secret manager** multi-select dropdown, select the configurations whose values Confluent Cloud should fetch from the Azure Key Vault. For each selected field, enter a secret identifier for that field instead of the literal credential value. For example: - For a plaintext secret, a secret identifier is the vault URI and the secret name delimited using a `::` in the format `::`. If you select the `database.password` plaintext secret, then the secret identifier for the `database.password` field is `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret`. - For a JSON secret, a secret identifier is the vault URI, the secret name, and the configuration key delimited using a `::` in the format `::::`. If you select `database.user` and `database.password` keys from a JSON secret: - The secret identifier for the `database.user` field is `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user`. - The secret identifier for the `database.password` field is `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password`. For more information, see [Configuration value format](#az-key-vault-secret-format). 5. Select your provider integration name from the **Provider integration** dropdown. 6. Complete your connector’s configuration. For detailed documentation on supported connectors, see [Supported connectors](overview.md#secret-manager-supported-connectors). ### Confluent CLI 1. Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON: - The authentication type that the connector supports. For exact authentication type, see the connector’s documentation. For example, it’s `authentication.method` for Postgres CDC Source V2 connector and `mongodb.auth.mechanism` for MongoDB Atlas Source connector. - `secret.manager.enabled` - `secret.manager` - `secret.manager.managed.configs` - `secret.manager.provider.integration.id` - The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the `database.password` and the `database.user` fields. For more information on the Azure Key Vault’s secret identifier format, see [Configuration value format](#az-key-vault-secret-format). ```json { "name": "SqlServerCdcSource1", "config": { "connector.class": "SqlServerCdcSourceV2", "name": "SqlServerCdcSource1", "kafka.auth.mode": "SERVICE_ACCOUNT", "kafka.service.account.id": "sa-xxxxxx", "authentication.method": "Password", "secret.manager.enabled": "true", "secret.manager": "AZURE_KEY_VAULT", "secret.manager.managed.configs": "database.password,database.user", "secret.manager.provider.integration.id": "cspi-devxxxxxx", "database.hostname": "203.0.113.0", "database.port": "1433", "database.user": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user", "database.password": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password", "database.names": "db123", "database.encrypt": "false", "output.data.format": "JSON", "output.key.format": "JSON", "table.include.list": "table123", "topic.prefix": "sqlcdc", "tasks.max": "1" } } ``` 2. Enter the following command to load the JSON and start the connector: ```bash confluent connect cluster create --config-file .json ``` For example, if your filename is `sqlcdc_with_key_vault.json`, then run: ```bash confluent connect cluster create --config-file sqlcdc_with_key_vault.json ``` Example output: ```bash Created connector SqlServerCdcSource1 lcc-ix4dl ``` ### Confluent REST API Include the following fields in the configuration payload of your [Create a connector](../connect-api-section.md#ccloud-connect-api-create-connector) API request. - The authentication type that the connector supports. For exact authentication type, see the connector’s documentation. For example, it’s `authentication.method` for Postgres CDC Source V2 connector and `mongodb.auth.mechanism` for MongoDB Atlas Source connector. - `secret.manager.enabled` - `secret.manager` - `secret.manager.managed.configs` - `secret.manager.provider.integration.id` - The secret-name whose value you want Confluent Cloud to fetch. In this example, it is `database.password` field’s value. For more information on the Azure Key Vault’s secret identifier format, see [Configuration value format](#az-key-vault-secret-format). ```bash curl --request POST \ --url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \ --header 'Authorization: Basic ' \ --header 'content-type: application/json' \ --data '{ "name": "SqlServerCdcSource1", "config": { "connector.class": "SqlServerCdcSourceV2", "name": "SqlServerCdcSource1", "kafka.auth.mode": "SERVICE_ACCOUNT", "kafka.service.account.id": "sa-xxxxxx", "authentication.method": "Password", "secret.manager.enabled": "true", "secret.manager": "AZURE_KEY_VAULT", "secret.manager.managed.configs": "database.password,database.user", "secret.manager.provider.integration.id": "cspi-devxxxxxx", "database.hostname": "203.0.113.0", "database.port": "1433", "database.user": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user", "database.password": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password", "database.names": "db123", "database.encrypt": "false", "output.data.format": "JSON", "output.key.format": "JSON", "table.include.list": "table123", "topic.prefix": "sqlcdc", "tasks.max": "1" } }' ``` ### Verify Azure Key Vault authentication After the connector is provisioned and running, verify the successful authorization through the credentials fetched from Azure Key Vault. ### View topic messages 1. Sign in to your Confluent Cloud account. 2. In the left navigation menu, click **Connectors**. 3. Click the connector you just created. 4. View the messages produced to or received from a topic. ### View provider integration resources 1. Go to the **Environments** page at [https://confluent.cloud/environments](https://confluent.cloud/environments) and select the environment in which you created your provider integration. 2. Click the **Provider integrations** tab. 3. Under the **Resources** column, view the resources. 4. Click the resource link to view the associated connector that you created. ### Configuration value format For the connector configurations that are managed through Azure Key Vault, you must specify the secret identifiers as configuration values in one of the following formats: 1. If your secret is stored as plaintext (non-JSON format), ensure that you specify its configuration value as `::`. For example, if you want Confluent Cloud to fetch the database password configuration, and: - The secret-name within Azure Key Vault is `sqlserver-db-secret` - The Azure Key Vault’s URI is `https://your-key-vault-name.vault.azure.net` Then, specify the value of the database password as `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret`. 2. If your secrets are inside a JSON object stored as an Azure Key Vault secret, then ensure that you specify each configuration value as `::::`. For example, if `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret` stores a JSON that looks like: ```json { "user": "dbadmin", "password": "$trongPassw0rdExamp1e" } ``` - To fetch the database user, ensure that you specify the connector configuration value as `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user`. - To fetch the database password, ensure that you specify the connector configuration value as `https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password`. ## Troubleshooting **Connector fails to start with authentication error** * Verify your Azure Key Vault name and secret name are correct. * Ensure the service principal associated with your provider integration has the required `Key Vault Secrets User` role on your Azure Key Vault. * Check that the secret exists in your Azure Key Vault and contains a valid value. * Validate your provider integration using the **Validate** option in Confluent Cloud. * Ensure that secret identifiers of the configurations that Confluent Cloud fetches from the secret manager follow the specified format. For more information, see [Configuration value format](#az-key-vault-secret-format).