Note

The secret manager integration feature is in early access.

An early access feature is a component of Confluent Cloud introduced to gain feedback. This feature should be used only for evaluation and non-production testing purposes or to provide feedback to Confluent, particularly as it becomes more widely available in follow-on preview editions. If you would like to participate in the Early Access Program, reach out to your account team.

Early Access Program features are intended for evaluation use in development and testing environments only, and not for production use. Early Access Program features are provided: (a) without support; (b) “AS IS”; and (c) without indemnification, warranty, or condition of any kind. No service level commitment will apply to Early Access Program features. Early Access Program features are considered to be a Proof of Concept as defined in the Confluent Cloud Terms of Service. Confluent may discontinue providing preview releases of the Early Access Program features at any time in Confluent’s sole discretion.

Create a secret manager integration in Confluent Cloud

This topic explains how to integrate a secret manager with Confluent Cloud to retrieve authentication related secrets for fully-managed connectors.

Overview

Fully-managed connectors in Confluent Cloud can integrate with secret managers from cloud service providers (CSPs) such as Azure Key Vault to retrieve sensitive information such as database credentials, access keys, and similar authentication related secrets. This feature ensures that your secrets never persist within Confluent’s boundary, and Confluent Cloud will always fetch them at runtime as needed.

Secret manager integration uses Confluent Provider Integration to authenticate with your CSP and securely access secrets stored in services like Azure Key Vault, AWS Secrets Manager, or Google Cloud Secret Manager. For more information about provider integration in Confluent Cloud, see Manage Provider Integration for Fully-Managed Connectors in Confluent Cloud.

Note

At the moment, secret manager supports Azure Key Vault integration only.

Prerequisites

Before you start integrating a secret manager with Kafka Connect in Confluent Cloud:

Limitations

  • Cross-cloud limitations: The secret manager must be in the same cloud provider as your Confluent Cloud cluster. For example, using Azure Key Vault with a Connect cluster on AWS is not supported.

  • Supported cluster type: Right now, you can integrate a secret manager within Dedicated clusters only.

Note

Limitations on integrating a CSP with Confluent Cloud are applicable to secret manager integration as well. For more information, see Provider integration limitations.

Microsoft Azure Key Vault integration

Supported connectors

The following connectors support secret manager integration with Azure Key Vault:

Supported regions

Microsoft Azure Key Vault is supported in the following tier-1 regions:

  • westeurope

  • eastus

  • eastus2

  • centralus

  • westus2

  • northeurope

  • southeastasia

  • brazilsouth

  • centralindia

  • australiaeast

  • uksouth

Step 1: Configure a Microsoft Azure Provider Integration to access Azure Key Vault

Secret manager integration requires a provider integration to establish secure access to your Azure Key Vault. You must configure an Azure Provider Integration in Confluent Cloud before you can create a connector that uses Azure Key Vault.

For detailed setup instructions, see Create an Azure Provider Integration in Confluent Cloud.

Step 2: Set up Azure Key Vault

Perform the following steps to create an Azure Key Vault.

  1. Create an Azure Key Vault and add your secrets such as database credentials, access keys, and similar authentication related secrets. For more information, see:

  2. Assign the following required permissions to the service principal associated with your Azure provider integration to access your Azure Key Vault:

    • Microsoft.KeyVault/vaults/secrets/getSecret/action (Get Secret)

    • Microsoft.KeyVault/vaults/secrets/readMetadata/action (List Secrets)

    These permissions are included in the built-in Key Vault Secrets User role. You can assign this role to your service principal using the Azure portal or Azure CLI.

  3. If you want to use connectors with secret manager within an Azure Private Link Confluent Cloud cluster, see Egress Private Link Endpoints Setup Guide: First-Party Services on Azure for Confluent Cloud.

Step 3: Use Azure Key Vault integration with a connector

Now that you have successfully integrated Azure Key Vault with Confluent Cloud, you can use it within a connector. For example, you can add a SQL Server CDC Source v2 connector, and use Azure Key Vault to fetch sensitive data such as passwords for authentication.

Tip

Use Read a provider integration API to fetch the id of the provider integration and use it as secret.manager.provider.integration.id in your connector configuration.

Authenticate using Azure Key Vault

At the Add Source/Sink connector screen, complete the following in the Authentication step:

  1. Choose an authentication method that uses static credentials like Password and SAS Key.

  2. Toggle the Use secret manager setting on to let Confluent Cloud fetch credentials from Azure Key Vault.

  3. Select AZURE_KEY_VAULT from the Secret manager dropdown.

  4. From the Configurations from Secret manager multi-select dropdown, select the configurations whose values Confluent Cloud should fetch. Ensure that the values for the selected configurations follow the format specified in the Configuration value format section.

  5. Select your provider integration name from the Provider integration dropdown.

  6. Complete your connector’s configuration. For detailed documentation on supported connectors, see Supported connectors.

Verify Azure Key Vault authentication

Once the connector is provisioned and running, verify the successful authorization through the credentials fetched from Azure Key Vault.

  1. Sign in to your Confluent Cloud account.

  2. In the left navigation menu, click Connectors.

  3. Click the connector you just created above.

  4. View the messages produced to or received from a topic.

  1. Go to the Environments page at https://confluent.cloud/environments and select the environment in which you just created a provider integration.

  2. Click the Provider integrations tab.

  3. Under the Resources column, view the resources.

  4. Click on the resource link to view the associated connector that you just created above.

Authenticate using secrets from Azure Key Vault

Create a JSON file that contains the connector configuration properties with provider integration. Add the following fields in the JSON:

  • authentication.method

  • secret.manager.enabled

  • secret.manager

  • secret.manager.managed.configs

  • secret.manager.provider.integration.id

  • The configuration whose value you want Confluent Cloud to fetch. In this example, it is the value of the database.password and the database.user fields. For more information on the Azure Key Vault’s secret-name format, see Configuration value format.

{
   "name": "SqlServerCdcSource1",
   "config": {
      "connector.class": "SqlServerCdcSourceV2",
      "name": "SqlServerCdcSource1",
      "kafka.auth.mode": "SERVICE_ACCOUNT",
      "kafka.service.account.id": "sa-xxxxxx",
      "authentication.method": "Password",
      "secret.manager.enabled": "true",
      "secret.manager": "AZURE_KEY_VAULT",
      "secret.manager.managed.configs": "database.password,database.user",
      "secret.manager.provider.integration.id": "cspi-devxxxxxx",
      "database.hostname": "203.0.113.0",
      "database.port": "1433",
      "database.user": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user",
      "database.password": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password",
      "database.names": "db123",
      "database.encrypt": "false",
      "output.data.format": "JSON",
      "output.key.format": "JSON",
      "table.include.list": "table123",
      "topic.prefix": "sqlcdc",
      "tasks.max": "1"
   },
}

Load the properties file

Enter the following command to load the JSON and start the connector:

confluent connect cluster create --config-file <file-name>.json

For example, if your filename is sqlcdc_with_key_vault.json, then run:

confluent connect cluster create --config-file sqlcdc_with_key_vault.json

Example output:

Created connector SqlServerCdcSource1 lcc-ix4dl

Validate Azure Key Vault authorization

After successful provisioning, verify messages are populating your Kafka topic.

To verify whether messages are populating your Kafka topics:

  1. Sign in to your Confluent Cloud account.

  2. In the left navigation menu, click Connectors.

  3. Click the connector you just created above.

  4. View the messages produced to or received from a topic.

Authenticate using Azure Key Vault

Include the following fields in the configuration payload of your Create a connector API request.

  • authentication.method

  • secret.manager.enabled

  • secret.manager

  • secret.manager.managed.configs

  • secret.manager.provider.integration.id

  • The secret-name whose value you want Confluent Cloud to fetch. In this example, it is database.password field’s value. For more information on the Azure Key Vault’s secret-name format, see Configuration value format.

curl --request POST \
--url 'https://api.confluent.cloud/connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'content-type: application/json' \
--data '{
   "name": "SqlServerCdcSource1",
   "config": {
      "connector.class": "SqlServerCdcSourceV2",
      "name": "SqlServerCdcSource1",
      "kafka.auth.mode": "SERVICE_ACCOUNT",
      "kafka.service.account.id": "sa-xxxxxx",
      "authentication.method": "Password",
      "secret.manager.enabled": "true",
      "secret.manager": "AZURE_KEY_VAULT",
      "secret.manager.managed.configs": "database.password,database.user",
      "secret.manager.provider.integration.id": "cspi-devxxxxxx",
      "database.hostname": "203.0.113.0",
      "database.port": "1433",
      "database.user": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user",
      "database.password": "https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password",
      "database.names": "db123",
      "database.encrypt": "false",
      "output.data.format": "JSON",
      "output.key.format": "JSON",
      "table.include.list": "table123",
      "topic.prefix": "sqlcdc",
      "tasks.max": "1"
   },
}'

Verify Microsoft Azure Key Vault based authentication

Your new connector is now processing the messages after successful provisioning. Use Confluent Cloud Console to view the messages.

  1. Sign in to your Confluent Cloud account.

  2. In the left navigation menu, click Connectors. Click the connector you just created in the Step 4.

  3. View the messages produced to a topic, for example, from the SQL Server CDC V2 Source connector.

This validates the successful Entra ID-based authorization through the secret manager integration.

Configuration value format

For the connector configurations that are managed through Azure Key Vault, you must specify the value of those configurations according to the following specified formats:

  1. If your secret is stored as plaintext (non-JSON format), ensure that you specify its configuration value as <AZURE_VAULT_URI>::<secret_name>.

    For example, if you want Confluent Cloud to fetch the database password configuration, and:

    • The secret-name within Azure Key Vault is sqlserver-db-secret

    • The Azure Key Vault’s URI is https://your-key-vault-name.vault.azure.net

    Then, specify the value of the database password as https://your-key-vault-name.vault.azure.net::sqlserver-db-secret.

  2. If your secrets are stored as a JSON object within the Azure Key Vault secret, then ensure that you specify each configuration value as <AZURE_VAULT_URI>::<secret_name>::<config-key>.

    For example, if https://your-key-vault-name.vault.azure.net::sqlserver-db-secret stores a JSON that looks like:

    {
   "user": "dbadmin",
   "password": "$trongPassw0rdExamp1e"
}

    • To fetch the database user, ensure that you specify the connector configuration value as https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::user.

    • To fetch the database password, ensure that you specify the connector configuration value as https://your-key-vault-name.vault.azure.net::sqlserver-db-secret::password.

Troubleshooting

Connector fails to start with authentication error

  • Verify your Azure Key Vault name and secret name are correct.

  • Ensure the service principal associated with your provider integration has the required Key Vault Secrets User role on your Azure Key Vault.

  • Check that the secret exists in your Azure Key Vault and contains a valid value.

  • Validate your provider integration using the Validate option in Confluent Cloud.

  • Ensure that the values of the configurations fetched from the secret manager follow the specified format. For more information, see Configuration value format.

Secret rotation and lifecycle

Secret manager integration retrieves secrets when a connector starts or restarts. To apply rotated credentials from your secret manager:

  1. Rotate the secret in your secret manager using your CSP’s secret management tools.

  2. Restart the connector in Confluent Cloud to fetch the updated secret. For example, navigate to the connector settings and click Restart.

Note

Connectors do not automatically poll for secret updates.

If you rotate credentials in your secret manager, you must manually restart the connector after waiting for at-least five minutes from the time you rotate your credentials. This ensures that previously cached credentials get invalidated.

Related content