1. Home
  2. Cloud
  3. Confluent Cloud Clusters
  4. Encrypt Kafka Clusters using Self-Managed Keys

Encrypt Confluent Cloud Dedicated Kafka Clusters using Self-Managed Keys on Azure

When you create a Confluent Cloud Dedicated Kafka cluster on Azure, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user to decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Azure using Azure Key Vault to create and manage encryption keys. Use of self-managed encryption keys for clusters in Confluent Cloud on Azure includes the following operations and limitations:

  • You can only enable self-managed encryption keys during creation of Dedicated Kafka clusters. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
  • Use Azure Key Vault to generate, use, rotate, and destroy your encryption keys.
  • Azure RBAC and purge protection must be enabled.
  • Customer-managed keys (CMKs) are supported.
  • Your customer-managed key (CMK) and the Confluent Cloud Kafka cluster must be colocated in the same region.
  • Available for all regions.
  • Only RSA “software-protected” encryption keys are supported.
  • Importing key material is not supported.
  • You can prevent access to your stored data by revoking access to the encryption key but the cluster is inaccessible only after you delete the cluster.
  • When a cluster is deleted, the encryption key is released after five days and is then available to reuse for cluster creation. As a security best practice, encryption keys should not be reused for production clusters.
  • Automatic key rotation is available using Azure Key Vault. Manual key rotation is not supported.
  • Infinite storage is supported.

Note

Confluent CLI v3.3.0 introduces confluent byok create|delete|describe|list commands for managing your self-managed encryption keys in Confluent Cloud. For details, see the Confluent CLI reference.

Confluent CLI v3.3.0 also adds a new --byok flag to the confluent kafka cluster create command for creating encrypted Dedicated clusters. For details, see the Confluent CLI reference.

Keys (byok/v1) API is available in Open Preview for Confluent Cloud on Azure. Use the Keys API to include self-managed encryption keys (aka BYOK) as part of your cluster creation workflow (including the ability to build policy profiles).

Create a Dedicated Kafka cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor Azure can regain access to your data.

To create an encrypted Confluent Cloud Dedicated Kafka cluster on Azure that uses a self-managed encryption key:

  1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

  2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

  3. For 2. Regions/zones under Create cluster, select Azure as the cloud service provider, select the Region and Availability, and then click Continue.

  4. For 3. Networking under Create cluster, select the networking type and click Continue.

  5. For 4. Security under Create cluster, select Self-managed to manage your own encryption key using Azure Key Vault. The Azure Vault Key section appears.

    Step 1: In a separate browser window, go to Key Vaults on your Azure Portal account, select the Azure Key vault key to use, and then enter the following information in the Confluent Cloud Console:

    Entry Description
    Azure Key Vault Resource ID The resource ID of the Azure Key Vault. To find the resource ID, go to Key Vaults on your Azure Portal account, select the Azure Key vault key to use, and then click Overview. Click JSON View (to the right of Essentials) and then copy the value for Resource ID.
    Azure Key Vault Key identifier without version The key ID of the Azure Key Vault.
    Azure Key Vault Tenant ID The tenant ID of the Azure Active Directory associated with your subscription. See Find tenant ID through the Azure portal.

    Click Create new if this is your first time, or click Use existing if you have an available key.

    Important

    • The encryption key and your cluster must be in the same region.
    • When you create keys in Azure Key Vaults, you must:
      • Use an RSA “software-protected” encryption key.
      • Enable purge protection (enforces a mandatory retention period for deleted vaults and vault objects).
      • Enable Azure RBAC for access to the key.
      • If you have network restrictions, enable “Allow trusted Microsoft services to bypass this firewall?”.

    Step 2: While signed in to the correct cluster, copy the CLI snippet, then run the command in your terminal.

    After running the command, return to Confluent Cloud Console and click Continue.

    The Confluent Cloud cluster is created using your encryption key and is ready to use after provisioning.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you get an error message. Close the modal; any invalid fields are highlighted in the original form. Reenter a valid value in the highlighted field.

Revoke Confluent access to a self-managed encryption cluster in Azure

Self-managed encryption keys used in Confluent Cloud with Azure Key Vault require access by Confluent for Confluent Cloud Kafka clusters to operate properly. You should only revoke access if you have a major security concern and need to completely remove Confluent access to your data.

Important

When you disable, or revoke, access to the encryption key for a cluster, the cluster is inaccessible only after you delete the cluster.

Before revoking access to the master key, your Confluent Cloud cluster must be deleted. This ensures that access to data is safely revoked.

To disable access by Confluent to your encryption key:

  1. In your Confluent Cloud environment, stop all clients (producers and consumers) connected to your Confluent Cloud cluster.
  2. Go to the Confluent Cloud Console and delete your cluster.
  3. Go to Azure Key Vault in your Azure Portal at https://portal.azure.com/ and delete the encryption key.

View the security settings

After you create a Dedicated cluster that uses a self-managed encryption key (and the cluster is provisioned), you can view the security settings. You can use these cluster security settings to verify the correct encryption key is used.

To view your Azure Dedicated cluster security settings:

  1. Select your Confluent Cloud cluster.
  2. Click Cluster settings and then Security.

Note that anyone authorized to view your Azure Dedicated clusters can view this data.