Encrypt Confluent Cloud Dedicated Kafka Clusters using Self-Managed Keys on Google Cloud

When you create a Confluent Cloud Dedicated Kafka cluster on Google Cloud, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user to decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Google Cloud using Google Cloud Key Management Service (KMS) to create and manage encryption keys. Use of self-managed encryption keys for clusters in Confluent Cloud on Google Cloud includes the following operations and limitations:

• You can only enable self-managed encryption keys during creation of Dedicated clusters. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
• Use Google Cloud Key Management Service (KMS) to generate, use, rotate, and destroy your encryption keys.
• Customer-managed encryption keys (CMEKs) are supported.
• The customer encryption key and the cloud cluster must be colocated in the same region.
• Available for all regions.
• Only symmetric keys are supported.
• Importing key material is not supported.
• Each encryption key is associated with a single cluster. You cannot use the same encryption key for multiple clusters.
• When a cluster is deleted, the encryption key is released after five days and is then available to reuse for cluster creation. As a security best practice, encryption keys should not be reused for production clusters.
• Automatic key rotation is available using the Google Cloud KMS console. Manual key rotation is not supported.

Note

Confluent CLI v3.3.0 introduces confluent byok create|delete|describe|list commands for managing your self-managed encryption keys in Confluent Cloud. For details, see the Confluent CLI reference.

Confluent CLI v3.3.0 adds a new --byok flag to the confluent kafka cluster create command for creating encrypted Dedicated clusters (on AWS and Azure). For Google Cloud, continue to use the --encryption-id flag instead. For details, see the Confluent CLI reference.

Create a Dedicated cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor Google Cloud can regain access to your data.

To create an encrypted Confluent Cloud Dedicated cluster on Google Cloud that uses a self-managed encryption key:

1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

3. For 2. Regions/zones under Create cluster, select Google Cloud as the cloud service provider, select the Region and Availability, and then click Continue.

4. For 3. Networking under Create cluster, select the networking type and click Continue.

5. For 4 - Security under Create cluster, select Self-managed to manage your own encryption key using Google Cloud Key Management Service. Additional steps appear.

   Note

   • Only symmetric keys are supported.
   • Importing key material is not supported.
   • The key must be for the zone selected in 2. Regions/zone under Create cluster.

   Step 1: Go to the Google Cloud KMS console, copy the resource name of the cryptographic key, return to Confluent Cloud, and paste it in the Google Cloud resource name field.

   Step 2: In the Google Cloud KMS console, complete the following tasks:

   • Create a custom role granted the following required permissions:
     • cloudkms.cryptoKeyVersions.useToDecrypt
     • cloudkms.cryptoKeyVersions.useToEncrypt
     • cloudkms.cryptoKeys.get
   • Copy the Google Group ID from the Confluent Cloud Console, return to the Google Cloud KMS console, select the key of the Google Cloud resource name (entered in Step 1) and in the Permissions tab, click ADD MEMBER and paste the Google Group ID you copied as a new member, assign the custom role, and click SAVE.
   • For more information, see:
     • REST Resource: group
     • Google Cloud Key Management Service documentation

   After completing the two steps above in Google Cloud KMS, return to Confluent Cloud and click Continue.

6. For 5. Review and launch under Create cluster. enter the Cluster name and click Launch Cluster.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Use the Confluent CLI to create an encrypted cluster with a self-managed key

To create a Dedicated cluster that uses a self-managed encryption key in Confluent Cloud on Google Cloud, run the following command, substituting your values for the cluster name (<cluster-name>), the number of Confluent Kafka units (<cku-number>), and the Google Cloud encryption key ID (<gcp-encryption-id>).

confluent kafka cluster create <cluster-name> --cloud "gcp" --region "<KMS-region>" --type "dedicated" --cku <cku-number> --encryption-key "<GCP-key-resource-namespace>"
Create a role with these permissions, add the identity as a member of your key, and grant your role to the member.

Permissions:
  - ``cloudkms.cryptoKeyVersions.useToDecrypt``
  - ``cloudkms.cryptoKeyVersions.useToDecrypt``

  - ``cloudkms.cryptoKeyVersions.get``

Identity:
  <Google-Group-ID>


 Please confirm you've authorized the key for this account: <Google-Group-ID> (y/n):y

When you specify the --encryption-key option, you are prompted to update your Google Cloud KMS policy.

For details, see:

• confluent kafka cluster create
• Google Cloud Key Management Service documentation

Revoke Confluent access to a self-managed encyption cluster in Google Cloud

Self-managed encryption keys used in Confluent Cloud with Google Cloud KMS require access by Confluent for Confluent Cloud Kafka clusters to operate properly. You should only revoke access if you have a major security concern and need to completely remove Confluent access to your data.

Important

When you disable, or revoke, access to the encryption key for a cluster, the cluster is inaccessible only after you delete the cluster.

Before revoking access to the master key, your Confluent Cloud cluster must be deleted. This ensures that access to data is safely revoked.

To disable access by Confluent to your encryption key:

1. In your Confluent Cloud environment, stop all clients (producers and consumers) connected to your Confluent Cloud Kafka cluster.
2. Go the Confluent Cloud Console and delete your cluster.
3. Go to the Key Management page in the Google Cloud console at https://console.cloud.google.com/security/kms/ and disable access to the master key.

View the Dedicated cluster security settings

After you create a Dedicated cluster that uses a self-managed encryption key (and the cluster is provisioned), you can view the security settings. You can use these cluster security settings to verify the correct encryption key is used.

To view your Google Cloud Dedicated cluster security settings:

1. Select your Confluent Cloud cluster.
2. Click Cluster settings and then Security.

Note that anyone authorized to view your Google Cloud Dedicated clusters can view this data.