Encrypt Confluent Cloud clusters using self-managed keys – Google Cloud Platform

When you create a Confluent Cloud Dedicated cluster on Google Cloud Platform, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user to decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Google Cloud Platform using Google Cloud Key Management Service (KMS) to create and manage encryption keys. Use of self-managed encryption keys for clusters in Confluent Cloud on Google Cloud Platform includes the following operations and limitations:

  • You can only enable self-managed encryption keys during creation of Dedicated clusters. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
  • Use Google Cloud Key Management Service (KMS) to generate, use, rotate, and destroy your encryption keys.
  • Customer-managed encryption keys (CMEKs) are supported.
  • The customer encryption key and the cloud cluster must be colocated in the same region.
  • Available for all regions, except us-east1 and us-west2.
  • Only symmetric keys are supported.
  • Importing key material is not supported.
  • You can revoke encryption keys to prevent access to stored data.
  • Automatic key rotation is available using the Google Cloud KMS console. Manual key rotation is not supported.
  • Infinite Storage is not available on BYOK-enabled clusters.

Create a Dedicated cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor GCP can regain access to your data.

To create an encrypted Confluent Cloud Dedicated cluster on Google Cloud Platform that uses a self-managed encryption key:

  1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click ** Add cluster** if other clusters exist.

  2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

  3. For 2. Regions/zones under Create cluster, select Google Cloud as the cloud service provider, select the Region and Availability, and then click Continue.

  4. For 3. Networking under Create cluster, select the networking type and click Continue.

  5. For 4 - Security under Create cluster, select Self-managed to manage your own encryption key using Google Cloud Key Management Service. Additional steps appear.

    Step 1: Go to the Google Cloud KMS console, copy the resource name of the cryptographic key, return to Confluent Cloud, and paste it in the Google Cloud resource name field.

    Step 2: In the Google Cloud KMS console, complete the following tasks:

    • Create a custom role granted the following required permissions:
      • cloudkms.cryptoKeyVersions.useToDecrypt
      • cloudkms.cryptoKeyVersions.useToEncrypt
      • cloudkms.cryptoKeys.get
    • Copy the service account from the Confluent Cloud UI, return to the Google Cloud KMS console, select the key of the Google Cloud resource name (entered in Step 1) and in the Permissions tab, click ADD MEMBER and paste the service account you copied as a new member, assign the custom role, and click SAVE.
    • For details, see the Google Cloud Key Management Service documentation.

    After completing the two steps above in Google Cloud KMS, return to Confluent Cloud and click Continue.

  6. For 5. Review and launch under Create cluster. enter the Cluster name and click Launch Cluster.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Use the Confluent Cloud CLI to create an encrypted cluster with a self-managed key

The following Confluent Cloud CLI example shows how to create a Confluent Cloud cluster that uses a self-managed encryption key. Any content in brackets (<>) must be customized for your environment.

ccloud kafka cluster create <cluster-name> --cloud "gcp" --region "<KMS-region>" --type "dedicated" --encryption-key "<GCP-key-resource-namespace>"
Create a role with these permissions, add the identity as a member of your key, and grant your role to the member.

Permissions:

  - cloudkms.cryptoKeyVersions.useToDecrypt

  - cloudkms.cryptoKeyVersions.useToDecrypt

  - cloudkms.cryptoKeyVersions.get

Identity:
  <service-account>


Please confirm you've authorized the key for this account: <service-account> (y/n):y

When you specify the --encryption-key option, you are prompted to update your GCP KMS policy.

For details, see:

View the Dedicated cluster security settings

You can view the Security settings for Dedicated clusters provisioned on GCP. In other words, if you used Automatic, Self-managed, or have an existing Dedicated cluster on GCP that you created prior to using the Self-managed encryption option, you can view the cluster security settings. The data in the cluster security settings is informational only and serves to identify the keys in use.

To view your dedicated GCP cluster security settings:

  1. Select your Confluent Cloud cluster.
  2. Click the Cluster settings tab and then Security.

Note that anyone authorized to view your GCP Dedicated clusters can view this data.