Encrypt Confluent Cloud Clusters using Self-Managed Keys – Google Cloud

When you create a Confluent Cloud Dedicated cluster on Google Cloud, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user to decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Google Cloud using Google Cloud Key Management Service (KMS) to create and manage encryption keys. Use of self-managed encryption keys for clusters in Confluent Cloud on Google Cloud includes the following operations and limitations:

  • You can only enable self-managed encryption keys during creation of Dedicated clusters. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
  • Use Google Cloud Key Management Service (KMS) to generate, use, rotate, and destroy your encryption keys.
  • Customer-managed encryption keys (CMEKs) are supported.
  • The customer encryption key and the cloud cluster must be colocated in the same region.
  • Available for all regions.
  • Only symmetric keys are supported.
  • Importing key material is not supported.
  • You can revoke encryption keys to prevent access to stored data.
  • Automatic key rotation is available using the Google Cloud KMS console. Manual key rotation is not supported.

Create a Dedicated cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor Google Cloud can regain access to your data.

To create an encrypted Confluent Cloud Dedicated cluster on Google Cloud that uses a self-managed encryption key:

  1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

  2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

  3. For 2. Regions/zones under Create cluster, select Google Cloud as the cloud service provider, select the Region and Availability, and then click Continue.

  4. For 3. Networking under Create cluster, select the networking type and click Continue.

  5. For 4 - Security under Create cluster, select Self-managed to manage your own encryption key using Google Cloud Key Management Service. Additional steps appear.

    Step 1: Go to the Google Cloud KMS console, copy the resource name of the cryptographic key, return to Confluent Cloud, and paste it in the Google Cloud resource name field.

    Step 2: In the Google Cloud KMS console, complete the following tasks:

    • Create a custom role granted the following required permissions:
      • cloudkms.cryptoKeyVersions.useToDecrypt
      • cloudkms.cryptoKeyVersions.useToEncrypt
      • cloudkms.cryptoKeys.get
    • Copy the service account from the Confluent Cloud Console, return to the Google Cloud KMS console, select the key of the Google Cloud resource name (entered in Step 1) and in the Permissions tab, click ADD MEMBER and paste the service account you copied as a new member, assign the custom role, and click SAVE.
    • For details, see the Google Cloud Key Management Service documentation.

    After completing the two steps above in Google Cloud KMS, return to Confluent Cloud and click Continue.

  6. For 5. Review and launch under Create cluster. enter the Cluster name and click Launch Cluster.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Use the Confluent Cloud CLI to create an encrypted cluster with a self-managed key

To create a Dedicated cluster that uses a self-managed encryption key in Confluent Cloud on Google Cloud, run the following command, substituting your values for the cluster name (<cluster-name>), the number of Confluent Kafka units (<cku-number>), and the Google Cloud encryption key ID (<gcp-encryption-id>).

ccloud kafka cluster create <cluster-name> --cloud "gcp" --region "<KMS-region>" --type "dedicated" --cku <cku-number> --encryption-key "<GCP-key-resource-namespace>"
Create a role with these permissions, add the identity as a member of your key, and grant your role to the member.

Permissions:

  - ``cloudkms.cryptoKeyVersions.useToDecrypt``

  - ``cloudkms.cryptoKeyVersions.useToDecrypt``

  - ``cloudkms.cryptoKeyVersions.get``

Identity:
  <service-account>


Please confirm you've authorized the key for this account: <service-account> (y/n):y

When you specify the --encryption-key option, you are prompted to update your Google Cloud KMS policy.

For details, see:

View the Dedicated cluster security settings

After you create a Dedicated cluster that uses a self-managed encryption key, you can view the security settings after the cluster is provisioned on Google Cloud. The cluster security settings can be used to verify the encryption key being used.

To view your dedicated Google Cloud cluster security settings:

  1. Select your Confluent Cloud cluster.
  2. Click Cluster settings and then Security.

Note that anyone authorized to view your Google Cloud Dedicated clusters can view this data.