# Use Self-Managed Encryption Keys in Confluent Cloud on Google Cloud Confluent Cloud clusters on Google Cloud are encrypted to protect your [data at rest](../../../_glossary.md#term-data-at-rest). As an option, when you create a [Dedicated Kafka cluster](../../../clusters/cluster-types.md#dedicated-cluster) or [Enterprise Kafka cluster](../../../clusters/cluster-types.md#enterprise-cluster) on Google Cloud, you can use self-managed encryption keys to protect your data, allowing only the appropriate entity or user to decrypt it. Self-managed keys provide you greater privacy and data security, and allow you to maintain control over your encryption keys. When you use self-managed encryption keys to encrypt Kafka clusters, Confluent does not control access to your encryption keys. You are responsible for managing your keys and using [Google Cloud Key Management Service (KMS)](https://cloud.google.com/security-key-management) to generate, use, rotate, and destroy your encryption keys. ## Requirements Self-managed encryption keys are supported only on Kafka clusters created with the Self-managed encryption mode. To use self-managed encryption keys on Google Cloud for supported Kafka cluster types, follow these requirements: ### Key creation and management Required RBAC role: [OrganizationAdmin](../../access-control/rbac/predefined-rbac-roles.md#organizationadmin-role) or [EnvironmentAdmin](../../access-control/rbac/predefined-rbac-roles.md#environmentadmin-role). - Create a Dedicated or Enterprise Kafka cluster on Google Cloud using the “Self-managed” encryption mode. After provisioning your Dedicated or Enterprise cluster, you cannot switch modes between Automatic (default) and Self-managed. - Review the [general requirements](overview.md#byok-general-requirements) that apply across all cloud service providers. - Manage encryption keys on Google Cloud using [Cloud Key Management](https://cloud.google.com/security-key-management) - [Cloud Key Management Service (Cloud KMS)](https://cloud.google.com/security-key-management) for generating, using, rotating, and destroying Cloud KMS encryption keys. - [Cloud External Key Manager (Cloud EKM)](https://cloud.google.com/kms/docs/ekm) for communicating with an external key manager used to manage encryption keys. - Key rotation: - Automatic key rotation is available using the Google Cloud KMS console, but manual key rotation is not supported. - For external key manager (EKM) keys, follow the key rotation process and policies of your external key manager and Google Cloud EKM. - WARNING: Deleting old keys or key versions is a permanent operation that cannot be undone and results in data loss. - [Customer-managed encryption keys (CMEKs)](https://cloud.google.com/storage/docs/encryption/customer-managed-keys) are supported. - Only “software-protected” keys are supported. Importing key material is not supported. - If you delete a cluster, you can delete the self-managed encryption key immediately. However, you can’t reuse a deleted key to create another cluster for five days. As a security best practice, don’t reuse encryption keys for production clusters. ### FIPS 140-2 certification - Software-protected keys (FIPS 140-2 Level 1): Google Cloud KMS encryption keys use the BoringCryptoModule (BCM). - Hardware Security Module (HSM) keys (FIPS 140-2 Level 3): Google Cloud CloudHSM is validated to FIPS 140-2 Level 3. - For details, see [FIPS 140-2 Validated](https://cloud.google.com/security/compliance/fips-140-2-validated). #### WARNING If you accidentally delete the master key, you are no longer able to access your encrypted data. Neither Confluent nor Google Cloud can regain access to your data. #### WARNING **Avoid updating key policies unless absolutely necessary.** Key policy misconfigurations can cause immediate cluster unavailability and service disruption. For details on how to manage your key policy, see [Manage Google Cloud key policies](#byok-gcp-manage-key-policies). ## Create a self-managed encryption key A self-managed encryption key can be created in two ways: 1. **From the global Encryption Keys page** (recommended) 2. **During cluster creation** ### Method 1: From the global Encryption Keys page To create a self-managed encryption key from the global Encryption Keys page: 1. In the Confluent Cloud Console, click the hamburger menu in the upper right corner. 2. Select **Encryption keys** from the menu. 3. Click **Add new key**. 4. **Step 1: Choose a cloud provider** - Select **Google Cloud**. 5. **Step 2: Enter key details** - Provide the following information: - **Key Alias** (optional): A meaningful name to identify the key - **Google Cloud resource name**: The full resource name of your cryptographic key from the Google Cloud KMS console 6. Click **Register key**. The key is created and will appear in the encryption keys table. 7. **Step 3: Configure permissions and policy** - Follow the Google Cloud-specific instructions to configure the required permissions. 8. Click **Finish**. The key enters an initializing state which runs asynchronously and may take up to 5 minutes. The key will show a status of “Initializing” until validation completes. Once validated, the key can be used when creating clusters. ### Method 2: Create a Kafka cluster with self-managed encryption ### Confluent Cloud Console To create an encrypted Kafka cluster on Confluent Cloud on Google Cloud that uses a self-managed encryption key: 1. Navigate to the **Clusters** page for your environment and click **Create cluster** if you are creating the first cluster in your environment, or click **Add cluster** if other clusters exist. 2. For **Select cluster type** under **Create cluster**, select a supported Kafka cluster type (Dedicated or Enterprise) and click **Begin Configuration**. 3. For **Regions/zones** under **Create cluster**, select Google Cloud as the cloud service provider, select the **Region** and **Availability**, and then click **Continue**. 4. For **Networking** under **Create cluster**, select the [networking](../../../networking/overview.md#cloud-networking) type and click **Continue**. 5. For **Security** under **Create cluster**, select **Self-managed** to manage your own encryption key using Google Cloud Key Management Service. You can either: - **Select an existing key**: Choose from the dropdown list of previously created and validated encryption keys from the global Encryption Keys page. - **Add a new key**: Create a new encryption key during cluster creation by providing the Google Cloud resource name. #### NOTE - Only symmetric keys are supported. - Importing key material is not supported. - Key validation during cluster creation is asynchronous and may take a few minutes. - The key must be in the same region as your cluster. **Step 1**: Go to the [Google Cloud KMS console](https://console.cloud.google.com/security/kms), copy the resource name of the cryptographic key, return to Confluent Cloud, and paste it in the **Google Cloud resource name** field. **Step 2**: In the [Google Cloud KMS console](https://console.cloud.google.com/security/kms), set the required permissions for your key as described in [Manage Google Cloud key policies](#byok-gcp-manage-key-policies). - For more information, see: - [REST Resource: group](https://developers.google.com/admin-sdk/directory/reference/rest/v1/groups) - [Google Cloud Key Management Service documentation](https://cloud.google.com/kms/docs) - [VPC Service Controls](https://docs.cloud.google.com/vpc-service-controls/docs/overview) After completing the two steps above in Google Cloud KMS, return to Confluent Cloud and click **Continue**. 6. For **5. Review and launch** under **Create cluster**. enter the **Cluster name** and click **Launch Cluster**. #### NOTE A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal dialog; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field. If the key is not valid or not authorized for Confluent, you can revisit the policy and authorization instructions from the global Encryption Keys page. Navigate to the Encryption Keys page, find your key, and click **View key details** to access the permissions and policy configuration instructions. ### Confluent CLI To create a Dedicated or Enterprise Kafka cluster that uses a self-managed encryption key in Confluent Cloud on Google Cloud, run the Confluent CLI `confluent kafka cluster create` command, substituting your values for the cluster name (``), the number of [CKUs](../../../_glossary.md#term-Confluent-Unit-for-Kafka-CKU) (``), and the Google Cloud encryption key ID (``). ```shell confluent kafka cluster create --cloud "gcp" \ --region "" \ --type "dedicated|enterprise" \ --cku \ --encryption-key "" Set the required permissions for your key as described in :ref:`byok-gcp-manage-key-policies`. Please confirm you've authorized the key for this account: (y/n):y ``` When you specify the `--encryption-key` option, you are prompted to update your Google Cloud KMS policy. The Google Group ID is unique per cluster and is created automatically during cluster creation. This ID remains static, but service accounts associated with the cluster can change over time. For details, see: - [confluent kafka cluster create](https://docs.confluent.io/confluent-cli/current/command-reference/kafka/cluster/confluent_kafka_cluster_create.html) - [Google Cloud Key Management Service documentation](https://cloud.google.com/kms/docs) ## Manage Google Cloud key policies You can manage Google Cloud key policies with a custom role or with VPC Service Controls. Google Cloud custom roles is the recommended approach. ### Manage key policies with a custom role (recommended) You can manage the key policies using a Google Cloud custom role. 1. Create a [custom role](https://cloud.google.com/kms/docs/reference/permissions-and-roles#custom_roles) and grant it the following required permissions: - `cloudkms.cryptoKeyVersions.useToDecrypt` - `cloudkms.cryptoKeyVersions.useToEncrypt` - `cloudkms.cryptoKeys.get` 2. Add the **Google Group ID** provided in the Confluent Cloud Console as a new member and assign the custom role to it. The Google Group ID is unique per cluster and is created automatically during cluster creation. This ID remains static, but service accounts associated with the cluster can change over time. ### Manage key policies with Google Cloud VPC Service Controls (advanced) Confluent supports using [Google Cloud VPC Service Controls](https://cloud.google.com/vpc-service-controls/docs) to protect access to Google Cloud services such as Cloud Storage that are used by your Confluent Cloud clusters. Confluent supports granting the per-cluster Google Group ID in both ingress and egress policies. Other VPC Service Controls approaches are not supported. To allow Confluent to access resources inside your service perimeter, add an ingress rule that allows the **Confluent Google Group ID** for your cluster. This is the same Google Group ID shown in the Confluent Cloud Console during cluster creation and is referenced above for the KMS key policy. #### Configure VPC Service Controls with the Google Cloud portal ### Google Cloud Console 1. In Access Context Manager, edit your service perimeter and add an **ingress rule**. 2. Configure the ingress rule. Here is an example: ![image](images/vpc-gcp-ingress.png) 3. Create a matching **egress rule** that allows the same Confluent Google Group ID. Here is an example: ![image](images/vpc-gcp-egress.png) ### Terraform You can use Terraform to manage your VPC Service Controls configuration. Replace the placeholder values with your specific configuration: - ``: Access Context Manager policy ID - ``: Name for your service perimeter - ``: Google Cloud project ID - ``: Cluster Google Group ID ```hcl terraform { required_version = ">= 1.0" required_providers { google = { source = "hashicorp/google" version = "~> 7.0" } } } resource "google_access_context_manager_service_perimeter" "confluent_vpc_sc" { parent = "accessPolicies/" name = "accessPolicies//servicePerimeters/" title = "confluent-cluster-vpc-sc" description = "VPC Service Controls perimeter for Confluent Cloud cluster access" status { # Protected project resources = [ "projects/" ] # GCP services to restrict restricted_services = [ "storage.googleapis.com", "cloudkms.googleapis.com" ] # Ingress Policy - Allow Confluent Cloud groups access ingress_policies { ingress_from { identities = [ "group:" ] sources { access_level = "*" } } ingress_to { resources = ["*"] operations { service_name = "*" } } } # Egress Policy - Allow Confluent Cloud groups access egress_policies { egress_from { identities = [ "group:" ] } egress_to { resources = ["*"] operations { service_name = "*" } } } } timeouts {} use_explicit_dry_run_spec = false } ``` Recommendations: : - Use `terraform plan` to preview changes before applying. - Consider using Terraform modules to reuse this configuration across multiple clusters. - The `restricted_services` list can be customized based on which Google Cloud services your workloads need to access. - For production deployments, consider using more restrictive `resources` and `operations` scoping instead of wildcards. - Replace `*` placeholders with explicit services and project resources if your organization requires stricter scoping. For details on configuring ingress and egress policies, see [Configure service perimeter rules](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules). For more information about CEL expressions in VPC Service Controls, see [Using CEL expressions](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#using-cel). ## Update a Google Cloud KMS key policy Follow these steps to update your Google Cloud KMS key policy without disrupting your Confluent Cloud cluster operations: Prerequisites : * Administrative access to your Google Cloud project and KMS service. * Current working backup of your key policy. * Planned maintenance window for policy updates. * Understanding of the required Confluent permissions. 1. Create a backup of your current working key policy. You can retrieve the current policy using the gcloud CLI: ```shell gcloud kms keys get-iam-policy \ \ --keyring \ --location \ > gcp-kms-key-policy-backup-$(date +%Y-%m-%d).json ``` 2. Apply the policy update during your planned maintenance window: ```shell gcloud kms keys set-iam-policy \ \ --keyring \ --location \ new-policy.json ``` For more information, see the [gcloud kms keys set-iam-policy](https://cloud.google.com/sdk/gcloud/reference/kms/keys/set-iam-policy) documentation. 3. Verify your cluster is still operational: * Check cluster status in the Confluent Cloud Console. * Verify producers and consumers are still functioning. * Monitor for any error messages or alerts. 4. Check Google Cloud audit logs for any access denied errors related to your KMS key: ```shell gcloud logging read \ "resource.type=\"cloudkms_cryptokey\" AND protoPayload.methodName=\"Decrypt\" AND status.message!=\"OK\"" \ --freshness=10m ``` 5. Monitor your cluster health for at least 30 minutes after the policy update: * Watch cluster metrics and health indicators. * Check for any encryption-related errors. * Verify that automatic operations (like scaling) continue to work. If you encounter problems after updating the key policy: 1. Immediately restore the backup policy. 2. Monitor cluster recovery for up to 30 minutes. 3. Contact [Confluent Support](https://support.confluent.io) if the cluster doesn’t recover after restoring the original policy.