Use Self-Managed Encryption Keys in Confluent Cloud on AWS

Protect the data at rest stored in your Confluent Cloud Enterprise or Dedicated Kafka clusters on AWS using AWS Key Management Service to create and manage encryption keys,

Requirements

Self-managed encryption keys can be used with supported Kafka clusters on Confluent Cloud created using the Self-managed encryption mode. To use self-managed encryption keys on AWS for supported Kafka cluster types, follow these requirements:

Key creation and management

Required RBAC role: OrganizationAdmin or EnvironmentAdmin.

• Create a Dedicated or Enterprise Kafka cluster on AWS using the “Self-managed” encryption mode. After provisioning your Dedicated or Enterprise cluster, you cannot switch modes between Automatic (default) and Self-managed.
• Use AWS Key Management Service (KMS) to generate, use, rotate, and destroy customer master keys (CMKs).
• Only symmetric, software-protected keys are supported.
• Importing key material is not supported.
• AWS KMS External Key Store (XKS) is supported.
  • Your External Key Manager (EKM) configuration (including any proxies), AWS External Key Store (XKS), AWS Key Management Service (KMS), and Kafka cluster must all be located in the same AWS region. Cross-region configurations are not supported.
  • Each XKS instance must be dedicated exclusively to Confluent. Using XKS keys for other services impacts Confluent’s ability to guarantee SLA performance due to quota conflicts and performance factors outside of Confluent’s control.
  • Each XKS instance supports a maximum of four Confluent encryption keys. If you need more than four keys, create additional XKS instances.
• Key rotation:
  • Automatic key rotation is available using the AWS KMS console, but manual key rotation is not supported.
  • WARNING: Deleting old keys is a permanent operation that cannot be undone and results in data loss.
• Use a unique encryption key for each active cluster.
• If you delete a cluster, the encryption key is released after five days and is available for reuse during cluster creation. As a security best practice, encryption keys should not be reused for production clusters.

FIPS 140-2 certification

• Self-managed keys created after March 2023 are FIPS 140-2 Level 3 certified except for AWS regions only offering FIPS 140-2 Level 2 certification. For details, see AWS KMS FAQs.
• You can also use AWS KMS customer managed keys (CMKs) backed by AWS-managed hardware security modules (HSMs) for FIPS 140-2 Level 3 certification.
• For more information, see FIPS 140-2.

Create a Kafka cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor AWS can regain access to your data.

Confluent Cloud ConsoleConfluent CLI

To create an encrypted Confluent Cloud cluster on AWS that uses a self-managed encryption key:

1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

2. For Select cluster type under Create cluster, select a supported Kafka cluster type, and click Begin Configuration.

3. For Regions/zones under Create cluster, select AWS as the cloud service provider, select the Region and Availability, and then click Continue.

4. For Networking under Create cluster, select the networking type and click Continue.

5. For Security under Create cluster, select Self managed to manage your own encryption key using AWS Key Management Service. Additional steps appear.

   Important

   • Only symmetric keys are supported.
   • Importing key material is not supported.
   • The key must be for the same region as the cluster.

6. Select Create new if you need a new key, or select Use existing if you already have a key.

   • If you select Create new, go to the AWS KMS (requires sign-in) console and create a new key. Copy the key ARN and then paste it into the Amazon Resource Name (key ID) field.

   • If you select Use existing, select your key ARN from the drop-down list, then ensure that the policy of the key in your AWS KMS includes the block of code provided in the Confluent Cloud Console.

     The key must be for the region selected earlier. The key ARN is a unique, fully-qualified identifier of a customer-managed key (CMK) and cannot be changed. Each Confluent Cloud cluster requires a unique key.

   Relevant AWS documentation:

   • Finding the key ID and key ARN
   • Creating keys

7. From the Confluent Cloud Console, copy the provided permission statements exactly as-is (do not edit or modify) into the existing Key policy of the ARN in your AWS KMS. The code block gives your AWS key policy authorization to access your Confluent Cloud cluster and authorizes Confluent access to your KMS. For details, see Key policies.

   Important

   Make sure to append the code block (that includes two new permission statements) immediately after the existing permission statement and remember to add the expected comma separator between the existing statement and the new statements.

   After you complete the cluster creation process, this cluster-key pairing is locked. You cannot change it for the lifetime of the cluster. You can still modify permissions related to the key, and also disable or delete it, as long as your AWS permissions allow for it.

   The updated key policy should look similar to the following example:

   {
     "Sid": "Allow Confluent account(s) (152535741197) to use the key",
     "Effect": "Allow",
     "Principal": {
       "AWS": [
         "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
       ]
     },
     "Action": [
       "kms:Encrypt",
       "kms:Decrypt",
       "kms:ReEncrypt*",
       "kms:GenerateDataKey*",
       "kms:DescribeKey"
     ],
     "Resource": "*"
   },
   {
     "Sid": "Allow Confluent account(s) (152535741197) to attach persistent resources",
     "Effect": "Allow",
     "Principal": {
       "AWS": [
         "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
       ]
     },
     "Action": [
       "kms:CreateGrant",
       "kms:ListGrants",
       "kms:RevokeGrant"
     ],
     "Resource": "*"
   }
   

8. Click Continue.

9. On the Review and launch page, enter a meaningful name in Cluster name and click Launch Cluster.

   A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Create a AWS key policy

KMS keys belong to the AWS account in which they are created.

• The IAM user who creates a KMS key is not automatically considered to be the key owner and does not automatically have permission to use or manage the KMS key they created. To get permission, you must specify a key policy that includes permission statements that control access. For details, see Permissions for creating KMS keys.

By default, the key policy grants every principal in an AWS account access to the AWS KMS customer-managed key (CMK). To restrict access to your encryption key, you can create your own AWS KMS key policy that specifies a permission set.

• The permission sets must be added to the key policy when you create the KMS key to ensure that you can control access to the KMS key.
• Aliases can be used to control access to KMS keys, following the best practice of granting least privileged access. For details, see Using aliases.

For details, see Key policies in AWS KMS.

Example of an AWS KMS key policy

Your AWS KMS key policy consists of two distinct permission statements that grant access for compute and storage requirements.

The first permission statement grants Confluent AWS accounts the permissions required to describe the KMS encryption key, list metadata information about the key, perform encryption, decryption, re-encryption, and data-key generation. In the example below, the first statement specifies the required Confluent AWS accounts that are granted these permissions using the two roles for blob (tiered) and block (disk) storage. As the current Confluent Cloud infrastructure continues to grow and scale, an increasing number of accounts is required to maintain scalability.

The second permission statement grants Confluent AWS accounts the permissions required to manage (create, list, and revoke) grants for data storage. These grants are internally used to provide access for performing cryptographic operations to the specified grantee principal. In this case, the permission statement creates a grant for the role that manages block storage, which in turn creates the grant for the AWS internal roles (for example aws:ec2-infrastructure) to persistently attach AWS EBS volumes (block storage) to the AWS EC2 instance where the Kafka broker (pod) runs.

{
  "Sid": "Allow Confluent account(s) (152535741197) to use the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
},
{
  "Sid": "Allow Confluent account(s) (152535741197) to attach persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}

For "Resource": "*" above, the wildcard character (*) indicates that the KMS key policy applies to the customer-managed key (CMK).