Encrypt Confluent Cloud Dedicated Kafka Clusters using Self-Managed Keys on AWS

When you create a Confluent Cloud Dedicated Kafka cluster on Amazon Web Services, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user can decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Amazon Web Services using AWS Key Management Service (KMS) to create and manage encryption keys.

Use of self-managed encryption keys for clusters in Confluent Cloud on AWS includes the following operations and limitations:

• Create Dedicated clusters that use self-managed encryption keys during cluster creation. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
• Use AWS Key Management Service (KMS) to generate, use, rotate, and destroy your encryption keys.
• Customer master keys (CMKs) are supported.
• The cluster and the encryption key must use the same region.
• Each cluster must have its own unique encryption key.
• Only symmetric keys are supported.
• Importing key material is not supported.
• To prevent access to stored data, revoke the encryption key.
• Automatic key rotation can be enabled using AWS KMS. Manual key rotation is not supported.
• Each encryption key is associated with a single cluster. You cannot use the same encryption key for multiple clusters.
• When a cluster is deleted, the encryption key is released after five days and is then available to reuse for cluster creation. As a security best practice, encryption keys should not be reused for production clusters.

Note

Confluent CLI v3.3.0 introduces confluent byok create|delete|describe|list commands for managing your self-managed encryption keys in Confluent Cloud. For details, see the Confluent CLI reference.

Confluent CLI v3.3.0 also adds a new --byok flag to the confluent kafka cluster create command for creating encrypted Dedicated clusters. For details, see the Confluent CLI reference.

Keys (byok/v1) API is available in Open Preview for Confluent Cloud on AWS. Use the Keys API to include self-managed encryption keys (aka BYOK) as part of your cluster creation workflow (including the ability to build policy profiles).

Create a Dedicated cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor AWS can regain access to your data.

Confluent Cloud ConsoleConfluent CLI

To create an encrypted Confluent Cloud Dedicated cluster on AWS that uses a self-managed encryption key:

1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

3. For 2. Regions/zones under Create cluster, select AWS as the cloud service provider, select the Region and Availability, and then click Continue.

4. For 3. Networking under Create cluster, select the networking type and click Continue.

5. For 4 - Security under Create cluster, select Self-managed to manage your own encryption key using AWS Key Management Service. Additional steps appear.

   Important

   • Only symmetric keys are supported.
   • Importing key material is not supported.
   • The key must be for the same region as the cluster.

6. Enter the Amazon Resource Name (key ID) for your encryption key. You can get the key ARN using the AWS KMS console at console.aws.amazon.com/kms/home. In the navigation bar, click Customer managed keys and then find your existing key or create a new key.

   Note

   • The key must be for the region selected in 2. Regions/zone under Create cluster.

   The key ARN is a unique, fully qualified identifier of a customer-managed key (CMK) and cannot be changed. Each Confluent Cloud cluster requires a unique key.

   Relevant AWS documentation:

   • Finding the key ID and key ARN
   • Creating keys

7. From the Confluent Cloud Console, copy the provided permission statements exactly as-is (do not edit or modify) into the existing Key policy of the ARN in your AWS KMS. The code block gives your AWS key policy authorization to access your Confluent Cloud cluster and authorizes Confluent access to your KMS. For details, see Key policies.

   Important

   Make sure to append the code block (that includes two new permission statements) immediately after the existing permission statement and remember to add the expected comma separator between the existing statement and the new statements.

   After you complete the cluster creation process, this cluster-key pairing is locked. You cannot change it for the lifetime of the cluster. You can still modify permissions related to the key, and also disable or delete it, as long as your AWS permissions allow for it.

   The updated key policy should look similar to the following example:

   {
       "Id": "key-consolepolicy-3",
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "Enable IAM User Permissions",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::626217718466:root"
               },
               "Action": "kms:*",
               "Resource": "*"
           },
           {
             "Sid": "Allow Confluent account(s) (050879227952, 563559260525, 259082970751, 390327825978, 457492987184, 417601102659, 479111403931, 121194963621, 152535741197) to use the key",
             "Effect": "Allow",
             "Principal": {
               "AWS": [
                 "arn:aws:iam::050879227952:role/block_storage_manager",
                 "arn:aws:iam::563559260525:role/block_storage_manager",
                 "arn:aws:iam::259082970751:role/block_storage_manager",
                 "arn:aws:iam::390327825978:role/block_storage_manager",
                 "arn:aws:iam::457492987184:role/block_storage_manager",
                 "arn:aws:iam::417601102659:role/block_storage_manager",
                 "arn:aws:iam::479111403931:role/block_storage_manager",
                 "arn:aws:iam::121194963621:role/block_storage_manager",
                 "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
               ]
             },
             "Action": [
               "kms:Encrypt",
               "kms:Decrypt",
               "kms:ReEncrypt*",
               "kms:GenerateDataKey*",
               "kms:DescribeKey"
             ],
             "Resource": "*"
           },
           {
             "Sid": "Allow Confluent account(s) (050879227952, 563559260525, 259082970751, 390327825978, 457492987184, 417601102659, 479111403931, 121194963621, 152535741197) to attach persistent resources",
             "Effect": "Allow",
             "Principal": {
               "AWS": [
                 "arn:aws:iam::050879227952:role/block_storage_manager",
                 "arn:aws:iam::563559260525:role/block_storage_manager",
                 "arn:aws:iam::259082970751:role/block_storage_manager",
                 "arn:aws:iam::390327825978:role/block_storage_manager",
                 "arn:aws:iam::457492987184:role/block_storage_manager",
                 "arn:aws:iam::417601102659:role/block_storage_manager",
                 "arn:aws:iam::479111403931:role/block_storage_manager",
                 "arn:aws:iam::121194963621:role/block_storage_manager",
                 "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
               ]
             },
             "Action": [
               "kms:CreateGrant",
               "kms:ListGrants",
               "kms:RevokeGrant"
             ],
             "Resource": "*"
           }
   

8. On the Review and launch page, enter the Cluster name and click Launch Cluster.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Create an AWS key policy

KMS keys belong to the AWS account in which they are created.

• The IAM user who creates a KMS key is not automatically considered to be the key owner and does not automatically have permission to use or manage the KMS key they created. To get permission, you must specify a key policy that includes permission statements that control access. For details, see Permissions for creating KMS keys.

By default, the key policy grants every principal in an AWS account access to the AWS KMS customer-managed key (CMK). To restrict access to your encryption key, you can create your own AWS KMS key policy that specifies a permission set.

Important

• The permission sets must be added to the key policy when you create the KMS key to ensure that you can control access to the KMS key.
• Aliases can be used to control access to KMS keys, following the best practice of granting least privileged access. For details, see Using aliases.

For details, see Key policies in AWS KMS.

Example of an AWS KMS key policy

Your AWS KMS key policy consists of two distinct permission statements that grant access for compute and storage requirements.

The first permission statement grants Confluent AWS accounts the permissions required to describe the KMS encryption key, list metadata information about the key, perform encryption, decryption, re-encryption, and data-key generation. In the example below, the first statement specifies the required Confluent AWS accounts that are granted these permissions using the two roles for blob (tiered) and block (disk) storage. As the current Confluent Cloud infrastructure continues to grow and scale, an increasing number of accounts is required to maintain scalability.

The second permission statement grants Confluent AWS accounts the permissions required to manage (create, list, and revoke) grants for data storage. These grants are used internally by AWS to provide access for performing cryptographic operations to a given grantee principal. In this example, AWS internally uses this permission to grant access to service-linked-roles, such as aws:ec2-infrastructure, to persistently attach AWS EBS volumes (block storage) to the AWS EC2 instance where the Kafka broker (pod) runs.

{
  "Sid": "Allow Confluent account(s) (050879227952, 563559260525, 259082970751, 390327825978, 457492987184, 417601102659, 479111403931, 121194963621, 152535741197) to use the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::050879227952:role/block_storage_manager",
      "arn:aws:iam::563559260525:role/block_storage_manager",
      "arn:aws:iam::259082970751:role/block_storage_manager",
      "arn:aws:iam::390327825978:role/block_storage_manager",
      "arn:aws:iam::457492987184:role/block_storage_manager",
      "arn:aws:iam::417601102659:role/block_storage_manager",
      "arn:aws:iam::479111403931:role/block_storage_manager",
      "arn:aws:iam::121194963621:role/block_storage_manager",
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
},
{
  "Sid": "Allow Confluent account(s) (050879227952, 563559260525, 259082970751, 390327825978, 457492987184, 417601102659, 479111403931, 121194963621, 152535741197) to attach persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::050879227952:role/block_storage_manager",
      "arn:aws:iam::563559260525:role/block_storage_manager",
      "arn:aws:iam::259082970751:role/block_storage_manager",
      "arn:aws:iam::390327825978:role/block_storage_manager",
      "arn:aws:iam::457492987184:role/block_storage_manager",
      "arn:aws:iam::417601102659:role/block_storage_manager",
      "arn:aws:iam::479111403931:role/block_storage_manager",
      "arn:aws:iam::121194963621:role/block_storage_manager",
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}

For "Resource": "*" above, the wildcard character (*) indicates that the KMS key policy applies to the customer-managed key (CMK).

Frequently asked questions

Can Confluent accounts read a customer-managed key?

The AWS KMS does not allows Confluent to read a customer-managed key (CMK), regardless of permissions. The permissions you add describe or perform only regardless of permissions. The permissions you add describe or perform only the defined cryptographic operations on the CMK key using the AWS KMS interface.

Why are we listing N accounts instead of 1?

At any point in the life cycle, the brokers can move around between various AWS accounts running the platform. Hence, there is a need to list all the accounts to ensure there is a continuity of service during such migrations.

Who can decrypt the block and blob storage of my clusters?

The IAM role used by the broker (pod) is restricted to:

• Access only buckets that belong to their own cluster.
• Access only the customer-managed key that is mapped to their own cluster.
• Not have IAM permissions to attach or detach block storage directly, but it is handled using a Kubernetes cluster-wide process. The pod has access to only the attached block storage device (EBS volume). This prevents brokers (pods) from accessing the block and blob storage of any other clusters keeping them isolated.

Revoke Confluent access to a self-managed encryption cluster in AWS

Self-managed encryption keys used in Confluent Cloud with AWS KMS require access by Confluent for Confluent Cloud Kafka clusters to operate properly. You should only revoke access if you have a major security concern and need to completely remove Confluent access to your data.

Important

When you disable, or revoke, access to the encryption key for a cluster, the cluster is inaccessible only after you delete the cluster.

Before revoking access to the master key, your Confluent Cloud cluster must be deleted. This ensures that access to data is safely revoked.

To revoke or disable access by Confluent to your encryption key:

1. In your Confluent Cloud environment, stop all clients (producers and consumers) connected to your Confluent Cloud Kafka cluster.
2. Go the Confluent Cloud Console at https://confluent.cloud/login and delete your cluster.
3. Go to the AWS KMS console at console.aws.amazon.com/kms/home and disable access to the master key.

View the Dedicated cluster security settings

After you create a Dedicated cluster that uses a self-managed encryption key (and the cluster is provisioned), you can view the security settings. You can use these cluster security settings to verify the correct encryption key is used.

To view your dedicated AWS cluster security settings:

1. Select your Confluent Cloud cluster.
2. Click the Cluster settings tab and then Security.

Note that anyone authorized to view your AWS Dedicated clusters can view this data.