Encrypt Confluent Cloud Clusters using Self-Managed Keys – AWS

When you create a Confluent Cloud Dedicated cluster on Amazon Web Services, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user can decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Amazon Web Services using AWS Key Management Service (KMS) to create and manage encryption keys.

Use of self-managed encryption keys for clusters in Confluent Cloud on Amazon Web Services includes the following operations and limitations:

  • Create Dedicated clusters that use self-managed encryption keys during cluster creation. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
  • Use AWS Key Management Service (KMS) to generate, use, rotate, and destroy your encryption keys.
  • Customer master keys (CMKs) are supported.
  • The encryption key used must match the region specified for the cluster.
  • Only symmetric keys are supported.
  • Importing key material is not supported.
  • Revoke encryption keys to prevent access to stored data.
  • Enable automatic key rotation using AWS KMS. Manual key rotation is not supported.
  • The Confluent Terraform provider currently does not support self-managed encryption keys.

Create a Dedicated cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor AWS can regain access to your data.

To create an encrypted Confluent Cloud Dedicated cluster on Amazon Web Services that uses a self-managed encryption key:

  1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

  2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

  3. For 2. Regions/zones under Create cluster, select AWS as the cloud service provider, select the Region and Availability, and then click Continue.

  4. For 3. Networking under Create cluster, select the networking type and click Continue.

  5. For 4 - Security under Create cluster, select Self-managed to manage your own encryption key using AWS Key Management Service. Additional steps appear.

    Note

    • Only symmetric keys are supported.
    • Importing key material is not supported.
    • The key must be for the region selected in 2. Regions/zone under Create cluster.

    Step 1: Enter an Amazon resource name (key ID) for your encryption key. To locate your ARN, sign in to the AWS KMS Console and create or locate the ARN.

    Note

    The key ARN is a unique, fully qualified identifier of a customer-managed key (CMK) and cannot be changed. Each Confluent Cloud cluster requires a unique key.

    Step 2: Copy the provided key policy code block exactly as-is (do not edit or modify) into the Key policy of the ARN in your AWS KMS system to authorize access for your Confluent Cloud cluster. Make sure to paste the code block (that includes two new permission statements) below the existing statement and add the expected comma separator between the existing and new statements.

    The code block gives your AWS key policy authorization to access your Confluent Cloud cluster and authorizes Confluent access to your KMS. For details, see Create an AWS key policy below.

    Note

    After you complete the cluster creation process, this cluster-key pairing is locked. You cannot change it for the lifetime of the cluster. You can still modify permissions related to the key, and also disable or delete it, as long as your AWS permissions allow for it.

  6. For 5. Review and launch under Create cluster. Enter the Cluster name and click Launch Cluster.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Using the Confluent CLI to encrypt clusters using your own key

As mentioned above, you must use AWS as the provider, and specify the dedicated cluster type.

The following Confluent CLI example shows how to create an encrypted Confluent Cloud cluster using your customer-managed key. Any content in brackets (<>) must be customized for your environment.

confluent kafka cluster create sales092020 --cloud "aws" --region "<KMS-region>" --type "dedicated" --cku <CKU-value> --encryption-key "<AWS-ARN-ID>"

Copy and append these permissions to the existing "Statements" array field i
n the key policy of your ARN to authorize access for Confluent:

{
    "Sid" : "Allow Confluent account (123456789101) to use the key",
    "Effect" : "Allow",
    "Principal" : {
      "AWS": [
       "arn:aws:iam::050879227952:role/block_storage_manager",
       "arn:aws:iam::563559260525:role/block_storage_manager",
       "arn:aws:iam::219109013385:role/cc-kafka-2416cad8-400c-11ed-becb-5a0536b760f1"
      ]
    },
    "Action" : [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ],
    "Resource" : "*"
  }, {
    "Sid" : "Allow Confluent account (123456789101) to attach persistent resources",
    "Effect" : "Allow",
    "Principal" : {
      "AWS": [
        "arn:aws:iam::050879227952:role/block_storage_manager",
        "arn:aws:iam::563559260525:role/block_storage_manager",
        "arn:aws:iam::219109013385:role/cc-kafka-2416cad8-400c-11ed-becb-5a0536b760f1"
      ]
    },
    "Action" : [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ],
    "Resource" : "*"
}

Please confirm you've authorized the key for this account: 123456789101 (y/n):y

When you specify the --encryption-key option, you are prompted to update your AWS KMS policy.

For details, see:

Create an AWS key policy

KMS keys belong to the AWS account in which they are created.

  • The IAM user who creates a KMS key is not automatically considered to be the key owner and does not automatically have permission to use or manage the KMS key they created. To get permission, you must specify a key policy that includes permission statements that control access. For details, see Permissions for creating KMS keys.
  • By default, the key policy grants every principal in an AWS account access to the AWS KMS customer-managed key (CMK). To restrict access to your encryption key, you can create an AWS KMS key policy that specifies permissions.

Important

  • The permission sets must be added to the key policy when you create the KMS key to ensure that you can control access to the KMS key.
  • Aliases can be used to control access to KMS keys, following the best practice of granting least privileged access. For details, see Using aliases.

For details, see Key policies in AWS KMS.

Example of an AWS KMS key policy

Your AWS KMS key policy consists of two distinct permission statements that grant access for compute and storage requirements. The first permission statement grants three Confluent AWS accounts permissions required to describe the KMS encryption key, list metadata information about the key, perform encryption, decryption, re-encryption, and data-key generation.

In the example below, the first statement specifies only three Confluent AWS accounts that are granted the aforementioned permissions using the two roles for blob (tiered) and block (disk) storage.

Note

Because the current Confluent Cloud infrastructure continues to grow and scale, an increasing number of accounts is required to maintain scalability.

Two accounts in the first permission statement host the Confluent Cloud compute infrastructure. These two accounts are required to access to the key in order to encrypt the block storage (disks) and to manage migrations and other operations.

In the second permission statement, three Confluent AWS accounts are used to manage (create, list, and revoke) grants for data storage. These grants are used internally by AWS to provide access for performing cryptographic operations to a given grantee principal. In this example, AWS internally uses this permission to grant access to service-linked-roles, such as aws:ec2-infrastructure, to persistently attach AWS EBS volumes (block storage) to the AWS EC2 instance where the Kafka broker (pod) runs.

{
  "Sid": "Allow Confluent account(s) (050879227952, 563559260525, 219109013385) to use the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
       "arn:aws:iam::050879227952:role/block_storage_manager",
       "arn:aws:iam::563559260525:role/block_storage_manager",
       "arn:aws:iam::219109013385:role/cc-kafka-2416cad8-400c-11ed-becb-5a0536b760f1"
     ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
]
},
{
  "Sid": "Allow Confluent account(s) (050879227952, 563559260525, 219109013385) to attach persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::050879227952:role/block_storage_manager",
      "arn:aws:iam::563559260525:role/block_storage_manager",
      "arn:aws:iam::219109013385:role/cc-kafka-2416cad8-400c-11ed-becb-5a0536b760f1"
    ]
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}

For "Resource": "*" above, the wildcard character (*) indicates that the KMS key policy applies to the customer-managed key (CMK).

Frequently asked questions

Can Confluent accounts read a customer-managed key?
The AWS KMS does not allows Confluent to read a customer-managed key (CMK), regardless of permissions. The permissions you add describe or perform only the defined cryptographic operations on the CMK key using the AWS KMS interface.
Why are we listing N accounts instead of 1?
At any point in the life cycle, the brokers can move around between various AWS accounts running the platform. Hence, there is a need to list all the accounts to ensure there is a continuity of service during such migrations.
Who can decrypt the block and blob storage of my clusters?

The IAM role used by the broker (pod) is restricted to:

  • Access only buckets that belong to their own cluster.
  • Access only the customer-managed key that is mapped to their own cluster.
  • Not have IAM permissions to attach or detach block storage directly, but it is handled using a Kubernetes cluster-wide process. The pod has access to only the attached block storage device (EBS volume). This prevents brokers (pods) from accessing the block and blob storage of any other clusters keeping them isolated.

Revoke Confluent access to a self-managed encryption cluster in AWS

Self-managed encryption keys used in Confluent Cloud with AWS KMS require access by Confluent for Confluent Cloud Kafka clusters to operate properly. You should only revoke access if you have a major security concern and need to completely remove Confluent access to your data.

Important

When you disable, or revoke, access to the encryption key for a cluster, the cluster is inaccessible only after you delete the cluster.

Before revoking access to the master key, your Confluent Cloud cluster must be deleted. This ensures that access to data is safely revoked.

To revoke or disable access by Confluent to your encryption key:

  1. In your Confluent Cloud environment, stop all clients (producers and consumers) connected to your Confluent Cloud Kafka cluster.
  2. Go the Confluent Cloud Console at https://confluent.cloud/login and delete your cluster.
  3. Go to the AWS KMS console at console.aws.amazon.com/kms/home and disable access to the master key.

View the Dedicated cluster security settings

You can view the Security settings for Dedicated clusters provisioned on AWS. In other words, if you used Automatic, Self-managed, or have an existing Dedicated cluster on AWS that you created prior to using the Self-managed encryption option, you can view the cluster security settings. The data in the cluster security settings is informational only and serves to identify the keys in use.

To view your dedicated AWS cluster security settings:

  1. Select your Confluent Cloud cluster.
  2. Click the Cluster settings tab and then Security.

Note that anyone authorized to view your AWS Dedicated clusters can view this data.