Encrypt Confluent Cloud Clusters using Self-Managed Keys – Amazon Web Services

When you create a Confluent Cloud Dedicated cluster on Amazon Web Services, you can optionally use self-managed encryption keys to protect data at rest, allowing only the appropriate entity or user can decrypt it. Also known as bring your own key (BYOK) encryption, self-managed keys provide you greater privacy and data integrity, which is frequently required by government, health, finance, and many other industries.

Protect the data at rest stored in your Confluent Cloud Dedicated clusters on Amazon Web Services using AWS Key Management Service (KMS) to create and manage encryption keys. Use of self-managed encryption keys for clusters in Confluent Cloud on Amazon Web Services includes the following operations and limitations:

  • Create Dedicated clusters that use self-managed encryption keys during cluster creation. You cannot switch between Automatic (default) and Self-managed encryption modes after the cluster has been provisioned.
  • Use AWS Key Management Service (KMS) to generate, use, rotate, and destroy your encryption keys.
  • Customer master keys (CMKs) are supported.
  • The encryption key used must match the region specified for the cluster.
  • Only symmetric keys are supported.
  • Importing key material is not supported.
  • Revoke encryption keys to prevent access to stored data.
  • Enable automatic key rotation using AWS KMS. Manual key rotation is not supported.

Create a Dedicated cluster with self-managed encryption

Warning

If you accidentally delete the master key, you will no longer be able to access your encrypted data. Neither Confluent nor AWS can regain access to your data.

To create an encrypted Confluent Cloud Dedicated cluster on Amazon Web Services that uses a self-managed encryption key:

  1. Navigate to the Clusters page for your environment and click Create cluster if you are creating the first cluster in your environment, or click Add cluster if other clusters exist.

  2. For 1. Select cluster type under Create cluster, select Dedicated and click Begin Configuration.

  3. For 2. Regions/zones under Create cluster, select AWS as the cloud service provider, select the Region and Availability, and then click Continue.

  4. For 3. Networking under Create cluster, select the networking type and click Continue.

  5. For 4 - Security under Create cluster, select Self-managed to manage your own encryption key using AWS Key Management Service. Additional steps appear.

    Note

    • Only symmetric keys are supported.
    • Importing key material is not supported.
    • The key must be for the region selected in 2. Regions/zone under Create cluster.

    Step 1: Enter the Amazon Resource Name (ARN) for your encryption key. To locate your ARN, log in to the AWS KMS Console and create or locate the ARN.

    Step 2: Copy the provided block of code exactly as-is (do not edit or modify) and paste it, appending the required permissions to the key policy of the ARN in your AWS key, and then click Continue.

    The code block gives your AWS key policy authorization to access your Confluent Cloud cluster and authorizes Confluent access to your KMS. For details, see the AWS KMS documentation.

    Note

    After you complete the cluster creation process, this cluster-key pairing is locked. You cannot change it for the lifetime of the cluster. You can still modify permissions related to the key, and also disable or delete it, as long as your AWS permissions allow for it.

  6. For 5. Review and launch under Create cluster. enter the Cluster name and click Launch Cluster.

Note

A successful validation results in the provisioning of your cluster. If the cluster configuration is invalid because the encryption key is not valid or not authorized for Confluent, then you will get an error message indicating so. Close the modal; any invalid fields will be highlighted in the original form. Reenter a valid value in the highlighted field.

Revoke Confluent access to self-managed clusters in AWS

Self-managed encryption keys used in Confluent Cloud with AWS KMS require access by Confluent for clusters to operate properly. When you revoke or disable access to the encryption keys, your clusters are inaccessible only after you delete the clusters.

To revoke or disable access by Confluent to your encryption key, notify Confluent support to remove your clusters.

Using the Confluent CLI to encrypt clusters using your own key

As mentioned above, you must use AWS as the provider, and specify the dedicated cluster type.

The following Confluent CLI example shows how to create an encrypted Confluent Cloud cluster using your customer-managed key. Any content in brackets (<>) must be customized for your environment.

confluent kafka cluster create sales092020 --cloud "aws" --region "<KMS-region>" --type "dedicated" --cku <CKU-value> --encryption-key "<AWS-ARN-ID>"

Copy and append these permissions to the existing "Statements" array field i
n the key policy of your ARN to authorize access for Confluent:

{
    "Sid" : "Allow Confluent account (123456789101) to use the key",
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : ["arn:aws:iam::123456789101:root"]
    },
    "Action" : [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ],
    "Resource" : "*"
  }, {
    "Sid" : "Allow Confluent account (123456789101) to attach persistent resources",
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : ["arn:aws:iam::123456789101:root"]
    },
    "Action" : [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ],
    "Resource" : "*"
}

Please confirm you've authorized the key for this account: 123456789101 (y/n):y

When you specify the --encryption-key option, you are prompted to update your AWS KMS policy.

For details, see:

View the Dedicated cluster security settings

You can view the Security settings for Dedicated clusters provisioned on AWS. In other words, if you used Automatic, Self-managed, or have an existing Dedicated cluster on AWS that you created prior to using the Self-managed encryption option, you can view the cluster security settings. The data in the cluster security settings is informational only and serves to identify the keys in use.

To view your dedicated AWS cluster security settings:

  1. Select your Confluent Cloud cluster.
  2. Click the Cluster settings tab and then Security.

Note that anyone authorized to view your AWS Dedicated clusters can view this data.

Create an AWS key policy

KMS keys belong to the AWS account in which they are created.

  • The IAM user who creates a KMS key is not automatically considered to be the key owner and does not automatically have permission to use or manage the KMS key they created. To get permission, you can specify a key policy that

    For details, see Permissions for creating KMS keys.

By default, the key policy grants every principal in an AWS account access to the AWS KMS customer-managed key (CMK). To restrict access to your encryption key, you can create your own AWS KMS key policy <https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html>`__ that specifies a permission set.

Important

  • You must add your permission set to the key policy while creating the KMS key to ensure that you can control access to the KMS key.
  • You can use aliases to control access to KMS keys and to follow the best practice of granting least privileged access.
  • When you add, delete, or update an alias for a KMS key, the permissions are

For details, see Key policies in AWS KMS.

Example of an AWS KMS key policy

The AWS KMS key policy consists of two distinct permission statements that grant access for compute and storage requirements. The first statement grants the specified Confluent AWS accounts permissions to describe the KMS encryption key, including the ability to list metadata information about the key and to perform encryption, decryption, re-encryption, and data-key generation.

In the example below, the first statement includes the list of Confluent AWS account IDs that are granted the aforementioned permissions.

Note

Because the current Confluent Cloud infrastructure continues to grow and scale, the number of accounts requires increases in correspondence to maintain scalability.

These two accounts in the statement host the Confluent Cloud compute infrastructure. Two accounts are required to access to the key in order to encrypt the block storage (disks) and to manage migrations and other operations. The other eight accounts in the list are storage accounts and require access to the KMS encryption key to perform encryption on the blob storage (AWS S3 buckets).

The second statement in the example below allows Confluent AWS accounts to manage (create, list, and revoke) grants for data storage. These grants are used internally by AWS to provide access for performing cryptographic operations to a given grantee principal. In this case, AWS internally uses this permission to grant access to service-linked-roles, such as aws:ec2-infrastructure, to persistently attach AWS EBS volumes (block storage) to the AWS EC2 instance where the Kafka broker (pod) runs.

{
  "Sid": "Allow Confluent account(s) (050879227952, 152535741197, 161406349951, 350927345273, 439989760662, 563559260525, 571452958320, 649317483566, 702463834586, 819787557123) to use the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::050879227952:root",
      "arn:aws:iam::563559260525:root"
      "arn:aws:iam::152535741197:root",
      "arn:aws:iam::161406349951:root",
      "arn:aws:iam::350927345273:root",
      "arn:aws:iam::439989760662:root",,
      "arn:aws:iam::571452958320:root",
      "arn:aws:iam::649317483566:root",
      "arn:aws:iam::702463834586:root",
      "arn:aws:iam::819787557123:root"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
},
{
  "Sid": "Allow Confluent account(s) (050879227952, 152535741197, 161406349951, 350927345273, 439989760662, 563559260525, 571452958320, 649317483566, 702463834586, 819787557123) to attach persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::050879227952:root",
      "arn:aws:iam::563559260525:root",
      "arn:aws:iam::152535741197:root",
      "arn:aws:iam::161406349951:root",
      "arn:aws:iam::350927345273:root",
      "arn:aws:iam::439989760662:root",
      "arn:aws:iam::571452958320:root",
      "arn:aws:iam::649317483566:root",
      "arn:aws:iam::702463834586:root",
      "arn:aws:iam::819787557123:root"
    ]
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}

For "Resource": "*" above, the wildcard character (*) indicates that the KMS key policy applies to the customer-managed key (CMK).

Frequently asked questions

Can Confluent accounts read a customer-managed key?
The AWS KMS does not allows Confluent to read a customer-managed key (CMK), regardless of permissions. The permission set describes or performs only the defined cryptographic operations on the CMK key using the AWS KMS interface.
Why are we using root principal?
Because the specific ARN used for the blob and block storage management is non-deterministic during the time of cluster creation.
Why are we listing N accounts instead of 1?
At any point in the life cycle, the brokers can move around between various AWS accounts running the platform. Hence, there is a need to list all the accounts to ensure there is a continuity of service during such migrations.
Who can decrypt the block and blob storage of my clusters?

The IAM role used by the broker (pod) is restricted to access only buckets that belong to their own cluster.

The IAM role used by the broker (pod) is restricted to access only the customer-managed key that is mapped to their own cluster.

The aforementioned IAM role does not have IAM permissions to attach or detach block storage directly, but it is handled using a Kubernetes cluster-wide process. The pod has access to only the attached block storage device (EBS volume).

This prevents brokers (pods) from accessing the block and blob storage of any other clusters keeping them isolated.