<a id="byok-encrypted-clusters-aws"></a>

# Use Self-Managed Encryption Keys in Confluent Cloud on AWS

Protect the data at rest stored in your Confluent Cloud Dedicated, Enterprise, and AWS Freight Kafka clusters
on AWS using [AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html)
to create and manage encryption keys,

<a id="byok-requirements-aws"></a>

## Requirements

Self-managed encryption keys can be used with supported Kafka clusters on
Confluent Cloud created using the Self-managed encryption mode. To use self-managed
encryption keys on AWS for supported Kafka cluster types, follow these requirements:

<a id="byok-requirements-aws-key-management"></a>

### Key creation and management

Required RBAC role: [OrganizationAdmin](../../access-control/rbac/predefined-rbac-roles.md#organizationadmin-role) or [EnvironmentAdmin](../../access-control/rbac/predefined-rbac-roles.md#environmentadmin-role).

- Create a Dedicated, Enterprise, or Freight Kafka cluster on AWS using the “Self-managed”
  encryption mode. After provisioning your Dedicated, Enterprise, or Freight cluster, you
  cannot switch modes between Automatic (default) and Self-managed.
- Use [AWS Key Management Service (KMS)](https://console.aws.amazon.com/kms/home)
  to generate, use, rotate, and destroy [customer master keys (CMKs)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html).
- Only symmetric, software-protected keys are supported.
- Using a [KMS key with imported key material](https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys-conceptual.html) is supported.
- [AWS KMS External Key Store (XKS)](https://docs.aws.amazon.com/kms/latest/developerguide/keystore-external.html) is supported.
  - Your External Key Manager (EKM) configuration (including any proxies),
    AWS External Key Store (XKS), AWS Key Management Service (KMS),
    and Kafka cluster must all be located in the same AWS region. Cross-region
    configurations are not supported.
  - Each XKS instance must be dedicated exclusively to Confluent. Using XKS keys
    for other services impacts Confluent’s ability to guarantee SLA performance
    due to quota conflicts and performance factors outside of Confluent’s control.
  - Each XKS instance supports a maximum of four Confluent encryption keys. If you
    need more than four keys, create additional XKS instances.
- Key rotation:
  - Automatic key rotation is available using the AWS KMS console.
  - On-demand key rotation is supported for KMS keys created with imported key material.
  - WARNING: There is a hard limit of 10 on-demand rotations imposed by AWS. While keys generated in KMS can still be rotated via automatic key rotation after reaching the on-demand key rotation limit, imported key material KMS keys cannot be rotated via automatic rotation. If the on-demand rotation limit is reached, no further rotation of any kind is possible.
  - WARNING: Deleting old keys is a permanent operation that cannot
    be undone and results in data loss.
  - Key rotations impact both Kafka data and any [Tableflow](../../../topics/tableflow/overview.md#cloud-tableflow)
    data stored in Confluent Managed Storage.
- Use a unique encryption key for each active cluster.
- If you delete a cluster, you can delete the self-managed encryption key immediately.
  However, you can’t reuse a deleted key to create another cluster for five days.
  As a security best practice, don’t reuse encryption keys for production clusters.

### FIPS 140-2 certification

- Self-managed keys created after March 2023 are FIPS 140-2 Level 3
  certified except for AWS regions only offering FIPS 140-2 Level 2
  certification. For details, see [AWS KMS FAQs](https://aws.amazon.com/kms/faqs/).
- You can also use AWS KMS customer managed keys (CMKs) backed by AWS-managed
  hardware security modules (HSMs) for FIPS 140-2 Level 3 certification.
- For more information, see [FIPS 140-2](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html).

### Tableflow integration with self-managed encryption keys

When you use [Tableflow](../../../topics/tableflow/overview.md#cloud-tableflow) with Confluent Managed Storage,
the BYOK encryption key used for your Kafka cluster is automatically reused for
Tableflow managed tables. This ensures consistent encryption across your Kafka
data and Tableflow tables.

#### IMPORTANT
- The encryption key cannot be changed separately for Tableflow Confluent
  Managed Storage.
- Key rotations impact both Kafka data and Tableflow data in Confluent
  Managed Storage.
- If you revoke access to the encryption key, both Kafka and Tableflow data
  in Confluent Managed Storage become inaccessible.

<a id="create-encryption-key-aws"></a>

## Create a self-managed encryption key

A self-managed encryption key can be created in two ways:

1. **From the global Encryption Keys page**
2. **During cluster creation**

<a id="create-encryption-key-global-aws"></a>

### Method 1: From the global Encryption Keys page

To create a self-managed encryption key from the global Encryption Keys page:

1. In the Confluent Cloud Console, click the hamburger menu in the upper right corner.
2. Select **Encryption keys** from the menu.
3. Click **Add key**.
4. **Step 1: Choose a cloud provider** - Select **AWS**.
5. **Step 2: Enter key details** - Provide the following information:
   - **Key Alias** (optional): A human-friendly name to identify the key
   - **Amazon Resource Name (ARN)**: The ARN of the key to use for encryption
6. Click **Register key**. The key is created and will appear in the
   encryption keys table.
7. **Step 3: Configure permissions and policy** - Follow the AWS-specific
   instructions to configure RBAC permissions.
8. Click **Finish**. The key enters an initializing state which runs
   asynchronously and may take up to 5 minutes.

The key will show a status of “Initializing” until validation completes. After
the key is validated, it can be used when creating clusters.

<a id="create-dedicated-cluster-with-encryption-aws"></a>

### Method 2: Create a Kafka cluster with self-managed encryption

#### WARNING
If you accidentally delete the master key, you are no longer able to access
your encrypted data. Neither Confluent nor AWS can regain access to your
data.

### Confluent Cloud Console

To create an encrypted Confluent Cloud cluster on AWS that uses a
self-managed encryption key:

1. Navigate to the **Clusters** page for your environment and click
   **Create cluster** if you are creating the first cluster in your environment,
   or click **Add cluster** if other clusters exist.
2. For **Select cluster type** under **Create cluster**, select a supported Kafka
   cluster type, and click **Begin Configuration**.
3. For **Regions/zones** under **Create cluster**, select **AWS** as the cloud
   service provider, select the **Region** and **Availability**, and then click
   **Continue**.
4. For **Networking** under **Create cluster**, select the [networking](../../../networking/overview.md#cloud-networking)
   type and click **Continue**.
5. For **Security** under **Create cluster**, select **Self managed** to
   manage your own encryption key using AWS Key Management Service.

   You can either:
   - **Select an existing key**: Choose from the dropdown list of previously
     created and validated encryption keys from the global Encryption Keys page.
   - **Add a new key**: Create a new encryption key during cluster creation by
     providing the Amazon Resource Name (ARN).

   #### IMPORTANT
   - Only symmetric keys are supported.
   - Key validation during cluster creation is asynchronous and may take
     a few minutes.
   - The key must be in the same region as your cluster.
6. If adding a new key during cluster creation:
   * Go to the [AWS KMS (requires sign-in)](https://console.aws.amazon.com/kms)
     console and create a new key. Copy the key ARN and then paste it into
     the **Amazon Resource Name (key ID)** field.
   * If selecting an existing key, choose your key from the dropdown list.

     The key ARN is a unique, fully-qualified identifier of a customer-managed
     key (CMK) and cannot be changed. Each Confluent Cloud cluster requires a unique key.

   Relevant AWS documentation:
   - [Finding the key ID and key ARN](https://docs.aws.amazon.com/kms/latest/developerguide/find-cmk-id-arn.html)
   - [Creating keys](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html)
7. From the Confluent Cloud Console, copy the provided permission statements exactly as-is
   (do not edit or modify) into the existing **Key policy** of the ARN in your AWS KMS.
   The code block gives your AWS key policy authorization to access your
   Confluent Cloud cluster and authorizes Confluent access to your KMS. For details on
   how to create and manage your key policy, see
   [Create an AWS key policy](#create-aws-kms-key-policy).

   #### IMPORTANT
   Make sure to append the code block (that includes two new permission statements) immediately
   after the existing permission statement and remember to add the expected comma separator
   between the existing statement and the new statements.

   After you complete the cluster creation process, this cluster-key pairing
   is locked. You cannot change it for the lifetime of the cluster. You can
   still modify permissions related to the key, and also disable or delete it,
   as long as your AWS permissions allow for it.

   The updated key policy should look similar to the following example:
   ```json
   {
     "Sid": "Allow Confluent account(s) (152535741197) to use the key",
     "Effect": "Allow",
     "Principal": {
       "AWS": [
         "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
       ]
     },
     "Action": [
       "kms:Encrypt",
       "kms:Decrypt",
       "kms:ReEncrypt*",
       "kms:GenerateDataKey*",
       "kms:DescribeKey"
     ],
     "Resource": "*"
   },
   {
     "Sid": "Allow Confluent account(s) (152535741197) to attach persistent resources",
     "Effect": "Allow",
     "Principal": {
       "AWS": [
         "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
       ]
     },
     "Action": [
       "kms:CreateGrant",
       "kms:ListGrants",
       "kms:RevokeGrant"
     ],
     "Resource": "*"
   }
   ```
8. Click **Continue**.
9. On the **Review and launch** page, enter a meaningful name in **Cluster name**
   and click **Launch Cluster**.

   A successful validation results in your cluster being provisioned. If the
   cluster configuration is invalid because the encryption key is not valid or
   not authorized for Confluent, then you will get an error message indicating
   so. Close the modal; any invalid fields will be highlighted in the original
   form. Reenter a valid value in the highlighted field.

   If the key is not valid or not authorized for Confluent, you can revisit
   the policy and authorization instructions from the global Encryption Keys
   page. Navigate to the Encryption Keys page, find your key, and click
   **View key details** to access the permissions and policy configuration
   instructions.

### Confluent CLI

1. Create a key in AWS Management Console or using the AWS CLI.
2. Create a key resource in Confluent Cloud using the following Confluent CLI command,
   replacing `<AWS-ARN-ID>` with your AWS KMS key ARN.
   ```shell
   confluent byok create <AWS-ARN-ID>
   ```

   The command returns the CCK ID (Confluent Cloud key ID) and permission statements
   to append to the AWS KMS key policy.

   For example:
   ```shell
   +------------+------------------------------------------------------------------------------+
   | ID         | cck-stgc7gd47                                                                |
   | Key        | arn:aws:kms:us-east-1:037803949979:key/45aade25-6395-4920-86a8-f52dbeab4639  |
   | Roles      | arn:aws:iam::811656834269:role/cc-kafka-03f3b3dd-924f-11ef-87a7-82d574be0321 |
   | Cloud      | AWS                                                                          |
   | State      | AVAILABLE                                                                    |
   | Created At | 2024-10-24 21:29:14.319331884                                                |
   |            | +0000 UTC                                                                    |
   +------------+------------------------------------------------------------------------------+
   ```

   Copy and append these permissions into the key policy “Statements” field of the ARN in your AWS key management system to authorize access for your Confluent Cloud cluster.
   ```json
   {
     "Sid" : "Allow Confluent accounts to use the key",
     "Effect" : "Allow",
     "Principal" : {
       "AWS" : [
         "arn:aws:iam::811656834269:role/cc-kafka-03f3b3dd-924f-11ef-87a7-82d574be0321"
       ]
     },
     "Action" : [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ],
     "Resource" : ""
   }, {
     "Sid" : "Allow Confluent accounts to attach persistent resources",
     "Effect" : "Allow",
     "Principal" : {
       "AWS" : [
         "arn:aws:iam::811656834269:role/cc-kafka-03f3b3dd-924f-11ef-87a7-82d574be0321"
       ]
     },
     "Action" : [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ],
     "Resource" : ""
   }
   ```

   Next, append the permissions to the existing key policy of the ARN in
   your AWS KMS using AWS console or the following AWS CLI command.
   Replace the `<key-ID>` with the key ID of your AWS KMS key and replace `<policy-file>`
   with the path to the file that contains the updated key policy:
   ```shell
   aws kms put-key-policy --key-id <key-ID> --policy <policy-file>
   ```

   For details, see [Updating key policies in AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html)
   and [put-key-policy command in AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/kms/put-key-policy.html).
3. Create a Confluent Cloud Dedicated cluster using your customer-managed key,
   replacing the placeholders with your values.
   ```shell
   confluent kafka cluster create <cluster-name> \
     --cloud "aws" \
     --region "<KMS-region>" \
     --type "dedicated|enterprise|freight" \
     --cku <CKU-value> \
     --byok <CCK-ID>
   ```

   You can find the CCK ID in the output of the `confluent byok list` command.

The following Confluent CLI example shows how to create an encrypted Confluent Cloud
Kafka cluster using your customer-managed key.

```shell
confluent kafka cluster create <cluster-name> \
  --cloud "aws" \
  --region "<KMS-region>" \
  --type "<dedicated|enterprise|freight>" \
  --cku <CKU-value> \
  --encryption-key "<AWS-ARN-ID>"
```

To authorize access for Confluent, copy and append the permission statements
to the existing “Statements” array field in the key policy of your ARN.

```json
{
  "Sid": "Allow Confluent account(s) (152535741197) to use the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
},
{
  "Sid": "Allow Confluent account(s) (152535741197) to attach persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}
```

After confirming that you’re authorizing this key, the cluster is created.

When you specify the `--encryption-key` option, you are prompted to
update your AWS KMS policy.

For details, see:

- [confluent kafka cluster create](https://docs.confluent.io/confluent-cli/current/command-reference/kafka/cluster/confluent_kafka_cluster_create.html)
- [Key policies in AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html)

<a id="create-aws-kms-key-policy"></a>

## Create an AWS key policy

KMS keys belong to the AWS account in which they are created.

- The IAM user who creates a KMS key is not automatically considered to be
  the key owner and does not automatically have permission to use or manage
  the KMS key they created. To get permission, you must specify a key policy
  that includes permission statements that control access. For details, see
  [Permissions for creating KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html#create-key-permissions).

By default, the key policy grants every principal in an AWS account access to
the AWS KMS customer-managed key (CMK). To restrict access to your encryption key,
you can create your own [AWS KMS key policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html)
that specifies a permission set.

- The permission sets must be added to the key policy when you create the KMS key
  to ensure that you can control access to the KMS key.
- Aliases can be used to control access to KMS keys, following the best practice
  of [granting least privileged access](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
  For details, see [Using aliases](https://docs.aws.amazon.com/kms/latest/developerguide/kms-alias.html).

For details, see [Key policies in AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html).

<a id="aws-kms-key-policy-example"></a>

### Example of an AWS KMS key policy

Your AWS KMS key policy consists of two distinct permission statements that grant
access for compute and storage requirements.

The **first permission statement** grants Confluent AWS accounts the permissions
required to describe the KMS encryption key, list metadata information about the
key, perform encryption, decryption, re-encryption, and data-key generation.
In the example below, the first statement specifies the required Confluent AWS
accounts that are granted these permissions using the two roles for blob (tiered)
and block (disk) storage. As the current Confluent Cloud infrastructure continues to grow
and scale, an increasing number of accounts is required to maintain scalability.

The **second permission statement** grants Confluent AWS accounts the permissions
required to manage (create, list, and revoke) grants for data storage. These
grants are internally used to provide access for performing cryptographic
operations to the specified grantee principal. In this case, the permission statement
creates a grant for the role that manages block storage, which in turn creates the
grant for the AWS internal roles (for example `aws:ec2-infrastructure`) to
persistently attach AWS EBS volumes (block storage) to the AWS EC2 instance
where the Kafka broker (pod) runs.

```json
{
  "Sid": "Allow Confluent account(s) (152535741197) to use the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
},
{
  "Sid": "Allow Confluent account(s) (152535741197) to attach persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::152535741197:role/cc-kafka-d45e6381-b878-11ed-bdff-028e28f108bd"
    ]
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}
```

For `"Resource": "*"` above, the wildcard character (`*`) indicates that
the KMS key policy applies to the customer-managed key (CMK).

<a id="update-aws-kms-key-policy"></a>

## Update an AWS KMS key policy

Follow these steps to safely update your AWS KMS key policy without
disrupting your Confluent Cloud cluster operations.

Policy update recommendations
: * Add new permissions rather than replacing existing ones.
  * Contact Confluent Support first to get the correct new ARNs before updating.
  * Add new principals to existing statements rather than creating separate statements.
  * Test thoroughly in non-production environments before applying to production keys.

Prerequisites
: * Administrative access to your AWS account and KMS service.
  * Current working backup of your key policy.
  * Planned maintenance window for policy updates.
  * Understanding of the required Confluent permissions.

Procedure
: 1. Create a backup of your current working key policy. You can retrieve
     the current policy using the AWS CLI:
     ```shell
     aws kms get-key-policy \
       --key-id <your-key-id> \
       --policy-name default \
       --output text > key-policy-backup-$(date +%Y-%m-%d).json
     ```
  <br/>
     Alternatively, you can use the AWS KMS console to view and copy the
     policy. For detailed steps, see [Viewing a key policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-viewing.html)
     in the AWS documentation.
  2. Validate the policy using AWS IAM Access Analyzer:
     ```shell
     aws accessanalyzer validate-policy \
       --policy-document file://new-policy.json \
       --policy-type RESOURCE_POLICY
     ```
  <br/>
     Review the findings and fix any issues with severity `ERROR` before
     proceeding. For more details, see [Validate policies with IAM Access
     Analyzer](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.html)
     and the [validate-policy CLI reference](https://docs.aws.amazon.com/cli/latest/reference/accessanalyzer/validate-policy.html).
  3. During your planned maintenance window, apply the policy update:
  <br/>
     ### Using the AWS CLI (recommended)
  <br/>
     Run this command in the AWS CLI:
     ```shell
     aws kms put-key-policy \
       --key-id <your-key-id> \
       --policy-name default \
       --policy file://updated-policy.json
     ```
  <br/>
     ### Using the AWS Console
  <br/>
     To update the policy using the AWS KMS console, follow
     [these steps](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html).
     Ensure you don’t remove any Confluent-required permissions during the update.
  4. Verify your cluster is still operational.
     * Check cluster status in the Confluent Cloud Console.
     * Verify producers and consumers are still functioning.
     * Monitor for any error messages or alerts.
     * Check AWS CloudTrail events for any access denied errors related to your KMS key:
       ```shell
       # For macOS (BSD date):
       aws cloudtrail lookup-events \
         --lookup-attributes AttributeKey=EventSource,AttributeValue=kms.amazonaws.com \
         --start-time "$(date -v-10M +%Y-%m-%dT%H:%M:%S)" \
         --end-time "$(date +%Y-%m-%dT%H:%M:%S)"
  <br/>
       # For Linux (GNU date):
       aws cloudtrail lookup-events \
         --lookup-attributes AttributeKey=EventSource,AttributeValue=kms.amazonaws.com \
         --start-time "$(date --date='10 minutes ago' +%Y-%m-%dT%H:%M:%S)" \
         --end-time "$(date +%Y-%m-%dT%H:%M:%S)"
       ```
  <br/>
       For more information, see [Logging AWS KMS API calls with AWS
       CloudTrail](https://docs.aws.amazon.com/kms/latest/developerguide/logging-using-cloudtrail.html)
       and [lookup-events CLI reference](https://docs.aws.amazon.com/cli/latest/reference/cloudtrail/lookup-events.html).
  5. Monitor your cluster for at least 30 minutes after the policy update:
     * Watch cluster metrics and health indicators.
     * Check for any encryption-related errors.
     * Verify that automatic operations (like scaling) continue to work.

### Troubleshoot AWS key policy updates

Follow these steps if you encounter problems after updating the key policy.

* Restore the backup policy:
  ```shell
  aws kms put-key-policy \
    --key-id <your-key-id> \
    --policy-name default \
    --policy file://key-policy-backup-YYYY-MM-DD.json
  ```
* Monitor cluster recovery for up to 30 minutes.
* Contact Confluent Support if the cluster doesn’t recover after restoring the original policy.

## Tableflow integration with self-managed encryption keys

Tableflow supports self-managed encryption keys (BYOK) for enhanced security
and compliance.

#### IMPORTANT
BYOK for Tableflow is currently only supported on AWS Dedicated clusters.

For complete information about using BYOK with Tableflow,
including setup workflows, validation procedures, and troubleshooting, see
[Use self-managed encryption keys with Tableflow](tableflow-byok.md#tableflow-byok-integration).