<a id="use-byok-api"></a>

# Use the BYOK API with Self-Managed Keys on Confluent Cloud

Required RBAC role: [OrganizationAdmin](../../access-control/rbac/predefined-rbac-roles.md#organizationadmin-role) or [EnvironmentAdmin](../../access-control/rbac/predefined-rbac-roles.md#environmentadmin-role).

You can use the Key Management API for Bring Your Own Key (BYOK) to upload and
retrieve self-managed keys on Dedicated, Enterprise, and Freight Kafka clusters in Confluent Cloud. Key objects
represent customer managed keys on Dedicated, Enterprise, and Freight Kafka clusters. Keys are used to
protect data at rest stored in your clusters on AWS and Azure.

In addition to managing these self-managed keys with the Confluent Cloud Console,
you can use a collection of REST APIs to manage keys for your Confluent Cloud
environments.

You can use the [Bring Your Own Key (BYOK) Management API](https://docs.confluent.io/cloud/current/ccloud/keys-byok-v-1/)
to perform the following operations on Key objects:

- [Create a BYOK Object](#byok-key-create-api)
- [Read a BYOK Key](#byok-key-read-api)
- [List BYOK objects](#byok-key-list-api)
- [Delete a BYOK object](#byok-key-delete-api)

#### IMPORTANT
If your Confluent Cloud cluster was provisioned on AWS before August 11, 2023, it
is a Pre-BYOK-API-V1 cluster and you cannot use the BYOK V1 API to manage its
self-managed encryption keys. For details, see
[Pre-BYOK-API-V1 Self-Managed encryption keys](#legacy-byok).

<a id="byok-api-basics"></a>

## Base URL

The base URL for a BYOK API request is

```bash
https://confluent.cloud/api
```

## Object model

All Confluent objects share the following set of common properties:

* `api_version`: Indicates the API version for the object. The version for the BYOK API is `byok/v1`.
* `kind`: Indicates what kind of object it is.
* `id`: An identifier for the object.

<a id="byok-cluster-authentication"></a>

## Authentication

Each request to the server must include an `Authorization: Basic {key}` header that contains
a Confluent Cloud API key. For more details on creating or accessing your Confluent Cloud API keys, see [Manage API Keys in Confluent Cloud](../../authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.md#cloud-cloud-api-keys).
API requests without a correct authentication header will fail. An API key grants access to a
single Confluent cluster.

These keys grant API access to many resources, and should not be shared publicly to GitHub, in client-side code,
and so forth.

HTTP Basic authorization requires the key to be colon-separated and base64-encoded.

For example, if your API Key is `abcp7DEFGH123456789` and the corresponding Secret
is `XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO`, generate a base64-encoded header with the following command on
MacOS or Linux:

```bash
echo -n "abcp7DEFGH123456789:XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO" | base64
```

The resulting authorization header will be:

```bash
Authorization: Basic QUJDREVGR0gxMjM0NTY3ODk6WE5DSVc5M0kyTDFTUVBKU0o4MjNLMUxTOTAyS0xERk1DWlBXRU8=
```

Alternatively, if testing using Postman, specify Basic Auth and provide the API Key as the Username and the API Secret
as the password and Postman will provide the encoding.

## Test the BYOK API with curl

The following sections provide examples of using [curl](https://curl.haxx.se/)
commands for HTTP GET and HTTP POST requests based on the BYOK API. Before using the curl
commands, you must sign in to Confluent Cloud. For more information about the Confluent CLI,
see [Confluent Cloud CLI Reference](https://docs.confluent.io/confluent-cli/current/command-reference-index.html).

For example:

1. Sign in to Confluent Cloud using the Confluent CLI.
   ```shell
   confluent login --url https://confluent.cloud
   ```
2. Provide username and password at prompts.
3. Create an API key and API secret.
   ```shell
   confluent api-key create --resource cloud
   ```
4. Store the API key and API secret in environment variables.
   ```shell
   export CCLOUD_KEY=<generated-API-key>
   ```

   ```shell
   export CCLOUD_SECRET=<generated-API-secret>
   ```
5. Store the URL for Confluent Cloud in a variable:
   ```shell
   export CCLOUD_URL=https://confluent.cloud/api/
   ```
6. Run the following API request to list byok objects:
   ```bash
   curl -u $CCLOUD_KEY:$CCLOUD_SECRET "$CCLOUD_URL/byok/v1/keys"
   ```

   The output should like the example response displayed below for [List BYOK objects](#byok-key-list-api).

## Request examples

<a id="byok-key-list-api"></a>

### List BYOK objects

API reference: [List of BYOK objects](https://docs.confluent.io/cloud/current/ccloud/list-byok-v-1-keys/)

#### Request example

The following examples show how to retrieve a sorted, filtered, paginated list of
all Kafka clusters.

```json
GET /byok/v1/keys
```

Your request can include the following query parameters:

| Parameter    | Required?   | Description                                                                                             |
|--------------|-------------|---------------------------------------------------------------------------------------------------------|
| `provider`   | No          | Filter the results by exact match for provider.                                                         |
| `state`      | No          | Filter the results by exact match for state.                                                            |
| `page_size`  | No          | Specifies an integer page size for the returned payload. Maximum value is `100`. Default value is `10`. |
| `page_token` | No          | Contains the opaque pagination token string.                                                            |

#### Response example

A successful request returns an HTTP `200 OK` response and a JSON payload that
includes a list of BYOK objects. The response describes each object, including
the following information:

* The associated cloud-specific key details.
* The provider for the BYOK object.
* The object status (`AVAILABLE, IN_USE`).

```json
{
  "api_version": "byok/v1",
  "kind": "KeyList",
  "metadata": {
    "first": "https://api.confluent.cloud/byok/v1/keys",
    "last": "https://api.confluent.cloud/byok/v1/keys?page_token=bcAOehAY8F16YD84Z1wT",
    "prev": "https://api.confluent.cloud/byok/v1/keys?page_token=YIXRY97wWYmwzrax4dld",
    "next": "https://api.confluent.cloud/byok/v1/keys?page_token=UvmDWOB1iwfAIBPj6EYb",
    "total_size": 123
  },
  "data": [
    {
      "api_version": "byok/v1",
      "kind": "Key",
      "id": "dlz-f3a90de",
      "metadata": {
        "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
        "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
        "created_at": "2006-01-02T15:04:05-07:00",
        "updated_at": "2006-01-02T15:04:05-07:00",
        "deleted_at": "2006-01-02T15:04:05-07:00"
      },
      "key": {
        "key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
        "roles": [
          "arn:aws:iam::123456789876:role/block_storage_manager",
          "arn:aws:iam::987654321234:role/cc-kafka-1111aaaa-11aa-11aa-11aa-111111aaaaaa"
        ],
        "kind": "AwsKey"
      },
      "provider": "AWS",
      "state": "IN_USE"
    }
  ]
}
```

<a id="byok-key-create-api"></a>

### Create a BYOK Object

> API reference: [Create a BYOK object](https://docs.confluent.io/cloud/current/ccloud/create-byok-v-1-key/)

To register a BYOK object, send an HTTP POST request call to the key endpoint.

Make sure you include the authorization header with your API key and API secret.

```json
POST /byok/v1/keys
```

#### Request

When you make the request, include a JSON payload that contains a `key` object
showing the cloud-specific key details. The content of the `key` object varies
for each cloud service provider.

#### Request specification for AWS

| Parameter   | Required   | Description                                        |
|-------------|------------|----------------------------------------------------|
| `key_arn`   | Required   | The Amazon Resource Name (ARN) of the AWS KMS key. |
| `kind`      | Required   | BYOK kind type (AwsKey).                           |

#### Request example

```json
{
  "key": {
    "key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
    "kind": "AwsKey"
  }
}
```

#### Response example

A successful request returns an HTTP `200 OK` response and a JSON payload that contains
a list of BYOK objects. The response describes each object, including the
following information:

* The associated cloud-specific key details.
* The provider for the BYOK object.
* The object status (`AVAILABLE, IN_USE`).

```json
{
  "api_version": "byok/v1",
  "kind": "Key",
  "id": "dlz-f3a90de",
  "metadata": {
    "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
    "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
    "created_at": "2006-01-02T15:04:05-07:00",
    "updated_at": "2006-01-02T15:04:05-07:00",
    "deleted_at": "2006-01-02T15:04:05-07:00"
  },
  "key": {
    "key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
    "roles": [
      "arn:aws:iam::123456789876:role/block_storage_manager",
      "arn:aws:iam::987654321234:role/cc-kafka-1111aaaa-11aa-11aa-11aa-111111aaaaaa"
    ],
    "kind": "AwsKey"
  },
  "provider": "AWS",
  "state": "IN_USE"
}
```

#### Request specification for Azure

| Parameter      | Required?   | Description                                                                     |
|----------------|-------------|---------------------------------------------------------------------------------|
| `key_id`       | Yes         | The unique Key Object Identifier URL without version of an Azure Key Vault key. |
| `key_vault_id` | Yes         | The Key Vault ID containing the key.                                            |
| `tenant_id`    | Yes         | Tenant ID (uuid) hosting the Key Vault containing the key                       |
| `kind`         | Yes         | BYOK kind type (AzureKey).                                                      |

#### Request example

```json
{
  "key": {
    "key_id": "https://vault-name.vault.azure.net/keys/key-name",
    "key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "kind": "AzureKey"
  }
}
```

#### Response example

A successful call returns an HTTP `200 OK` and a JSON payload that contains a list of BYOK objects.
The response describes each object including the following information:

- The associated cloud-specific key details.
- The provider for the BYOK object.
- The object status (AVAILABLE, IN_USE).

```json
{
  "api_version": "byok/v1",
  "kind": "Key",
  "id": "dlz-f3a90de",
  "metadata": {
    "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
    "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
    "created_at": "2006-01-02T15:04:05-07:00",
    "updated_at": "2006-01-02T15:04:05-07:00",
    "deleted_at": "2006-01-02T15:04:05-07:00"
  },
  "key": {
    "key_id": "https://vault-name.vault.azure.net/keys/key-name",
    "key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
    "application_id": "00000000-0000-0000-0000-000000000000",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "kind": "AzureKey"
  },
  "provider": "Azure",
  "state": "IN_USE"
}
```

<a id="byok-key-read-api"></a>

### Read a BYOK Key

API reference: [Read a BYOK object](https://docs.confluent.io/cloud/current/ccloud/get-byok-v-1-key/)

To read a BYOK object, make a GET call to the keys endpoint to retrieve details about a
specified key. You must specify the id in the request.
This call does not require a payload. Make sure you include the authorization header with your API key and API secret.

#### Request example

```json
GET /byok/v1/keys/{id}
```

Your request must include the following parameters:

| Parameter   | Required?   | Description                        |
|-------------|-------------|------------------------------------|
| `id`        | No          | The unique identifier for the key. |

#### Response example

A successful request returns an HTTP `200 OK` response and a JSON payload
that contains a list of BYOK objects. The response describes each object,
including the following information:

* The associated cloud-specific key details.
* The provider for the BYOK object.
* The object status (`AVAILABLE, IN_USE`).

```json
{
  "api_version": "byok/v1",
  "kind": "Key",
  "id": "dlz-f3a90de",
  "metadata": {
    "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
    "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
    "created_at": "2006-01-02T15:04:05-07:00",
    "updated_at": "2006-01-02T15:04:05-07:00",
    "deleted_at": "2006-01-02T15:04:05-07:00"
  },
  "key": {
    "key_id": "https://vault-name.vault.azure.net/keys/key-name",
    "key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
    "application_id": "00000000-0000-0000-0000-000000000000",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "kind": "AzureKey"
  },
  "provider": "Azure",
  "state": "IN_USE"
}
```

<a id="byok-key-delete-api"></a>

### Delete a BYOK object

API reference: [Delete a Cluster](https://docs.confluent.io/cloud/current/ccloud/get-byok-v-1-key/)

You can delete an encrypted cluster with a DELETE request to the keys
endpoint. You specify `id` in the request, and the call does not
require a payload. Ensure that you include the authorization header
with your API key and API secret.

#### IMPORTANT
After the encryption key (BYOK key object) is deleted, it cannot be restored.
For example, if you delete the BYOK key object in Confluent Cloud on az| and
plan to reuse the key for testing, you must run the Confluent CLI commands again
to create a new BYOK key object.

#### Request example

```json
DELETE /byok/v1/keys/{id}
```

| Parameter   | Required?   | Description                        |
|-------------|-------------|------------------------------------|
| `id`        | Yes         | The unique identifier for the key. |

#### Response example

Success returns an HTTP `204 No Content` response.

If you delete a Kafka cluster and try to delete the BYOK key object associated with it,
this operation fails until the encryption key is available for reuse (after five days).
For details on the types of responses you can expect when a call fails,
see [Failures](#byok-cloud-cluster-api-failures).

<a id="byok-cloud-cluster-api-failures"></a>

## Failures

When a failure occurs, you should receive one of the following statuses:

`400 Bad Request`
: Typically a malformed or missing parameter or a parameter value that is not allowed.

`401 Unauthorized`
: Typically invalid or missing authentication credentials.

`403 Forbidden`
: Indicates the credentials provided are not valid for the resource. Make sure your cluster and environment identifiers are correct
  and match the credentials provided.

`404 Not Found`
: Indicates the resource could not be found at the specified URL.

`429 Rate Limit Exceeded`
: You have sent too many requests.

`500 Oops something went wrong`
: An issue not covered by other errors.

The failure message body will provide more detail about the failure reason and an identifier to use with Confluent
support if you need help solving the problem. The following example shows a `400 Bad Request`
error with details about why the call failed.

```bash
HTTP/1.1 400 BAD REQUEST
Content-Type: application/json

{
    "errors": [
      {
           "id": "c9cb03c2878ca172bbd0072bd99a9aaa",
            "status": "400",
            "code": "parameter_missing",
            "detail": "Missing Parameter",
            "source": {
               "parameter": "environment"
             }
      }
   ]
 }
```

<a id="legacy-byok"></a>

## Pre-BYOK-API-V1 Self-Managed encryption keys

#### IMPORTANT
This section applies only to Confluent Cloud clusters that were provisioned
on AWS before August 11, 2023. If your cluster was provisioned on AWS on
or after August 11, 2023, use the BYOK V1 API described in the preceding
sections to manage your self-managed encryption keys.

If you are using Confluent Cloud clusters provisioned on AWS before August 11, 2023,
follow the information in the following sections to manage your Pre-BYOK-API-V1
self-managed encryption keys to encrypt data at rest for your Kafka clusters.

<a id="legacy-byok-prerequisites"></a>

### Prerequisites for managing Pre-BYOK-API-V1 clusters

To manage your Pre-BYOK-API-V1 Confluent Cloud cluster on AWS, you need the following:

* An AWS account with access to AWS KMS.
* A Confluent Cloud cluster provisioned on AWS before August 11, 2023.
* The Confluent CLI installed and configured.
* Appropriate AWS IAM permissions to manage KMS keys.

<a id="legacy-byok-requirements"></a>

### Identify a Pre-BYOK-API-V1 Confluent Cloud cluster

If your Confluent Cloud cluster was created and provisioned on AWS before August 11, 2023,
then it is likely a Pre-BYOK-API-V1 cluster. To verify that your cluster is
a Pre-BYOK-API-V1 cluster, run the following Confluent CLI command to inspect
the cluster and verify that the cluster cannot use the BYOK V1 API:

```shell
confluent kafka cluster describe <cluster-id> --output json
```

If you don’t know the cluster ID, you can use the following Confluent CLI command
to list all your clusters and get the ID for the cluster you want to inspect:

```shell
confluent kafka cluster list
```

For a Pre-BYOK-API-V1 cluster, the output includes a `key_id` field within the
`encryption_key_id` section, like this:

```json
{
  "is_current": true,
  "id": "lkc-abc123",
  "name": "test-byok-cluster",
  "type": "DEDICATED",
  "cluster_size": 1,
  "ingress_limit": 50,
  "egress_limit": 150,
  "storage": "Infinite",
  "cloud": "aws",
  "region": "us-west-2",
  "availability": "single-zone",
  "status": "UP",
  "endpoint": "SASL_SSL://pkc-19m9ov.us-west-2.aws.confluent.cloud:9092",
  "encryption_key_id": "arn:aws:kms:us-west-2:111122223333:key/mrk-1234abcd-12ab-34cd-56ef-1234567890ab",
  "rest_endpoint": "https://pkc-19m9ov.us-west-2.aws.confluent.cloud:443"
}
```

If you see this structure in your cluster’s output, it confirms that you have a
Pre-BYOK-API-V1 cluster. Newer Confluent Cloud clusters do not include the `key_id`
field in this format.

<a id="manage-legacy-byok"></a>

### Manage Pre-BYOK-API-V1 self-managed encryption keys

Pre-BYOK-API-V1 clusters differ from newer Confluent Cloud clusters in the following ways:

* A distinct CCK ID (Confluent Cloud key ID) is not associated with the cluster. This
  means you cannot use the BYOK V1 API to manage your keys through the
  Confluent Cloud Console or Confluent CLI.
* You cannot retrieve your KMS key policy directly using the Confluent Cloud APIs or
  the Confluent CLI for Pre-BYOK-API-V1 clusters. You must manage your KMS key
  policies directly in the AWS Console, but you must include the policy
  statement shown in
  [Replace a lost IAM policy](#replace-lost-iam-policy) in your policy.

To manage your Pre-BYOK-API-V1 self-managed encryption keys:

1. Use the AWS Console or AWS CLI to manage your KMS keys.
2. Keep track of your key ARNs and policies manually.
3. Ensure your IAM policies are properly configured as shown in
   [Replace a lost IAM policy](#replace-lost-iam-policy).

<a id="replace-lost-iam-policy"></a>

### Replace a lost IAM policy

#### WARNING
Do not reuse or re-onboard encryption keys that were used for a Pre-BYOK-API-V1
cluster into a new Confluent Cloud cluster. Unexpected results might occur.

If you lose your IAM policy for a Pre-BYOK-API-V1 cluster, append the following
JSON statement to your existing policy to ensure that you can continue to
manage your Kafka cluster:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow Confluent account(s) to use the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::050451403612:root",
                    "arn:aws:iam::050879227952:root",
                    "arn:aws:iam::121194963621:root",
                    "arn:aws:iam::259082970751:root",
                    "arn:aws:iam::266735807085:root",
                    "arn:aws:iam::390327825978:root",
                    "arn:aws:iam::417601102659:root",
                    "arn:aws:iam::457492987184:root",
                    "arn:aws:iam::479111403931:root",
                    "arn:aws:iam::563559260525:root",
                    "arn:aws:iam::727646498665:root",
                    "arn:aws:iam::891612556359:root",
                    "arn:aws:iam::897722682106:root",
                    "arn:aws:iam::908027390566:root",
                    "arn:aws:iam::980921717494:root",
                    "arn:aws:iam::982081071159:root",
                    "arn:aws:iam::649317483566:root",
                    "arn:aws:iam::439989760662:root",
                    "arn:aws:iam::819787557123:root",
                    "arn:aws:iam::571452958320:root",
                    "arn:aws:iam::702463834586:root",
                    "arn:aws:iam::350927345273:root",
                    "arn:aws:iam::152535741197:root",
                    "arn:aws:iam::161406349951:root",
                    "arn:aws:iam::219109013385:root",
                    "arn:aws:iam::777099487581:root",
                    "arn:aws:iam::590184035729:root",
                    "arn:aws:iam::637423370172:root",
                    "arn:aws:iam::058264492257:root",
                    "arn:aws:iam::851725567684:root"
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow Confluent account(s) to attach persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::050451403612:root",
                    "arn:aws:iam::050879227952:root",
                    "arn:aws:iam::121194963621:root",
                    "arn:aws:iam::259082970751:root",
                    "arn:aws:iam::266735807085:root",
                    "arn:aws:iam::390327825978:root",
                    "arn:aws:iam::417601102659:root",
                    "arn:aws:iam::457492987184:root",
                    "arn:aws:iam::479111403931:root",
                    "arn:aws:iam::563559260525:root",
                    "arn:aws:iam::727646498665:root",
                    "arn:aws:iam::891612556359:root",
                    "arn:aws:iam::897722682106:root",
                    "arn:aws:iam::908027390566:root",
                    "arn:aws:iam::980921717494:root",
                    "arn:aws:iam::982081071159:root",
                    "arn:aws:iam::649317483566:root",
                    "arn:aws:iam::439989760662:root",
                    "arn:aws:iam::819787557123:root",
                    "arn:aws:iam::571452958320:root",
                    "arn:aws:iam::702463834586:root",
                    "arn:aws:iam::350927345273:root",
                    "arn:aws:iam::152535741197:root",
                    "arn:aws:iam::161406349951:root",
                    "arn:aws:iam::219109013385:root",
                    "arn:aws:iam::777099487581:root",
                    "arn:aws:iam::590184035729:root",
                    "arn:aws:iam::637423370172:root",
                    "arn:aws:iam::058264492257:root",
                    "arn:aws:iam::851725567684:root"
                ]
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*"
        }
    ]
}
```

## Related content

- [BYOK (byok/v1)](https://docs.confluent.io/cloud/current/ccloud/keys-byok-v-1/)