Use the BYOK API with Self-Managed Keys on Confluent Cloud¶

Required RBAC role: OrganizationAdmin or EnvironmentAdmin.

You can use the Key Management API for BYOK (Bring Your Own Key) to upload and retrieve self-managed keys on Dedicated Kafka clusters in Confluent Cloud. Key objects represent customer managed keys on Dedicated Kafka clusters. Keys are used to protect data at rest stored in your clusters on AWS and Azure.

In addition to managing these self-managed keys with the Confluent Cloud Console, you can use a collection of REST APIs to manage keys for your Confluent Cloud environments.

You can use the Bring Your Own Key (BYOK) Management API to perform the following operations on Key objects:

Create a BYOK Object
Read a BYOK Key
List BYOK objects
Delete a BYOK object

Base URL¶

The base URL for a BYOK API request is:

https://confluent.cloud/api

Object model¶

All Confluent objects share the following set of common properties:

api_version: Indicates the API version for the object. The version for the BYOK API is byok/v1.
kind: Indicates what kind of object it is.
id: An identifier for the object.

Authentication¶

Each request to the server must include an Authorization: Basic {key} header that contains a Confluent Cloud API key. For more details on creating or accessing your Confluent Cloud API keys, see Manage API Keys in Confluent Cloud. API requests without a correct authentication header will fail. An API key grants access to a single Confluent cluster.

These keys grant API access to many resources, and should not be shared publicly to GitHub, in client-side code, and so forth.

HTTP Basic authorization requires the key to be colon-separated and base64-encoded.

For example, if your API Key is abcp7DEFGH123456789 and the corresponding Secret is XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO, generate a base64-encoded header with the following command on MacOS or Linux:

echo -n "abcp7DEFGH123456789:XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO" | base64

The resulting authorization header will be:

Authorization: Basic QUJDREVGR0gxMjM0NTY3ODk6WE5DSVc5M0kyTDFTUVBKU0o4MjNLMUxTOTAyS0xERk1DWlBXRU8=

Alternatively, if testing using Postman, specify Basic Auth and provide the API Key as the Username and the API Secret as the password and Postman will provide the encoding.

Test the BYOK API with curl¶

The following sections provide examples of using curl commands for HTTP GET and HTTP POST requests based on the BYOK API. Before using the curl commands, you must sign in to Confluent Cloud. For more information about the Confluent CLI, see Confluent Cloud CLI Reference.

For example:

Sign in to Confluent Cloud using the Confluent CLI. (Provide username and password at prompts.)
```
confluent login --url https://confluent.cloud
```

Create an API key and API secret.

confluent api-key create --resource cloud

Store the API key and API secret in environment variables.

export CCLOUD_KEY=<generated-API-key>

export CCLOUD_SECRET=<generated-API-secret>

Store the URL for Confluent Cloud in a variable:

export CCLOUD_URL=https://confluent.cloud/api/

Run the following API request to list byok objects:
```
curl -u $CCLOUD_KEY:$CCLOUD_SECRET "$CCLOUD_URL/byok/v1/keys"
```
The output should like the example response displayed below for List BYOK objects.

Request examples¶

List BYOK objects¶

API reference: List of BYOK objects

Request example¶

The following examples show how to retrieve a sorted, filtered, paginated list of all Kafka clusters.

GET /byok/v1/keys

Your request can include the following query parameters:

Parameter	Required?	Description
`provider`	No	Filter the results by exact match for provider.
`state`	No	Filter the results by exact match for state.
`page_size`	No	Specifies an integer page size for the returned payload. Maximum value is `100`. Default value is `10`.
`page_token`	No	Contains the opaque pagination token string.

Response example¶

A successful request returns an HTTP 200 OK response and a JSON payload that includes a list of BYOK objects. The response describes each object, including the following information:

The associated cloud-specific key details.
The provider for the BYOK object.
The object status (AVAILABLE, IN_USE).

{
  "api_version": "byok/v1",
  "kind": "KeyList",
  "metadata": {
    "first": "https://api.confluent.cloud/byok/v1/keys",
    "last": "https://api.confluent.cloud/byok/v1/keys?page_token=bcAOehAY8F16YD84Z1wT",
    "prev": "https://api.confluent.cloud/byok/v1/keys?page_token=YIXRY97wWYmwzrax4dld",
    "next": "https://api.confluent.cloud/byok/v1/keys?page_token=UvmDWOB1iwfAIBPj6EYb",
    "total_size": 123
  },
  "data": [
    {
      "api_version": "byok/v1",
      "kind": "Key",
      "id": "dlz-f3a90de",
      "metadata": {
        "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
        "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
        "created_at": "2006-01-02T15:04:05-07:00",
        "updated_at": "2006-01-02T15:04:05-07:00",
        "deleted_at": "2006-01-02T15:04:05-07:00"
      },
      "key": {
        "key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
        "roles": [
          "arn:aws:iam::123456789876:role/block_storage_manager",
          "arn:aws:iam::987654321234:role/cc-kafka-1111aaaa-11aa-11aa-11aa-111111aaaaaa"
        ],
        "kind": "AwsKey"
      },
      "provider": "AWS",
      "state": "IN_USE"
    }
  ]
}

Create a BYOK Object¶

API reference: Create a BYOK object

To register a BYOK object, send an HTTP POST request call to the key endpoint.

Make sure you include the authorization header with your API key and API secret.

POST /byok/v1/keys

Request¶

When you make the request, include a JSON payload that contains a key object showing the cloud-specific key details. The content of the key object varies for each cloud service provider.

Request specification for AWS¶

Parameter	Required	Description
`key_arn`	Required	The Amazon Resource Name (ARN) of the AWS KMS key.
`kind`	Required	BYOK kind type (AwsKey).

Request example¶

{
  "key": {
    "key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
    "kind": "AwsKey"
  }
}

Response example¶

A successful request returns an HTTP 200 OK response and a JSON payload that contains a list of BYOK objects. The response describes each object, including the following information:

The associated cloud-specific key details.
The provider for the BYOK object.
The object status (AVAILABLE, IN_USE).

{
  "api_version": "byok/v1",
  "kind": "Key",
  "id": "dlz-f3a90de",
  "metadata": {
    "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
    "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
    "created_at": "2006-01-02T15:04:05-07:00",
    "updated_at": "2006-01-02T15:04:05-07:00",
    "deleted_at": "2006-01-02T15:04:05-07:00"
  },
  "key": {
    "key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
    "roles": [
      "arn:aws:iam::123456789876:role/block_storage_manager",
      "arn:aws:iam::987654321234:role/cc-kafka-1111aaaa-11aa-11aa-11aa-111111aaaaaa"
    ],
    "kind": "AwsKey"
  },
  "provider": "AWS",
  "state": "IN_USE"
}

Request specification for Azure¶

Parameter	Required?	Description
`key_id`	Yes	The unique Key Object Identifier URL without version of an Azure Key Vault key.
`key_vault_id`	Yes	The Key Vault ID containing the key.
`tenant_id`	Yes	Tenant ID (uuid) hosting the Key Vault containing the key
`kind`	Yes	BYOK kind type (AzureKey).

Request example¶

{
  "key": {
    "key_id": "https://vault-name.vault.azure.net/keys/key-name",
    "key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "kind": "AzureKey"
  }
}

Response example¶

A successful call returns an HTTP 200 OK and a JSON payload that contains a list of BYOK objects. The response describes each object including the following information:

The associated cloud-specific key details.
The provider for the BYOK object.
The object status (AVAILABLE, IN_USE).

{
  "api_version": "byok/v1",
  "kind": "Key",
  "id": "dlz-f3a90de",
  "metadata": {
    "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
    "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
    "created_at": "2006-01-02T15:04:05-07:00",
    "updated_at": "2006-01-02T15:04:05-07:00",
    "deleted_at": "2006-01-02T15:04:05-07:00"
  },
  "key": {
    "key_id": "https://vault-name.vault.azure.net/keys/key-name",
    "key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
    "application_id": "00000000-0000-0000-0000-000000000000",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "kind": "AzureKey"
  },
  "provider": "Azure",
  "state": "IN_USE"
}

Read a BYOK Key¶

API reference: Read a BYOK object

To read a BYOK object, make a GET call to the keys endpoint to retrieve details about a specified key. You must specify the id in the request. This call does not require a payload. Make sure you include the authorization header with your API key and API secret.

Request example¶

GET /byok/v1/keys/{id}

Your request must include the following parameters:

Parameter	Required?	Description
`id`	No	The unique identifier for the key.

Response example¶

A successful request returns an HTTP 200 OK response and a JSON payload that contains a list of BYOK objects. The response describes each object, including the following information:

The associated cloud-specific key details.
The provider for the BYOK object.
The object status (AVAILABLE, IN_USE).

{
  "api_version": "byok/v1",
  "kind": "Key",
  "id": "dlz-f3a90de",
  "metadata": {
    "self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
    "resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
    "created_at": "2006-01-02T15:04:05-07:00",
    "updated_at": "2006-01-02T15:04:05-07:00",
    "deleted_at": "2006-01-02T15:04:05-07:00"
  },
  "key": {
    "key_id": "https://vault-name.vault.azure.net/keys/key-name",
    "key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
    "application_id": "00000000-0000-0000-0000-000000000000",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "kind": "AzureKey"
  },
  "provider": "Azure",
  "state": "IN_USE"
}

Delete a BYOK object¶

API reference: Delete a Cluster

You can delete an encrypted cluster with a DELETE request to the keys endpoint. You specify id in the request, and the call does not require a payload. Ensure that you include the authorization header with your API key and API secret.

Important

After the encryption key (BYOK key object) is deleted, it cannot be restored. For example, if you delete the BYOK key object in Confluent Cloud on az| and plan to reuse the key for testing, you must run the Confluent CLI commands again to create a new BYOK key object.

Request example¶

DELETE /byok/v1/keys/{id}

Parameter	Required?	Description
`id`	Yes	The unique identifier for the key.

Response example¶

Success returns an HTTP 204 No Content response.

If you delete a Kafka cluster and try to delete the BYOK key object associated with it, this operation fails until the encryption key is available for reuse (after five days). For details on the types of responses you can expect when a call fails, see Failures.

Failures¶

When a failure occurs, you should receive one of the following statuses:

400 Bad Request: Typically a malformed or missing parameter or a parameter value that is not allowed.
401 Unauthorized: Typically invalid or missing authentication credentials.
403 Forbidden: Indicates the credentials provided are not valid for the resource. Make sure your cluster and environment identifiers are correct and match the credentials provided.
404 Not Found: Indicates the resource could not be found at the specified URL.
429 Rate Limit Exceeded: You have sent too many requests.
500 Oops something went wrong: An issue not covered by other errors.

The failure message body will provide more detail about the failure reason and an identifier to use with Confluent support if you need help solving the problem. The following example shows a 400 Bad Request error with details about why the call failed.

HTTP/1.1 400 BAD REQUEST
Content-Type: application/json

{
    "errors": [
      {
           "id": "c9cb03c2878ca172bbd0072bd99a9aaa",
            "status": "400",
            "code": "parameter_missing",
            "detail": "Missing Parameter",
            "source": {
               "parameter": "environment"
             }
      }
   ]
 }

Use the BYOK API with Self-Managed Keys on Confluent Cloud¶

Base URL¶

Object model¶

Authentication¶

Test the BYOK API with curl¶

Request examples¶

List BYOK objects¶

Request example¶

Response example¶

Create a BYOK Object¶

Request¶

Request specification for AWS¶

Request example¶

Response example¶

Request specification for Azure¶

Request example¶

Response example¶

Read a BYOK Key¶

Request example¶

Response example¶

Delete a BYOK object¶

Request example¶

Response example¶

Failures¶

Related content¶