Use the BYOK API with Self-Managed Keys on Confluent Cloud
Required RBAC role: OrganizationAdmin or EnvironmentAdmin.
You can use the Key Management API for Bring Your Own Key (BYOK) to upload and retrieve self-managed keys on Dedicated, Enterprise, and Freight Kafka clusters in Confluent Cloud. Key objects represent customer managed keys on Dedicated, Enterprise, and Freight Kafka clusters. Keys are used to protect data at rest stored in your clusters on AWS and Azure.
In addition to managing these self-managed keys with the Confluent Cloud Console, you can use a collection of REST APIs to manage keys for your Confluent Cloud environments.
You can use the Bring Your Own Key (BYOK) Management API to perform the following operations on Key objects:
Important
If your Confluent Cloud cluster was provisioned on AWS before August 11, 2023, it is a Pre-BYOK-API-V1 cluster and you cannot use the BYOK V1 API to manage its self-managed encryption keys. For details, see Pre-BYOK-API-V1 Self-Managed encryption keys.
Base URL
The base URL for a BYOK API request is
https://confluent.cloud/api
Object model
All Confluent objects share the following set of common properties:
api_version: Indicates the API version for the object. The version for the BYOK API isbyok/v1.kind: Indicates what kind of object it is.id: An identifier for the object.
Authentication
Each request to the server must include an Authorization: Basic {key} header that contains
a Confluent Cloud API key. For more details on creating or accessing your Confluent Cloud API keys, see Manage API Keys in Confluent Cloud.
API requests without a correct authentication header will fail. An API key grants access to a
single Confluent cluster.
These keys grant API access to many resources, and should not be shared publicly to GitHub, in client-side code, and so forth.
HTTP Basic authorization requires the key to be colon-separated and base64-encoded.
For example, if your API Key is abcp7DEFGH123456789 and the corresponding Secret
is XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO, generate a base64-encoded header with the following command on
MacOS or Linux:
echo -n "abcp7DEFGH123456789:XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO" | base64
The resulting authorization header will be:
Authorization: Basic QUJDREVGR0gxMjM0NTY3ODk6WE5DSVc5M0kyTDFTUVBKU0o4MjNLMUxTOTAyS0xERk1DWlBXRU8=
Alternatively, if testing using Postman, specify Basic Auth and provide the API Key as the Username and the API Secret as the password and Postman will provide the encoding.
Test the BYOK API with curl
The following sections provide examples of using curl commands for HTTP GET and HTTP POST requests based on the BYOK API. Before using the curl commands, you must sign in to Confluent Cloud. For more information about the Confluent CLI, see Confluent Cloud CLI Reference.
For example:
Sign in to Confluent Cloud using the Confluent CLI.
confluent login --url https://confluent.cloud
Provide username and password at prompts.
Create an API key and API secret.
confluent api-key create --resource cloud
Store the API key and API secret in environment variables.
export CCLOUD_KEY=<generated-API-key>
export CCLOUD_SECRET=<generated-API-secret>
Store the URL for Confluent Cloud in a variable:
export CCLOUD_URL=https://confluent.cloud/api/
Run the following API request to list byok objects:
curl -u $CCLOUD_KEY:$CCLOUD_SECRET "$CCLOUD_URL/byok/v1/keys"
The output should like the example response displayed below for List BYOK objects.
Request examples
List BYOK objects
API reference: List of BYOK objects
Request example
The following examples show how to retrieve a sorted, filtered, paginated list of all Kafka clusters.
GET /byok/v1/keys
Your request can include the following query parameters:
Parameter |
Required? |
Description |
|---|---|---|
|
No |
Filter the results by exact match for provider. |
|
No |
Filter the results by exact match for state. |
|
No |
Specifies an integer page size for the returned payload. Maximum value is |
|
No |
Contains the opaque pagination token string. |
Response example
A successful request returns an HTTP 200 OK response and a JSON payload that
includes a list of BYOK objects. The response describes each object, including
the following information:
The associated cloud-specific key details.
The provider for the BYOK object.
The object status (
AVAILABLE, IN_USE).
{
"api_version": "byok/v1",
"kind": "KeyList",
"metadata": {
"first": "https://api.confluent.cloud/byok/v1/keys",
"last": "https://api.confluent.cloud/byok/v1/keys?page_token=bcAOehAY8F16YD84Z1wT",
"prev": "https://api.confluent.cloud/byok/v1/keys?page_token=YIXRY97wWYmwzrax4dld",
"next": "https://api.confluent.cloud/byok/v1/keys?page_token=UvmDWOB1iwfAIBPj6EYb",
"total_size": 123
},
"data": [
{
"api_version": "byok/v1",
"kind": "Key",
"id": "dlz-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
"resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
"created_at": "2006-01-02T15:04:05-07:00",
"updated_at": "2006-01-02T15:04:05-07:00",
"deleted_at": "2006-01-02T15:04:05-07:00"
},
"key": {
"key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"roles": [
"arn:aws:iam::123456789876:role/block_storage_manager",
"arn:aws:iam::987654321234:role/cc-kafka-1111aaaa-11aa-11aa-11aa-111111aaaaaa"
],
"kind": "AwsKey"
},
"provider": "AWS",
"state": "IN_USE"
}
]
}
Create a BYOK Object
API reference: Create a BYOK object
To register a BYOK object, send an HTTP POST request call to the key endpoint.
Make sure you include the authorization header with your API key and API secret.
POST /byok/v1/keys
Request
When you make the request, include a JSON payload that contains a key object
showing the cloud-specific key details. The content of the key object varies
for each cloud service provider.
Request specification for AWS
Parameter |
Required |
Description |
|---|---|---|
|
Required |
The Amazon Resource Name (ARN) of the AWS KMS key. |
|
Required |
BYOK kind type (AwsKey). |
Request example
{
"key": {
"key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"kind": "AwsKey"
}
}
Response example
A successful request returns an HTTP 200 OK response and a JSON payload that contains
a list of BYOK objects. The response describes each object, including the
following information:
The associated cloud-specific key details.
The provider for the BYOK object.
The object status (
AVAILABLE, IN_USE).
{
"api_version": "byok/v1",
"kind": "Key",
"id": "dlz-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
"resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
"created_at": "2006-01-02T15:04:05-07:00",
"updated_at": "2006-01-02T15:04:05-07:00",
"deleted_at": "2006-01-02T15:04:05-07:00"
},
"key": {
"key_arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"roles": [
"arn:aws:iam::123456789876:role/block_storage_manager",
"arn:aws:iam::987654321234:role/cc-kafka-1111aaaa-11aa-11aa-11aa-111111aaaaaa"
],
"kind": "AwsKey"
},
"provider": "AWS",
"state": "IN_USE"
}
Request specification for Azure
Parameter |
Required? |
Description |
|---|---|---|
|
Yes |
The unique Key Object Identifier URL without version of an Azure Key Vault key. |
|
Yes |
The Key Vault ID containing the key. |
|
Yes |
Tenant ID (uuid) hosting the Key Vault containing the key |
|
Yes |
BYOK kind type (AzureKey). |
Request example
{
"key": {
"key_id": "https://vault-name.vault.azure.net/keys/key-name",
"key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
"tenant_id": "00000000-0000-0000-0000-000000000000",
"kind": "AzureKey"
}
}
Response example
A successful call returns an HTTP 200 OK and a JSON payload that contains a list of BYOK objects.
The response describes each object including the following information:
The associated cloud-specific key details.
The provider for the BYOK object.
The object status (AVAILABLE, IN_USE).
{
"api_version": "byok/v1",
"kind": "Key",
"id": "dlz-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
"resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
"created_at": "2006-01-02T15:04:05-07:00",
"updated_at": "2006-01-02T15:04:05-07:00",
"deleted_at": "2006-01-02T15:04:05-07:00"
},
"key": {
"key_id": "https://vault-name.vault.azure.net/keys/key-name",
"key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
"application_id": "00000000-0000-0000-0000-000000000000",
"tenant_id": "00000000-0000-0000-0000-000000000000",
"kind": "AzureKey"
},
"provider": "Azure",
"state": "IN_USE"
}
Read a BYOK Key
API reference: Read a BYOK object
To read a BYOK object, make a GET call to the keys endpoint to retrieve details about a specified key. You must specify the id in the request. This call does not require a payload. Make sure you include the authorization header with your API key and API secret.
Request example
GET /byok/v1/keys/{id}
Your request must include the following parameters:
Parameter |
Required? |
Description |
|---|---|---|
|
No |
The unique identifier for the key. |
Response example
A successful request returns an HTTP 200 OK response and a JSON payload
that contains a list of BYOK objects. The response describes each object,
including the following information:
The associated cloud-specific key details.
The provider for the BYOK object.
The object status (
AVAILABLE, IN_USE).
{
"api_version": "byok/v1",
"kind": "Key",
"id": "dlz-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/byok/v1/keys/cck-12345",
"resource_name": "crn://confluent.cloud/organization=9bb441c4-edef-46ac-8a41-c49e44a3fd9a/key=cck-12345",
"created_at": "2006-01-02T15:04:05-07:00",
"updated_at": "2006-01-02T15:04:05-07:00",
"deleted_at": "2006-01-02T15:04:05-07:00"
},
"key": {
"key_id": "https://vault-name.vault.azure.net/keys/key-name",
"key_vault_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourcegroup-name/providers/Microsoft.KeyVault/vaults/vault-name",
"application_id": "00000000-0000-0000-0000-000000000000",
"tenant_id": "00000000-0000-0000-0000-000000000000",
"kind": "AzureKey"
},
"provider": "Azure",
"state": "IN_USE"
}
Delete a BYOK object
API reference: Delete a Cluster
You can delete an encrypted cluster with a DELETE request to the keys
endpoint. You specify id in the request, and the call does not
require a payload. Ensure that you include the authorization header
with your API key and API secret.
Important
After the encryption key (BYOK key object) is deleted, it cannot be restored. For example, if you delete the BYOK key object in Confluent Cloud on az| and plan to reuse the key for testing, you must run the Confluent CLI commands again to create a new BYOK key object.
Request example
DELETE /byok/v1/keys/{id}
Parameter |
Required? |
Description |
|---|---|---|
|
Yes |
The unique identifier for the key. |
Response example
Success returns an HTTP 204 No Content response.
If you delete a Kafka cluster and try to delete the BYOK key object associated with it, this operation fails until the encryption key is available for reuse (after five days). For details on the types of responses you can expect when a call fails, see Failures.
Failures
When a failure occurs, you should receive one of the following statuses:
400 Bad RequestTypically a malformed or missing parameter or a parameter value that is not allowed.
401 UnauthorizedTypically invalid or missing authentication credentials.
403 ForbiddenIndicates the credentials provided are not valid for the resource. Make sure your cluster and environment identifiers are correct and match the credentials provided.
404 Not FoundIndicates the resource could not be found at the specified URL.
429 Rate Limit ExceededYou have sent too many requests.
500 Oops something went wrongAn issue not covered by other errors.
The failure message body will provide more detail about the failure reason and an identifier to use with Confluent
support if you need help solving the problem. The following example shows a 400 Bad Request
error with details about why the call failed.
HTTP/1.1 400 BAD REQUEST
Content-Type: application/json
{
"errors": [
{
"id": "c9cb03c2878ca172bbd0072bd99a9aaa",
"status": "400",
"code": "parameter_missing",
"detail": "Missing Parameter",
"source": {
"parameter": "environment"
}
}
]
}
Pre-BYOK-API-V1 Self-Managed encryption keys
Important
This section applies only to Confluent Cloud clusters that were provisioned on AWS before August 11, 2023. If your cluster was provisioned on AWS on or after August 11, 2023, use the BYOK V1 API described in the preceding sections to manage your self-managed encryption keys.
If you are using Confluent Cloud clusters provisioned on AWS before August 11, 2023, follow the information in the following sections to manage your Pre-BYOK-API-V1 self-managed encryption keys to encrypt data at rest for your Kafka clusters.
Prerequisites for managing Pre-BYOK-API-V1 clusters
To manage your Pre-BYOK-API-V1 Confluent Cloud cluster on AWS, you need the following:
An AWS account with access to AWS KMS.
A Confluent Cloud cluster provisioned on AWS before August 11, 2023.
The Confluent CLI installed and configured.
Appropriate AWS IAM permissions to manage KMS keys.
Identify a Pre-BYOK-API-V1 Confluent Cloud cluster
If your Confluent Cloud cluster was created and provisioned on AWS before August 11, 2023, then it is likely a Pre-BYOK-API-V1 cluster. To verify that your cluster is a Pre-BYOK-API-V1 cluster, run the following Confluent CLI command to inspect the cluster and verify that the cluster cannot use the BYOK V1 API:
confluent kafka cluster describe <cluster-id> --output json
If you don’t know the cluster ID, you can use the following Confluent CLI command to list all your clusters and get the ID for the cluster you want to inspect:
confluent kafka cluster list
For a Pre-BYOK-API-V1 cluster, the output includes a key_id field within the
encryption_key_id section, like this:
{
"is_current": true,
"id": "lkc-abc123",
"name": "test-byok-cluster",
"type": "DEDICATED",
"cluster_size": 1,
"ingress_limit": 50,
"egress_limit": 150,
"storage": "Infinite",
"cloud": "aws",
"region": "us-west-2",
"availability": "single-zone",
"status": "UP",
"endpoint": "SASL_SSL://pkc-19m9ov.us-west-2.aws.confluent.cloud:9092",
"encryption_key_id": "arn:aws:kms:us-west-2:111122223333:key/mrk-1234abcd-12ab-34cd-56ef-1234567890ab",
"rest_endpoint": "https://pkc-19m9ov.us-west-2.aws.confluent.cloud:443"
}
If you see this structure in your cluster’s output, it confirms that you have a
Pre-BYOK-API-V1 cluster. Newer Confluent Cloud clusters do not include the key_id
field in this format.
Manage Pre-BYOK-API-V1 self-managed encryption keys
Pre-BYOK-API-V1 clusters differ from newer Confluent Cloud clusters in the following ways:
A distinct CCK ID (Confluent Cloud key ID) is not associated with the cluster. This means you cannot use the BYOK V1 API to manage your keys through the Confluent Cloud Console or Confluent CLI.
You cannot retrieve your KMS key policy directly using the Confluent Cloud APIs or the Confluent CLI for Pre-BYOK-API-V1 clusters. You must manage your KMS key policies directly in the AWS Console, but you must include the policy statement shown in Replace a lost IAM policy in your policy.
To manage your Pre-BYOK-API-V1 self-managed encryption keys:
Use the AWS Console or AWS CLI to manage your KMS keys.
Keep track of your key ARNs and policies manually.
Ensure your IAM policies are properly configured as shown in Replace a lost IAM policy.
Replace a lost IAM policy
Warning
Do not reuse or re-onboard encryption keys that were used for a Pre-BYOK-API-V1 cluster into a new Confluent Cloud cluster. Unexpected results might occur.
If you lose your IAM policy for a Pre-BYOK-API-V1 cluster, append the following JSON statement to your existing policy to ensure that you can continue to manage your Kafka cluster:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Allow Confluent account(s) to use the key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::050451403612:root",
"arn:aws:iam::050879227952:root",
"arn:aws:iam::121194963621:root",
"arn:aws:iam::259082970751:root",
"arn:aws:iam::266735807085:root",
"arn:aws:iam::390327825978:root",
"arn:aws:iam::417601102659:root",
"arn:aws:iam::457492987184:root",
"arn:aws:iam::479111403931:root",
"arn:aws:iam::563559260525:root",
"arn:aws:iam::727646498665:root",
"arn:aws:iam::891612556359:root",
"arn:aws:iam::897722682106:root",
"arn:aws:iam::908027390566:root",
"arn:aws:iam::980921717494:root",
"arn:aws:iam::982081071159:root",
"arn:aws:iam::649317483566:root",
"arn:aws:iam::439989760662:root",
"arn:aws:iam::819787557123:root",
"arn:aws:iam::571452958320:root",
"arn:aws:iam::702463834586:root",
"arn:aws:iam::350927345273:root",
"arn:aws:iam::152535741197:root",
"arn:aws:iam::161406349951:root",
"arn:aws:iam::219109013385:root",
"arn:aws:iam::777099487581:root",
"arn:aws:iam::590184035729:root",
"arn:aws:iam::637423370172:root",
"arn:aws:iam::058264492257:root",
"arn:aws:iam::851725567684:root"
]
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow Confluent account(s) to attach persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::050451403612:root",
"arn:aws:iam::050879227952:root",
"arn:aws:iam::121194963621:root",
"arn:aws:iam::259082970751:root",
"arn:aws:iam::266735807085:root",
"arn:aws:iam::390327825978:root",
"arn:aws:iam::417601102659:root",
"arn:aws:iam::457492987184:root",
"arn:aws:iam::479111403931:root",
"arn:aws:iam::563559260525:root",
"arn:aws:iam::727646498665:root",
"arn:aws:iam::891612556359:root",
"arn:aws:iam::897722682106:root",
"arn:aws:iam::908027390566:root",
"arn:aws:iam::980921717494:root",
"arn:aws:iam::982081071159:root",
"arn:aws:iam::649317483566:root",
"arn:aws:iam::439989760662:root",
"arn:aws:iam::819787557123:root",
"arn:aws:iam::571452958320:root",
"arn:aws:iam::702463834586:root",
"arn:aws:iam::350927345273:root",
"arn:aws:iam::152535741197:root",
"arn:aws:iam::161406349951:root",
"arn:aws:iam::219109013385:root",
"arn:aws:iam::777099487581:root",
"arn:aws:iam::590184035729:root",
"arn:aws:iam::637423370172:root",
"arn:aws:iam::058264492257:root",
"arn:aws:iam::851725567684:root"
]
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*"
}
]
}