Protect Data at Rest Using Self-managed Encryption Keys on Confluent Cloud

Basic encryption of data at rest is provided to all Confluent Cloud clusters by default. This encryption is transparent to the user and is managed by Confluent. As a stronger security option, Confluent Cloud provides optional support for self-managed encryption keys when you create Enterprise or Dedicated Kafka clusters in Confluent Cloud. Also known as bring-your-own-key (BYOK) encryption, this option might be preferable for organizations that want to use their encryption key to encrypt data at rest or require the option to disable access by Confluent. This provides greater privacy and data integrity, which is often required for compliance by government, health, finance, and many other industries.

Benefits of self-managed encryption keys

By using self-managed encryption keys, you gain the following benefits:

  • Full control over encryption keys: You have full control over your master key in your cloud service provider’s key management service, as well as the data at rest stored in your Confluent Cloud account.
  • Ability to revoke access: You can revoke access to your encryption keys during a security breach to prevent access to your data at rest. For details, see Revoke access to data at rest.
  • Rotate your encryption keys: You can enable automatic key rotation on your key management service to increase the security of your data at rest.

Encryption of data at rest on Kafka clusters

Confluent Cloud supports encryption of data with self-managed encryption keys for supported Kafka clusters on AWS, Azure, or Google Cloud. Supported cluster types can be encrypted using the key management service (KMS) on your cloud service provider, but there are some differences between the cluster types.

For Dedicated Confluent Cloud clusters and Enterprise clusters (currently, only available on AWS):

  • Configure and manage encryption of data at rest on your clusters using the same Confluent Platform process described for supported cloud service providers ( AWS, Azure, or Google Cloud)
  • Use the key management service (KMS) on your cloud service provider to create and manage self-managed encryption keys.
  • Your data at rest is always stored and encrypted using your self-managed encryption keys.
  • (Optional) Use client-side field level encryption (CSFLE) to encrypt the most sensitive data on your clients before it is sent to Confluent Cloud. CSFLE ensures that if your data is compromised, only users with access to the encryption keys for sensitive data can access the data. For more information, see Protect Sensitive Data Using Client-Side Field Level Encryption on Confluent Cloud.

Summary of the encryption process:

  1. Consumed data is temporarily loaded into local storage on compute nodes after being encrypted using a self-managed encryption key for Dedicated clusters and a Confluent-owned encryption key for Enterprise clusters.
  2. For both Dedicated and Enterprise clusters, your data is then encrypted with your self-managed encryption key and stored in Infinite Storage.
  3. Temporary storage is then overwritten with zeros to ensure that the previously stored information is completely erased and unrecoverable before reallocation and reuse.

Requirements

To use self-managed encryption keys for your clusters on Confluent Cloud, you must meet the following requirements:

  • Supported Kafka cluster types: Enterprise (currently, available only on AWS) and Dedicated.

  • Create a Kafka cluster on Confluent Cloud for a supported cluster type, selecting the Use self-managed encryption keys option. You cannot change this option after the cluster is provisioned.

  • Use a key management service (KMS) that supports the Key Management Interoperability Protocol (KMIP) version 1.1 or later.

    Supported key management services include:

    • Amazon Web Services Key Management Service (AWS KMS)
    • Azure Key Vault
    • Google Cloud Key Management Service (Cloud KMS)

  • Importing key material is not supported.

  • You can revoke access to data on a cluster by disabling access to the encryption key, but the cluster is inaccessible only after you delete the cluster. For details, see Revoke Access to Data at Rest.

  • After you delete a cluster, the encryption key associated with the cluster is released after five days and can be reused to create a new encrypted cluster. As a security best practice, encryption keys should not be reused for production clusters.

  • Each cluster requires a unique encryption key. Using unique encryption keys helps ensure that if one cluster is compromised, the attacker cannot use the same key to access data from other clusters.

  • Both the Kafka cluster and your KMS must be colocated in the same region.

  • Self-managed encryption keys are supported on the following cloud service providers:

  • External key managers (EKM) are currently supported only on Google Cloud. For details, see Key creation and management.

  • Self-managed encryption keys currently apply only to Confluent Cloud Dedicated and Enterprise (available only on AWS) Kafka clusters, and not to any consumer, including Confluent Cloud for Apache Flink®.

List the service quota for self-managed encryption keys

The service quota for self-managed encryption keys (BYOK keys) is applied at the Organization scope instead of the Kafka Cluster scope because self-managed encryption keys must be created before you create an encrypted Kafka cluster using those encryption keys. When you use the Confluent CLI and the Scopes (service-quota/v1) API, the service quota for self-managed encryption keys is displayed as Max BYOK keys per organization and the quota code is byok.max_keys.per_org.

The default service quota for self-managed encryption keys is listed in organization service quotas.

To use the Confluent CLI to view the currently applied service quota for self-managed encryption keys in your organization, run the following command:

confluent service-quota list organization

To use the Scopes (service-quota/v1) API to display the currently applied service quota for self-managed encryption keys in your organization, see Organization quotas.

View the cluster security settings

After you create and provision a Kafka cluster that uses a self-managed encryption key, you can view the security settings. You can use these cluster security settings to verify the correct encryption key is used.

To view your cluster security settings:

  1. Select your Confluent Cloud cluster.
  2. Click Cluster settings and then Security.

Note that anyone authorized to view your clusters can view this information.

Related content