Configure Network Encryption

Secure network communications between Confluent components using Confluent for Kubernetes (CFK). CFK supports Transport Layer Security (TLS), an industry-standard encryption protocol, which relies on keys and certificates to establish trusted connections. This section describes how to manage keys and certificates when you configure TLS encryption for Confluent Platform. For security concepts in Confluent Platform, see Security. For information about rotating and managing existing certificates, see Manage TLS Certificates for Confluent Platform Using Confluent for Kubernetes.

CFK supports the following mechanisms to enable TLS encryption:

Auto-generated certificates

CFK auto-generates the server certificates, using the certificate authority (CA) that you provide.

Use auto-generated certificates if all access and communication to Confluent services stays within the Kubernetes network.

User-provided certificates

You provide the private key, public key, and CA.

Use user-provided certificates if you need to enable access to Confluent services from a domain outside Kubernetes.

Separate certificates for internal and external communications

You provide separate TLS certificates for the internal and external communications so that you do not mix external and internal domains in the certificate Subject Alternative Name (SAN).

This feature is supported for Apache Kafka®, ksqlDB, Schema Registry, Metadata Service (MDS), and Kafka REST services, starting in CFK 2.6.0 and Confluent Platform 7.4.0 release.

Dynamic Kafka certificate updates

When you rotate certificates by providing new server certificates, CFK automatically updates the configurations to use them. By default, this triggers a rolling restart of the affected Confluent Platform pod.

To minimize disruption during rolling restarts of Kafka brokers, you can enable dynamic certificate update for Kafka and Kafka REST service. CFK updates TLS private keys and certificates without rolling the Kafka cluster.

This feature is only supported at the individual listener level.

Disable TLS hostname verification for the internal client

If your internal listener certificate SAN cannot include the Kubernetes internal DNS name, you can disable TLS hostname verification for the CFK operator’s own internal Kafka client.

Use auto-generated TLS certificates

You can use auto-generated certificates to encrypt internal traffic within a Kubernetes network. You bring the certificate authority (CA) you want Confluent for Kubernetes (CFK) to use, and CFK automates provisioning, securing, and operating certificates for internal networking communications.

The generated server certificates expire in 365 days.

For a tutorial scenario on using auto-generated certs, see the quickstart tutorial.

Configure auto-generated certificates using one of the following methods, in order of preference:

1. Use annotations for CA certificates in component custom resource (CR).

2. Configure managed certificates at the CFK operator level using Helm values.

3. Use the legacy CA pair (ca-pair-sslcerts) if none of the preceding options are configured.

   Use the TLS secret named ca-pair-sslcerts for the auto-generated certificate feature.

Configure auto-generated certificates using annotations

Apply the following managed certificate annotations at the component CR level. These configurations override all other CA certificate configurations, such as the other CA certificates set at the CFK level.

platform.confluent.io/managed-cert-ca-pair-secret
platform.confluent.io/managed-cert-duration-in-days
platform.confluent.io/managed-cert-renew-before-in-days
platform.confluent.io/managed-cert-add-sans

For example:

kubectl annotate kafka kafka platform.confluent.io/managed-cert-ca-pair-secret="my-custom-ca-pair"
kubectl annotate kafka kafka platform.confluent.io/managed-cert-duration-in-days=60
kubectl annotate kafka kafka platform.confluent.io/managed-cert-renew-before-in-days=30
kubectl annotate kafka kafka platform.confluent.io/managed-cert-add-sans="kafka.operator,kafka.operator.local"

Configure auto-generated certificates using Helm values

Starting in CFK 2.4.0, you can customize auto-generated certificates in the CFK Helm values. Define the certificate rotation and renewal policy using the following settings:

• The secret name or Vault directory path for the certificates

• The validity period of the certificates

• The renewal time for the certificates

• The certificate SANs

For upgrading from the legacy, default auto-generated certificates to the configurable certificates, see Upgrade auto-generated certificates to configurable auto-generated certificates.

To configure auto-generated certificates:

1. Use a certificate manager to create a CA certificate and key with the file names tls.crt and tls.key, respectively.

2. Define the secret for the CA certificate pairs.

   Create a Kubernetes secret with keys, tls.crt and tls.key. CFK uses these CA key pairs to sign the certificates.

   • To create a secret with a CA certificate and key for auto-generated certificates:

     kubectl create secret tls <secret_name> \
       --cert=/path/to/ca.pem \
       --key=/path/to/ca-key.pem
     

     You can assign any name to the secret (<secret_name>). There is no requirement for the name.

     If you select this option, you must follow the next step and set managedCerts.enabled: true and set managedCerts.caCertificate.secretRef=<secret_name>.

   • To use Vault for auto-generated secrets, set up Vault and specify the path to the CA pair certificates in the next step. The tls.crt and tls.key files must be present in the directory.

     If you select this option, you must follow the next step and set managedCerts.enabled: true and set managedCerts.caCertificate.directoryPathInContainer=<path>.

3. If you want to customize certificate settings, configure the settings in the CFK Helm values (in values.yaml).

   Edit the Helm values file, and then apply the changes with the helm upgrade command:

   managedCerts:
     enabled:                       --- [1]
     caCertificate:                 --- [2]
       secretRef:                   --- [3]
       directoryPathInContainer:    --- [4]
     certDurationInDays:            --- [5]
     renewBeforeInDays:             --- [6]
     sans:                          --- [7]
   

   • [1] Set to true to enable and configure CFK-managed certificates at the Helm level.

     When set to true, CFK uses the settings under managedCerts: for auto-generated certificates.

     Enabling this setting rolls the clusters.

   • [2] CA certificate pair for auto-generated certificates in this CFK deployment. Use secretRef or directoryPathInContainer to provide the certificates.

   • [3] Set to the CA pair secret reference name.

     The expected keys are tls.crt and tls.key for the CA certificate and CA certificate key, respectively.

     When you use the Kubernetes secrets method to provide TLS certificates, CFK automates creating and configuring the keystore and truststore.

   • [4] Set to the path where CA pair certificates are mounted. The tls.crt and tls.key files must be present in the Vault directory path.

     When you use Vault directory path (directoryPathInContainer), CFK does not automate the creation of keystore and truststore. You need to create the keystore and truststore first.

     directoryPathInContainer overrides secretRef if both are defined.

   • [5] Set to the number of days for which the auto-generated certificates are valid. The default value is 60 (days).

     After CFK is deployed, you can update this setting at the CR level with an annotation. For details, see Manage auto-generated certificates.

   • [6] Set to the renewal time for auto-generated certificates. The default value is 30 (days).

     After CFK is deployed, you can update this setting at the CR level with an annotation. For details, see Manage auto-generated certificates.

   • [7] SANs to be added for all auto-generated certificates generated by this CFK. Use this only for adding wildcard SANs.

     Modifying this setting rolls all Confluent clusters and regenerates the certificates for all Confluent Platform clusters managed by CFK.

   For example:

   managedCerts:
     enabled: true
     caCertificate:
       secretRef: my-capair-secert
     renewBeforeInDays: 20
     certDurationInDays: 50
     sans: "*.global"
   

4. (Optional) Alternative to the previous step, you can pass the settings in the helm upgrade command. For example:

   helm upgrade --install confluent-operator ./confluent-for-kubernetes \
     --set managedCerts.caCertificate.secretRef=my-capair-secret \
     --set managedCerts.certDurationInDays=50 \
     --set managedCerts.renewBeforeInDays=20 \
     -n confluent
   

5. Configure each component’s custom resource (CR) to use auto-generated certificates:

   spec:
     tls:
       autoGeneratedCerts: true
   

CFK creates the required server certificates and stores them as Kubernetes secrets for Confluent components to use:

kubectl get secrets
NAME                             TYPE
...
zookeeper-generated-jks          kubernetes.io/tls
kafka-generated-jks              kubernetes.io/tls
...

The generated server certificates expire in 365 days for the default auto-generated certificates. The customized auto-generated certificates expire as set in certDurationInDays in the Helm value.

For a tutorial scenario on using auto-generated certs, see the quickstart tutorial.

Use the legacy CA pair for auto-generated certificates

If neither of the preceding two preferred options is configured, you can use the TLS secret, ca-pair-sslcerts, for auto-generated certificates.

1. Provide a root certificate authority as a Kubernetes Secret named ca-pair-sslcerts. Provide the certificate authority public and private key in the following format:

   kubectl create secret tls ca-pair-sslcerts \
     --cert=/path/to/ca.pem \
     --key=/path/to/ca-key.pem
   

2. Configure each component to use auto-generated certificates.

   spec:
     tls:
       autoGeneratedCerts: true
   

CFK creates the required server certificates and stores them as Kubernetes secrets for Confluent components to use. For example:

kubectl get secrets
NAME                             TYPE
...
zookeeper-generated-jks          kubernetes.io/tls
kafka-generated-jks              kubernetes.io/tls
...

Use user-provided TLS certificates

When you provide TLS certificates, CFK takes the provided files and configures Confluent components accordingly.

For each component, provide the following TLS certificate information:

• The CA for the component to trust, including the CAs used to issue server certificates for any Confluent component cluster

  CFK requires these so that peer-to-peer communication (for example, between Kafka brokers) and communication between components (for example, from Connect workers to Kafka) works.

• The component’s server certificate (public key)

• The component’s server private key

CFK supports the following TLS certificate formats:

TLS certificate format groups

Group	Files	Description
TLS Group 1: PEM-encoded files	fullchain.pem, privkey.pem, cacerts.pem	CFK automatically generates the key store and trust store from these PEM files.
TLS Group 2: Java KeyStore files	keystore.jks, truststore.jks, optional jksPassword.txt	CFK expects the .jks files in PKCS12 format. This group provides a direct trust store and key store, giving you more control over certificate alias naming.
TLS Group 3: PEM-encoded files	tls.crt, tls.key, ca.crt	Similar to TLS Group 1, but expects these specific file names. CFK converts these files into the required key store and trust store structures.

You can provide custom TLS certificates in the following formats.

• When using Kubernetes secrets for the certificates:

  • Use PEM-encoded files: TLS Group 1 or TLS Group 3.

  • Use Java KeyStore files: TLS Group 2.

• When using HashiCorp Vault directory path for the certificates:

  • Use Java KeyStore files: TLS Group 2.

Important

Use a unique TLS secret for each Confluent Platform component. With separate secrets, you have more control over restarts of the components when you update certificates. You can avoid potential startup failures due to a parallel restart.

Define SAN

The certificate’s SAN list must cover every hostname that clients use to access the Confluent component:

• If TLS for internal communication network encryption is enabled, include the internal network, <component>.<namespace>.svc.cluster.local, in the SAN list.

• If TLS for external network communication is enabled, include the external domain name in the SAN list.

The following are the internal and external SANs of each Confluent component that need to be included in the component certificate SAN. The examples use the default component prefixes.

Kafka

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

  • Example: kafka.confluent.svc.cluster.local

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of brokers, 0 to (number of brokers - 1).

  • Example: kafka-0.kafka.confluent.svc.cluster.local

  • The range can be handled through a wildcard domain, for example, *.kafka.confluent.svc.cluster.local.

• External bootstrap domain SAN: <bootstrap_prefix>.my-external-domain

  • Example: kafka-bootstrap.acme.com

• External broker SAN: <broker_prefix><x>.my-external-domain

  • Example: b0.acme.com

  • The range can be handled through a wildcard domain, for example, *.acme.com

MDS

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of brokers, 0 to (number of brokers - 1).

  • Example: kafka-0.kafka.confluent.svc.cluster.local

• External domain SAN: <mds_prefix>.my-external-domain

  • Example: mds.my-external-domain

KRaft

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of the KRaft controller, 0 to (number of servers - 1).

  • Example: kraftcontroller-0.kraftcontroller.confluent.svc.cluster.local

• External domain SAN: <kraftcontroller_prefix>.my-external-domain

ZooKeeper

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of ZooKeeper servers, 0 to (number of servers - 1).

  • Example: zookeeper-0.zookeeper.confluent.svc.cluster.local

• No external access domain

Important

Starting with Confluent Platform version 8.0, ZooKeeper is no longer part of Confluent Platform.

Schema Registry

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of Schema Registry servers, 0 to (number of servers - 1).

  • Example: schemaregistry-0.schemaregistry.confluent.svc.cluster.local

• External domain SAN: <schemaregistry_prefix>.my-external-domain

REST Proxy

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of REST Proxy servers, 0 to (number of servers - 1).

  • Example: kafkarestproxy-0.kafkarestproxy.confluent.svc.cluster.local

• External domain SAN: <kafkarestproxy_prefix>.my-external-domain

Connect

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

• Internal SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of Connect servers, 0 to (number of servers - 1).

  • Example: connect-0.connect.confluent.svc.cluster.local

• External domain SAN: <connect_prefix>.my-external-domain

ksqlDB

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

• Internal access SAN: <customResourceName>-<x>.<customResourceName>.<namespace>.svc.cluster.local

  <x> is the ordinal number of ksqlDB servers, 0 to (number of servers - 1).

  • Example: ksqldb-0.ksqldb.confluent.svc.cluster.local

• External domain SAN: <ksqldb_prefix>.my-external-domain

Control Center (Legacy)

• Internal bootstrap access SAN: <customResourceName>.<namespace>.svc.cluster.local

• Internal access SAN: <customResourceName>-0.<customResourceName>.<namespace>.svc.cluster.local

  • Example: controlcenter-0.controlcenter.confluent.svc.cluster.local

• External domain SAN: <controlcenter_prefix>.my-external-domain

For an example of how to create certificates with appropriate SAN configurations, see the Create your own certificates tutorial.

Provide TLS keys and certificates in PEM format

1. Prepare the following files:

   • ca.pem: This contains the list of CAs to trust, in PEM-encoded format. List the certificates by concatenating them, one after the other, for example:

     -----BEGIN CERTIFICATE-----
     ...
     -----END CERTIFICATE-----
     -----BEGIN CERTIFICATE-----
     ...
     -----END CERTIFICATE-----
     

   • server.pem: This contains the full server certificate chain in PEM-encoded format.

   • server-key.pem: This contains the PEM-encoded server certificate private key.

2. Create a Kubernetes secret with the following keys, referred to as TLS Group 1 in CFK:

   kubectl create secret generic kafka-tls \
   --from-file=fullchain.pem=server.pem \
   --from-file=cacerts.pem=ca.pem \
   --from-file=privkey.pem=server-key.pem
   

   Alternatively, you can create a Kubernetes secret with the following keys, referred to as TLS Group 3 in CFK:

   kubectl create secret generic kafka-tls \
   --from-file=tls.crt=server.pem \
   --from-file=ca.crt=ca.pem \
   --from-file=tls.key=server-key.pem
   

   The tls.crt, ca.crt, and tls.key keys are typically present in secrets created by cert-manager, a popular open source tool for managing certificates. CFK supports this convention for convenience. The expected file contents, and how Confluent Platform uses them, are identical whether you use the *.pem keys or the *.crt and *.key keys.

   Cluster Linking and Schema Exporter require their TLS secret key to be in the PKCS 8 key format. You can convert the key to PKCS8 with the following command:

   openssl pkcs8 -topk8 \
     -in <input_key> \
     -out <output_key_pkcs8> -nocrypt
   

3. Configure the component CR to use the secret:

   spec:
     tls:
       secretRef: kafka-tls
   

Provide TLS keys and certificates in Java KeyStore format

For the Confluent Platform component custom resources (CRs) that create statefulset pods, you can provide Java KeyStore formatted TLS keys and certificates using the CR API.

The application resource CRDs, such as the ClusterLink CRD and SchemaExporter CRD, don’t support Java KeyStore format keys and certificates directly.

To provide TLS keys and certificates for Confluent component CRs:

1. Prepare the following files:

   • keystore.jks: PKCS12 format keystore, containing the component server key.

   • truststore.jks: PKCS12 format truststore, containing the certificates to trust.

   • jksPassword.txt: Password that CFK uses for both JKS keystore and truststore.

     Create the jksPassword.txt file with jksPassword=<password_for_jks>:

     echo -n "jksPassword=<password_for_jks>" > jksPassword.txt
     

2. Create a Kubernetes secret with the following keys, referred to as TLS Group 2 in CFK:

   kubectl create secret generic kafka-tls \
   --from-file=keystore.jks=keystore.jks \
   --from-file=truststore.jks=truststore.jks \
   --from-file=jksPassword.txt=jksPassword.txt
   

3. Configure in the component CR:

   spec:
     tls:
       secretRef: kafka-tls
   

You can find an example script that generates JKS truststore and keystore in the CFK example GitHub repository. Don’t use the example script in a production environment.

Use separate TLS certificates for internal and external communications

When you enable TLS for a Confluent component, CFK uses the global certificate for internal and external traffic by default, as follows:

kind: <component>
spec:
  tls:
    secretRef: <certificate_secret>

For Kafka, ksqlDB, Schema Registry, MDS, and Kafka REST services, you have an option to configure the components and services to use separate TLS certificates for internal and external listeners.

After you configure listeners as described next, review the spec.dependencies section in other Confluent Platform component CRs. If another Confluent Platform component depends on the component you updated to use separate certificates, provide a matching TLS secret and endpoint on that dependent component to establish the connection successfully.

Provide a TLS certificate for the internal listener

Configure Kafka, ksqlDB, Schema Registry, MDS, or Kafka REST service to use separate TLS certificates for its internal listener.

1. For Kafka, ksqlDB or Schema Registry, specify the certificate in the internal listener section in its CR:

   kind: <component>
   spec:
     listeners:
       internal:
         tls:
           enabled: true
           secretRef: <internal_tls_cert>
   

2. For MDS or Kafka REST service, specify the certificate in the internal listener section in Kafka CR:

   kind: Kafka
   spec:
     services:
       mds or kafkaRest:
         listeners:
           internal:
             tls:
               enabled: true
               secretRef: <internal_tls_cert>
   

3. You can configure the external listener to use the global TLS certificate or its own external listener certificate.

   The global certificates are specified in:

   • spec.tls.secretRef for ksqlDB and Schema Registry

   • spec.services.<mds_or_kafka_rest>.tls.secretRef for MDS and Kafka REST services

Provide a TLS certificate for the external listener

Configure Kafka, ksqlDB, Schema Registry, MDS, or Kafka REST service to use separate TLS certificates for its external listener.

1. For Kafka, ksqlDB and Schema Registry, specify the certificate in the external listener section in its CR:

   kind: <component>
   spec:
     listeners:
       external:
         tls:
           enabled: true
           secretRef: <external_tls_cert>
   

2. For MDS or Kafka REST service, specify the certificate in the external listener section in Kafka CR:

   kind: Kafka
   spec:
     services:
       mds or kafkaRest:
         listeners:
           external:
             tls:
               enabled: true
               secretRef: <external_tls_cert>
   

3. You can configure the internal listener to use the global TLS certificate or its own internal listener certificate.

   The global certificates are specified in:

   • spec.tls.secretRef for ksqlDB and Schema Registry

   • spec.services.<mds_or_kafka_rest>.tls.secretRef for MDS and Kafka REST services

Enable dynamic Kafka certificate update

You can enable dynamic TLS certificate update for Kafka listeners and the Kafka REST service so that the Kafka cluster doesn’t roll when certificates change.

This feature is only supported:

• With certificates configured with Kubernetes secret (secretRef)

• With PEM certificates

• At the individual Kafka listener level

  You can’t use global, autogenerated, or other component certificates for dynamic certificate update.

  The following are example snippets of the supported scenarios:

  spec:
    listeners:
      internal:
        authentication:
          type: mtls
        tls:
          enabled: true
          secretRef: tls-certs
  

  services:
    kafkaRest:
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: tls-certs-rest
  

To enable dynamic certificate update:

1. Configure the password encoder secret for Kafka as described in Manage Password Encoder Secrets.

   This is a requirement to enable dynamic certificate update.

2. Create the _confluent-operator topic as described in Manage Topics.

   The following is an example KafkaTopic CR:

   apiVersion: platform.confluent.io/v1beta1
   kind: KafkaTopic
   metadata:
     name: confluent-operator
     namespace: operator
   spec:
     name: _confluent-operator
     kafkaRestClassRef:
       name: kafkarestclass
   

3. Apply the annotations to enable dynamic certificate update and to force reconcile:

   kubectl annotate kafka kafka platform.confluent.io/enable-dynamic-configs="true"
   

   kubectl annotate kafka kafka platform.confluent.io/force-reconcile="true"
   

Kafka pod rolls the first time the feature is enabled.

Disable SNI validation

If you need to run clients without SNI validation in Confluent Platform 8.x, set the specified property using configuration overrides in the component custom resource, as shown for each component below. Starting in Confluent Platform 8.0, CFK enables Server Name Indication (SNI) validation by default, and clients that don’t pass SNI headers can’t communicate with Confluent Platform components. CFK doesn’t enable SNI validation by default in Confluent Platform 7.x.

• Kafka

  kind: Kafka
  spec:
    configOverrides:
      server:
        - confluent.http.server.sni.host.check.enabled=false
  

• MDS

  kind: Kafka
  spec:
    configOverrides:
      server:
        - confluent.metadata.server.sni.host.check.enabled=false
  

• Control Center

  kind: ControlCenter
  spec:
    configOverrides:
      server:
        - confluent.controlcenter.rest.sni.host.check.enabled=false
  

• REST Proxy

  kind: KafkaRestProxy
  spec:
    configOverrides:
      server:
        - confluent.http.server.sni.host.check.enabled=false
  

• Schema Registry, Connect, any other Confluent Platform components

  kind: <component>
  spec:
    configOverrides:
      server:
        - sni.host.check.enabled=false
  

Disable TLS hostname verification for the Kafka internal client

Starting in CFK 3.4.0, you can disable TLS hostname verification for the CFK operator’s internal Kafka client. CFK uses this client for internal operator-to-broker communication over the Kafka internal listener, including the dynamic configuration workflow.

By default, this internal client verifies that the internal listener certificate’s SAN includes the Kubernetes internal DNS name of the broker it connects to. If your organization’s security policy does not allow Kubernetes internal domain names in certificate SANs, this verification fails and the operator cannot communicate with the internal listener.

To disable hostname verification for the operator’s internal client, set endpointIdentificationAlgorithm to none under spec.internalClientConfig in the Kafka CR:

kind: Kafka
spec:
  internalClientConfig:
    endpointIdentificationAlgorithm: none

The default value is https, which keeps hostname verification enabled and preserves existing behavior for clusters that do not set this field. Certificate chain validation always applies. This setting affects only hostname matching.

Consider the following when you set this field:

• Changing internalClientConfig alone does not trigger a rolling restart. You must roll or restart already-running Kafka pods before they pick up the new value.

• This field affects only the operator’s own internal client. It does not change TLS verification behavior for the Kafka broker’s own internal clients, such as its bootstrap AdminClient and its metrics and telemetry reporters, which enforce hostname verification independently.

  If your internal listener certificate does not have the Kubernetes internal SAN, setting internalClientConfig.endpointIdentificationAlgorithm alone is not enough to bring up the cluster. You must also disable hostname verification for the broker’s own internal clients using configuration overrides:

  kind: Kafka
  spec:
    configOverrides:
      server:
        - ssl.endpoint.identification.algorithm
  

  Unlike internalClientConfig, changing configOverrides.server triggers the rolling restart needed to apply the change.

Configure Client-Side Field Level Encryption

Client-Side Field Level Encryption (CSFLE) encrypts and decrypts sensitive data fields at the client level before sending the data to Kafka. Your client application performs the actual encryption and decryption. Confluent Platform, including deployment through Confluent for Kubernetes (CFK), only handles key management and schema configuration. It doesn’t perform encryption or decryption itself.

For an overview of CSFLE in Confluent Platform, see Client-Side Field Level Encryption Overview.

For detailed CSFLE configuration steps, see Use Client-Side Field Level Encryption on Confluent Platform.

For example CSFLE configurations in Confluent Platform, see the Client-Side Field Level Encryption example configuration.

CSFLE requirements and considerations

• Confluent Platform 8.0 or later with the CSFLE add-on enabled.

• CSFLE in Confluent Platform is supported only in non-shared mode.

  • Schema Registry supports CSFLE mainly for data encryption key (DEK) permissions checks.

  • The client performs all key management service (KMS) interactions, including encryption and decryption.

• CSFLE supports Java clients for producing and consuming encrypted messages.

• CSFLE is not available in Confluent CLI, Connect, ksqlDB, Apache Flink®, or non-Java clients at this time.

• CSFLE is not integrated with Control Center.

• Only string and byte type Apache Avro® fields are supported for CSFLE tagging and encryption.

• The kafka-avro-console-producer and kafka-avro-console-consumer tools work with CSFLE.

• Supported KMS types include a local key encryption key (KEK), AWS KMS, and HashiCorp Vault. For the full list, see Supported KMS types.

• The CSFLE API is protected using Confluent Platform role-based access control (RBAC).

Configure CSFLE in CFK

To deploy CSFLE with CFK:

1. Deploy Confluent Platform components.

   Specifically, configure Schema Registry with the required configuration for CSFLE using the configOverrides.server property in the SchemaRegistry custom resource (CR) YAML.

   For example:

   kind: SchemaRegistry
   spec:
     replicas: 1
     image:
       application: confluentinc/cp-schema-registry
       init: confluentinc/confluent-init-container
     configOverrides:
       server:
       - resource.extension.class=io.confluent.kafka.schemaregistry.rulehandler.RuleSetResourceExtension,io.confluent.dekregistry.DekRegistryResourceExtension
       - confluent.license.addon.csfle=<cp-enterprise-license-key>
   

   The value for confluent.license.addon.csfle is the same as your main Confluent Platform Enterprise license key.

2. Grant necessary RBAC permissions for users and key resources (topics, subjects, and KEKs). For example:

   • Give the ResourceOwner role to Schema Registry’s internal Kafka client for the _dek_registry_keys topic.

   • Grant user roles for topics, subjects, and KEK resources.

     You can use the ConfluentRolebinding CR or the confluent iam rbac role-binding create command. For example:

     confluent iam rbac role-binding create \
       --principal User:sr \
       --role ResourceOwner \
       --resource Topic:_dek_registry_keys \
       --kafka-cluster <cluster_id>
     

   • For enabling RBAC for DEK Registry, see Access control (RBAC) for CSFLE.

3. Register Schemas and RuleSets using the REST API, tagging fields for encryption.

   • Define the schema for the topic and add tags to the schema fields in the schema that you want to encrypt.

   • Define an encryption policy that specifies rules to use to encrypt the tags.

4. Register the KEK resource using the Schema Registry REST API or the register-deks command.

   An example JSON payload for the REST API:

   {
     "name": "my-kek",
     "kmsType": "local-kms",
     "kmsKeyId": "mykey",
     "shared": false
   }
   

   For details on managing CSFLE keys, see Manage CSFLE keys.

5. Configure the Java client with KMS credentials or local secret, and produce messages with encrypted fields.

   • Clients must be configured to provide the secret or KMS credentials at runtime. For local KEK, this is usually a base64 string; for AWS KMS, environment variables for credentials must be set.

     Example Java client properties:

     props.put("rule.executors._default_.param.secret", "pgbju8SjcaWJOtTSgeBckA==");
     props.put("schema.registry.basic.auth.user.info", "testadmin:testadmin");
     

   • Only fields of type string or byte with the correct tag are supported for encryption.

   • When a message is produced with CSFLE, the tagged field is encrypted using the configured KEK.

   • Consumers must provide the correct key or credential to decrypt and read the tagged field.

Troubleshooting

• Encryption and decryption problems: Check client logs and Schema Registry logs to help with DEK permissions. Most issues involve missing or incorrect KEK or client KMS setup.

• RBAC issues: Create the correct role-bindings for all required resources (topics, subjects, KEKs).