Manage TLS Certificates for Confluent Platform Using Confluent for Kubernetes

This topic describes how to manage Transport Layer Security (TLS) certificates for Confluent Platform using Confluent for Kubernetes (CFK).

Requirements and considerations

  • Plan rotation before certificate expiry.

  • There should be an overlap window between new and old certificates during the renewal process.

  • Test the rotation process first in a lower-tier, non-production environment.

Rotate user-provided server certificates

In CFK, to rotate user-provided certificates that have been provided as Kubernetes secrets, you can update the contents of the Kubernetes secrets.

For example, to rotate the server certificates that you provided in .pem files:

  1. Update the server.pem certificate and server-key.pem certificate private key.

  2. Update the Kubernetes secret using the following command:

    kubectl create secret generic component-tls \
    --from-file=fullchain.pem=server.pem \
    --from-file=cacerts.pem=ca.pem \
    --from-file=privkey.pem=server-key.pem \
    --save-config --dry-run=client -oyaml | \
    kubectl apply -f -
    

CFK will watch for the Kubernetes secret changes, will notice that the content has changed, and will then perform a rolling restart of the component to configure the new server certificate.

Note that CFK only watches the secrets provided in the .pem format. This feature is not supported for the secrets provided in the .jks format.

Rotate certificate authority for user-provided certificates

You can rotate the certificate authority (CA) that has been provided as Kubernetes secrets using the steps described in this section.

The examples use the .pem files for providing certificates.

Step 1. Append new certificate authority

  1. Append the new intermediate/root CA to the ca.pem file. It will co-exist with the old intermediate/root CA.

    file name: ca.pem
    content:
    -----BEGIN CERTIFICATE-----
    <old root CA certificate>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <new root CA certificate>
    -----END CERTIFICATE-----
    
  2. Update the secret:

    kubectl create secret generic component-tls \
    --from-file=fullchain.pem=server.pem \
    --from-file=cacerts.pem=ca.pem \
    --from-file=privkey.pem=server-key.pem \
    --save-config --dry-run=client -oyaml | \
    kubectl apply -f -
    

CFK will watch for the Kubernetes secret changes, notice that the content has changed, and then perform a rolling restart for the component to configure the new root CA for use alongside the old root CA.

If this CA is for Kafka and the dynamic loading is enabled, the cluster will not roll.

Step 2. Generate server certificates with new certificate authority

  1. Generate the new server certificates with the new CA, and replace the server.pem certificate and the server-key.pem certificate private key.

  2. Update the Kubernetes secret:

    kubectl create secret generic component-tls \
    --from-file=fullchain.pem=server.pem \
    --from-file=cacerts.pem=ca.pem \
    --from-file=privkey.pem=server-key.pem \
    --save-config --dry-run=client -oyaml | \
    kubectl apply -f -
    

CFK will watch for the Kubernetes secret changes, will notice that the content has changed, and will then rolling restart the component to configure the new server certificate.

If this certificate is for Kafka and the dynamic loading is enabled, the cluster will not roll.

Rotate certificate authority used for auto-generated certificates

In CFK, you can provide a certificate authority (CA) for Confluent Platform components to use for auto-generated certificates.

CA certificate rotation is a critical security operation that replaces an existing CA with a new one while maintaining service availability.

Follow the steps described in this section to renew the truststore and auto-generated certificates. This process maintains service availability throughout without downtime.

Wait for each step to complete before moving to the next step.

Important considerations

  • Do not apply both force-managed-truststore-renewal and force-managed-cert-renewal annotations simultaneously.

  • Always renew truststores for all components before renewing certificates.

  • Wait for each component’s rollout to complete before proceeding.

  • Plan rotation before certificate expiry.

  • Securely store and manage CA private keys.

  • Test the rotation process in a non-production environment first.

Step 1. Update CFK configuration

  1. Create CA certificates.

    Create your old (if it does not already exist) and new CA certificates.

  2. Create Kubernetes or Vault secrets for the CA certificates.

    For example:

    # Create secret for new CA
    kubectl -n <namespace> create secret tls ca-pair-sslcerts-new \
      --cert=new/ca.pem --key=new/ca-key.pem
    
  3. Update your CFK configuration to include both old and new CA certificates.

    caCertificate and oldCaCertificate support both secretRef (Kubernetes secret) and directoryPathInContainer (Vault directory path). When both are provided, directoryPathInContainer will be used.

    For example:

    managedCerts:
      enabled: true
      caCertificate:
        secretRef: ca-pair-sslcerts-new  # Your new CA secret
      oldCaCertificate:
        secretRef: ca-pair-sslcerts      # Your old CA secret
    
  4. Apply the configuration.

    helm upgrade --install -f values.yaml operator confluent-for-kubernetes \
      --namespace <namespace>
    

Step 2. Renew truststore

  1. Apply the truststore renewal annotation to each Confluent Platform component.

    kubectl annotate <Component CR type> <Component CR name> platform.confluent.io/force-managed-truststore-renewal="true"
    
  2. Verify that all components truststores now have dual CAs, both old and new CA.

    kubectl exec <pod-name> -- keytool -list \
      -keystore /mnt/sslcerts/truststore.jks \
      -storepass mystorepassword | grep -c "Certificate fingerprint"
    

    You should see two certificates in each truststore.

Step 3. Renew certificates

  1. Apply the certificate renewal annotation to each Confluent Platform component.

    kubectl annotate <Component CR type> <Component CR name> platform.confluent.io/force-managed-cert-renewal="true"
    
  2. Verify that all certificates are now signed by the new CA:

    kubectl exec <pod-name> -- openssl x509 -in /mnt/sslcerts/tls.crt -text -noout | grep "Issuer:"
    

    The issuer should show your new CA.

Rotate LDAP server certificates used by MDS

If you use LDAP or LDAPS for role-based access control (RBAC) with Metadata Service (MDS), the Kafka brokers trust the LDAP server certificate through the LDAP truststore. Use the guidance in this section to rotate the LDAP server certificate or its CA without breaking authentication to MDS.

For more information about configuring MDS with LDAP, see Configure authentication to access MDS.

Renew the LDAP server certificate under the same CA

If the broker truststore contains the issuing or root CA chain, renewing the LDAP server leaf certificate shouldn’t require a truststore change.

Trust the CA chain instead of pinning the LDAP leaf certificate.

If the truststore contains only the current LDAP server certificate, the brokers don’t trust a renewed certificate and MDS authentication fails. In that case, add the CA chain to the truststore secret and wait for the roll to finish before you renew the certificate. For more details, see Rotate the LDAP CA.

Rotate the LDAP CA

To rotate the CA that issues the LDAP server certificate, keep the old and new CAs trusted during the transition:

  1. Update the LDAP truststore secret to contain both the current CA certificate and the new CA certificate.

    Don’t change the LDAP server certificate yet.

  2. Wait for CFK to roll the brokers and load the updated truststore. Keep the LDAP server on the old certificate while the roll runs.

  3. Confirm the roll is healthy. Check that all broker pods restarted and are ready.

  4. Switch the LDAP server to the certificate issued by the new CA.

    The certificate’s subject alternative names must include the hostname in the LDAP address configured for MDS. MDS verifies the hostname, and a mismatch fails authentication on every broker.

  5. In a later, separate change, remove the old CA certificate from the LDAP truststore secret and roll the brokers again.

    Don’t combine this step with the previous steps.

Manage auto-generated certificates

To manage auto-generated certificates in Confluent for Kubernetes (CFK), use the following annotations in the Confluent Platform component CRs.

  • Force a renewal of auto-generated certificates:

    kubectl annotate <Component CR type> <Component CR name> \
      platform.confluent.io/force-managed-cert-renewal="true"
    

    The annotation triggers a reconciliation on the Confluent Platform component. The process causes the certificates to be renewed, and the component to be rolled.

    After the certificates are renewed, the annotation is reset to false.

  • Add additional custom SAN for auto-generated certificates. This will be added to SANs defined at the global level in managedCerts.sans in the CFK Helm values (in values.yaml).

    kubectl annotate secret ca-pair-sslcerts \
      platform.confluent.io/managed-cert-add-sans="<Comma separated list of SANs>"
    

    For example, the following annotation will add the SANs, kafka.operator, kafka.operator.local, to auto-generated certificates for Kafka:

    kubectl annotate secret ca-pair-sslcerts \
      platform.confluent.io/managed-cert-add-sans="kafka.operator, kafka.operator.local"
    

    After the annotation is set, use the platform.confluent.io/force-managed-cert-renewal="true" annotation to force a renewal of certificates. If not forced, SANs will be updated during the next reconciliation.

  • Specify the Kubernetes secret name for the CA certificates at the CR level. This will override the global config defined in managedCerts.caCertificate in the CFK Helm values (in values.yaml). This can be used even if managedCerts.enable is set to false.

    kubectl annotate <Component CR type> <Component CR name> \
      platform.confluent.io/managed-cert-ca-pair-secret=<Secret name>
    

    The secret should have tls.crt and tls.key with the CA certificate and key.

  • Configure the optional settings for auto-generated certificates at the CR level:

    kubectl annotate <Component CR type> <Component CR name> \
      platform.confluent.io/managed-cert-duration-in-days=<Number of days>
    
    kubectl annotate <Component CR type> <Component CR name> \
      platform.confluent.io/managed-cert-renew-before-in-days=<Number of days>
    

    For example:

    kubectl annotate kafka kafka platform.confluent.io/managed-cert-duration-in-days=60
    
    kubectl annotate kafka kafka platform.confluent.io/managed-cert-renew-before-in-days=30
    

    After the annotations are set, use the platform.confluent.io/force-managed-cert-renewal="true" to force a renewal of certificates.