Manage ZooKeeper TLS in a Running Confluent Platform Environment Using Ansible Playbooks

This topic describes the processes of enabling and disabling ZooKeeper TLS after it is deployed.

Enable ZooKeeper client-to-server TLS encryption and authentication

The process of enabling ZooKeeper client-to-server TLS involves turning on both a secure port and a non-secure port until the change is fully rolled out.

Before turning on TLS, your original inventory would have:

all:
  vars:
    zookeeper_ssl_enabled: false
    zookeeper_client_authentication_type: none
    zookeeper_quorum_authentication_type: none

To enable client-to-server TLS:

1. Generate keystores for Kafka broker so that it can communicate to the TLS-enabled ZooKeeper:

   ansible-playbook -i hosts.yml playbooks/all.yml \
     --e zookeeper_ssl_enabled=true \
     --tags kafka_broker,ssl \
     --skip-tags configuration,package
   

2. Set the following properties in your inventory to:

   • Turn on ZooKeeper TLS encryption.
   • Turn on client mutual TLS (mTLS) authentication.
   • Enable a non-secure ZooKeeper client port.

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: none
       deployment_strategy: rolling
       zookeeper_custom_properties:
         clientPort: 2181
   

   Roll out the above settings:

   ansible-playbook -i hosts.yml playbooks/all.yml \
     --tags kafka_broker,zookeeper \
     --skip-tags package
   

   This will keep a non-secure port open on ZooKeeper, while the ZooKeeper reconfiguration is running, and then update Kafka to use the secure port.

3. Now that Kafka is on the secure connection, remove the non-secure client port, clientPort:

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: none
       deployment_strategy: rolling
   

   Roll out the above settings:

   ansible-playbook -i hosts.yml playbooks/all.yml \
     --tags kafka_broker,zookeeper \
     --skip-tags package
   

4. (Optional) If you intend to set zookeeper.set.acl=true in Kafka’s server.properties, run the following script to add ACLs on ZooKeeper node:

   /usr/bin/zookeeper-security-migration \
     --zookeeper.acl=secure \
     --zookeeper.connect=zookeeper1:2182 \
     --zk-tls-config-file /etc/kafka/server.properties
   

Enable ZooKeeper server-to-server TLS encryption and authentication

Enabling ZooKeeper server-to-server encryption and authentication involves a similar process of multiple ZooKeeper cluster rolls.

1. Enable server-to-server mTLS authentication, but using custom properties, don’t enforce mTLS (sslQuorum: "false") and allow both TLS and non-TLS connection on the same port (portUnification: "true").

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: mtls
       deployment_strategy: rolling
       zookeeper_custom_properties:
         portUnification: "true"
         sslQuorum: "false"
   

   Roll out the above settings:

   ansible-playbook -i hosts.yml playbooks/all.yml --tags zookeeper --skip-tags package
   

2. Turn on mTLS by removing the sslQuorum custom property. The port unification is necessary because, as this change rolls out, some ZooKeeper nodes will be over TLS and some not:

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: mtls
       deployment_strategy: rolling
       zookeeper_custom_properties:
         portUnification: "true"
   

   Roll out the above settings:

   ansible-playbook -i hosts.yml playbooks/all.yml --tags zookeeper --skip-tags package
   

3. After the previous step completes, all ZooKeeper nodes will be using mTLS for their connection, and there is no need for port unification. Remove the portUnification: "true" custom property and do one more roll out.

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: mtls
       deployment_strategy: rolling
   

   Roll out the above settings:

   ansible-playbook -i hosts.yml playbooks/all.yml --tags zookeeper --skip-tags package
   

Disable ZooKeeper TLS encryption and authentication

This section describes how to disable TLS from both server-to-server and client-to-server connections.

Before turning off TLS, your original inventory would have:

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls

1. (Optional) If zookeeper.set.acl=true is set in Kafka’s server.properties, run the following script to remove ACLs on the ZooKeeper node. Also set zookeeper.set.acl: false in your kafka_broker_custom_properties:

   /usr/bin/zookeeper-security-migration --zookeeper.acl=unsecure \
     --zookeeper.connect=zookeeper1:2182 \
     --zk-tls-config-file /etc/kafka/server.properties
   

2. Update your inventory to:

   • Add a non-secure client port to ZooKeeper
   • Tell Kafka that ZooKeeper TLS is disabled.
   • Add ZooKeeper quorum port unification, meaning server-to-server connections can be either TLS or non-TLS:

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: mtls
       deployment_strategy: rolling
       zookeeper_custom_properties:
         clientPort: 2181
         portUnification: "true"
   
   kafka_broker:
     vars:
       zookeeper_ssl_enabled: false
       zookeeper_client_authentication_type: none
   

   Roll out the previous settings:

   ansible-playbook -i hosts.yml playbooks/all.yml \
     --tags kafka_broker,zookeeper \
     --skip-tags package
   

3. Stop enforcing ZooKeeper server-to-server SSL quorum:

   all:
     vars:
       zookeeper_ssl_enabled: true
       zookeeper_client_authentication_type: mtls
       zookeeper_quorum_authentication_type: mtls
       deployment_strategy: rolling
       zookeeper_custom_properties:
         clientPort: 2181
         portUnification: "true"
         sslQuorum: "false"
   kafka_broker:
     vars:
       zookeeper_ssl_enabled: false
       zookeeper_client_authentication_type: none
   

   Roll out the previous settings:

   ansible-playbook -i hosts.yml playbooks/all.yml \
     --tags kafka_broker,zookeeper \
     --skip-tags package
   

4. At this point, ZooKeeper’s client (Kafka) and ZooKeeper are using non-encrypted traffic, so you can remove the custom properties and do one more roll out:

   all:
     vars:
       zookeeper_ssl_enabled: false
       zookeeper_client_authentication_type: none
       zookeeper_quorum_authentication_type: none
       deployment_strategy: rolling
   

   ansible-playbook -i hosts.yml playbooks/all.yml \
     --tags kafka_broker,zookeeper \
     --skip-tags package