Reconfigure ZooKeeper TLS

This topic describes the processes of enabling and disabling ZooKeeper TLS after it is deployed.

Enable ZooKeeper client-to-server TLS encryption and authentication

The process of enabling ZooKeeper client-to-server TLS involves turning on both a secure port and a non-secure port until the change is fully rolled out.

Before turning on TLS, your original inventory would have:

all:
  vars:
    zookeeper_ssl_enabled: false
    zookeeper_client_authentication_type: none
    zookeeper_quorum_authentication_type: none

To enable client-to-server TLS:

Generate keystores for Kafka broker so that it can communicate to the TLS-enabled ZooKeeper:

ansible-playbook -i hosts.yml playbooks/all.yml \
  --e zookeeper_ssl_enabled=true \
  --tags kafka_broker,ssl

Set the following properties in your inventory to:

Turn on ZooKeeper TLS encryption.
Turn on client mutual TLS (mTLS) authentication.
Enable a non-secure ZooKeeper client port.

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: none
    deployment_strategy: rolling
    zookeeper_custom_properties:
      clientPort: 2181

Roll out the above settings:

ansible-playbook -i hosts.yml playbooks/all.yml \
  --tags kafka_broker,zookeeper \
  --skip-tags package

This will keep a non-secure port open on ZooKeeper, while the ZooKeeper reconfiguration is running, and then update Kafka to use the secure port.

Now that Kafka is on the secure connection, remove the non-secure client port, clientPort:

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: none
    deployment_strategy: rolling

Roll out the above settings:

ansible-playbook -i hosts.yml playbooks/all.yml \
  --tags kafka_broker,zookeeper \
  --skip-tags package

(Optional) If you intend to set zookeeper.set.acl=true in Kafka’s server.properties, run the following script to add ACLs on ZooKeeper node:

/usr/bin/zookeeper-security-migration \
  --zookeeper.acl=secure \
  --zookeeper.connect=zookeeper1:2182 \
  --zk-tls-config-file /etc/kafka/server.properties

Enable ZooKeeper server-to-server TLS encryption and authentication

Enabling ZooKeeper server-to-server encryption and authentication involves a similar process of multiple ZooKeeper cluster rolls.

Enable server-to-server mTLS authentication, but using custom properties, don’t enforce mTLS (sslQuorum: "false") and allow both TLS and non-TLS connection on the same port (portUnification: "true").

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls
    deployment_strategy: rolling
    zookeeper_custom_properties:
      portUnification: "true"
      sslQuorum: "false"

Roll out the above settings:

ansible-playbook -i hosts.yml playbooks/all.yml --tags zookeeper --skip-tags package

Turn on mTLS by removing the sslQuorum custom property. The port unification is necessary because, as this change rolls out, some ZooKeeper nodes will be over TLS and some not:

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls
    deployment_strategy: rolling
    zookeeper_custom_properties:
      portUnification: "true"

Roll out the above settings:

ansible-playbook -i hosts.yml playbooks/all.yml --tags zookeeper --skip-tags package

After the previous step completes, all ZooKeeper nodes will be using mTLS for their connection, and there is no need for port unification. Remove the portUnification: "true" custom property and do one more roll out.

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls
    deployment_strategy: rolling

Roll out the above settings:

ansible-playbook -i hosts.yml playbooks/all.yml --tags zookeeper --skip-tags package

Disable ZooKeeper TLS encryption and authentication

This section describes how to disable TLS from both server-to-server and client-to-server connections.

Before turning off TLS, your original inventory would have:

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls

(Optional) If zookeeper.set.acl=true is set in Kafka’s server.properties, run the following script to remove ACLs on the ZooKeeper node. Also set zookeeper.set.acl: false in your kafka_broker_custom_properties:
```
/usr/bin/zookeeper-security-migration --zookeeper.acl=unsecure \
  --zookeeper.connect=zookeeper1:2182 \
  --zk-tls-config-file /etc/kafka/server.properties
```

Update your inventory to:

Add a non-secure client port to ZooKeeper
Tell Kafka that ZooKeeper TLS is disabled.
Add ZooKeeper quorum port unification, meaning server-to-server connections can be either TLS or non-TLS:

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls
    deployment_strategy: rolling
    zookeeper_custom_properties:
      clientPort: 2181
      portUnification: "true"

kafka_broker:
  vars:
    zookeeper_ssl_enabled: false
    zookeeper_client_authentication_type: none

Roll out the previous settings:

ansible-playbook -i hosts.yml playbooks/all.yml \
  --tags kafka_broker,zookeeper \
  --skip-tags package

Stop enforcing ZooKeeper server-to-server SSL quorum:

all:
  vars:
    zookeeper_ssl_enabled: true
    zookeeper_client_authentication_type: mtls
    zookeeper_quorum_authentication_type: mtls
    deployment_strategy: rolling
    zookeeper_custom_properties:
      clientPort: 2181
      portUnification: "true"
      sslQuorum: "false"
kafka_broker:
  vars:
    zookeeper_ssl_enabled: false
    zookeeper_client_authentication_type: none

Roll out the previous settings:

ansible-playbook -i hosts.yml playbooks/all.yml \
  --tags kafka_broker,zookeeper \
  --skip-tags package

At this point, ZooKeeper’s client (Kafka) and ZooKeeper are using non-encrypted traffic, so you can remove the custom properties and do one more roll out:

all:
  vars:
    zookeeper_ssl_enabled: false
    zookeeper_client_authentication_type: none
    zookeeper_quorum_authentication_type: none
    deployment_strategy: rolling

ansible-playbook -i hosts.yml playbooks/all.yml \
  --tags kafka_broker,zookeeper \
  --skip-tags package