Upgrade Confluent Platform with Ansible Playbooks

Ansible Playbooks for Confluent Platform (Confluent Ansible) can upgrade the Confluent Platform components. To safely upgrade your hosts, make use of the Rolling Deployment Strategy, which will go host by host, shutting down the component, upgrading packages, restarting the service, and validating service health before moving onto the next one.

You must complete the upgrades in the following order:

  1. Upgrade ZooKeeper.
  2. Upgrade Kafka brokers.
  3. Upgrade (in any order):
    • Schema Registry
    • REST Proxy
    • Connect
  4. Upgrade Confluent Control Center.
  5. Upgrade external clients.
  6. Upgrade Kafka log format. This step ensures that the log is formatted properly for the new version of Confluent Platform after all upgrades have been completed.

Confluent Platform does not guarantee the clusters and Confluent Platform components in different major versions will be functional. An upgrade process is complete when all the components are upgraded.

Requirements

The upgrade playbooks have the following requirements:

  • Confluent Platform components must have been originally installed and configured using Ansible Playbooks for Confluent Platform.
  • You must have the same hosts.yml file used during the installation.

Upgrade notes

Before you start the upgrade process, review the following changes and make any necessary updates.

  • SASL/SCRAM default version

    The default SASL/SCRAM version was changed from 256 to 512.

    If the version of SSL/SCRAM is specified as 256 in your server.properties, you must update your inventory and change sasl_protocol: scram to sasl_protocol: scram256.

  • ZooKeeper TLS

    TLS is now enabled by default for ZooKeeper when ssl_enabled: true is set.

    If your current ZooKeeper deployment does not use TLS, set zookeeper_ssl_enabled: false in your inventory.

  • ZooKeeper server-to-server TLS

    Improved logic and variables were added around ZooKeeper server to server authentication. If you have sslQuorum=true in your zookeeper.properties, set zookeeper_quorum_authentication_type: mtls in your inventory.

  • Enable Admin REST APIs

    When upgrading from 5.5.x to 6.2.x, you must enable Admin REST APIs by setting the following property in your inventory file. If Admin REST APIs is not enabled, component upgrades will fail:

    kafka_broker_rest_proxy_enabled: true

  • Disable canonicalization

    If canonicalization has not been enabled during the Confluent Platform cluster creation, explicitly set the following property in the hosts.yml inventory file. An example scenario is when you upgrade a Confluent Platform cluster that uses Kerberos to authenticate Kafka brokers to ZooKeeper.

    kerberos:
  canonicalize: false

  • Variable name updates in hosts.yaml

    Misspelled variable names were corrected in the 7.2.2 version.

    If upgrading from a version, earlier than 7.2.2, to a version, 7.2.2 or later, make the following updates in your inventory file:

    • From: kakfa_connect_replicator_<property_name>
    • To: kafka_connect_replicator_<property_name>

  • ZooKeeper version dependency

    The ZooKeeper dependency has been upgraded to 3.8.1 due to 3.6 reaching end-of-life. To bring both your Kafka and ZooKeeper clusters to the latest versions:

    (Case A) If upgrading from a Kafka version 2.4 (Confluent Platform version 5.4) or later:

    1. Kafka clusters can be updated directly.
    2. ZooKeeper clusters that are running binaries bundled with Kafka versions 2.4 or later can be updated normally as specified in the following sections in this document.

    (Case B) If upgrading from a Kafka version older than 2.4 (Confluent Platform version older than 5.4):

    1. Kafka clusters first need to be upgraded to a version 2.4 or later, and earlier than 3.6.
    2. ZooKeeper clusters that are running binaries bundled with a Kafka version that are older than 2.4 need to be updated to the binaries bundled with Kafka versions later than 2.4 and earlier than 3.6.
    3. You could then continue following the steps in Case A above.

Upgrade Confluent Platform

Step 1. Download Ansible Playbooks for Confluent Platform

Download the Confluent Ansible for the target version of Confluent Platform that you are upgrading to:

ansible-galaxy collection install confluent.platform:<version>

For example, to upgrade to Confluent Platform 7.3.11:

ansible-galaxy collection install confluent.platform:7.3.11

To upgrade to the latest version of Confluent Platform, you can omit <version>:

ansible-galaxy collection install --upgrade confluent.platform

Step 2. Set Rolling Deployment strategy

In order to avoid component outages, set the Deployment Strategy to rolling as below:

deployment_strategy: rolling

Step 3. Upgrade ZooKeeper

To upgrade ZooKeeper, run the provisioning playbook with the zookeeper tag:

ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags zookeeper

Step 4. Upgrade Kafka

Upgrading Kafka takes a few steps because you need to apply extra care to the inter.broker.protocol.version and log.message.format.version properties.

  1. Enter the following command and note the current version of confluent-kafka or confluent-server installed on your hosts.

    On Red Hat hosts:

    rpm -qa | grep confluent

    An example output:

    confluent-server-6.0.1-1.noarch

    On Debian hosts:

    apt list --installed confluent-server

  2. Set the version properties.

    • For 6.0.x Kafka package version, set these custom properties in your inventory:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.6
  log.message.format.version: 2.6

    • For 6.1.x Kafka package version, set these custom properties in your inventory:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.7
  log.message.format.version: 2.7

    • For 6.2.x Kafka package version, set these custom properties in your inventory:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.8
  log.message.format.version: 2.8

    • For other versions, review the Kafka upgrade documentation, and set the following custom properties in your inventory with the versions listed in the version table.

      kafka_broker_custom_properties:
  inter.broker.protocol.version:
  log.message.format.version:

      For example, for the 3.3.x Kafka package version:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 3.3
  log.message.format.version: 3.3

  3. To upgrade and set the inter.broker.protocol.version and log.message.format.version properties, run the provisioning playbook with the kafka_broker tag:

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_broker

  4. At this point the packages or archive have been upgraded, but the two properties are set to the starting version. Update inter.broker.protocol.version as below:

    kafka_broker_custom_properties:
  inter.broker.protocol.version: 3.3

    And run the provisioning playbook again:

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_broker \
  --skip-tags package

Important

Do not update the log.message.format.version until all Kafka clients have been upgraded.

Step 5. Upgrade other components

Note

If you need to upgrade specific hosts instead of all of them, you can limit the upgrade. This can be useful when your components are behind a load balancer. In this case, remove a specific host from the load balancer pool, upgrade it, then add it back. This ensures no traffic is disrupted. Enter the following command to limit the upgrade to one or more specific hosts.

ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
  --tags <component> \
  --limit "<host1>,<host2>"

Use the commands below to update the other components:

  • Schema Registry

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags schema_registry

  • Connect

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_connect

  • ksqlDB

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags ksql

  • REST Proxy

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_rest

  • Confluent Control Center

    ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags control_center

Step 6. Upgrade clients

Confluent Platform Ansible playbooks do not currently support upgrading clients. Review the Confluent Platform upgrade preparation for additional information.

Step 7. Update the Kafka log format

After all of the applicable clients have been updated, update your inventory variables for the log.message.format.version property:

kafka_broker_custom_properties:
  inter.broker.protocol.version: 3.3
  log.message.format.version: 3.3

And run the provisioning playbook again:

ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_broker \
  --skip-tags package

Upgrade co-located Confluent Platform components

When your Confluent Platform services are colocated and the installation method is set to package, the upgrade process can leave your hosts in an unstable state. This is because Confluent components all share the same packages. For example if you upgrade ZooKeeper where ZooKeeper and Kafka are running on the same host, the binaries and JARs Kafka depends on, such as JAR files, will get upgraded/replaced as the service is running.

This section describes how to manually upgrade a colocated Kafka and ZooKeeper installation. The process is applicable to co-locating other Confluent Platform components, as well.

Note

This upgrade method only applies to installation_method: package where multiple Confluent Platform components are running on the same host.

Step 1. Set broker_id variable on each Kafka host

Confirm each kafka_broker host has the broker_id variable set. For example:

kafka_broker:
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
      broker_id: 0
    ip-172-31-34-247.us-east-2.compute.internal:
      broker_id: 1
    ip-172-31-34-248.us-east-2.compute.internal:
      broker_id: 2

If the broker_id variable is not set on each kafka_broker host, use the following command to query the meta.properties file on the Kafka hosts. If you have customized the logs directory, replace /var/lib/kafka/data/ with the log.dirs property value:

ansible -i /path/to/hosts.yml \
  -m shell \
  -a "grep broker.id /var/lib/kafka/data/meta.properties" \
  kafka_broker

In the output similar to below, retrieve the applicable broker.id value.

ip-172-31-34-247.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=2
ip-172-31-34-246.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=1
ip-172-31-34-248.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=3

Step 2. Take note of Kafka controller and ZooKeeper leader

In the upgrade process, you must upgrade the Kafka controller and ZooKeeper leader last.

To query the Kafka controller, run the following command.

ansible -i /path/to/hosts.yml kafka_broker \
  -m import_role \
  -a "name=confluent.platform.kafka_broker tasks_from=dynamic_groups.yml"

In the sample output below, the controller is host: ip-172-31-34-246.us-east-2.compute.internal.

ip-172-31-34-246.us-east-2.compute.internal | SUCCESS => {
    "msg": "Broker ID: 1 and Controller ID: 1"
}
ip-172-31-34-247.us-east-2.compute.internal | SUCCESS => {
    "msg": "Broker ID: 2 and Controller ID: 1"
}
ip-172-31-34-248.us-east-2.compute.internal | SUCCESS => {
    "msg": "Broker ID: 3 and Controller ID: 1"
}

To query the ZooKeeper leader, run the following command.

ansible -i /path/to/hosts.yml zookeeper \
  -m import_role \
  -a "name=confluent.platform.zookeeper tasks_from=dynamic_groups.yml"

In the sample output below, the leader is host: ip-172-31-34-248.us-east-2.compute.internal.

ip-172-31-34-247.us-east-2.compute.internal | SUCCESS => {
    "msg": "Mode: follower"
}
ip-172-31-34-246.us-east-2.compute.internal | SUCCESS => {
    "msg": "Mode: follower"
}
ip-172-31-34-248.us-east-2.compute.internal | SUCCESS => {
    "msg": "Mode: leader"
}

Step 3. Upgrade using Limits

You can upgrade all Confluent Platform components on a given host using Ansible Limits.

Using the hosts discovered in the first step, select a host that is not a Kafka controller or ZooKeeper leader and follow the below steps:

  1. Set the inter.broker.protocol.version and log.message.format.version properties as described in Step 4. Upgrade Kafka.

  2. Stop all Confluent services on the host. For example:

    ansible -i /path/to/hosts.yml \
  -m shell \
  -a "systemctl stop confluent-*" \
  ip-172-31-34-247.us-east-2.compute.internal

  3. Upgrade and start all Confluent components on the host.

    ansible-playbook \
  -i /path/to/hosts.yml confluent.platform.all \
  --limit ip-172-31-34-247.us-east-2.compute.internal

Step 4. Repeat on rest of the hosts

Repeat the tasks in Step 3 for:

  1. The remaining of the colocated hosts.
  2. The ZooKeeper leader.
  3. The Kafka controller.

Important

You must upgrade the Kafka controller last, and the ZooKeeper leader second to last.