Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Upgrading

Ansible Playbooks for Confluent Platform includes playbooks to upgrade all Confluent Platform components. The playbooks go host by host, shutting down the component, upgrading packages, and validating application health before moving onto the next one.

You must complete the upgrades in the following order:

  1. Upgrade Zookeeper.
  2. Upgrade Kafka brokers.
  3. Upgrade (in any order):
    • Schema Registry
    • REST Proxy
    • Connect
  4. Upgrade Confluent Control Center.
  5. Upgrade external clients.
  6. Upgrade Kafka log format. This step ensures that the log is formatted properly for the new version of Confluent Platform after all upgrades have been completed.

Requirements

The upgrade playbooks have the following requirements:

  • Component must have been originally installed and configured using Ansible Playbooks for Confluent Platform.
  • You must have the same hosts.yml file used during the installation.
  • Upgrade playbooks are supported in Confluent Platform version 5.3.0 and later.
  • The Kafka broker upgrade playbook requires the jolokia agent be running on all Kafka brokers. Jolokia is required for the upgrade to work.

Step 1. Check out the upgrade branch

Check out the branch of cp-ansible you want to upgrade to. Branches will be in the format <version>-post. For example 5.4.0-post.

cd /path/to/cp-ansible
git fetch
git checkout 5.4.0-post

Step 2. Upgrade ZooKeeper

The ZooKeeper upgrade is designed to run on Kafka clusters with three nodes or more. As it upgrades the hosts it checks for quorum and health on each host before moving on to the next. Enter the command below to upgrade ZooKeeper:

ansible-playbook -i /path/to/hosts.yml upgrade_zookeeper.yml

Step 3. Upgrade Kafka

The Kafka broker upgrade playbook upgrades each broker in sequence. The upgrade takes special care around the inter.broker.protocol.version and log.message.format.version properties. First, enter the following command and note the current version of confluent-kafka-2.12 or confluent-server installed on your hosts:

# rpm -qa | grep confluent
confluent-server-5.3.0-1.noarch

Next, run the upgrade_kafka_broker playbook and pass in the version installed:

ansible-playbook -i /path/to/hosts.yml upgrade_kafka_broker.yml -e kafka_broker_upgrade_start_version=5.3.0

This playbook upgrades the Kafka brokers in sequence.

Important

The log.message.format.version property is not updated when the Kafka upgrade completes. This property is changed at the end of this procedure, after all clients have been upgraded.

Step 4. Upgrade other components

Note

If you need to upgrade specific hosts instead of all of them, you can limit the upgrade. This can be useful when your components are behind a load balancer. In this case, remove a specific host from the load balancer pool, upgrade it, then add it back. This ensures no traffic is disrupted. Enter the following command to limit the upgrade to one or more specific hosts.

ansible-playbook -i /path/to/hosts.yml upgrade_<component>.yml --limit "<host1>,<host2>"

Use the commands below to update the other components:

  • Schema Registry

    ansible-playbook -i /path/to/hosts.yml upgrade_schema_registry.yml

  • Connect

    ansible-playbook -i /path/to/hosts.yml upgrade_kafka_connect.yml

  • KSQL

    ansible-playbook -i /path/to/hosts.yml upgrade_ksql.yml

  • REST Proxy

    ansible-playbook -i /path/to/hosts.yml upgrade_kafka_rest.yml

  • Confluent Control Center

    ansible-playbook -i /path/to/hosts.yml upgrade_control_center.yml

Step 5. Upgrade clients

Confluent Platform Ansible playbooks do not currently support upgrading clients. Review Preparation for additional information.

Step 6. Update the Kafka log format

After all of the applicable clients have been updated, run the following command to update update the log format:

ansible-playbook -i /path/to/hosts.yml upgrade_kafka_broker_log_format.yml -e kafka_broker_upgrade_end_version=5.4