Upgrade Confluent Platform with Ansible Playbooks

Ansible Playbooks for Confluent Platform can also upgrade the Confluent Platform components. To safely upgrade your hosts, make use of the Rolling Deployment Strategy, which will go host by host, shutting down the component, upgrading packages, restarting the service, and validating service health before moving onto the next one.

You must complete the upgrades in the following order:

  1. Upgrade ZooKeeper.
  2. Upgrade Kafka brokers.
  3. Upgrade (in any order):
    • Schema Registry
    • REST Proxy
    • Connect
  4. Upgrade Confluent Control Center.
  5. Upgrade external clients.
  6. Upgrade Kafka log format. This step ensures that the log is formatted properly for the new version of Confluent Platform after all upgrades have been completed.

Requirements

The upgrade playbooks have the following requirements:

  • Confluent Platform components must have been originally installed and configured using Ansible Playbooks for Confluent Platform.
  • You must have the same hosts.yml file used during the installation.
  • Check and download the community.general collection if using Ansible 2.9.

Upgrade notes

Before you start the upgrade process, review the following changes and make any necessary updates.

  • SASL/SCRAM default version

    The default SASL/SCRAM version was changed from 256 to 512.

    If the version of SSL/SCRAM is specified as 256 in your server.properties, you must update your inventory and change sasl_protocol: scram to sasl_protocol: scram256.

  • ZooKeeper TLS

    TLS is now enabled by default for ZooKeeper when ssl_enabled: true is set.

    If your current ZooKeeper deployment does not use TLS, set zookeeper_ssl_enabled: false in your inventory.

  • ZooKeeper server-to-server TLS

    Improved logic and variables were added around ZooKeeper server to server authentication. If you have sslQuorum=true in your zookeeper.properties, set zookeeper_quorum_authentication_type: mtls in your inventory.

  • Enable Admin REST APIs

    When upgrading from 5.5.x to 6.2.x, you must enable Admin REST APIs by setting the following property in your inventory file. If Admin REST APIs is not enabled, component upgrades will fail:

    kafka_broker_rest_proxy_enabled: true

Step 1. Check out the upgrade branch

Check out the branch of cp-ansible you want to upgrade to. Branches are in the format <version>-post, for example, 6.2.7-post.

cd /path/to/cp-ansible
git fetch
git checkout 6.2.7-post

Step 2. Set Rolling Deployment strategy

In order to avoid component outages, you should set the Deployment Strategy to rolling as below:

deployment_strategy: rolling

Step 3. Upgrade ZooKeeper

To upgrade ZooKeeper, run the provisioning playbook with the zookeeper tag:

ansible-playbook -i /path/to/hosts.yml all.yml --tags zookeeper

Step 4. Upgrade Kafka

Upgrading Kafka takes a few steps because you need to apply extra care to the inter.broker.protocol.version and log.message.format.version properties.

  1. Enter the following command and note the current version of confluent-kafka or confluent-server installed on your hosts.

    On Red Hat hosts:

    rpm -qa | grep confluent

    An example output:

    confluent-server-6.0.1-1.noarch

    On Debian hosts:

    apt list --installed confluent-server

  2. Set the version properties.

    • For 6.0.x Kafka package version, set these custom properties in your inventory:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.6
  log.message.format.version: 2.6

    • For 6.1.x Kafka package version, set these custom properties in your inventory:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.7
  log.message.format.version: 2.7

    • For 6.2.x Kafka package version, set these custom properties in your inventory:

      kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.8
  log.message.format.version: 2.8

    • For other versions, review the Kafka upgrade documentation.

  3. To upgrade and set the inter.broker.protocol.version and log.message.format.version properties, run the provisioning playbook with the kafka_broker tag:

    ansible-playbook -i /path/to/hosts.yml all.yml --tags kafka_broker

  4. At this point the packages or archive have been upgraded, but the two properties are set to the starting version. To update the inter.broker.protocol.version, update your inventory as below:

    kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.7
  log.message.format.version: 2.6

    And run the provisioning playbook again:

    ansible-playbook -i /path/to/hosts.yml all.yml --tags kafka_broker \
  --skip-tags package

Important

Do not update the log.message.format.version until all Kafka clients have been upgraded.

Step 5. Upgrade other components

Note

If you need to upgrade specific hosts instead of all of them, you can limit the upgrade. This can be useful when your components are behind a load balancer. In this case, remove a specific host from the load balancer pool, upgrade it, then add it back. This ensures no traffic is disrupted. Enter the following command to limit the upgrade to one or more specific hosts.

ansible-playbook -i /path/to/hosts.yml all.yml \
  --tags <component> \
  --limit "<host1>,<host2>"

Use the commands below to update the other components:

  • Schema Registry

    ansible-playbook -i /path/to/hosts.yml all.yml --tags schema_registry

  • Connect

    ansible-playbook -i /path/to/hosts.yml all.yml --tags kafka_connect

  • ksqlDB

    ansible-playbook -i /path/to/hosts.yml all.yml --tags ksql

  • REST Proxy

    ansible-playbook -i /path/to/hosts.yml all.yml --tags kafka_rest

  • Confluent Control Center

    ansible-playbook -i /path/to/hosts.yml all.yml --tags control_center

Step 6. Upgrade clients

Confluent Platform Ansible playbooks do not currently support upgrading clients. Review the Confluent Platform upgrade preparation for additional information.

Step 7. Update the Kafka log format

After all of the applicable clients have been updated, update your inventory variables for the log.message.format.version property:

kafka_broker_custom_properties:
  inter.broker.protocol.version: 2.8
  log.message.format.version: 2.8

And run the provisioning playbook again:

ansible-playbook -i /path/to/hosts.yml all.yml --tags kafka_broker \
  --skip-tags package

Upgrade co-located Confluent Platform components

When your Confluent Platform services are colocated and the installation method is set to package, the upgrade process can leave your hosts in an unstable state. This is because Confluent components all share the same packages. For example if you upgrade ZooKeeper where ZooKeeper and Kafka are running on the same host, the binaries and JARs Kafka depends on, such as JAR files, will get upgraded/replaced as the service is running.

This section describes how to manually upgrade a colocated Kafka and ZooKeeper installation. The process is applicable to co-locating other Confluent Platform components, as well.

Note

This upgrade method only applies to installation_method: package where multiple Confluent Platform components are running on the same host.

Step 1. Set broker_id variable on each Kafka host

Confirm each kafka_broker host has the broker_id variable set. For example:

kafka_broker:
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
      broker_id: 0
    ip-172-31-34-247.us-east-2.compute.internal:
      broker_id: 1
    ip-172-31-34-248.us-east-2.compute.internal:
      broker_id: 2

If the broker_id variable is not set on each kafka_broker host, use the following command to query the meta.properties file on the Kafka hosts. If you have customized the logs directory, replace /var/lib/kafka/data/ with the log.dirs property value:

ansible -i /path/to/hosts.yml \
  -m shell \
  -a "grep broker.id /var/lib/kafka/data/meta.properties" \
  kafka_broker

In the output similar to below, retrieve the applicable broker.id value.

ip-172-31-34-247.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=2
ip-172-31-34-246.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=1
ip-172-31-34-248.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=3

Step 2. Take note of Kafka controller and ZooKeeper leader

In the upgrade process, you must upgrade the Kafka controller and ZooKeeper leader last.

To query the Kafka controller, run the following command:

ansible -i /path/to/hosts.yml kafka_broker \
  -m import_role \
  -a "name=confluent.kafka_broker tasks_from=dynamic_groups.yml"

In the sample output below, the controller is host: ip-172-31-34-246.us-east-2.compute.internal.

ip-172-31-34-246.us-east-2.compute.internal | SUCCESS => {
    "msg": "Broker ID: 1 and Controller ID: 1"
}
ip-172-31-34-247.us-east-2.compute.internal | SUCCESS => {
    "msg": "Broker ID: 2 and Controller ID: 1"
}
ip-172-31-34-248.us-east-2.compute.internal | SUCCESS => {
    "msg": "Broker ID: 3 and Controller ID: 1"
}

To query the ZooKeeper leader, run the following command:

ansible -i /path/to/hosts.yml zookeeper \
  -m import_role \
  -a "name=confluent.zookeeper tasks_from=dynamic_groups.yml"

In the sample output below, the leader is host: ip-172-31-34-248.us-east-2.compute.internal.

ip-172-31-34-247.us-east-2.compute.internal | SUCCESS => {
    "msg": "Mode: follower"
}
ip-172-31-34-246.us-east-2.compute.internal | SUCCESS => {
    "msg": "Mode: follower"
}
ip-172-31-34-248.us-east-2.compute.internal | SUCCESS => {
    "msg": "Mode: leader"
}

Step 3. Upgrade using Limits

You can upgrade all Confluent Platform components on a given host using Ansible Limits.

Using the hosts discovered in the first step, select a host that is not a Kafka controller or ZooKeeper leader and follow the below steps:

  1. Set the inter.broker.protocol.version and log.message.format.version properties as described in Step 4. Upgrade Kafka.

  2. Stop all Confluent services on the host. For example:

    ansible -i /path/to/hosts.yml \
  -m shell \
  -a "systemctl stop confluent-*" \
  ip-172-31-34-247.us-east-2.compute.internal

  3. Upgrade and start all Confluent components on the host. For example:

    ansible-playbook \
  -i /path/to/hosts.yml \
  all.yml \
  --limit ip-172-31-34-247.us-east-2.compute.internal

Step 4. Repeat on rest of the hosts

Repeat the tasks in Step 3 for:

  1. The remaining of the colocated hosts.
  2. The ZooKeeper leader.
  3. The Kafka controller.

Important

You must upgrade the Kafka controller last, and the ZooKeeper leader second to last.