Upgrade Confluent Platform with Ansible Playbooks¶
Ansible Playbooks for Confluent Platform (Confluent Ansible) can upgrade the Confluent Platform components. To safely upgrade your hosts, make use of the Rolling Deployment Strategy, which will go host by host, shutting down the component, upgrading packages, restarting the service, and validating service health before moving onto the next one.
You must complete the upgrades in the following order:
- Upgrade ZooKeeper.
- Upgrade Kafka brokers.
- Upgrade (in any order):
- Schema Registry
- REST Proxy
- Connect
- Upgrade Confluent Control Center.
- Upgrade external clients.
- Upgrade Kafka log format. This step ensures that the log is formatted properly for the new version of Confluent Platform after all upgrades have been completed.
Confluent Platform does not guarantee the clusters and Confluent Platform components in different major versions will be functional. An upgrade process is complete when all the components are upgraded.
Requirements¶
The upgrade playbooks have the following requirements:
- Confluent Platform components must have been originally installed and configured using Ansible Playbooks for Confluent Platform.
- You must have the same
hosts.yml
file used during the installation.
Upgrade notes¶
Before you start the upgrade process, review the following changes and make any necessary updates.
SASL/SCRAM default version
The default SASL/SCRAM version was changed from 256 to 512.
If the version of SSL/SCRAM is specified as 256 in your
server.properties
, you must update your inventory and changesasl_protocol: scram
tosasl_protocol: scram256
.ZooKeeper TLS
TLS is now enabled by default for ZooKeeper when
ssl_enabled: true
is set.If your current ZooKeeper deployment does not use TLS, set
zookeeper_ssl_enabled: false
in your inventory.ZooKeeper server-to-server TLS
Improved logic and variables were added around ZooKeeper server to server authentication. If you have
sslQuorum=true
in yourzookeeper.properties
, setzookeeper_quorum_authentication_type: mtls
in your inventory.Enable Admin REST APIs
When upgrading from 5.5.x to 6.2.x, you must enable Admin REST APIs by setting the following property in your inventory file. If Admin REST APIs is not enabled, component upgrades will fail:
kafka_broker_rest_proxy_enabled: true
Disable canonicalization
If canonicalization has not been enabled during the Confluent Platform cluster creation, explicitly set the following property in the
hosts.yml
inventory file. An example scenario is when you upgrade a Confluent Platform cluster that uses Kerberos to authenticate Kafka brokers to ZooKeeper.kerberos: canonicalize: false
Variable name updates in
hosts.yaml
Misspelled variable names were corrected in the
7.2.2
version.If upgrading from a version, earlier than
7.2.2
, to a version,7.2.2
or later, make the following updates in your inventory file:- From:
kakfa_connect_replicator_<property_name>
- To:
kafka_connect_replicator_<property_name>
- From:
ZooKeeper version dependency
The ZooKeeper dependency has been upgraded to 3.8.1 due to 3.6 reaching end-of-life. To bring both your Kafka and ZooKeeper clusters to the latest versions:
(Case A) If upgrading from a Kafka version 2.4 (Confluent Platform version 5.4) or later:
- Kafka clusters can be updated directly.
- ZooKeeper clusters that are running binaries bundled with Kafka versions 2.4 or later can be updated normally as specified in the following sections in this document.
(Case B) If upgrading from a Kafka version older than 2.4 (Confluent Platform version older than 5.4):
- Kafka clusters first need to be upgraded to a version 2.4 or later, and earlier than 3.6.
- ZooKeeper clusters that are running binaries bundled with a Kafka version that are older than 2.4 need to be updated to the binaries bundled with Kafka versions later than 2.4 and earlier than 3.6.
- You could then continue following the steps in Case A above.
Upgrade Confluent Platform¶
Step 1. Download Ansible Playbooks for Confluent Platform¶
Download the Confluent Ansible for the target version of Confluent Platform that you are upgrading to:
ansible-galaxy collection install confluent.platform:<version>
For example, to upgrade to Confluent Platform 7.3.12:
ansible-galaxy collection install confluent.platform:7.3.12
To upgrade to the latest version of Confluent Platform, you can omit <version>
:
ansible-galaxy collection install --upgrade confluent.platform
Step 2. Set Rolling Deployment strategy¶
In order to avoid component outages, set the Deployment Strategy to rolling as below:
deployment_strategy: rolling
Step 3. Upgrade ZooKeeper¶
To upgrade ZooKeeper, run the provisioning playbook with the zookeeper
tag:
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags zookeeper
Step 4. Upgrade Kafka¶
Upgrading Kafka takes a few steps because you need to apply extra care to the
inter.broker.protocol.version
and log.message.format.version
properties.
Enter the following command and note the current version of
confluent-kafka
orconfluent-server
installed on your hosts.On Red Hat hosts:
rpm -qa | grep confluent
An example output:
confluent-server-6.0.1-1.noarch
On Debian hosts:
apt list --installed confluent-server
Set the version properties.
For 6.0.x Kafka package version, set these custom properties in your inventory:
kafka_broker_custom_properties: inter.broker.protocol.version: 2.6 log.message.format.version: 2.6
For 6.1.x Kafka package version, set these custom properties in your inventory:
kafka_broker_custom_properties: inter.broker.protocol.version: 2.7 log.message.format.version: 2.7
For 6.2.x Kafka package version, set these custom properties in your inventory:
kafka_broker_custom_properties: inter.broker.protocol.version: 2.8 log.message.format.version: 2.8
For other versions, review the Kafka upgrade documentation, and set the following custom properties in your inventory with the versions listed in the version table.
kafka_broker_custom_properties: inter.broker.protocol.version: log.message.format.version:
For example, for the 3.3.x Kafka package version:
kafka_broker_custom_properties: inter.broker.protocol.version: 3.3 log.message.format.version: 3.3
To upgrade and set the
inter.broker.protocol.version
andlog.message.format.version
properties, run the provisioning playbook with thekafka_broker
tag:ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_broker
At this point the packages or archive have been upgraded, but the two properties are set to the starting version. Update
inter.broker.protocol.version
as below:kafka_broker_custom_properties: inter.broker.protocol.version: 3.3
And run the provisioning playbook again:
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_broker \ --skip-tags package
Important
Do not update the log.message.format.version
until all Kafka clients have
been upgraded.
Step 5. Upgrade other components¶
Note
If you need to upgrade specific hosts instead of all of them, you can limit the upgrade. This can be useful when your components are behind a load balancer. In this case, remove a specific host from the load balancer pool, upgrade it, then add it back. This ensures no traffic is disrupted. Enter the following command to limit the upgrade to one or more specific hosts.
ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
--tags <component> \
--limit "<host1>,<host2>"
Use the commands below to update the other components:
Schema Registry
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags schema_registry
Connect
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_connect
ksqlDB
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags ksql
REST Proxy
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_rest
Confluent Control Center
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags control_center
Step 6. Upgrade clients¶
Confluent Platform Ansible playbooks do not currently support upgrading clients. Review the Confluent Platform upgrade preparation for additional information.
Step 7. Update the Kafka log format¶
After all of the applicable clients have been updated, update your inventory
variables for the log.message.format.version
property:
kafka_broker_custom_properties:
inter.broker.protocol.version: 3.3
log.message.format.version: 3.3
And run the provisioning playbook again:
ansible-playbook -i /path/to/hosts.yml confluent.platform.all --tags kafka_broker \
--skip-tags package
Upgrade co-located Confluent Platform components¶
When your Confluent Platform services are colocated and the installation method is set to package, the upgrade process can leave your hosts in an unstable state. This is because Confluent components all share the same packages. For example if you upgrade ZooKeeper where ZooKeeper and Kafka are running on the same host, the binaries and JARs Kafka depends on, such as JAR files, will get upgraded/replaced as the service is running.
This section describes how to manually upgrade a colocated Kafka and ZooKeeper installation. The process is applicable to co-locating other Confluent Platform components, as well.
Note
This upgrade method only applies to installation_method: package
where
multiple Confluent Platform components are running on the same host.
Step 1. Set broker_id variable on each Kafka host¶
Confirm each kafka_broker
host has the broker_id
variable set. For
example:
kafka_broker:
hosts:
ip-172-31-34-246.us-east-2.compute.internal:
broker_id: 0
ip-172-31-34-247.us-east-2.compute.internal:
broker_id: 1
ip-172-31-34-248.us-east-2.compute.internal:
broker_id: 2
If the broker_id
variable is not set on each kafka_broker
host, use the
following command to query the meta.properties
file on the Kafka hosts. If
you have customized the logs directory, replace /var/lib/kafka/data/
with
the log.dirs
property value:
ansible -i /path/to/hosts.yml \
-m shell \
-a "grep broker.id /var/lib/kafka/data/meta.properties" \
kafka_broker
In the output similar to below, retrieve the applicable broker.id
value.
ip-172-31-34-247.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=2
ip-172-31-34-246.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=1
ip-172-31-34-248.us-east-2.compute.internal | CHANGED | rc=0 >>
broker.id=3
Step 2. Take note of Kafka controller and ZooKeeper leader¶
In the upgrade process, you must upgrade the Kafka controller and ZooKeeper leader last.
To query the Kafka controller, run the following command.
ansible -i /path/to/hosts.yml kafka_broker \
-m import_role \
-a "name=confluent.platform.kafka_broker tasks_from=dynamic_groups.yml"
In the sample output below, the controller is host:
ip-172-31-34-246.us-east-2.compute.internal
.
ip-172-31-34-246.us-east-2.compute.internal | SUCCESS => {
"msg": "Broker ID: 1 and Controller ID: 1"
}
ip-172-31-34-247.us-east-2.compute.internal | SUCCESS => {
"msg": "Broker ID: 2 and Controller ID: 1"
}
ip-172-31-34-248.us-east-2.compute.internal | SUCCESS => {
"msg": "Broker ID: 3 and Controller ID: 1"
}
To query the ZooKeeper leader, run the following command.
ansible -i /path/to/hosts.yml zookeeper \
-m import_role \
-a "name=confluent.platform.zookeeper tasks_from=dynamic_groups.yml"
In the sample output below, the leader is host:
ip-172-31-34-248.us-east-2.compute.internal
.
ip-172-31-34-247.us-east-2.compute.internal | SUCCESS => {
"msg": "Mode: follower"
}
ip-172-31-34-246.us-east-2.compute.internal | SUCCESS => {
"msg": "Mode: follower"
}
ip-172-31-34-248.us-east-2.compute.internal | SUCCESS => {
"msg": "Mode: leader"
}
Step 3. Upgrade using Limits¶
You can upgrade all Confluent Platform components on a given host using Ansible Limits.
Using the hosts discovered in the first step, select a host that is not a Kafka controller or ZooKeeper leader and follow the below steps:
Set the
inter.broker.protocol.version
andlog.message.format.version
properties as described in Step 4. Upgrade Kafka.Stop all Confluent services on the host. For example:
ansible -i /path/to/hosts.yml \ -m shell \ -a "systemctl stop confluent-*" \ ip-172-31-34-247.us-east-2.compute.internal
Upgrade and start all Confluent components on the host.
ansible-playbook \ -i /path/to/hosts.yml confluent.platform.all \ --limit ip-172-31-34-247.us-east-2.compute.internal
Step 4. Repeat on rest of the hosts¶
Repeat the tasks in Step 3 for:
- The remaining of the colocated hosts.
- The ZooKeeper leader.
- The Kafka controller.
Important
You must upgrade the Kafka controller last, and the ZooKeeper leader second to last.