<a id="ansible-upgrade"></a>

# Upgrade Confluent Platform with Ansible Playbooks

Ansible Playbooks for Confluent Platform (Confluent Ansible) can upgrade the Confluent Platform components. To safely upgrade
your hosts, make use of the Rolling Deployment Strategy, which will go host by
host, shutting down the component, upgrading packages, restarting the service,
and validating service health before moving on to the next one.

Confluent Platform does not guarantee the clusters and Confluent Platform components in different major
versions will be functional. An upgrade process is complete when all the
components are upgraded.

## Requirements

The upgrade playbooks have the following requirements:

* Confluent Platform components must have been originally installed and configured using
  Ansible Playbooks for Confluent Platform.
* You must have the same `hosts.yml` file used during the installation.

## Upgrade notes

Before you start the upgrade process, review the following changes and make any
necessary updates.

* ZooKeeper removal in Confluent Platform 8.0

  ZooKeeper was removed in Confluent Platform 8.0 and is no longer supported with the 8.0 version.
  Follow the steps in  [Upgrade ZooKeeper-based Confluent Platform deployment](#ansible-upgrade-zk) to migrate your ZooKeeper-based
  deployment to KRaft before you upgrade to Confluent Platform 8.0.
* Upgrade Control Center from 2.0 or 2.1 to 2.2 in Confluent Ansible 8.0

  Confluent Platform 8.0 does not work with Control Center 2.0 or 2.1. So, when upgrading Confluent Platform to 8.0
  and Control Center to 2.2, you must upgrade Control Center before upgrading Kafka. This
  dependency is only specific to when you upgrade to Confluent Platform 8.0.
* Upgrade Confluent Control Center (Legacy) alerts to Control Center in Confluent Ansible 8.0

  Starting in the 8.0 release, Confluent Ansible and Confluent Platform no longer support Confluent Control Center (Legacy).

  If you have alerts you need to migrate from Confluent Control Center (Legacy) to Control Center, before you upgrade
  your Confluent Platform deployment to 8.0, upgrade the Confluent Platform to 7.9.1 and migrate your alerts
  as described in [Control Center Alert Migration](https://docs.confluent.io/control-center/current/installation/alert-migrate.html).

  Your Confluent Control Center (Legacy) and Control Center must be configured with the same Kafka bootstrap endpoint
  to point to the same Kafka cluster for alert migration.

  Upgrading Confluent Control Center (Legacy) to Control Center or upgrading Confluent Control Center (Legacy) metrics to Control Center are not
  supported.
* Upgrade Log4j to Log4j 2 in Confluent Ansible 8.0

  Starting in the 8.0 release, Confluent Ansible and Confluent Platform only support Log4j2.

  When upgrading from Confluent Ansible 7.x to 8.x, the custom Log4j
  configurations on your 7.x cluster are not automatically
  converted to Log4j 2 configurations. You need to explicitly define the
  variables for Log4j 2 as described in [Configure Log4j 2](ansible-configure.md#ansible-log4j2). In 8.x, by
  default, Confluent Ansible  sets up Log4j 2 with the default values mentioned in
  [VARIABLES.md](https://github.com/confluentinc/cp-ansible/blob/master/docs/VARIABLES.md).
* SASL/SCRAM default version

  The default SASL/SCRAM version was changed from 256 to 512.

  If the version of SASL/SCRAM is specified as 256 in your `server.properties`,
  you must update your inventory and change `sasl_protocol: scram` to
  `sasl_protocol: scram256`.
* Enable Admin REST APIs

  When upgrading from 5.5.x to 6.2.x, you must enable Admin REST APIs by setting
  the following property in your inventory file. If Admin REST APIs is not enabled,
  component upgrades will fail:
  ```yaml
  kafka_broker_rest_proxy_enabled: true
  ```
* Disable canonicalization

  If canonicalization has not been enabled during the Confluent Platform cluster creation,
  explicitly set the following property in the `hosts.yml` inventory file.
  ```yaml
  kerberos:
    canonicalize: false
  ```
* Variable name updates in `hosts.yaml`

  Misspelled variable names were corrected in the `7.2.2` version.

  If upgrading from a version, earlier than `7.2.2`, to a version, `7.2.2`
  or later, make the following updates in your inventory file:
  * From: `kafka_connect_replicator_<property_name>`
  * To: `kafka_connect_replicator_<property_name>`
* Archive installation path for archive upgrades

  For archive installations (`installation_method: archive`), keep
  `archive_destination_path` constant across versions and do not include the
  Confluent Platform version in the path. The upgrade playbooks use this path to detect the
  installed version. If you change it between releases, archive-based upgrades
  can fail.

  For example:
  * Do: `archive_destination_path: "/opt/confluent"` (same value for 7.8 to
    7.9)
  * Do not: Change from `"/data/confluent-platform-7.8.x"` to
    `"/data/confluent-platform-7.9.x"` between versions.

### Upgrade Confluent Platform with mTLS authentication

Starting in 7.8.0, Confluent Ansible uses the `ssl_client_authentication`
variable to enable mTLS in Confluent Platform. In 7.7.x and earlier, Confluent Ansible used
the `ssl_mutual_auth_enabled: true` variable to enable mTLS authentication.

When upgrading Confluent Platform components with mTLS enabled from Confluent Ansible 7.7.x or
earlier to a 7.8+ version, you must update your mTLS configuration:

* Set both `ssl_client_authentication` and `ssl_mutual_auth_enabled` together.
  * If `ssl_mutual_auth_enabled: true`, set
    `ssl_client_authentication: required` or `ssl_client_authentication: requested`.

    If the component (Kafka listener, Confluent Platform component, or MDS) doesn’t have mTLS
    and you want to enable it in this upgrade, set `ssl_client_authentication`
    to `requested`.

    If the component already has mTLS and you want to upgrade to
    Confluent Platform 8.3.0, you don’t need to use `requested` as mTLS is already
    enabled.
  * If `ssl_mutual_auth_enabled: false`, set `ssl_client_authentication: none`.

This applies at the `all.vars` level and at the  Kafka listeners and other
Confluent Platform components levels.

For example, if we have the following settings in 7.7 or below:

```yaml
all:
  vars:
    ssl_mutual_auth_enabled: true
    kafka_broker_custom_listeners:
      client:
        name: CLIENT
        port: <port>
        ssl_enabled: true
        ssl_mutual_auth_enabled: true
schema_registry:
  ssl_mutual_auth_enabled: true
```

Update the settings as below when upgrading to 7.8+:

```yaml
all:
  vars:
    ssl_mutual_auth_enabled: true
    ssl_client_authentication: required
    kafka_broker_custom_listeners:
      client:
        name: CLIENT
        port: <port>
        ssl_enabled: true
        ssl_mutual_auth_enabled: true
        ssl_client_authentication: required
schema_registry:
  ssl_mutual_auth_enabled: true
  ssl_client_authentication: required
```

<a id="ansible-upgrade-zk"></a>

## Upgrade ZooKeeper-based Confluent Platform deployment

ZooKeeper has been deprecated and is no longer included in Confluent Platform 8.0 and later.

To upgrade to Confluent Platform 8.0, first migrate your ZooKeeper-based deployment to KRaft, and
then upgrade that KRaft-based cluster to 8.0.

**To upgrade a Zookeeper-based Confluent Platform deployment 7.6.x and earlier to the 8.0 release:**

1. [Upgrade your Confluent Platform to 7.9](https://docs.confluent.io/ansible/7.9/ansible-upgrade.html).

   If you are on Confluent Platform 7.6.x or earlier, first, you need to upgrade your Confluent Platform to
   a version where production-level KRaft migration is supported, such as 7.9.
2. [Migrate the Zookeeper-based Confluent Platform to KRaft in 7.9](https://docs.confluent.io/ansible/7.9/ansible-migrate-kraft.html).
3. [Upgrade the KRaft-based Confluent Platform to 8.0](#ansible-upgrade-kraft).

**To upgrade a Zookeeper-based Confluent Platform deployment 7.7.x and later to the 8.0 release:**

According to the [Confluent Platform KRaft migration guide](https://docs.confluent.io/platform/current/installation/migrate-zk-kraft.html#migrate-from-zk-to-kraft-on-cp),
it is recommended that you use Confluent Platform 7.7.0 or laster version for production
migration. If you are using Confluent Platform 7.7.x or later, you can migrate the
current version of Confluent Platform to KRaft, and then upgrade that KRaft-based Confluent Platform to
8.0.

1. [Migrate the Zookeeper-based Confluent Platform to KRaft in the current version](https://docs.confluent.io/ansible/7.9/ansible-migrate-kraft.html).
2. [Upgrade the KRaft-based Confluent Platform to 8.0](#ansible-upgrade-kraft).

<a id="ansible-upgrade-kraft"></a>

## Upgrade KRaft-based Confluent Platform deployment

If you are running a Confluent Platform version later than 7.4.0 and your  clusters are
running in KRaft mode, you can upgrade your KRaft-based Confluent Platform deployment as
described in this section.

Perform the upgrade in the following order:

1. Upgrade KRaft.
2. Upgrade Kafka brokers.
3. Upgrade (in any order):
   * Schema Registry
   * REST Proxy
   * Connect

1. Upgrade Control Center.
2. Upgrade external clients.
3. Upgrade Kafka metadata version.

### Step 1. Download Ansible Playbooks for Confluent Platform

Download the Confluent Ansible for the target version of Confluent Platform that you are
upgrading to:

```bash
ansible-galaxy collection install confluent.platform:<version>
```

For example, to upgrade to Confluent Platform 8.3.0:

```bash
ansible-galaxy collection install confluent.platform:8.3.0
```

To upgrade to the latest version of Confluent Platform, you can omit `<version>`:

```bash
ansible-galaxy collection install --upgrade confluent.platform
```

### Step 2. Set rolling deployment strategy

In order to avoid component outages, set the [Deployment Strategy to
rolling](ansible-reconfigure.md#ansible-rolling-deployment) as below:

```yaml
deployment_strategy: rolling
```

### Step 3. Upgrade KRaft controller and Kafka brokers

1. If upgrading a deployment with RBAC and mTLS authentication configured:
   1. Add all controller’s and broker’s principals in `super.users` of
      `kafka_controller_custom_properties` before starting upgrade. For example:
      ```yaml
      kafka_controller_custom_properties:
             super.users: User:my_controller_admin;User:my_broker_admin
      ```
   2. Upgrade both KRaft and Kafka:
      ```bash
      ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
        --tags kafka_controller,kafka_broker
      ```
   3. Skip the next two steps that upgrade KRaft and Kafka separately.
2. To upgrade KRaft, run the provisioning playbook with the
   `kafka_controller` tag:
   ```bash
   ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
     --tags kafka_controller
   ```
3. To upgrade Kafka, run the provisioning playbook with the `kafka_broker` tag:
   ```bash
   ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
     --tags kafka_broker
   ```
4. Once you are satisfied that the broker performance meets your expectations,
   increment the `metadata.version` for the broker by running the
   `kafka-features` tool with the upgrade argument:
   ```bash
   ./bin/kafka-features --bootstrap-server <server>:<port> \
     --command-config <path to controller's client.properties> \
     upgrade --metadata 4.3
   ```

   The path of the `client.properties` file is:
   * `/opt/confluent/etc/controller/client.properties` in case of archive
     installation
   * `/etc/controller/client.properties` in case of package
     installation

   See [Configure listeners](ansible-configure.md#ansible-security-listeners) for where the bootstrap server host and
   port are configured.

<a id="ansible-upgrade-cp-components"></a>

### Step 4. Upgrade other components

#### NOTE
If you need to upgrade specific hosts instead of all of them, you can [limit
the upgrade](https://ansible-tips-and-tricks.readthedocs.io/en/latest/ansible/commands/#limit-to-one-or-more-hosts).
This is useful when your components are behind a load balancer. In this
case, remove a specific host from the load balancer pool, upgrade it, then
add it back. This ensures no traffic is disrupted. Enter the following
command to limit the upgrade to one or more specific hosts.

```bash
ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
  --tags <component> \
  --limit "<host1>,<host2>"
```

Use the commands below to update the other components:

* Schema Registry
  ```bash
  ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
    --tags schema_registry
  ```
* Connect
  ```bash
  ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
    --tags kafka_connect
  ```
* ksqlDB
  ```bash
  ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
    --tags ksql
  ```
* REST Proxy
  ```bash
  ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
    --tags kafka_rest
  ```
* Control Center
  ```bash
  ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
    --tags control_center_next_gent
  ```

### Step 5. Upgrade clients

Confluent Platform Ansible playbooks do not currently support upgrading clients. Review
[the Confluent Platform upgrade
preparation](/platform/current/installation/upgrade.html#upgrade-preparation) for additional
information.

<a id="ansible-upgrade-clone-kraft"></a>

## Upgrade co-located KRaft-based Confluent Platform deployments

When your Confluent Platform services are colocated, the upgrade process
can leave your hosts in an unstable state as Confluent components all share the
same packages.

This section describes how to manually upgrade colocated Kafka, KRaft, and
other Confluent Platform components using [Ansible Limits](https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html#patterns-and-ansible-playbook-flags).

Use this upgrade method when the package installation is used
(`installation_method: package`) and when multiple Confluent Platform components are running
on the same hosts.

### Step 1: Upgrade KRaft controller

1. Get a list of KRaft hosts where `node_id` is set:
   ```bash
   ansible -i /path/to/hosts.yml \
     -m shell -a "grep node.id /var/lib/controller/data/meta.properties" \
     kafka_controller
   ```
2. For each KRaft host you retrieved in the previous step, repeat:
   1. Stop the KRaft controller using the command. For example:
      ```bash
      ansible -i /path/to/hosts.yml \
        -m shell \
        -a "systemctl stop confluent-kcontroller" \
        ip-172-31-34-247.us-east-2.compute.internal
      ```
   2. Upgrade and start the KRaft controller on the host. For example:
      ```bash
      ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
        --limit ip-172-31-34-247.us-east-2.compute.internal
        --tags kafka_controller
      ```

### Step 2. Upgrade Kafka brokers

1. Get a list of Kafka broker hosts where `broker_id` is set:
   ```bash
   ansible -i /path/to/hosts.yml \
     -m shell -a "grep node.id /var/lib/kafka/data/meta.properties" \
     kafka_broker
   ```
2. For each Kafka host you retrieved in the previous step, repeat:
   1. Stop the Kafka broker host using the command. For example:
      ```bash
      ansible -i /path/to/hosts.yml \
        -m shell \
        -a "systemctl stop confluent-server" \
        ip-172-31-34-247.us-east-2.compute.internal
      ```
   2. Upgrade and start Kafka brokers on the host using the command. For example:
      ```bash
      ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
        --limit ip-172-31-34-247.us-east-2.compute.internal
        --tags kafka_broker
      ```
3. Once you are satisfied that the broker performance meets your expectations,
   increment the `metadata.version` for the broker by running the
   `kafka-features` tool with the upgrade argument:
   ```bash
   ./bin/kafka-features --bootstrap-server <server>:<port> \
     --command-config <path to controller's client.properties> \
     upgrade --metadata 4.3
   ```

   The path of the `client.properties` file is:
   * `/opt/confluent/etc/controller/client.properties` in case of archive
     installation
   * `/etc/controller/client.properties` in case of package
     installation

   See [Configure listeners](ansible-configure.md#ansible-security-listeners) for where the bootstrap server host and port are configured.

### Step 3. Upgrade other Confluent components

Repeat the steps described below to upgrade each Confluent host.

1. Stop all Confluent services on the host. For example:
   ```bash
   ansible -i /path/to/hosts.yml \
     -m shell \
     -a "systemctl stop confluent-*" \
     ip-172-31-34-247.us-east-2.compute.internal
   ```
2. Upgrade and start all Confluent components on the host. For example:
   ```bash
   ansible-playbook \
     -i /path/to/hosts.yml confluent.platform.all \
     --limit ip-172-31-34-247.us-east-2.compute.internal
   ```

<a id="ansible-upgrade-c3"></a>

## Upgrade Confluent Control Center

Confluent Control Center is available as a separate package and follows independent versions.

The Confluent Ansible inventory file has the default version of Control Center for the
specific version of Confluent Ansible.

To upgrade Control Center to that default version
: Follow the steps in [Step 4. Upgrade other components](#ansible-upgrade-cp-components).

To override the default version, and upgrade Control Center to a different version
: 1. Set `confluent_control_center_next_gen_package_version:<release-version>`
     in the inventory file.
     ```yaml
     all:
       vars:
         confluent_control_center_next_gen_package_version:<release-version>
     ```
  2. Run the following command to upgrade Control Center:
     ```bash
     ansible-playbook -i /path/to/hosts.yml confluent.platform.all \
       --tags control_center_next_gen
     ```