Upgrade Confluent Platform

Use these instructions to upgrade earlier versions of Confluent Platform to the latest.

Note

If you are a Confluent Platform subscriber, and you have questions about upgrades or need help, you can contact Confluent using the Support Portal.

Preparation

Follow these guidelines when you prepare to upgrade.

  • Before upgrading, always back up all configuration and unit files with their file permissions, ownership, and customizations. Confluent Platform may not run if the proper ownership isn’t preserved on configuration files. By default, configuration files are located in the $CONFLUENT_HOME/etc directory and are organized by component.

  • The Confluent Platform package includes Confluent Server by default and requires a confluent.license key in your server.properties file. Starting with Confluent Platform 5.4.x, the Confluent Server broker checks for a license during start-up. You must supply a license string in each broker’s properties file using the confluent.license property as below:

    confluent.license=LICENCE_STRING_HERE_NO_QUOTES

    If you want to use the Kafka broker, download the confluent-community package. The Kafka broker is the default in all Debian or RHEL and CentOS packages.

  • If you are running a Confluent Platform version later than 7.4.0 and your clusters are running in KRaft mode, you may not need to upgrade ZooKeeper. See Upgrade procedures.

  • Upgrade your entire platform deployment so that all components are running the same version. Do not bridge component versions.

  • Read the documentation and draft an upgrade plan that matches your specific requirements and environment before starting the upgrade process. In other words, don’t start working through this guide on a live cluster. Read the guide entirely, make a plan, then execute the plan.

  • Give careful consideration to the order in which components are upgraded. Java clients (producer and consumer) can communicate with older brokers so you should plan to upgrade all brokers before clients. Clients include any application that uses Kafka producer or consumer, command line tools, Schema Registry, REST Proxy, Kafka Connect and Kafka Streams.

    Important

    Determine if clients are colocated with brokers. Although not recommended, some deployments have clients co-located with brokers (on the same node). In these cases, both the broker and clients share the same packages. If they are colocated, then ensure all client processes are not upgraded until all Kafka brokers have been upgraded.

  • Determine and install the appropriate Java version. Java 8 is now the minimum version supported, and it is deprecated and will be removed in a future release. See Supported Java Versions for a list of Confluent Platform versions and the corresponding Java version support before you upgrade. For complete compatibility information, see the Supported Versions and Interoperability for Confluent Platform.

  • The LDAP Authorizer is deprecated. To configure group-based authorization using LDAP, you must migrate to the commercial confluent-server package when upgrading to 7.6.0. See Migrate Confluent Platform to Confluent Server for details.

  • Consider using Confluent Control Center to monitor broker status during the rolling restart.

  • Decide on performing a rolling upgrade or a downtime upgrade. Confluent Platform supports both rolling upgrades (upgrade one broker at a time to avoid cluster downtime) and downtime upgrades (take down the entire cluster, upgrade it, and bring everything back up).

  • Read the Release Notes for Confluent Platform 7.6. They contain important information about noteworthy features, and changes to configurations that may impact your upgrade.

Upgrade procedures

Following is the recommended order of upgrades:

  1. Upgrade brokers.

  2. Upgrade the rest of Confluent Platform components as described in the later part of this topic.

    In most cases, we recommend you upgrade Confluent Control Center last among the Confluent Platform components. However, if you are upgrading from 6.0.1 or 6.0.2 to 6.1.0 or later, you should upgrade Confluent Control Center first, and then upgrade your Kafka brokers to avoid Control Center instability. If you have already upgraded your Kafka brokers and Confluent Control Center crashes, upgrade Control Center and the situation should resolve itself.

  3. If it makes sense, build applications that use Kafka producers and consumers against the new 7.6.x libraries and deploy the new versions. See Schemas, Serializers, and Deserializers for Confluent Platform for more details about using the 7.6.x libraries.

Upgrade ZooKeeper

Important

As of Confluent Platform 7.5, ZooKeeper is deprecated for new deployments. Confluent recommends KRaft mode for new deployments. For more information, see KRaft Overview for Confluent Platform.

You may not need to upgrade ZooKeeper if you started using Confluent Platform with version 7.4 or later.

Use the following guidelines to prepare for the upgrade:

  • ZooKeeper has been upgraded to version 3.8.3 from Confluent Platform version 6.1.x to Confluent Platform 7.6 to mitigate CVE-2023-44981 and because ZooKeeper version 3.6.3 has reached its end-of-life. To upgrade both Kafka and ZooKeeper clusters to the latest versions, note the following:
    • Confluent Platform version 5.4.x and later (Kafka clusters version >= 2.4) can be updated with no special steps. This means that ZooKeeper-based clusters that are running binaries bundled with Confluent Platform 5.4.6 or later can be updated directly.
    • Confluent Platform versions lower than 5.4.x (Kafka version 2.4) first need to be updated to a version equal to or greater than Confluent Platform 5.4.x and earlier than Confluent Platform 7.6. This means that ZooKeeper-based clusters which are running binaries bundled with Confluent Platform versions earlier than 5.4.x need to be updated to any binaries bundled with Confluent Platform versions 5.4.6 or later and earlier than 7.6. You can then update the clusters to version Confluent Platform 7.6.
  • Back up all configuration files and customizations to your configuration and unit files before upgrading.
  • Back up ZooKeeper data from the leader. It will get you back to the latest committed state in case of a failure.
  • Read through the documentation and draft an upgrade plan that matches your specific requirements and environment before starting the upgrade process.

Rolling upgrade of ZooKeeper

Perform the following steps to gather information needed for a rolling upgrade:

  1. To find who is the leader, run the following command:

    echo mntr | nc localhost 2181 | grep zk_server_state

    Verify that there is only one leader in the entire ZooKeeper ensemble.

  2. To find how many nodes are in sync with the leader, run the following command:

    echo mntr | nc localhost 2181 | grep zk_synced_followers

  3. Verify that all the followers are in sync with the leader:

    echo mntr | nc localhost 2181 | grep zk_synced_followers

For each ZooKeeper server, repeat the following steps. The leader ZooKeeper server should be upgraded last:

  1. Stop the ZooKeeper process gracefully.

  2. Upgrade the ZooKeeper binary.

  3. Start the ZooKeeper process.

  4. Wait until all the followers are in sync with the leader:

    echo mntr | nc localhost 2181 | grep zk_synced_followers

If there is an issue during an upgrade, you can rollback using the same steps.

The AdminServer

An embedded Jetty-based AdminServer was added in ZooKeeper 3.5.

The AdminServer is disabled by default in ZooKeeper distributed as part of Confluent Platform. To enable the AdminServer, set admin.enableServer=true in your local zookeeper.properties file.

The AdminServer is enabled by default (on port 8080) in ZooKeeper provided by the Apache Kafka® distribution. To configure the AdminServer, see the AdminServer configuration.

Four letter words whitelist in ZooKeeper

Starting in ZooKeeper 3.5.3, the Four Letter Words commands must be explicitly white listed in the zookeeper.4lw.commands.whitelist setting for ZooKeeper server to enable the commands. By default the whitelist only contains the srvr command which zkServer.sh uses. The rest of the Four Letter Words commands are disabled by default.

An example to whitelist stat, ruok, conf, and isro commands while disabling the rest of Four Letter Words command:

4lw.commands.whitelist=stat, ruok, conf, isro

An example to whitelist all Four Letter Words commands:

4lw.commands.whitelist=*

When running ZooKeeper in a Docker container, use the Java system property, -e KAFKA_OPTS=-'Dzookeeper.4lw.commands.whitelist=', in the docker run command. For example:

docker run -d \
 --net=host
 --name=zookeeper \
 -e ZOOKEEPER_CLIENT_PORT=32181 \
 -e ZOOKEEPER_TICK_TIME=2000 \
 -e ZOOKEEPER_SYNC_LIMIT=2 \
 -e KAFKA_OPTS='-Dzookeeper.4lw.commands.whitelist=*' \
 confluentinc/cp-zookeeper:7.6.3

See The Four Letter Words for more information.

Upgrade issue with missing snapshot file

The ZooKeeper upgrade from 3.4.X to 3.5.X can fail with the following error if there are no snapshot files in the 3.4 data directory.

ERROR Unable to load database on disk
(org.apache.zookeeper.server.quorum.QuorumPeer) java.io.IOException: No
snapshot found, but there are log entries. Something is broken! at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:222)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at
org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:919)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:905) at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

This usually happens in test upgrades where ZooKeeper 3.5.X is trying to load an existing 3.4 data directory in which no snapshot file has been created. For more details about this issue, refer to ZOOKEEPER-3056.

The recommended workaround is:

  1. Take a backup of the current ZooKeeper data directory.
  2. Look for snapshot files (file names starting with snapshot) in the ZooKeeper data directory.

For more details about the workaround, refer to the ZooKeeper Upgrade FAQ.

Upgrade Kafka brokers

In a rolling upgrade scenario, upgrade one Kafka broker at a time, taking into consideration the recommendations for doing rolling restarts to avoid downtime for end users.

In a downtime upgrade scenario, take the entire cluster down, upgrade each Kafka broker, then start the cluster.

Steps to upgrade for any fix pack release

Any fix pack release can perform a rolling upgrade (for example, 6.1.7 to 6.1.7) by simply upgrading each broker one at a time.

To upgrade each broker:

  1. Stop the broker.
  2. Upgrade the software (see below for your packaging type).
  3. Start the broker.

Steps for upgrading to 7.6.x (ZooKeeper mode)

In a rolling upgrade scenario, upgrading to Confluent Platform 7.6.x (Kafka 3.6.x) requires special steps, because Kafka 3.6.x includes changes to wire protocol and the inter-broker protocol.

Follow these steps for a rolling upgrade:

  1. Update server.properties on all Kafka brokers by modifying the inter.broker.protocol.version and log.message.format.version properties to match the currently installed version.

    In Confluent Platform version 7.0.0 and later, the broker configuration log.message.format.version and topic configuration message.format.version parameters are deprecated. The values of both configurations are assumed to be 3.0 if inter.broker.protocol.version is 3.0 or later. If log.message.format.version or message.format.version is set, you should clear them when inter.broker.protocol.version is upgraded to 3.0, to avoid potential compatibility issues if the inter.broker.protocol.version is downgraded. For more information, see KIP-724.

    For Confluent Platform versions before 7.0.0:

    • After upgrading from earlier versions to 6.1.x, customers with high partition counts (greater than 100,000) may experience cluster instability. For this reason, Confluent recommends against using the version 2.7 inter broker protocol (IBP).

    If you use Confluent Server, you must provide a valid Confluent license key, for example, confluent.license=123.

    Note

    You don’t need to restart the broker to activate the modified properties.

    The following table shows the broker protocol and log message format versions that correspond with each Confluent Platform version.

    Confluent Platform version Broker protocol version Log message format version
    7.6.x inter.broker.protocol.version=3.6 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    7.5.x inter.broker.protocol.version=3.5 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    7.4.x inter.broker.protocol.version=3.4 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    7.3.x inter.broker.protocol.version=3.3 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    7.2.x inter.broker.protocol.version=3.2 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    7.1.x inter.broker.protocol.version=3.1 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    7.0.x inter.broker.protocol.version=3.0 log.message.format.version parameter is ignored and the value is inferred from inter.broker.protocol.version
    6.2.x inter.broker.protocol.version=2.8 log.message.format.version=2.8
    6.1.x inter.broker.protocol.version=2.7 log.message.format.version=2.7
    • Confluent recommends against setting inter.broker.protocol.version to 2.7.

  2. Upgrade each Kafka broker, one at a time.

    1. Stop the broker.

    2. Upgrade the software as necessary for your packaging type.

    3. Start the broker.

      Important

      The active controller should be the last broker you restart. This ensures that the active controller isn’t moved on each broker restart, which would slow down the restart. To identify which broker in the cluster is the active controller, check the kafka.controller:type=KafkaController,name=ActiveControllerCount metric. The active controller reports 1 and the remaining brokers report 0. For more information, see Broker metrics.

  3. After all Kafka brokers have been upgraded, make the following update in server.properties: inter.broker.protocol.version=3.6.

  4. Restart each Kafka broker, one at a time, to apply the configuration change.

Steps for upgrading to 7.6.x (KRaft-mode)

You can upgrade KRaft clusters for any Confluent Platform version 7.4.0 and later to the latest Confluent Platform version.

You should upgrade all of the brokers, one at a time. Follow these steps for a rolling upgrade:

  1. Shut down the broker.

  2. Upgrade the software for the broker per the packaging type.

  3. Restart the broker.

  4. Verify that the cluster behavior and performance meets your expectations.

  5. After you are satisfied that the broker performance meets your expectations, increment the metadata.version for the broker by running the kafka-features tool with the upgrade argument:

    ./bin/kafka-features upgrade --bootstrap-server <server:port> --metadata 3.6

    For more information, see MetadataVersion. Note that IV means “internal version” and each metadata version after 0.10.0 also provides an internal version. In addition, each metadata version later than 3.2.x provides a boolean parameter, which when true indicates there are breaking metadata changes. After you have changed the metadata.version to the latest version, you can only downgrade if there are no metadata changes between the current and earlier version. The following table lists the metadata versions supported for each platform version.

    Confluent Platform version Metadata versions
    7.6.x 3.6IV0
    7.5.x 3.5IV0-3.5IV2
    7.4.x 3.4IV0
    7.3.x 3.3IV0-3.3IV3

Confluent license

Add the confluent.license configuration parameter to server.properties. Confluent Platform 5.4.x and later, requires confluent.license to start. For more information, see confluent-server.

Advertised listeners

When you upgrade from 5.x or earlier version to 6.x or later version, you need to include the following properties in the server.properties file:

Security

Starting with 5.4.x, the new authorizer class kafka.security.authorizer.AclAuthorizer replaces kafka.security.auth.SimpleAclAuthorizer. You must manually change existing instances of kafka.security.auth.SimpleAclAuthorizer to kafka.security.authorizer.AclAuthorizer in the server.properties file. For more information, see ACL concepts.

Replication factor for Self-Balancing Clusters

The confluent.balancer.topic.replication.factor setting was added in Confluent Platform 6.0.0 for Self-Balancing configuration.

If the value of confluent.balancer.topic.replication.factor is greater than the total number of brokers, the brokers will not start.

The default value of confluent.balancer.topic.replication.factor is 3.

For details on the setting, see confluent.balancer.topic.replication.factor.

Upgrade DEB packages using APT

  1. Back up all configuration files from /etc, including, for example, /etc/kafka, /etc/kafka-rest, and /etc/schema-registry.

  2. Stop the services and remove the existing packages and their dependencies. This can be done on one server at a time for a rolling upgrade.

    # The example below removes the Kafka package
sudo kafka-server-stop
sudo apt-get remove confluent-kafka

# To remove Confluent Platform and all its dependencies at once,
# run the following command after all services are stopped.
sudo apt-get autoremove confluent-platform-*

    Important

    If you’re running Confluent Server, the previous autoremove may not work. If not, try the following command:

    # Remove Confluent Platform
sudo apt-get remove confluent-platform

# And then remove the dependencies
sudo apt-get autoremove

  3. Remove the older GPG key and import the updated key. If you have already imported the updated 8b1da6120c2bf624 key, then you can skip this step. However, if you still have the old 670540c841468433 key installed, now is the time to remove it and import the 8b1da6120c2bf624 key:

    sudo apt-key del 41468433
wget -qO - https://packages.confluent.io/deb/7.6/archive.key | sudo apt-key add -

  4. Remove the repository files of the previous version.

    sudo add-apt-repository -r "deb https://packages.confluent.io/deb/<currently installed version> stable main"

  5. Add the 7.6 repository to /etc/apt/sources.list.

    Attention

    After Confluent Platform 8.0, the librdkafka, Avro, and libserdes C/C++ client packages will NOT be available from the https://packages.confluent.io/deb location. You will need to obtain those client packages from https://packages.confluent.io/clients after the Confluent Platform 8.0 release.

    For the clients repository, you must obtain your Debian distribution’s release “Code Name”, such as buster, focal, etc. You can do this by calling lsb_release -cs. The following example makes this call with $(lsb_release -cs), which should work in most cases. If it does not, you must pick the closest Debian or Ubuntu code name for your Debian Linux distribution that matches the supported Debian & Ubuntu Operating Systems supported by Confluent Platform.

    sudo add-apt-repository "deb https://packages.confluent.io/deb/7.6 stable main"
sudo add-apt-repository "deb https://packages.confluent.io/clients/deb $(lsb_release -cs) main"

  6. Refresh repository metadata.

    sudo apt-get update

  7. If you modified the configuration files, apt will prompt you to resolve the conflicts. Be sure to keep your original configuration. Install the new version:

      sudo apt-get install confluent-platform

# Or install the packages you need one by one. For example, to install only Kafka:
  sudo apt-get install confluent-kafka

    Tip

    You can view all available Confluent Platform builds with this command:

    apt-cache show confluent-platform

    You can install specific Confluent Platform builds by appending the version (<version>) to the install command:

    sudo apt-get install confluent-platform-<version>

  8. Start Confluent Platform components.

    kafka-server-start -daemon /etc/kafka/server.properties

Upgrade RPM packages by using YUM

  1. Back up all configuration files from /etc, including, for example, /etc/kafka (ZooKeeper mode) or etc/kafka/kraft (KRaft mode), /etc/kafka-rest, and /etc/schema-registry. You should also back up all customizations to your configuration and unit files.

  2. Stop the services and remove the existing packages and their dependencies. This can be done on one server at a time for a rolling upgrade (refer to Steps for upgrading to 7.6.x (ZooKeeper mode)).

    # The example below removes the Kafka package
  sudo kafka-server-stop
  sudo yum remove confluent-kafka

# To remove Confluent Platform and all its dependencies at once, run the following after stopping all services
  sudo yum autoremove confluent-platform-5.5.0

    Note

    If you’re running Confluent Server, the previous autoremove command won’t work. Instead, run the following command:

    sudo yum remove confluent-*

    To upgrade from previous minor versions of 6.0, run the following command instead:

    sudo yum autoremove confluent-platform

  3. Remove the repository files of the previous version.

    sudo rm /etc/yum.repos.d/confluent.repo

  4. Remove the older GPG key. This step is optional if you haven’t removed Confluent’s older (670540c841468433) GPG key. Confluent’s newer (8b1da6120c2bf624) key would appear in the RPM Database as gpg-pubkey-0c2bf624-60904208

    sudo rpm -e gpg-pubkey-41468433-54d512a8
sudo rpm --import https://packages.confluent.io/rpm/7.6/archive.key

  5. Add the repository to your /etc/yum.repos.d/ directory in a file named confluent-7.6.repo.

    Attention

    After Confluent Platform 8.0, the librdkafka, Avro, and libserdes C/C++ client packages will NOT be available from the https://packages.confluent.io/rpm location. You will need to obtain those client packages from https://packages.confluent.io/clients after the Confluent Platform 8.0 release.

    The $releasever and $basearch are Yum placeholder variables that change depending on what release version of the OS and CPU Architecture the OS is running. These are meant to be literal $releasever $basearch values in the Yum configuration, not shell variables.

    [confluent-7.6]
name=Confluent repository for 7.6.x packages
baseurl=https://packages.confluent.io/rpm/7.6
gpgcheck=1
gpgkey=https://packages.confluent.io/rpm/7.6/archive.key
enabled=1

[Confluent-Clients]
name=Confluent Clients repository
baseurl=https://packages.confluent.io/clients/rpm/centos/$releasever/$basearch
gpgcheck=1
gpgkey=https://packages.confluent.io/clients/rpm/archive.key
enabled=1

  6. Refresh repository metadata.

    sudo yum clean all

  7. Install the new version. Note that yum may override your existing configuration files, so you will need to restore them from the backup after installing the packages.

    sudo yum install confluent-platform

# Or install the packages you need one by one. For example, to install just Kafka:
sudo yum install confluent-kafka

    Important

    If you are running Confluent Server, you install confluent-server instead of confluent-kafka. Both packages can’t co-exist on the same system, so install each desired package individually, for example, run yum install confluent-server instead of yum install confluent-platform.

    Tip

    You can install the entire platform or the individual component packages. For a listing of packages, see Confluent Platform Packages.

  8. Start services.

    kafka-server-start -daemon /etc/kafka/server.properties

Upgrade using TAR or ZIP archives

For ZIP and TAR archives, you can delete the old archives directory after the new archive folder has been created and any previous configuration files have been copied into it as described in the following steps.

  1. Return to the directory where you installed Confluent Platform.

  2. Back up all configuration files from ./etc, including, for example, ./etc/kafka, ./etc/kafka-rest, ./etc/schema-registry, and ./etc/confluent-control-center.

  3. Stop the services and remove the existing packages and their dependencies. This can be done on one server at a time for a rolling upgrade (refer to Steps for upgrading to 7.6.x (ZooKeeper mode)).

      ./bin/control-center-stop
  ./bin/kafka-rest-stop
  ./bin/schema-registry-stop
  ./bin/kafka-server-stop
  ./bin/zookeeper-server-stop

# To remove Confluent Platform and all its dependencies at once, run the following after stopping all services
  cd ..
  rm -R confluent-6.2.1 (use the installed version number)

  4. Unpack the new archive. Note that YUM may override your existing configuration files, so you will need to restore them from the backup after installing the packages.

      tar xzf confluent-7.6.3.tar.gz

# Or for ZIP archives:

  unzip confluent-7.6.3.zip

  5. Start services.

    sudo confluent-7.6.3/bin/zookeeper-server-start -daemon /etc/kafka/zookeeper.properties
sudo confluent-7.6.3/bin/kafka-server-start -daemon /etc/kafka/server.properties

Upgrade Schema Registry

You can upgrade Schema Registry after all Kafka brokers have been upgraded.

The following are a few of the version-specific upgrade considerations:

  • To live update a Schema Registry cluster using Kafka election to version 6.0, first make sure that the previous instances are already updated to version 5.0.1 or newer.

    Instances running Confluent Platform 5.0.1 or later support expansion of Kafka election protocol as a part of upgrades without breaking live upgrade compatibility.

  • Starting with Confluent Platform 6.2.2, Schema Registry provides a new endpoint for Confluent Control Center to filter visible Schema Registry clusters which supports Control Center interaction with an RBAC enabled Schema Registry, and Control Center is upgraded to call this endpoint. Therefore, post-6.2.2 versions of Control Center are not compatible with pre-6.2.2 versions of Schema Registry because Control Center will try to call an endpoint which does not exist in pre-6.2.2 schema registries. If you are upgrading your deployments from Confluent Platform 6.2.1 or older, you must upgrade both Control Center and Schema Registry together to 6.2.2 or newer in order for Control Center to work properly with RBAC enabled Schema Registry clusters. Specifically, if you upgrade Control Center from pre-6.2.2, you must also upgrade Schema Registry.

To upgrade Schema Registry:

  1. For RBAC-enabled environments only: Add ResourceOwner for the REST Proxy user for the Confluent license topic resource (default name is _confluent-command). For example:

    confluent iam rbac role-binding create \
--role ResourceOwner \
--principal User:<service-account-id> \
--resource Topic:_confluent-command \
--kafka-cluster <kafka-cluster-id>

  2. Stop Schema Registry:

    schema-registry-stop

  3. Back up the configuration file. For example:

    cp SR.properties /back-up/SR.properties

  4. Remove the old version:

    yum remove confluent-schema-registry-<old release number>

    For example:

    yum remove confluent-schema-registry-5.1.2

  5. Install the new version:

    yum install -y confluent-schema-registry-<new release number>

  6. Restart Schema Registry:

    schema-registry-start SR.properties

Upgrade Confluent REST Proxy

You can upgrade the Confluent REST Proxy service after all Kafka brokers have been upgraded.

To upgrade the REST Proxy:

  1. For RBAC-enabled environments only: Add ResourceOwner for the REST Proxy user for the Confluent license topic resource (default name is _confluent-command). For example:

    confluent iam rbac role-binding create \
--role ResourceOwner \
--principal User:<service-account-id> \
--resource Topic:_confluent-command \
--kafka-cluster <kafka-cluster>

  2. Follow the same steps as described above to upgrade the package (back up configuration files, remove packages, install upgraded packages, etc.).

  3. Restart the Confluent REST Proxy service.

Upgrade Kafka Streams applications

You can upgrade Kafka Streams applications independently, without requiring Kafka brokers to be upgraded first.

Follow the instructions in the Kafka Streams Upgrade Guide to upgrade your applications to use the latest version of Kafka Streams.

Upgrade Kafka Connect

You can upgrade Kafka Connect in either standalone or distributed mode.

Note

The Confluent Replicator version must match the Connect version it is deployed on. For example, Replicator 7.6 should only be deployed to Connect 7.6, so if you upgrade Connect, you must upgrade Replicator.

Upgrade Kafka Connect standalone mode

You can upgrade Kafka Connect in standalone mode after all Kafka brokers have been upgraded.

To upgrade Kafka Connect, follow the same steps above to upgrade the package (back up config files, remove packages, install upgraded packages, etc.). Then, restart the client processes.

Upgrade Kafka Connect distributed mode

A new required configuration, status.storage.topic was added to Kafka Connect in 0.10.0.1. To upgrade a Kafka Connect cluster, add this configuration before updating to the new version. The setting will be ignored by older versions of Kafka Connect.

  1. Back up worker configuration files.
  2. Modify your configuration file to add the status.storage.topic setting. You can safely modify the configuration file while the worker is running. Note that you should create this topic manually. See Distributed Mode Configuration in the Connect User Guide for a detailed explanation.
  3. Perform a rolling restart of the workers.

Upgrade ksqlDB

To upgrade Confluent Platform ksqlDB to the latest version, follow the steps in Upgrading ksqlDB.

Upgrade Confluent Control Center

Follow the instructions in the Confluent Control Center Upgrade Guide.

Upgrade other client applications

Review Cross-component compatibility before you upgrade your client applications.

Confluent clients (C/C++, Python, Go and .NET) support all released Kafka broker versions, but not all features may be available on all broker versions because some features rely on newer broker functionality. See Build Client Applications for Confluent Platform for the list of Kafka features supported in the latest versions of clients.

If it makes sense, build applications that use Kafka producers and consumers against the new 3.6.x libraries and deploy the new versions. See Schemas, Serializers, and Deserializers for Confluent Platform for details about using the 3.6.x libraries.

Related content