.. _upgrade:

Upgrade
=======

.. note:: If you are a |cp| subscriber, and you have questions about upgrades (or need help), please
   feel free to contact us through our `Support Portal <https://support.confluent.io/>`__.

Preparation
-----------

Consider the below guidelines in preparation for the upgrade.

* Always back up all configuration files before upgrading. This includes, for example, ``/etc/kafka``, ``/etc/kafka-rest``, and ``/etc/schema-registry``.
* Read through the documentation and draft an upgrade plan that matches your specific requirements and environment before starting the upgrade process. Put differently, don't start working through this guide on a live cluster. Read the guide entirely, make a plan, then execute the plan.
* Pay careful consideration to the order in which components are upgraded. **Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers.** Version 0.10.2 clients can talk to version 0.10.0 or newer brokers. However, if your brokers are older than 0.10.0, you must upgrade all the brokers in the |ak-tm| cluster before upgrading your clients. Version 0.10.2 brokers support 0.8.x and newer clients. Before 0.10.2, |ak| is backward compatible, which means that clients from |ak| 0.8.x releases (|cp| 1.0.x) will work with brokers from |ak| release 0.8.x through |kafka_branch| (|cp| 2.0.x through |version|.x), but not vice-versa. This means you always need to plan upgrades such that *all* brokers are upgraded before clients. Clients include any application that uses |ak| producer or consumer, command line tools, Camus, |sr|, REST Proxy, |kconnect-long| and |kstreams|.

  * **IMPORTANT:** Due to a bug introduced in |ak| 0.9.0.0 (|cp| 2.0.0), clients that depend on |zk| (old Scala high-level Consumer and MirrorMaker if used with the old consumer) will not work with 0.10.x.x or newer brokers. Therefore, |ak| 0.9.0.0 (|cp| 2.0.0) clients should be upgraded to 0.9.0.1 (|cp| 2.0.1) **before** brokers are upgraded to 0.10.x.x or newer. This step is not necessary for 0.8.x or 0.9.0.1 clients.

  * **IMPORTANT:** Although not recommended, some deployments have clients co-located with brokers (on the same node). In these cases, both the broker and clients share the same packages. This is problematic because *all* brokers must be upgraded before clients are upgraded. Pay careful attention to this when upgrading.

* |ak| 2.0.0 contains changes with potential compatibility impact and deprecations with respect to previous major versions (i.e. 0.8.x.x, 0.9.x.x, 0.10.x.x, 0.11.x.x and 1.0.x). Refer to the `Apache Kafka documentation <https://kafka.apache.org/20/documentation/#upgrade_200_notable>`__ to understand how they affect applications using |ak|.

  * **IMPORTANT:** Support for Java 7 has been dropped, Java 8 is now the minimum version required. For complete compatibility information, see the :ref:`interoperability-versions`.

* Read the :ref:`release_notes`. They contain not only information about noteworthy features, but also changes to configurations that may impact your upgrade.

Step-by-step Guide
------------------

#. Consider using :ref:`Confluent Control Center <controlcenter_install>` to monitor broker status during the :ref:`rolling restart <rolling_restart>`.
#. Determine and install the appropriate Java version. See :ref:`Supported Java Versions <java-sys-req>` to determine which versions are supported for the |cp| version you are upgrading to.
#. Determine if clients are co-located with brokers. If yes, ensure all client processes are not upgraded until *all* |ak| brokers have been upgraded.
#. Decide on performing a rolling upgrade or a downtime upgrade. |cp| supports both rolling upgrades (upgrade one broker at a time to avoid cluster downtime) and downtime upgrades (take down the entire cluster, upgrade it, and bring everything back up).
#. Upgrade *all* |ak| brokers (more below).
#. Upgrade |sr|, REST Proxy and Camus (more below).
#. If it makes sense, build applications that use |ak| producers and consumers against the new |version|.x libraries and deploy the new versions. See :ref:`app_development` documentation for more details on using the |version|.x libraries.

Upgrade All |ak| Brokers
^^^^^^^^^^^^^^^^^^^^^^^^^

In a rolling upgrade scenario, upgrade one |ak| broker at a time, taking into consideration the recommendations for doing :ref:`rolling restarts <rolling_restart>` to avoid downtime for end users. In a downtime upgrade scenario, take the entire cluster down, upgrade each |ak| broker, then start the cluster.

**Steps to upgrade for any fix pack release (e.g. 3.1.1 to 3.1.2):** Any fix pack release will be able to perform a rolling upgrade by simply upgrading each broker one at a time. For each broker:

#. Stop the broker.
#. Upgrade the software (see below for your packaging type).
#. Start the broker.

**Steps for upgrading previous versions to 5.1.x:** In a rolling upgrade scenario, upgrading to |cp| |version|.x (|ak| |kafka_branch|.x) requires special steps because |ak| |kafka_branch|.x includes a change to the on-disk data format (unless the upgrade is from 3.3.x or newer) and the inter-broker protocol. Follow the below steps for a rolling upgrade:

#.  Update ``server.properties`` on all |ak| brokers by modifying the properties ``inter.broker.protocol.version`` and ``log.message.format.version`` to match the currently installed version:

    * For |cp| 2.0.x, use ``inter.broker.protocol.version=0.9.0`` and ``log.message.format.version=0.9.0``
    * For |cp| 3.0.x, use ``inter.broker.protocol.version=0.10.0`` and ``log.message.format.version=0.10.0``
    * For |cp| 3.1.x, use ``inter.broker.protocol.version=0.10.1`` and ``log.message.format.version=0.10.1``
    * For |cp| 3.2.x, use ``inter.broker.protocol.version=0.10.2`` and ``log.message.format.version=0.10.2``
    * For |cp| 3.3.x, use ``inter.broker.protocol.version=0.11.0`` and ``log.message.format.version=0.11.0``
    * For |cp| 4.0.x, use ``inter.broker.protocol.version=1.0`` and ``log.message.format.version=1.0``
    * For |cp| 4.1.x, use ``inter.broker.protocol.version=1.1`` and ``log.message.format.version=1.1``
    * For |cp| 5.0.x, use ``inter.broker.protocol.version=2.0`` and ``log.message.format.version=2.0``
#.  Upgrade each |ak| broker, one at a time.
#.  Once all |ak| brokers have been upgraded, modify ``server.properties`` again by changing the following property: ``inter.broker.protocol.version=2.1``.
#.  Restart each |ak| broker, one at a time, to apply the configuration change.
#.  Once most clients are using |version|.x, modify ``server.properties`` by changing the following property:  ``log.message.format.version=2.1``. Since the message format is the same in |cp| 3.3.x through |version|.x, this step is optional if the upgrade is from |cp| 3.3.x or newer.
#.  If you have overriden the message format as instructed above, you need to do one more rolling restart. Restart each |ak| broker, one at a time, to apply the configuration change.

Instructions for both deb packages and rpm packages are below. For ZIP and TAR archives, the old archives directory can be simply deleted after the new archive folder has been created and any old configuration files copied over.

**deb packages via apt**

#. Backup all configuration files from /etc, including, for example, ``/etc/kafka``, ``/etc/kafka-rest``, and ``/etc/schema-registry``.
#.  Stop the services and remove the existing packages and their dependencies. As mentioned above, this can be done on one server at a time for rolling upgrade.

    .. codewithvars:: bash

        # The example below removes the Kafka package (for Scala |scala_version|)
          sudo kafka-server-stop
          sudo apt-get remove confluent-kafka-|scala_version|

        # To remove Confluent Platform and all its dependencies at once, run the following after stopping all services
          sudo apt-get autoremove confluent-platform-|scala_version|

#. Remove the older GPG key and import the updated key. If you have already imported the updated ``8b1da6120c2bf624`` key, then you can skip this step. However, if you still have the old ``670540c841468433`` key installed, now is the time to remove it and import the ``8b1da6120c2bf624`` key:

   .. codewithvars:: bash

        sudo apt-key del 41468433
        wget -qO - https://packages.confluent.io/deb/|version|/archive.key | sudo apt-key add -

#. Remove the repository files of the previous version

   .. codewithvars:: bash

          sudo add-apt-repository -r "deb https://packages.confluent.io/deb/|version| stable main"

#. Add the |version| repository to /etc/apt/sources.list

   .. codewithvars:: bash

          sudo add-apt-repository "deb https://packages.confluent.io/deb/|version| stable main"

#. Refresh repository metadata

   .. codewithvars:: bash

          sudo apt-get update

#. Install the new version: (Note that if you modified the configuration files, apt will prompt you to resolve the conflicts. You will want to keep your original configuration).

   .. codewithvars:: bash

          sudo apt-get install confluent-platform-|scala_version|

        # Or install the packages you need one by one. For example, to install just Kafka:
          sudo apt-get install confluent-kafka-|scala_version|

#. Start |cp| components.

   .. codewithvars:: bash

          kafka-server-start -daemon /etc/kafka/server.properties


**rpm packages via yum**

#. Backup all configuration files from /etc, including, for example, ``/etc/kafka``, ``/etc/kafka-rest``, and ``/etc/schema-registry``.
#. Stop the services and remove the existing packages and their dependencies. As mentioned above, this can be done on one server at a time for rolling upgrade.

   .. codewithvars:: bash

        # The example below removes the Kafka package (for Scala |scala_version|)
          sudo kafka-server-stop
          sudo yum remove confluent-kafka-|scala_version|

        # To remove Confluent-Platform and all its dependencies at once, run the following after stopping all services
          sudo yum autoremove confluent-platform-|scala_version|

#. Remove the repository files of the previous version

   .. codewithvars:: bash

          sudo rm /etc/yum.repos.d/confluent.repo

#. Remove the older GPG key. This step is optional if you haven't removed Confluent's older (``670540c841468433``) GPG key. Confluent's newer (``8b1da6120c2bf624``) key would appear in the RPM Database as ``gpg-pubkey-0c2bf624-60904208``

   .. codewithvars:: bash

          sudo rpm -e gpg-pubkey-41468433-54d512a8
          sudo rpm --import https://packages.confluent.io/rpm/|version|/archive.key

#. Add the repository to your ``/etc/yum.repos.d/`` directory in a file named :litwithvars:`confluent-|version|.repo`.

   .. codewithvars:: ini

          [confluent-|version|]
          name=Confluent repository for |version|.x packages
          baseurl=https://packages.confluent.io/rpm/|version|
          gpgcheck=1
          gpgkey=https://packages.confluent.io/rpm/|version|/archive.key
          enabled=1

#. Refresh repository metadata

   .. codewithvars:: bash

          sudo yum clean all

#. Install the new version. Note that yum may override your existing configuration files, so you will need to restore them from backup after installing the packages:

   .. codewithvars:: bash

          sudo yum install confluent-platform-|scala_version|

        # Or install the packages you need one by one. For example, to install just Kafka:
          sudo yum install confluent-kafka-|scala_version|

#. Start services.

   .. codewithvars:: bash

          kafka-server-start -daemon /etc/kafka/server.properties

**TAR or ZIP archives**

#.  Go to the directory where you installed |cp|.
#.  Backup all configuration files from ./etc, including, for example, ``./etc/kafka``, ``./etc/kafka-rest``, ``./etc/schema-registry``, and ``./etc/confluent-control-center``.
#.  Stop the services and remove the existing packages and their dependencies. As mentioned above, this can be done on one server at a time for rolling upgrade.

    .. codewithvars:: bash

          ./bin/control-center-stop
          ./bin/kafka-rest-stop
          ./bin/schema-registry-stop
          ./bin/kafka-server-stop
          ./bin/zookeeper-server-stop

        # To remove Confluent Platform and all its dependencies at once, run the following after stopping all services
          cd ..
          rm -R confluent-3.3.1 (use the installed version number)

#.  Unpack the new archive. Note that yum may override your existing configuration files, so you will need to restore them from backup after installing the packages:

    .. codewithvars:: bash

          tar xzf confluent-|release|-|scala_version|.tar.gz

        # Or for ZIP archives:

          unzip confluent-|release|-|scala_version|.zip

#.  Start services.

    .. codewithvars:: bash

          sudo confluent-|release|/bin/zookeeper-server-start -daemon /etc/kafka/zookeeper.properties
          sudo confluent-|release|/bin/kafka-server-start -daemon /etc/kafka/server.properties


Upgrade |sr|
^^^^^^^^^^^^^^^^^^^^^^^

|sr| can be upgraded once *all* |ak| brokers have been upgraded.

To upgrade |sr|, follow the same steps above to upgrade the package (backup config files, remove packages, install upgraded packages, etc.).  Then, restart |sr|.


Upgrade |crest-long|
^^^^^^^^^^^^^^^^^^^^^^^^

The |crest-long| service can be upgraded once *all* Kafka brokers have been upgraded.

To upgrade the |crest| service, follow the same steps above to upgrade the package (backup config files, remove packages, install upgraded packages, etc.).  Then, restart the |crest-long| service.


Upgrade |kstreams| applications
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

|kstreams| applications can be upgraded independently and without requiring
Kafka brokers to have been upgraded first.

Please follow the instructions in the :ref:`Kafka Streams Upgrade Guide <streams_upgrade-guide>` to upgrade your applications to use the latest version of |kstreams|.

.. _upgrade_connect:

Upgrade |kconnect-long|
^^^^^^^^^^^^^^^^^^^^^^^

Upgrade |kconnect-long| Standalone Mode
"""""""""""""""""""""""""""""""""""""""

|kconnect-long| in standalone mode can be upgraded once *all* Kafka brokers have been upgraded.

To upgrade |kconnect-long|, follow the same steps above to upgrade the package (backup config files, remove packages, install upgraded packages, etc.).  Then, restart the client processes.

.. _upgrade_connect_distributed:

Upgrade |kconnect-long| Distributed Mode
""""""""""""""""""""""""""""""""""""""""

A new required configuration, ``status.storage.topic`` was added to |kconnect-long| in 0.10.0.1. To upgrade a |kconnect-long| cluster, this configuration should be added before updating to the new version. The setting will be ignored by older versions of |kconnect-long|.

#. Backup worker configuration files.

#. Modify your configuration file to add the ``status.storage.topic`` setting. You can safely modify the
   configuration file while the worker is running. Note that you should create this topic manually.
   See the :ref:`Distributed Mode Configuration <connect_userguide_distributed_config>` section of the
   :ref:`Connect User Guide <connect_userguide>` for a detailed explanation.

#. Perform a rolling restart of the workers.

Upgrade Camus
^^^^^^^^^^^^^

Camus was deprecated in |cp| 3.0.0 and has been removed in |cp| 5.0.0.


|c3|
^^^^^^^^^^^^^^^^^^^^^^^^

Control Center can be upgraded once *all* Kafka brokers have been upgraded.

Please follow the instructions in the :ref:`Control Center Upgrade Guide <controlcenter_upgrade>`.


Upgrade Other Client Applications
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Any other client application (e.g. applications that use Kafka's Java client, or Confluent's C/C++, Python,
Go or .NET clients) can be upgraded once *all* Kafka brokers have been upgraded.

As mentioned above, if it makes sense, build applications that use Kafka producers and consumers against the new
|kafka_branch|.x libraries and deploy the new versions. See :ref:`app_development` documentation for more details on using
the |kafka_branch|.x libraries.

* **NOTE:** the `Consumer API has changed <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61337336>`_ between Kafka 0.9.0.x and 0.10.0.0.

**NOTES:**

* The `Consumer API has changed <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61337336>`_
  between Kafka 0.9.0.x and 0.10.0.0.

* In librdkafka version 0.11.0, the default value for the ``api.version.request`` configuration property
  has changed from ``false`` to ``true``, meaning that librdkafka will make use of the latest protocol
  features of the broker without the need to set this property to ``true`` explicitly. Due to a bug in
  |ak| 0.9.0.x, this will cause client-broker connections to stall for 10 seconds during
  connection-startup on 0.9.0.x brokers. The workaround for this is to explicitly configure ``api.version.request``
  to ``false`` on clients communicating with <=0.9.0.x brokers.