Upgrade

General Upgrade Notes:

  • Always back up all configuration files before upgrading. This includes /etc/kafka, /etc/kafka-rest, /etc/schema-registry and /etc/camus.
  • It is highly recommended to read through the documentation and draft an upgrade plan that matches your specific requirements and environment before starting the upgrade process.
  • Apache Kafka is backward compatible, which means that clients from 0.8.x releases (Confluent Platform 1.0) will work with brokers from 0.9.x releases (Confluent Platform 2.0), but not vice-versa. This means you always need to plan upgrades so that brokers are upgraded before clients. Clients include any application that uses Kafka producer or consumer, Camus, Schema Registry and Rest Proxy.

The general order of upgrade should be:

  1. Upgrade all Kafka Brokers (rolling upgrade or with downtime)
  2. Upgrade Schema Registry, Rest Proxy and Camus
  3. If it makes sense, build applications that use Kafka producers and consumers against the new 0.9.0.0 libraries and deploy the new versions. See Application Development documentation for more details on using the 0.9.0.0 libraries.
  • Confluent Platform supports both rolling upgrade (upgrade one broker at a time to avoid cluster downtime) or downtime upgrade (take down the entire cluster, upgrade it, and bring everything back up).
  • While planning the upgrade it is important to read the specific steps for upgrading Apache Kafka , especially if you are planning on performing a rolling upgrade. There are specific configuration parameters you will need to know about and modify.

Upgrade Servers From CP 1.0.x to CP 2.0.0

You can upgrade from CP 1.0.x to CP 2.0.0 by upgrading the installed packages and restarting the respective processes. The procedure below will work for Kafka brokers, Schema Registry, Rest Proxy and Camus.

deb packages via apt

  1. Backup all configuration files from /etc, including /etc/kafka, /etc/kafka-rest, /etc/schema-registry and /etc/camus.

  2. Stop the services and remove the existing packages and their dependencies. As mentioned above, this can be done on one server at a time for rolling upgrade.

    # The example below removes the Kafka package (for Scala 2.10.4)
    $ sudo kafka-server-stop
    $ sudo apt-get remove confluent-kafka-2.10.4
    
    # To remove Confluent-Platform and all its dependencies at once, run the following after stopping all services
    $ sudo apt-get autoremove confluent-platform-2.10.4
    
  3. Remove the repository files of the previous version

    $ sudo add-apt-repository -r "deb http://packages.confluent.io/deb/1.0 stable main"
    
  4. Add the 2.0 repository to /etc/apt/sources.list

    $ sudo add-apt-repository "deb http://packages.confluent.io/deb/2.0 stable main"
    
  5. Refresh repository metadata

    $ sudo apt-get update
    
  6. Install the new version: (Note that if you modified the configuration files, apt will promote you to resolve the conflicts. You will want to keep your original configuration).

    $ sudo apt-get install confluent-platform-2.11.7
    
    
    # Or install the packages you need one by one. For example, to install just Kafka:
    $ sudo apt-get install confluent-kafka-2.11.7
    

rpm packages via yum

  1. Backup all configuration files from /etc, including /etc/kafka, /etc/kafka-rest, /etc/schema-registry and /etc/camus.

  2. Stop the services and remove the existing packages and their dependencies. As mentioned above, this can be done on one server at a time for rolling upgrade.

    # The example below removes the Kafka package (for Scala 2.10.4)
    $ sudo kafka-server-stop
    $ sudo yum remove confluent-kafka-2.10.4
    
    # To remove Confluent-Platform and all its dependencies at once, run the following after stopping all services
    $ sudo yum autoremove confluent-platform-2.10.4
    
  3. Remove the repository files of the previous version

    $ sudo rm /etc/yum.repos.d/confluent.repo
    
  4. Add the repository to your /etc/yum.repos.d/ directory in a file named confluent-2.0.repo

    [confluent-2.0]
    name=Confluent repository for 2.0.x packages
    baseurl=http://packages.confluent.io/rpm/2.0
    gpgcheck=1
    gpgkey=http://packages.confluent.io/rpm/2.0/archive.key
    enabled=1
    
  5. Refresh repository metadata

    $ sudo yum clean all
    
  6. Install the new version (Note that yum may override your existing configuration files, so you will need to restore them from backup after installing the packages):

    $ sudo yum install confluent-platform-2.11.7
    
    # Or install the packages you need one by one. For example, to install just Kafka:
    $ sudo yum install confluent-kafka-2.11.7
    

After successfully installing the packages, you will need to upgrade Apache Kafka. Refer to the Kafka documentation for guidance.

Only after Kafka was successfully upgraded, you can start the rest of the services such as schema registry or the Kafka REST proxy.