Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Migrate to Confluent Server

Confluent Platform includes a commercial component named confluent-server. This includes a Kafka broker with support for commercial features, Kafka Java client, Kafka Streams, Kafka Connect. This also includes proprietary plugins for security features including Confluent LDAP Authorizer and Authorization using Role-Based Access Control. Confluent Platform also includes a version of confluent-kafka, which is an open source Apache 2.0 licensed component. Both confluent-community and confluent-platform packages have a dependency on confluent-kafka. You can replace confluent-kafka with confluent-server package to use commercial features not available in confluent-kafka. For more information about the packages, see Confluent Platform Packages.

Note

If you have already installed some packages using confluent-kafka, you must remove this package before installing confluent-server. You may need to run yum autoremove or apt autoremove to remove packages installed as dependencies by a previous install.

You can migrate from confluent-kafka to confluent-server using the instructions in the following sections.

Note

The installation package names end with the Scala version that the Kafka is built on. For example, the confluent-platform-2.12 package is for Confluent Platform 5.5.15 and is based on Scala 2.12.

The Zip and Tar packages contain the Confluent Platform version followed by the Scala version. For example, a Zip package, confluent-5.5.15-2.12.zip denotes Confluent Platform version 5.5.15 and Scala version 2.12.

Important

The Confluent Platform package includes Confluent Server by default and requires a confluent.license key in your server.properties file. Starting with Confluent Platform 5.4.x, the Confluent Server broker checks for a license during start-up. You must supply a license string in each broker’s properties file using the confluent.license property as below:

confluent.license=LICENCE_STRING_HERE_NO_QUOTES

If you want to use the Kafka broker, download the confluent-community package. The Kafka broker is the default in all Debian or RHEL and CentOS packages.

Red Hat Enterprise Linux

If your cluster was installed on Red Hat Enterprise Linux (RHEL), you can migrate from confluent-kafka to confluent-server or from confluent-server to confluent-kafka.

Migrating from confluent-kafka to confluent-server

Complete the following steps to migrate to confluent-server when using Red Hat Enterprise Linux (RHEL).

  1. Install your cluster using cp-ansible. This installs confluent-kafka by default.

  2. Start your cluster and verify that it is working as expected.

  3. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka

  4. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  5. Use yum swap to change to confluent-server:

    yum swap confluent-kafka-2.12 confluent-server

  6. Copy your Kafka broker configuration backup to the broker configuration directory:

    sudo cp /tmp/server.properties /etc/kafka/

  7. Copy your override file using the following command:

    sudo cp /tmp/override.con /etc/systemd/system/confluent-server.service.d/

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  8. Copy your JAAS file to the broker configuration directory:

    sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/

  9. Start your Kafka brokers:

    sudo systemctl start confluent-server

Migrating from confluent-server to confluent-kafka

Complete the following steps to change from confluent-server back to confluent-kafka when using Red Hat Enterprise Linux (RHEL).

  1. Log in to each Kafka broker host and stop each broker:

    sudo systemctl stop confluent-server

  2. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-server.service.d/override.conf /tmp

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  3. Use yum swap to change to confluent-kafka:

    yum swap confluent-kafka-2.12 confluent-kafka

  4. Copy your configuration files to the broker configuration directory:

    sudo cp /tmp/server.properties /etc/kafka/

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  5. Copy your JAAS file to the broker configuration directory:

    sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/

  6. Start your Kafka brokers:

    sudo systemctl start confluent-kafka

Ubuntu

If your cluster was installed on Ubuntu, you can migrate from confluent-kafka to confluent-server or from confluent-server to confluent-kafka.

Migrating from confluent-kafka to confluent-server

Complete the following steps to migrate to confluent-server when using Ubuntu.

  1. Install your cluster using cp-ansible. This installs confluent-kafka by default.

  2. Start your cluster and verify that it is working as expected.

  3. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka

  4. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  5. Remove confluent-kafka from your Kafka broker hosts.

    sudo apt-get remove confluent-kafka-2.12

  6. Install confluent-server on your Kafka broker hosts.

    sudo apt-get install confluent-server

  7. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  8. Copy your override file using the following command:

    sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf
/etc/systemd/system/confluent-server.service.d

  9. Start your Kafka brokers:

    sudo systemctl start confluent-server

Migrating from confluent-server to confluent-kafka

Complete the following steps to change from confluent-server back to confluent-kafka when using Ubuntu.

  1. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-server

  2. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp

    If running with Kerberos, you need to back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  3. Remove confluent-server from your Kafka broker hosts.

    sudo apt-get remove confluent-server

  4. Install confluent-kafka on your Kafka broker hosts.

    sudo apt-get install confluent-kafka-2.12

  5. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.

  6. Start your Kafka brokers.

    sudo systemctl start confluent-kafka

Upgrade from 5.2.x to 5.3.x

If using confluent-kafka 5.2.x only

If you are currently using confluent-kafka 5.2.x and are not using commercial features like LDAP authorizer, you can upgrade confluent-kafka using the regular Upgrade Confluent Platform process.

If using confluent-kafka 5.2.x and confluent-security 5.2.x

Important

Note that confluent-security 5.2.x is not compatible with confluent-kafka 5.3.x, so you must uninstall or upgrade confluent-security when upgrading confluent-kafka.

If you are using the existing commercially-licensed LDAP authorizer from confluent-security 5.2.x and want to upgrade to 5.3.x, you must install confluent-server, which contains the LDAP authorizer. You must first uninstall 5.2.x versions of confluent-kafka and confluent-security and then install the new confluent-server package. This can be done on one server at a time for rolling upgrades.

Debian packages via APT

  1. Backup all configuration files from /etc, including, for example, /etc/kafka.

  2. Stop the broker and remove the existing packages and their dependencies.

    # stop the current kafka broker process
  sudo kafka-server-stop

# uninstall existing 5.2.x confluent-kafka
  sudo apt-get remove confluent-kafka-2.12

# uninstall existing 5.2.x confluent-security
  sudo apt-get remove confluent-security

# To remove Confluent Platform and all its dependencies at once, run the following after stopping all services
  sudo apt-get autoremove confluent-platform-2.12

  3. Remove the repository files of the previous version

    sudo add-apt-repository -r "deb https://packages.confluent.io/deb/5.2 stable main"

  4. Add the 5.5 repository to /etc/apt/sources.list.

    sudo add-apt-repository "deb https://packages.confluent.io/deb/5.5 stable main"

  5. Refresh repository metadata.

    sudo apt-get update

  6. Install confluent-server. (Note that if you modified the configuration files, apt will prompt you to resolve the conflicts. You should keep your original configuration.)

    sudo apt-get install confluent-server

  7. Start the Kafka broker. If no additional features are being used (e.g., RBAC), no changes are required for the existing server.properties file.

    kafka-server-start -daemon /etc/kafka/server.properties

RPM packages via Yum

  1. Backup all configuration files from /etc, including /etc/kafka.

  2. Stop the broker and remove the existing packages and their dependencies.

    # stop the current kafka broker process
  sudo kafka-server-stop

# uninstall existing 5.2.x confluent-kafka
  sudo yum remove confluent-kafka-2.12

# uninstall existing 5.2.x confluent-security
  sudo yum remove confluent-security

# To remove Confluent Platform and all its dependencies at once, run the following after stopping all services
  sudo yum autoremove confluent-platform-2.12

  3. Remove the repository files of the previous version

    sudo rm /etc/yum.repos.d/confluent.repo

  4. Add the 5.5 repository to directory in a file named confluent-5.5.repo.

    [confluent-5.5]
name=Confluent repository for 5.5.x packages
baseurl=https://packages.confluent.io/rpm/5.5
gpgcheck=1
gpgkey=https://packages.confluent.io/rpm/5.5/archive.key
enabled=1

  5. Refresh repository metadata.

    sudo yum clean all

  6. Install confluent-server. Note that yum may override your existing configuration files, so you must restore them from backup after installing the packages.

    sudo yum install confluent-server

  7. Restore any updated configuration files and start Kafka broker. If no additional features are being used (e.g., RBAC), no changes are required for the existing server.properties file.

    kafka-server-start -daemon /etc/kafka/server.properties

TAR or ZIP archives

Follow Upgrade Confluent Platform instructions to upgrade your broker using TAR or ZIP archives. Use confluent-5.5.15-2.12.tar.gz or the zip archive confluent-5.5.15-2.12.zip to install Kafka when using commercial features like LDAP authorizer.

Upgrade confluent-kafka and Remove confluent-security

If you are currently using confluent-kafka with LDAP authorizer from confluent-security 5.2.x, but do not intend to upgrade to the commercially licensed confluent-server package, you can upgrade confluent-kafka and use it without the LDAP authorizer. You can use the built-in SimpleAclAuthorizer for authorization using ACLs as described in Authorization using ACLs. You must replace group-based ACLs with equivalent user-based ACLs to ensure that all your existing authorization rules are applied. Note that you must remove or upgrade confluent-security when you upgrade confluent-kafka.