Migrate from Kafka or Confluent Platform Community Edition

If you are already running Apache Kafka® or the community features that Confluent offers, and want to use some of Confluent’s enterprise services, you can easily migrate from Kafka or Confluent Platform Community Edition to Confluent Server.

The next sections show you the steps to migrate to Confluent Server.

Note that Confluent Server requires that the confluent.license property be set to a valid license string in each broker properties file, and this license is checked at broker start up. For more information, see Manage Confluent Platform Licenses.

Migrate an Existing Kafka Deployment to Confluent Platform

This section describes migrating from open-source Kafka to Confluent Platform. You should start the migration by assessing your current Kafka deployment; the version of Kafka, and Kafka components and configuration files you are currently using.

The version of Kafka in Confluent Platform is fully compatible with the matching open source version and only contains additional patches for critical bugs when Confluent Platform and Kafka release schedules do not align. For version compatibility, see Confluent Platform and Apache Kafka compatibility. You should migrate to a compatible version, and if you want to upgrade, do so after you migrate.

An existing cluster can typically be upgraded easily by performing a rolling restart of Kafka brokers.

Download the Confluent Platform packages

Download and install the Confluent Platform archive that contains the entire platform. The next steps describe downloading a TAR or ZIP archive. For all of the installation options, see Install Confluent Platform On-Premises.

  1. Download the correct version of the Confluent Platform TAR or ZIP archive from https://www.confluent.io/download/.

  2. Extract the contents of the archive into a new Confluent Platform install directory. For ZIP files, use a GUI to extract the contents or run this command in a terminal:

    unzip confluent-7.6.3.zip

    For TAR files run this command:

    tar xzf confluent-7.6.3.tar.gz

  3. Copy all configuration files from the Kafka ./config directory into the ./etc/kafka folder under the new Confluent Platform directory you created in the previous step.

  4. Stop all Kafka services running in the Kafka directory. This might include kafka-server and zookeeper-server.

  5. Start the corresponding services in the Confluent Platform directory, and in addition start any Confluent enterprise services you want to use, for example confluent-control-center. Similar to Kafka, the start and stop scripts for all services are found in the ./bin directory.

  6. Repeat these steps on each server, one server at a time, to perform rolling migration.

  7. If at any time you want to move back to Kafka, simply stop the services in the Confluent Platform installation directory and start them in the Apache Kafka® directory.

Add additional tools and features

Remaining Confluent Platform services can be added incrementally. For example, start by adding Schema Registry and updating your applications to use the Avro serializer. You can also add the REST Proxy to support applications that may not have access to good Kafka clients or Avro libraries. Run the HDFS connector to load data from Kafka into HDFS continuously.

For an overview of all the features that Confluent Platform provides, see Confluent Platform Features.

Migrate from Confluent Community to full Confluent Platform

This section covers migrating from Confluent Community to full Confluent Platform.

Download new packages

The first step of migration is to install additional components, or download new Confluent Platform packages.

DEB Packages via apt or RPM Packages via yum

If you installed Confluent Platform using only Confluent Community components with yum or apt-get, you can use a yum command to install additional enterprise components. This is because the Confluent Platform packages that contain everything in Confluent Platform using only Confluent Community components plus additional enterprise packages.

Install the additional enterprise packages in your existing community deployment by running:

sudo yum install confluent-platform

or

sudo apt-get install confluent-platform

TAR or ZIP archives

If you installed Confluent Platform using only Confluent Community components from TAR or ZIP archives, you must download and install a new Confluent Platform archive that contains the entire platform.

  1. Download the Confluent Platform TAR or ZIP archive from https://www.confluent.io/download/.
  2. Extract the contents of the archive into a new Confluent Platform install directory. For ZIP files, use a GUI to extract the contents or run this command in a terminal:
unzip confluent-7.6.3.zip

tar xzf confluent-7.6.3.tar.gz

Finish the migration

Use the following steps to finish the migration process.

  1. Copy all configuration files from ./etc, including ./etc/kafka, ./etc/kafka-rest, ./etc/schema-registry, and ./etc/confluent-control-center into the new directory you created in previous step.
  2. Stop all Apache Kafka® services running in the Confluent Platform using only Confluent Community components directory. Depending on what was running, this will include kafka-rest, schema-registry, connect-distributed, kafka-server and zookeeper-server.
  3. Start the corresponding services in the new Confluent Platform directory, and in addition start any new Enterprise services you want to use - for example confluent-control-center.
  4. Repeat these steps on all Confluent servers, one server at a time, to perform rolling migration.
  5. If at any time you want to move back to Confluent Platform using only Confluent Community components, simply stop the services in the Confluent Platform installation directory and start them in the Confluent Platform using only Confluent Community components directory.

Related content