Migrate from Kafka or Confluent Platform Community Edition

Confluent is committed to supporting and developing the open source Apache Kafka® project. You can easily migrate from Apache Kafka® or Confluent Platform Community Edition to Confluent Server.

Migrate an Existing Kafka Deployment to Confluent Platform

This section describes migrating from open-source Kafka to Confluent Platform. You should start the migration by assessing your current Kafka deployment; the version of Kafka, and Kafka components and configuration files you are currently using.

The version of Kafka in Confluent Platform is fully compatible with the matching open source version and only contains additional patches for critical bugs when Confluent Platform and Kafka release schedules do not align. For version compatibility, see Confluent Platform and Apache Kafka compatibility. You should migrate to a compatible version, and if you want to upgrade, do so after you migrate.

An existing cluster can typically be upgraded easily by performing a rolling restart of Kafka brokers.

Download the packages

Download and install a new Confluent Platform archive that contains the entire platform. The next steps describe downloading a TAR or ZIP archive. For all of the installation options, see Install Confluent Platform On-Premises.

  1. Download the correct version of the Confluent Platform TAR or ZIP archive from https://www.confluent.io/download/.

  2. Extract the contents of the archive into a new Confluent Platform install directory. For ZIP files, use a GUI to extract the contents or run this command in a terminal:

    unzip confluent-7.6.0.zip
    

    For TAR files run this command:

    tar xzf confluent-7.6.0.tar.gz
    
  3. Copy all configuration files from ./config into the ./etc folder in the new directory you created in previous step.

  4. Stop all Kafka services running in the Kafka directory. This might include kafka-server and zookeeper-server.

  5. Start the corresponding services in the Confluent Platform directory, and in addition start any new Enterprise services you wish you use, for example confluent-control-center.

  6. Repeat these steps on all Confluent servers, one server at a time, to perform rolling migration.

  7. If at any time you want to move back to Apache Kafka®, simply stop the services in the Confluent Platform installation directory and start them in the Apache Kafka® directory.

Migrate developer tools

The rest of Confluent Platform can be added incrementally. Start by adding Schema Registry and updating your applications to use the Avro serializer. Next, add the REST Proxy to support applications that may not have access to good Kafka clients or Avro libraries. Run the HDFS connector to load data from Kafka into HDFS continuously.

Migrate to Control Center

You can also migrate an existing cluster to take advantage of Control Center. Simply deploy Control Center on a server, configure the server to communicate with your cluster, add interceptors to your Connect instance, producers, and consumers, and you’re ready to go.

Migrate from Confluent Community to full Confluent Platform

This section covers migrating from Confluent Community to full Confluent Platform.

Download new packages

The first step of migration is to install additional components, or download new Confluent Platform packages.

DEB Packages via apt or RPM Packages via yum

If you installed Confluent Platform using only Confluent Community components with yum or apt-get, you can use a yum command to install additional enterprise components. This is because the Confluent Platform packages that contain everything in Confluent Platform using only Confluent Community components plus additional enterprise packages.

Install the additional enterprise packages in your existing community deployment by running:

sudo yum install confluent-platform

or

sudo apt-get install confluent-platform

TAR or ZIP archives

If you installed Confluent Platform using only Confluent Community components from TAR or ZIP archives, you must download and install a new Confluent Platform archive that contains the entire platform.

  1. Download the Confluent Platform TAR or ZIP archive from https://www.confluent.io/download/.
  2. Extract the contents of the archive into a new Confluent Platform install directory. For ZIP files, use a GUI to extract the contents or run this command in a terminal:
unzip confluent-7.6.0.zip
tar xzf confluent-7.6.0.tar.gz

Finish the migration

Use the following steps to finish the migration process.

  1. Copy all configuration files from ./etc, including ./etc/kafka, ./etc/kafka-rest, ./etc/schema-registry, and ./etc/confluent-control-center into the new directory you created in previous step.
  2. Stop all Apache Kafka® services running in the Confluent Platform using only Confluent Community components directory. Depending on what was running, this will include kafka-rest, schema-registry, connect-distributed, kafka-server and zookeeper-server.
  3. Start the corresponding services in the Confluent Platform directory, and in addition start any new Enterprise services you wish you use - for example confluent-control-center.
  4. Repeat these steps on all Confluent servers, one server at a time, to perform rolling migration.
  5. If at any time you want to move back to Confluent Platform using only Confluent Community components, simply stop the services in the Confluent Platform installation directory and start them in the Confluent Platform using only Confluent Community components directory.