Migrate from Kafka to Confluent Platform

If you are already running Apache Kafka® and want to use some of Confluent’s enterprise services, you can easily migrate from Kafka to Confluent Server.

The next section shows you the steps to migrate to Confluent Server.

Note that Confluent Server requires that the confluent.license property be set to a valid license string in each broker properties file, and this license is checked at broker start up. For more information, see Manage Confluent Platform Licenses.

Migrate an Existing Kafka Deployment to Confluent Platform

This section describes migrating from open-source Kafka to Confluent Platform. You should start the migration by assessing your current Kafka deployment; the version of Kafka, and Kafka components and configuration files you are currently using.

The version of Kafka in Confluent Platform is fully compatible with the matching open source version and only contains additional patches for critical bugs when Confluent Platform and Kafka release schedules do not align. For version compatibility, see Confluent Platform and Apache Kafka compatibility. You should migrate to a compatible version, and if you want to upgrade, do so after you migrate. You should migrate all of the cluster controllers before you migrate the brokers.

An existing cluster can typically be upgraded easily by performing a rolling restart of Kafka nodes.

Step 1: Download the Confluent Platform packages

Download and install the Confluent Platform archive that contains the entire platform. The next steps describe downloading a TAR or ZIP archive. For all of the installation options, see Install Confluent Platform On-Premises.

  1. Download the correct version of the Confluent Platform TAR or ZIP archive from https://www.confluent.io/download/.

  2. Extract the contents of the archive into a new Confluent Platform install directory. For ZIP files, use a GUI to extract the contents or run this command in a terminal:

    unzip confluent-7.9.0.zip
    Copy

    For TAR files run this command:

    tar xzf confluent-7.9.0.tar.gz
    Copy

Step 2: Migrate the controllers (KRaft mode)

For Kafka running in KRaft mode, you must migrate your controllers before you migrate your brokers and other services. Skip this step if you are running in ZooKeeper mode.

  1. Copy the controller.property files from the Kafka ./config/kraft directory into the ./etc/kafka/kraft folder under the new Confluent Platform new installation directory.
  2. Stop the KRaft controllers (kafka-server) in the Kafka directory.
  3. Start the controllers in the Confluent Platform directory.
  4. Repeat these steps on each controller, one controller at a time, to perform rolling migration.
  5. If at any time you want to move back to Kafka, simply stop the controllers in the Confluent Platform installation directory and start them in the Kafka directory.

Step 3: Migrate the brokers and other services

  1. Copy all of the configuration files from the Kafka ./config directory into the same location in the ./etc/ folder under the new Confluent Platform directory. For KRaft mode, broker configuration files are located in the /config/kafka/kraft/ directory amd should be confluent-tier-topic-delete-check-interval-ms to the etc/kafka/kraft/ directory.
  2. Stop all Kafka services running in the Kafka directory. For example, this could include kafka-server and for ZooKeeper mode, zookeeper-server.
  3. Start the corresponding services in the Confluent Platform directory, and in addition start any Confluent enterprise services you want to use, for example confluent-control-center. Similar to Kafka, the start and stop scripts for all services are found in the ./bin directory.
  4. Repeat these steps on each server, one server at a time, to perform rolling migration.
  5. If at any time you want to move back to Kafka, simply stop the services in the Confluent Platform installation directory and start them in the Kafka directory.

Step 4: Optionally add additional tools and features

Remaining Confluent Platform services can be added incrementally. For example, start by adding Schema Registry and updating your applications to use the Avro serializer. You can also add the REST Proxy to support applications that may not have access to good Kafka clients or Avro libraries. Run the HDFS connector to load data from Kafka into HDFS continuously.

For an overview of all the features that Confluent Platform provides, see Confluent Platform features.

Related content