Migrate from ZooKeeper to KRaft on Confluent Platform

Migrating from ZooKeeper to KRaft means migrating existing metadata from Kafka brokers that are using ZooKeeper to manage metadata, to brokers that are using a KRaft quorum controller to manage metadata in Apache Kafka®. This topic will walk you through how to perform the migration. To learn more about KRaft, see KRaft Overview for Confluent Platform.

Important

Migration of production clusters is generally available in Confluent Platform version 7.6.1; however, it is recommended that you migrate Confluent Platform version 7.7.0 or later. You should not attempt production migration on any earlier versions of Confluent Platform or Kafka version 3.6.1 or earlier.

Prerequisites

  • Review the KRaft Limitations and Known Issues before you migrate.

  • Upgrade to Confluent Platform 7.7.2. Before you migrate from ZooKeeper to KRaft you should upgrade to the latest Confluent Platform version. See Upgrade Confluent Platform for the considerations and steps to do this.

  • To help with debugging, enable TRACE level logging for metadata migration. Add the following line to the log4j.properties file found in the CONFLUENT_HOME/etc/kafka/ directory:

    log4j.logger.org.apache.kafka.metadata.migration=TRACE

Step 1: Retrieve the cluster ID

You must format storage for your Kafka cluster with the ID of the existing cluster. You can get this ID with the zookeeper-shell tool. For example:

./bin/zookeeper-shell localhost:2181

Connecting to localhost:2181
Welcome to ZooKeeper!

get /cluster/id

{"version":"1","id":"WZEKwK-bS62oT3ZOSU0dgw"}

Save this ID, you will use it later after you configure the KRaft controllers.

Step 2: Configure a KRaft controller quorum

Deploy a set of KRaft controllers that will take over from ZooKeeper. Configure each of the KRaft controllers with the following:

  • A node.id that is unique across all brokers and controllers.
  • Migration enabled with zookeeper.metadata.migration.enable=true.
  • ZooKeeper connection configuration.
  • Other KRaft-mode required properties, such as controller.quorum.voters, and controller.listener.names.

Following is an example controller.properties file, for a controller listening on port 9093:

process.roles=controller
node.id=3000
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093

# Enable the migration
  zookeeper.metadata.migration.enable=true

# ZooKeeper client configuration
  zookeeper.connect=localhost:2181

# Enable migrations for cluster linking
  confluent.cluster.link.metadata.topic.enable=true

# Other configuration entries ...

Step 3: Format storage with the ID you saved previously

Format storage on each node with the ID and the controller configuration file. For example:

./bin/kafka-storage format --config ./etc/kafka/kraft/controller.properties --cluster-id WZEKwK-bS62oT3ZOSU0dgw

You might see output like the following:

Formatting /tmp/kraft-controller-logs with metadata version 3.7-IV0

Step 4: Start each controller

Start each controller, specifying the configuration file with migration enabled.

./bin/kafka-server-start.sh ./etc/kafka/kraft/controller.properties

Step 5: Enable migration on the brokers

Once the KRaft controllers are started, you will reconfigure each broker for KRaft migration and restart the broker. You can do a rolling restart to help ensure cluster availability during the migration. Metadata migration automatically starts when all of the brokers have been restarted.

Set the following for brokers:

  • inter.broker.protocol.version to version 3.7.
  • Enable migration with zookeeper.metadata.migration.enable=true.
  • Enable the cluster linking metadata topic with confluent.cluster.link.metadata.topic.enable=true.
  • The ZooKeeper connection configuration (zookeeper.connect).
  • Any other KRaft-mode required properties, such as controller.quorum.voters and controller.listener.names.

Following is an example configuration file for a broker that is ready for the KRaft migration.

broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

# Set the IBP
  inter.broker.protocol.version=3.7

# Enable the migration
  zookeeper.metadata.migration.enable=true

# Cluster linking metadata topic enabled
  confluent.cluster.link.metadata.topic.enable=true

# ZooKeeper client configuration
  zookeeper.connect=localhost:2181

# KRaft controller quorum configuration
  controller.quorum.voters=3000@localhost:9093
  controller.listener.names=CONTROLLER

When all of the Kafka brokers running with ZooKeeper for metadata management are restarted with migration properties set, the migration automatically begins. When migration is complete, you should see the following entry in a log at INFO level.

Completed migration of metadata from ZooKeeper to KRaft.

Step 6: Migrate the brokers

At this point, the metadata migration should be complete, but the brokers are still running in ZooKeeper mode.

The KRaft controller is running in migration mode, and it will send controller remote procedure calls (RPCs) such as UpdateMetadata and LeaderAndIsr to the ZooKeeper-mode brokers.

To migrate the brokers to KRaft, you need to reconfigure them as KRaft brokers and restart them.

Using the migration broker configuration as an example, replace the broker.id with node.id (maintaining the same identifier) and add process.roles=broker, which signals KRaft mode.

To finish the broker migration:

  • Remove the ZooKeeper configuration and cluster linking entries.
  • If you are using ACLs, change the authorizer class. For more information, see ACL concepts.
  • Restart the broker.

Following is an example of how a server.properties file for a migrated broker might look. Note that ZooKeeper-specific properties are commented out.

process.roles=broker
node.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

# Remove the IBP, KRaft uses "metadata.version" feature flag
# inter.broker.protocol.version=3.7

# Remove the migration enabled flag
# zookeeper.metadata.migration.enable=true

# Remove the cluster linking metadata topic setting
# confluent.cluster.link.metadata.topic.enable=true

# Remove ZooKeeper client configuration
# zookeeper.connect=localhost:2181

# Keep the KRaft controller quorum configuration
  controller.quorum.voters=3000@localhost:9093
  controller.listener.names=CONTROLLER

# If using ACLs, change the authorizer from AclAuthorizer used for ZooKeeper to the StandardAuthorizer used for KRaft.
  authorizer.class.name=kafka.security.authorizer

When you have finished updating the configuration and restarting each broker, the cluster is running in KRaft mode.

Step 7: Take KRaft controllers out of migration mode

The final step of the migration is to remove the zookeeper.metadata.migration.enable property to take the controllers out of migration mode, and to remove the ZooKeeper configuration entry. Following is an example controller.properties file for a controller that is migrated to KRaft mode and is listening on port 9093. Note that the ZooKeeper-specific properties are commented out.

process.roles=controller
node.id=3000
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093

# Disable migration
# zookeeper.metadata.migration.enable=true

# Remove ZooKeeper client configuration.
# zookeeper.connect=localhost:2181

After this step, restart each controller and your cluster should be migrated to KRaft mode.

Related content