Migrate a Multi-Region Cluster from ZooKeeper to KRaft

A multi-region cluster (MRC) is a Kafka deployment that spans more than one Kubernetes cluster or region, with one KRaftController quorum per region. This topic provides the complete sequence for migrating an MRC from ZooKeeper to KRaft, one region at a time, by using Confluent for Kubernetes. The per-region CRs and commands are identical to a single-cluster migration, so each step links to the matching single-cluster step for the detailed configuration. For the single-cluster procedure, see Migrate a Single Cluster from ZooKeeper to KRaft.

Important

Migrate, finalize, and roll back the regions strictly one at a time. This sequential procedure is the required approach for all multi-region clusters. Wait for each region to finish its broker rolls before you start the next region. Do not trigger more than one region at the same time.

Each region’s operator rolls brokers independently, with no coordination across regions. If you trigger more than one region at the same time, brokers that hold replicas of the same partition can restart together. This can make topics unavailable, even if the replication factor is high enough to survive one region restarting on its own.

You can migrate regions in parallel only if your cluster meets the redundancy conditions in Migrate regions in parallel (optional). If you are not sure, use this sequential procedure.

The following table lists the status that each region must reach before you move to the next region:

Stage	Wait for each region to reach
Migration	MIGRATE phase, subphase SubPhaseMigrateMonitorMigrationProgress
Finalization	COMPLETE phase
Rollback	RollbackToZk phase, subphase SubPhaseRollbackToZkComplete

If your MRC is a 2.5DC (two-and-a-half datacenter) topology, it has two full-mode datacenters plus a 0.5DC that runs only a KRaftController quorum and no Kafka brokers. The 0.5DC follows the same one-region-at-a-time sequence as the other regions, with extra configuration called out in Step 1 and Step 2.

Before you start, ensure you have met all the prerequisites in KRaft Migration Prerequisites.

Step 1: Prepare each region

In every region, complete the single-cluster preparation steps. These steps do not roll brokers, so you can complete them in all regions before you start the migration. Each of the following subsections covers a preparation step for the full-mode datacenters first, then the additional configuration the 0.5DC needs in a 2.5DC topology.

Configure the IBP version

In the full-mode datacenters, configure the IBP version as you would for a single cluster. For details, see Step 1.

The 0.5DC has no Kafka brokers, so IBP derivation works differently from full-mode datacenters:

• For standard images, for example, confluentinc/cp-server:7.9.x, the lite-mode KRaftMigrationJob derives the IBP version automatically from the KRaftController image tag. No manual annotation is needed.

• For custom images, the IBP version cannot be auto-derived. Add the platform.confluent.io/kraft-migration-ibp-version annotation to the KRaftMigrationJob CR, given there is no Kafka CR in the 0.5DC. Use the IBP version table in Step 1 to determine the value for your Confluent Platform version:

  kubectl annotate kraftmigrationjob <migration-job-name> \
    platform.confluent.io/kraft-migration-ibp-version="<ibp-version>" \
    -n <0.5dc-namespace>
  

Configure the KRaftController

In the full-mode datacenters, create and apply the KRaftController CR, then confirm it reaches the HOLD state in each region. For details, see Step 2. In CFK 3.3.0 and later, CFK derives the ZooKeeper endpoint for each region using Kafka, so no manual zookeeper.connect configuration is required in the KRaftController.

In a 2.5DC topology, the 0.5DC KRaftController requires configuration that full-mode datacenters do not need. Apply this configuration when you create the KRaftController CR.

ZooKeeper endpoint (required): Set spec.dependencies.migration.zookeeper.endpoint to the full ZooKeeper connection string for all ZooKeeper nodes across all datacenters, including the chroot path. In full-mode datacenters, the KRaftMigrationJob reads ZooKeeper configuration from the Kafka CR. The 0.5DC has no Kafka CR, so set the ZooKeeper endpoint directly on the KRaftController.

Cluster ID (required): Set spec.clusterID manually. In full-mode datacenters, the KRaftMigrationJob derives the cluster ID from the Kafka CR. Fetch the cluster ID from an existing Kafka broker and set it on the KRaftController before you deploy the KRaftMigrationJob:

CLUSTER_ID=$(kubectl exec <kafka-name>-0 -n <kafka-namespace> -- \
  grep cluster.id /mnt/data/data0/logs/meta.properties | cut -d= -f2)

Set the following fields based on your security configurations:

• spec.dependencies.migration.zookeeper.tls: Required if ZooKeeper uses TLS. Set tls.enabled: true and use the TLS port, for example, 2182 instead of 2181.

• spec.dependencies.migration.zookeeper.authentication: Required if ZooKeeper uses authentication, for example, digest authentication.

• spec.dependencies.mdsKafkaCluster: Required if you use RBAC authorization. Point it to a Kafka cluster in a full-mode datacenter, because the 0.5DC has no local Kafka for Metadata Service (MDS) token validation.

The platform.confluent.io/kraft-migration-hold-krc-creation: "true" annotation works the same as in full-mode datacenters. The lite-mode KRaftMigrationJob releases this hold.

For a complete KRaftController example, see the CFK examples repository. Beyond a standard KRaftController, the 0.5DC adds the spec.clusterID and spec.dependencies.migration.zookeeper fields:

spec:
  clusterID: <cluster-id-from-kafka-broker>
  dependencies:
    migration:
      zookeeper:
        endpoint: <zk-node-1>:2182,<zk-node-2>:2182,<zk-node-3>:2182/<chroot>
        tls:
          enabled: true

Step 2: Migrate each region

Migrate the regions one at a time.

1. Apply the KRaftMigrationJob CR in the first region. For details, see Step 3. For a 0.5DC, use a lite-mode KRaftMigrationJob, as described in the following subsection.

2. Monitor the migration job until it reaches the MIGRATE phase with the subphase SubPhaseMigrateMonitorMigrationProgress. At this point, all broker rolls for this region are complete. For details, see Step 4.

3. Repeat the previous steps for each remaining region. Wait for the same status before you move to the next region.

Migrate the 0.5DC with a lite-mode KRaftMigrationJob

Migrate the 0.5DC with a lite-mode KRaftMigrationJob. Create the KRaftMigrationJob without the kafka dependency. Omitting the kafka dependency triggers lite mode, which skips all Kafka-specific migration sub-phases, such as the broker rolling restart and the Kafka finalize RPC.

apiVersion: platform.confluent.io/v1beta1
kind: KRaftMigrationJob
metadata:
  name: <migration-job-name>
  namespace: <0.5dc-namespace>
spec:
  dependencies:
    # No kafka dependency. This omission triggers lite mode.
    zookeeper:
      name: <zookeeper-name>
      namespace: <0.5dc-namespace>
    kRaftController:
      name: <kraftcontroller-name>
      namespace: <0.5dc-namespace>

The lite-mode KRaftMigrationJob progresses through the same phases (SETUP, MIGRATE, DUAL-WRITE, and COMPLETE) but completes faster than full-mode jobs because it has no Kafka brokers to roll.

When the final region completes its broker rolls, the KRaftController quorums across all regions detect that all voters have registered and the cluster transitions to the DUAL-WRITE phase.

Note

DUAL-WRITE is a cluster-wide state, not a per-region state. The last region to finish its broker rolls determines when the cluster reaches DUAL-WRITE.

Step 3: Verify the cluster reaches DUAL-WRITE

Confirm that every region reports the DUAL-WRITE phase before you finalize. For details, see Step 5.1.

Then validate cluster health across all regions. For details, see Step 5.2.

Note

If validation fails, roll back instead of finalizing. For details, see Step 5. Rollback is supported during the SETUP, MIGRATE, and DUAL-WRITE phases.

Step 4: Finalize each region

Finalize the regions one at a time.

Warning

Finalization is irreversible per region. After ZooKeeper is removed from a region’s brokers, that region cannot roll back to ZooKeeper mode.

1. Trigger finalization in the first region. This removes the ZooKeeper dependency from the Kafka brokers and the migration configuration from the KRaftController, each of which requires a roll. For details, see Step 5.3.

2. Wait for the region to reach the COMPLETE phase.

3. Repeat the previous steps for each remaining region. Wait for the COMPLETE phase before you move to the next region.

After all regions report COMPLETE, complete the post-migration tasks. For details, see Step 6.

Step 5: Roll back each region (if needed)

If you need to return to ZooKeeper, roll back the regions one at a time. Rollback involves up to multiple Kafka broker rolls per region, plus a manual step to delete ZooKeeper znodes. For the full rollback procedure, see Roll Back to ZooKeeper.

1. Trigger rollback in the first region. Monitor its status until it reaches SubPhaseRollbackToZkWaitForManualNodeRemovalFromZk, which indicates that the first broker roll is complete and the job is waiting for manual intervention. For details, see Step 1.

2. Delete the /controller and /migration znodes from ZooKeeper as directed by the status condition, then apply the continue annotation to resume the rollback. For details, see Step 2 and Step 3. Wait for the region to reach SubPhaseRollbackToZkComplete.

3. Repeat the previous steps for each remaining region. Wait for SubPhaseRollbackToZkComplete before you move to the next region.

Migrate regions in parallel (optional)

If minimizing migration time is a priority and your cluster can handle multiple brokers restarting at the same time across regions, you can trigger all regions in parallel. Before you do, ensure that both of the following conditions are met:

KRaft quorum availability

The KRaftController quorum must be large enough to keep a majority even when one controller per region restarts at the same time. A two-region deployment needs at least six KRaft controllers (three per region). A three-region deployment needs at least seven controllers. This keeps the quorum majority intact even with one controller down in every region at the same time.

Topic availability

The replication factor for all topics must be greater than the number of regions. Set min.insync.replicas low enough that a partition keeps enough in-sync replicas to accept writes when one replica in each region is offline during its broker restart. For example, in a three-region deployment, RF >= 4 with min.insync.replicas <= RF - 3 ensures that one broker restarting per region does not cause topic unavailability.

If either condition is not met, use the sequential, one-region-at-a-time approach described in this topic.

Related content

• For the single-cluster migration procedure, see Migrate a Single Cluster from ZooKeeper to KRaft.

• For multi-region cluster prerequisites, see Multi-region cluster considerations.

• To understand what happens at each migration phase, see ZooKeeper to KRaft Migration Phases and Sub-phases.

• To roll back the migration, see Roll Back to ZooKeeper.

• For current limitations and workarounds, see Known Issues in ZooKeeper to KRaft Migration.