Migrate a Multi-Region Cluster from ZooKeeper to KRaft

A multi-region cluster (MRC) is a Kafka deployment that spans more than one Kubernetes cluster or region. This topic provides the complete sequence for migrating from ZooKeeper to KRaft in a multi-region cluster by using Confluent for Kubernetes. The per-region CRs and commands are identical to a single-cluster migration, so each step links to the matching single-cluster step for the detailed configuration. For the single-cluster procedure, see Migrate a Single Cluster from ZooKeeper to KRaft.

Important

The procedure in this topic applies only to multi-region clusters that run entirely on Kubernetes and are deployed and managed by Confluent for Kubernetes.

This procedure applies to both supported MRC topologies:

A full multi-region cluster, where every region runs Kafka brokers. The most common form is a three-datacenter (3DC) deployment.
A 2.5DC (two-and-a-half datacenter) cluster, which has two full-mode datacenters plus a 0.5DC that does not run Kafka brokers. The 0.5DC acts as the tiebreaker for the quorum.

Both topologies follow the same procedure. If a step or sequencing rule differs between a full MRC and a 2.5DC, it is called out on this page.

The version requirements depend on your topology and quorum mode:

Topology	Quorum mode	Supported CFK version	Supported Confluent Platform version
Full multi-region cluster (3DC)	Static	2.9.2 and later	Confluent Platform 7.6.1 through 7.9.x
Full multi-region cluster (3DC)	Dynamic	3.2 and later	Confluent Platform 7.9.6 through 7.9.x
2.5DC	Static	3.3.0 and later	Confluent Platform 7.6.1 through 7.9.x
2.5DC	Dynamic	3.3.0 and later	Confluent Platform 7.9.6 through 7.9.x

Your source cluster must run a ZooKeeper-based Confluent Platform version from Confluent Platform 7.6 through Confluent Platform 7.9.x. Confluent Platform 7.9.x is the last release that supports ZooKeeper, which is removed in Confluent Platform 8.0.

For the supporting examples, see the MRC migration scenarios.

Before you migrate, choose a static or a dynamic controller quorum for the target KRaft cluster. Dynamic quorum is the preferred target for MRC. The quorum mode, your topology, and your security configuration together determine how you sequence the migration across regions. If you use static quorum, ensure to keep the KRaft quorum healthy and your topics available while regions roll. For details, see Choose a quorum mode.

Important

Sequence the migration so that Kafka brokers roll in only one region at a time whenever your topology and quorum mode allow it.

If brokers roll in more than one region at the same time, brokers that hold replicas of the same partition can restart together. This can make topics unavailable, even if the replication factor is high enough to survive one region restarting on its own.

How you sequence the rolls depends on your topology, quorum mode, and security configuration. For details, see Choose a quorum mode. One case requires two regions to roll at the same time. On a full MRC with a static quorum and RBAC, two regions must form the controller majority together, so their brokers roll at the same time. The operator gates each roll on a cluster-wide under-replicated-partitions count of zero, which keeps at most one replica of any partition offline at a time. To avoid this constraint and keep rolls fully serialized, use a dynamic quorum.

You can also roll regions in parallel on purpose if your cluster meets the redundancy conditions in Migrate regions in parallel (optional). If you are not sure, use the sequential procedure for your topology and quorum mode.

The following table lists the status that each region must reach before you move to the next region:

Stage	Wait for each region to reach
Migration	`MIGRATE` phase, subphase `SubPhaseMigrateMonitorMigrationProgress`
Finalization	`COMPLETE` phase
Rollback	`RollbackToZk` phase, subphase `SubPhaseRollbackToZkComplete`

Choose a quorum mode

Decide whether the target KRaft cluster should use a dynamic or a static controller quorum. The quorum mode determines how the quorum forms during migration and how you sequence the regions.

Dynamic quorum (recommended)

With dynamic quorum (kraft.version=1, KIP-853), the bootstrap voter forms the quorum on its own, and the controllers in the other regions join as observers. Quorum formation does not wait for a cross-region voter majority, so the migration is less sensitive to cross-region timing, and you can apply the migration one region at a time.

Dynamic quorum also lets you add and remove controllers and reshape the voter set without rewriting controller.quorum.voters on every node and rolling, and it enables KRaft disaster-recovery tooling such as force-standalone and controller re-join. For a multi-region cluster, this means a lost region is recoverable. For these reasons, dynamic quorum is the preferred target for any multi-region cluster, especially a full MRC where every region runs brokers.

Dynamic quorum requires CFK 3.2 or later and Confluent Platform 7.9.6 or later. On Confluent Platform 7.9.6 and later, you can land directly on kraft.version=1 in a single migration. For the full version and infrastructure requirements, see Prerequisites and requirements.

Static quorum

With static quorum (kraft.version=0), every controller across all regions is a voter, and the quorum needs a majority of those voters to elect a leader and complete the migration. You must apply the migration job in enough regions to form that majority. How this affects broker-roll sequencing depends on your topology and security configuration, as described in Step 2. A lost region cannot be recovered with remove-controller. Use static quorum only if you are not adopting dynamic quorum yet.

In both modes, you must apply the KRaftMigrationJob in every region for the migration to complete. Each region’s migration job releases the platform.confluent.io/kraft-migration-hold-krc-creation hold on that region’s KRaftController. Until you apply the job in every region, the controllers in the remaining regions stay in the HOLD state, and the migration cannot finish. The quorum mode and your topology change only the order in which you apply the jobs, not whether every region needs one.

In a 2.5DC topology, the 0.5DC needs extra configuration that the full-mode datacenters do not. This configuration is called out inline in Step 1 and Step 2.

Before you start, ensure you have met all the prerequisites in KRaft Migration Prerequisites.

Step 1: Prepare each region

In every region, complete the single-cluster preparation steps. These steps do not roll brokers, so you can complete them in all regions before you start the migration. Each of the following subsections covers a preparation step for the full-mode datacenters first, then the additional configuration the 0.5DC needs in a 2.5DC topology.

Configure the IBP version

In the full-mode datacenters, configure the IBP version as you would do for a single cluster. For details, see Step 1.

The 0.5DC has no Kafka brokers, so IBP derivation works differently from full-mode datacenters:

For standard images, for example, confluentinc/cp-server:7.9.x, the lite-mode KRaftMigrationJob derives the IBP version automatically from the KRaftController image tag. No manual annotation is needed.
For custom images, the IBP version cannot be auto-derived. Add the platform.confluent.io/kraft-migration-ibp-version annotation to the KRaftMigrationJob CR, given there is no Kafka CR in the 0.5DC. Use the IBP version table in Step 1 to determine the value for your Confluent Platform version:
```
kubectl annotate kraftmigrationjob <migration-job-name> \
  platform.confluent.io/kraft-migration-ibp-version="<ibp-version>" \
  -n <0.5dc-namespace>
```

Configure the KRaftController

In the full-mode datacenters, create and apply the KRaftController CR, then confirm it reaches the HOLD state in each region. For details, see Step 2. In CFK 3.3.0 and later, CFK derives the ZooKeeper endpoint for each region using Kafka, so no manual zookeeper.connect configuration is required in the KRaftController.

In a 2.5DC topology, the 0.5DC KRaftController requires configuration that full-mode datacenters do not need. Apply this configuration when you create the KRaftController CR.

ZooKeeper endpoint (required): Set spec.dependencies.migration.zookeeper.endpoint to the full ZooKeeper connection string for all ZooKeeper nodes across all datacenters, including the chroot path. In full-mode datacenters, the KRaftMigrationJob reads ZooKeeper configuration from the Kafka CR. The 0.5DC has no Kafka CR, so set the ZooKeeper endpoint directly on the KRaftController.

Cluster ID (required): Set spec.clusterID manually. In full-mode datacenters, the KRaftMigrationJob derives the cluster ID from the Kafka CR. Fetch the cluster ID from an existing Kafka broker and set it on the KRaftController before you deploy the KRaftMigrationJob:

CLUSTER_ID=$(kubectl exec <kafka-name>-0 -n <kafka-namespace> -- \
  grep cluster.id /mnt/data/data0/logs/meta.properties | cut -d= -f2)

Set the following fields based on your security configurations:

spec.dependencies.migration.zookeeper.tls: Required if ZooKeeper uses TLS. Set tls.enabled: true and use the TLS port, for example, 2182 instead of 2181.
spec.dependencies.migration.zookeeper.authentication: Required if ZooKeeper uses authentication, for example, digest authentication.
spec.dependencies.mdsKafkaCluster: Required if you use RBAC authorization. Point it to a Kafka cluster in a full-mode datacenter, because the 0.5DC has no local Kafka for Metadata Service (MDS) token validation.

The platform.confluent.io/kraft-migration-hold-krc-creation: "true" annotation works the same as in full-mode datacenters. The lite-mode KRaftMigrationJob releases this hold.

For a complete KRaftController example, see the CFK examples repository. Beyond a standard KRaftController, the 0.5DC adds the spec.clusterID and spec.dependencies.migration.zookeeper fields:

spec:
  clusterID: <cluster-id-from-kafka-broker>
  dependencies:
    migration:
      zookeeper:
        endpoint: <zk-node-1>:2182,<zk-node-2>:2182,<zk-node-3>:2182/<chroot>
        tls:
          enabled: true

Configure dynamic quorum (recommended)

To migrate to dynamic quorum (kraft.version=1), apply the following configuration when you create the KRaftController CRs. If you are migrating to static quorum, skip this section.

Set dynamicQuorumConfig.enabled: true on the KRaftController in every region.
List every controller across all regions in controllerQuorumVoters on each KRaftController. The migration driver uses the full list to determine the expected quorum size. If the list is incomplete, the migration stays in the PRE_MIGRATION phase with a single-voter quorum.
Choose one region’s KRaftController as the bootstrap voter by setting dynamicQuorumConfig.bootstrapPod: 0 on it. Apply the migration job in this region first.
Set listeners.advertisedListenersEnabled: true on the KRaftController in every region. This is required so that voters advertise their external DNS endpoints. Without it, voters advertise internal svc.cluster.local addresses and cross-cluster controller promotion fails.
Create the bootstrap ConfigMap and the associated Kubernetes RBAC role bindings in the bootstrap voter’s namespace, and set podTemplate.serviceAccountName on the bootstrap voter’s KRaftController so its pod can read and update the ConfigMap.

Every non-bootstrap controller starts as an observer and joins the quorum as a voter later, during the DUAL-WRITE phase. For details, see Step 4.

For a complete dynamic quorum example, see the dynamic-quorum scenario in the CFK examples repository.

Step 2: Migrate each region

Apply the KRaftMigrationJob in every region. How you sequence the regions depends on the quorum mode you chose in Choose a quorum mode, your topology (full MRC or 2.5DC), and, for a static quorum, whether the cluster uses RBAC.

Dynamic quorum (recommended)

With a dynamic quorum, the bootstrap voter forms the quorum on its own, so you do not need a cross-region voter majority before brokers roll. Apply the KRaftMigrationJob one region at a time, starting with the bootstrap voter’s region. This sequence is the same for a full MRC and a 2.5DC, and for both secured and non-secured clusters.

Apply the KRaftMigrationJob CR in the bootstrap voter’s region. For details, see Step 3 (Start migration) in the single-cluster procedure.
Monitor the migration job until it reaches the MIGRATE phase with the subphase SubPhaseMigrateMonitorMigrationProgress. At this point, all broker rolls for this region are complete. For details, see Step 4 (Monitor migration) in the single-cluster procedure.
Repeat the previous steps for each remaining region, one at a time. Wait for the same status before you move to the next region.

Applying one region at a time sequences the broker rolls so that only one region rolls at a time. This eliminates the window where brokers that hold replicas of the same partition could restart in two regions at once.

2.5DC: Migrate the 0.5DC with a lite-mode KRaftMigrationJob, as described in the following section. Because the 0.5DC has no brokers, you can apply its lite-mode job alongside the bootstrap voter’s region without affecting broker-roll sequencing.

Static quorum

With a static quorum, the quorum needs a majority of voters to elect a leader, so you must apply the KRaftMigrationJob in enough regions to form that majority. Whether you can keep broker rolls fully serialized depends on your topology and, for a full MRC, on whether the cluster uses RBAC.

2.5DC: Use the 0.5DC as the tiebreaker. The 0.5DC has a controller voter but no brokers, so you can form the majority with the 0.5DC plus one full-mode region, and only that full-mode region rolls brokers. You can always apply these two regions together, on both secured and non-secured clusters.

Apply the KRaftMigrationJob in the 0.5DC (lite mode) and in the first full-mode region. Their voters form a majority, so the quorum forms and only the full-mode region rolls brokers.
Wait for the full-mode region to reach the MIGRATE phase with the subphase SubPhaseMigrateMonitorMigrationProgress.
Apply the KRaftMigrationJob in each remaining full-mode region, one at a time. Wait for the same status before you move to the next region.

Full MRC, non-secured: Every region runs brokers, and there is no broker-less tiebreaker. On a non-secured cluster, a controller becomes ready on a listener check alone, so each region’s migration job advances to the MIGRATE phase on its own, without a cross-region quorum. Apply one region at a time, and wait for each region to reach the MIGRATE phase with the subphase SubPhaseMigrateMonitorMigrationProgress before you apply the next. Broker rolls stay fully serialized.

Full MRC, with RBAC: The RBAC authorizer must read metadata from the quorum when a controller starts, so a region’s controller cannot become ready until a voter majority is up. The smallest majority spans two broker-bearing regions, which must come up together and roll brokers at the same time. You cannot fully serialize the rolls in this case. To minimize concurrency:

Apply the two regions that form the voter majority. Stagger the two applies by a few minutes to narrow the window in which both regions roll. The operator gates each roll on a cluster-wide under-replicated-partitions count of zero, which keeps at most one replica of any partition offline at a time. With an adequate replication factor, this remains a zero-downtime migration.
Wait for both regions to reach the MIGRATE phase with the subphase SubPhaseMigrateMonitorMigrationProgress.
Apply the KRaftMigrationJob in each remaining region, one at a time. Wait for each to reach the MIGRATE phase with the subphase SubPhaseMigrateMonitorMigrationProgress before you apply the next.

To keep broker rolls fully serialized on a secured full MRC, use a dynamic quorum instead.

Migrate the 0.5DC with a lite-mode KRaftMigrationJob

Migrate the 0.5DC with a lite-mode KRaftMigrationJob. Create the KRaftMigrationJob without the kafka dependency. Omitting the kafka dependency triggers lite mode, which skips all Kafka-specific migration sub-phases, such as the broker rolling restart and the Kafka finalize RPC.

apiVersion: platform.confluent.io/v1beta1
kind: KRaftMigrationJob
metadata:
  name: <migration-job-name>
  namespace: <0.5dc-namespace>
spec:
  dependencies:
    # No kafka dependency. This omission triggers lite mode.
    zookeeper:
      name: <zookeeper-name>
      namespace: <0.5dc-namespace>
    kRaftController:
      name: <kraftcontroller-name>
      namespace: <0.5dc-namespace>

The lite-mode KRaftMigrationJob progresses through the same phases (SETUP, MIGRATE, DUAL-WRITE, and COMPLETE) but completes faster than full-mode jobs because it has no Kafka brokers to roll.

When the final region completes its broker rolls, the KRaftController quorums across all regions detect that all voters have registered and the cluster transitions to the DUAL-WRITE phase.

Note

DUAL-WRITE is a cluster-wide state, not a per-region state. The last region to finish its broker rolls determines when the cluster reaches DUAL-WRITE.

Step 3: Verify the cluster reaches DUAL-WRITE

Confirm that every region reports the DUAL-WRITE phase before you finalize. For details, see Step 5.1.

Then validate cluster health across all regions. For details, see Step 5.2.

Note

If validation fails, roll back instead of finalizing. For details, see Step 6. Rollback is supported during the SETUP, MIGRATE, and DUAL-WRITE phases.

Step 4: Promote observer controllers to voters (dynamic quorum)

This step applies only to a dynamic quorum migration. If you are migrating to static quorum, proceed to Step 5.

With dynamic quorum, only the bootstrap voter starts as a voter. The KRaftController in every other region starts as an observer. While the cluster is in the DUAL-WRITE phase, promote each observer controller to a voter so that the final quorum spans all regions.

Run kafka-metadata-quorum ... add-controller against each observer controller pod, one region at a time. The command promotes the controller pod it runs on, so run it on every observer pod and skip the bootstrap voter, which is already a voter. Point --bootstrap-controller at the bootstrap voter’s external DNS endpoint:

BOOTSTRAP=<bootstrap-voter-external-dns>:9074

kubectl --context <context> exec <observer-controller-pod> -n <namespace> -- \
  kafka-metadata-quorum --bootstrap-controller $BOOTSTRAP add-controller

For a plaintext cluster, no --command-config is needed. For a secured cluster, on CFK 3.3.x and later, add --command-config/opt/confluentinc/etc/kafka/kafka-client.properties to the command.

Confirm that the voter set includes a controller from every region before you finalize:

kubectl --context <context> exec <bootstrap-voter-pod> -n <namespace> -- \
  kafka-metadata-quorum --bootstrap-controller localhost:9074 \
  describe --replication

For the complete dynamic quorum example, including the bootstrap ConfigMap and RBAC, see the dynamic-quorum scenario in the CFK examples repository.

Step 5: Finalize each region

Finalize the regions one at a time. Finalization is always sequential, regardless of your topology, quorum mode, or security configuration. Wait for each region to reach the COMPLETE phase before you finalize the next.

Warning

Finalization is irreversible per region. After ZooKeeper is removed from a region’s brokers, that region cannot roll back to ZooKeeper mode.

Trigger finalization in the first region. This removes the ZooKeeper dependency from the Kafka brokers and the migration configuration from the KRaftController, each of which requires a roll. For details, see Step 5.3.
Wait for the region to reach the COMPLETE phase.
Repeat the previous steps for each remaining region. Wait for the COMPLETE phase before you move to the next region.

After all regions report COMPLETE, complete the post-migration tasks. For details, see Step 6 (Post-migration tasks) in the single-cluster procedure.

Step 6: Roll back each region (if needed)

If you need to return to ZooKeeper, roll back the regions one at a time. Like finalization, rollback is always sequential, regardless of your topology, quorum mode, or security configuration. Rollback involves multiple Kafka broker rolls per region, plus a manual step to delete ZooKeeper znodes. For the full rollback procedure, see Roll Back to ZooKeeper.

Trigger rollback in the first region. Monitor its status until it reaches SubPhaseRollbackToZkWaitForManualNodeRemovalFromZk, which indicates that the first broker roll is complete and the job is waiting for manual intervention. For details, see Step 1.
Delete the /controller and /migration znodes from ZooKeeper as directed by the status condition, then apply the continue annotation to resume the rollback. For details, see Step 2 and Step 3. Wait for the region to reach SubPhaseRollbackToZkComplete.
Repeat the previous steps for each remaining region. Wait for SubPhaseRollbackToZkComplete before you move to the next region.

Migrate regions in parallel (optional)

Migrating regions in parallel is a deliberate optimization in which brokers roll in more than one region at the same time. This differs from the 2.5DC tiebreaker pairing and the RBAC majority pairing in Step 2, where only one broker-bearing region rolls at a time.

If minimizing migration time is a priority and your cluster can handle multiple brokers restarting at the same time across regions, you can trigger all regions in parallel. This applies to both quorum modes, although a dynamic quorum migration already sequences the regions safely when you apply the jobs one at a time. Before you migrate in parallel, ensure that both of the following conditions are met:

KRaft quorum availability: The KRaftController quorum must be large enough to keep a majority even when one controller per region restarts at the same time. A two-region deployment needs at least six KRaft controllers (three per region). A three-region deployment needs at least seven controllers. This keeps the quorum majority intact even with one controller down in every region at the same time.
Topic availability: The replication factor for all topics must be greater than the number of regions. Set min.insync.replicas low enough that a partition keeps enough in-sync replicas to accept writes when one replica in each region is offline during its broker restart. For example, in a three-region deployment, RF >= 4 with min.insync.replicas <= RF - 3 ensures that one broker restarting per region does not cause topic unavailability.

If either condition is not met, use the sequential, one-region-at-a-time guidelines for your topology and quorum mode in Step 2.