# Migrate a Multi-Region Cluster from ZooKeeper to KRaft A multi-region cluster (MRC) is a Kafka deployment that spans more than one Kubernetes cluster or region. This topic provides the complete sequence for migrating from ZooKeeper to KRaft in a multi-region cluster by using Confluent for Kubernetes. The per-region CRs and commands are identical to a single-cluster migration, so each step links to the matching single-cluster step for the detailed configuration. For the single-cluster procedure, see [Migrate a Single Cluster from ZooKeeper to KRaft](co-migrate-kraft-procedure.md#co-migrate-kraft-procedure-v2). #### IMPORTANT The procedure in this topic applies only to multi-region clusters that run entirely on Kubernetes and are deployed and managed by Confluent for Kubernetes. This procedure applies to both supported MRC topologies: * A full multi-region cluster, where every region runs Kafka brokers. The most common form is a three-datacenter (3DC) deployment. * A 2.5DC (two-and-a-half datacenter) cluster, which has two full-mode datacenters plus a 0.5DC that does not run Kafka brokers. The 0.5DC acts as the tiebreaker for the quorum. Both topologies follow the same procedure. If a step or sequencing rule differs between a full MRC and a 2.5DC, it is called out on this page. The version requirements depend on your topology and quorum mode: | Topology | Quorum mode | Supported CFK version | Supported Confluent Platform version | |---------------------------------|---------------|-------------------------|----------------------------------------| | Full multi-region cluster (3DC) | Static | 2.9.2 and later | Confluent Platform 7.6.1 through 7.9.x | | Full multi-region cluster (3DC) | Dynamic | 3.2 and later | Confluent Platform 7.9.6 through 7.9.x | | 2.5DC | Static | 3.3.0 and later | Confluent Platform 7.6.1 through 7.9.x | | 2.5DC | Dynamic | 3.3.0 and later | Confluent Platform 7.9.6 through 7.9.x | Your source cluster must run a ZooKeeper-based Confluent Platform version from Confluent Platform 7.6 through Confluent Platform 7.9.x. Confluent Platform 7.9.x is the last release that supports ZooKeeper, which is removed in Confluent Platform 8.0. For the supporting examples, see the [MRC migration scenarios](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/migration/KRaftMigration/multi-region-cluster). Before you migrate, choose a static or a dynamic controller quorum for the target KRaft cluster. Dynamic quorum is the preferred target for MRC. The quorum mode, your topology, and your security configuration together determine how you sequence the migration across regions. If you use static quorum, ensure to keep the KRaft quorum healthy and your topics available while regions roll. For details, see [Choose a quorum mode](#co-migrate-kraft-mrc-quorum-mode-v2). #### IMPORTANT Sequence the migration so that Kafka brokers roll in only one region at a time whenever your topology and quorum mode allow it. If brokers roll in more than one region at the same time, brokers that hold replicas of the same partition can restart together. This can make topics unavailable, even if the replication factor is high enough to survive one region restarting on its own. How you sequence the rolls depends on your topology, quorum mode, and security configuration. For details, see [Choose a quorum mode](#co-migrate-kraft-mrc-quorum-mode-v2). One case requires two regions to roll at the same time. On a full MRC with a static quorum and RBAC, two regions must form the controller majority together, so their brokers roll at the same time. The operator gates each roll on a cluster-wide under-replicated-partitions count of zero, which keeps at most one replica of any partition offline at a time. To avoid this constraint and keep rolls fully serialized, use a dynamic quorum. You can also roll regions in parallel on purpose if your cluster meets the redundancy conditions in [Migrate regions in parallel (optional)](#co-migrate-kraft-mrc-parallel-v2). If you are not sure, use the sequential procedure for your topology and quorum mode. The following table lists the status that each region must reach before you move to the next region: | Stage | Wait for each region to reach | |--------------|---------------------------------------------------------------------| | Migration | `MIGRATE` phase, subphase `SubPhaseMigrateMonitorMigrationProgress` | | Finalization | `COMPLETE` phase | | Rollback | `RollbackToZk` phase, subphase `SubPhaseRollbackToZkComplete` | ## Choose a quorum mode Decide whether the target KRaft cluster should use a dynamic or a static controller quorum. The quorum mode determines how the quorum forms during migration and how you sequence the regions. Dynamic quorum (recommended) : With dynamic quorum (`kraft.version=1`, KIP-853), the bootstrap voter forms the quorum on its own, and the controllers in the other regions join as observers. Quorum formation does not wait for a cross-region voter majority, so the migration is less sensitive to cross-region timing, and you can apply the migration one region at a time.
Dynamic quorum also lets you add and remove controllers and reshape the voter set without rewriting `controller.quorum.voters` on every node and rolling, and it enables KRaft disaster-recovery tooling such as `force-standalone` and controller re-join. For a multi-region cluster, this means a lost region is recoverable. For these reasons, dynamic quorum is the preferred target for any multi-region cluster, especially a full MRC where every region runs brokers.
Dynamic quorum requires CFK 3.2 or later and Confluent Platform 7.9.6 or later. On Confluent Platform 7.9.6 and later, you can land directly on `kraft.version=1` in a single migration. For the full version and infrastructure requirements, see [Prerequisites and requirements](co-configure-kraft-dynamic-quorum.md#co-kraft-dynamic-quorum-prerequisites). Static quorum : With static quorum (`kraft.version=0`), every controller across all regions is a voter, and the quorum needs a majority of those voters to elect a leader and complete the migration. You must apply the migration job in enough regions to form that majority. How this affects broker-roll sequencing depends on your topology and security configuration, as described in [Step 2](#co-migrate-kraft-mrc-migrate-step-v2). A lost region cannot be recovered with `remove-controller`. Use static quorum only if you are not adopting dynamic quorum yet. In both modes, you must apply the `KRaftMigrationJob` in every region for the migration to complete. Each region’s migration job releases the `platform.confluent.io/kraft-migration-hold-krc-creation` hold on that region’s `KRaftController`. Until you apply the job in every region, the controllers in the remaining regions stay in the `HOLD` state, and the migration cannot finish. The quorum mode and your topology change only the order in which you apply the jobs, not whether every region needs one. In a 2.5DC topology, the 0.5DC needs extra configuration that the full-mode datacenters do not. This configuration is called out inline in [Step 1](#co-migrate-kraft-mrc-0-5dc-v2) and [Step 2](#co-migrate-kraft-mrc-0-5dc-job-v2). Before you start, ensure you have met all the prerequisites in [KRaft Migration Prerequisites](co-migrate-kraft-prerequisites.md#co-migrate-kraft-prerequisites-v2). ## Step 1: Prepare each region In every region, complete the single-cluster preparation steps. These steps do not roll brokers, so you can complete them in all regions before you start the migration. Each of the following subsections covers a preparation step for the full-mode datacenters first, then the additional configuration the 0.5DC needs in a 2.5DC topology. ### Configure the IBP version In the full-mode datacenters, configure the IBP version as you would do for a single cluster. For details, see [Step 1](co-migrate-kraft-procedure.md#co-migrate-kraft-configure-crs). The 0.5DC has no Kafka brokers, so IBP derivation works differently from full-mode datacenters: * For standard images, for example, `confluentinc/cp-server:7.9.x`, the lite-mode `KRaftMigrationJob` derives the IBP version automatically from the `KRaftController` image tag. No manual annotation is needed. * For custom images, the IBP version cannot be auto-derived. Add the `platform.confluent.io/kraft-migration-ibp-version` annotation to the `KRaftMigrationJob` CR, given there is no Kafka CR in the 0.5DC. Use the IBP version table in [Step 1](co-migrate-kraft-procedure.md#co-migrate-kraft-configure-crs) to determine the value for your Confluent Platform version: ```bash kubectl annotate kraftmigrationjob \ platform.confluent.io/kraft-migration-ibp-version="" \ -n <0.5dc-namespace> ``` ### Configure the KRaftController In the full-mode datacenters, create and apply the `KRaftController` CR, then confirm it reaches the `HOLD` state in each region. For details, see [Step 2](co-migrate-kraft-procedure.md#co-migrate-kraft-deploy-kraft-kraft-cr-v2). In CFK 3.3.0 and later, CFK derives the ZooKeeper endpoint for each region using Kafka, so no manual `zookeeper.connect` configuration is required in the `KRaftController`. In a 2.5DC topology, the 0.5DC `KRaftController` requires configuration that full-mode datacenters do not need. Apply this configuration when you create the `KRaftController` CR. **ZooKeeper endpoint (required):** Set `spec.dependencies.migration.zookeeper.endpoint` to the full ZooKeeper connection string for all ZooKeeper nodes across all datacenters, including the `chroot` path. In full-mode datacenters, the `KRaftMigrationJob` reads ZooKeeper configuration from the Kafka CR. The 0.5DC has no Kafka CR, so set the ZooKeeper endpoint directly on the `KRaftController`. **Cluster ID (required):** Set `spec.clusterID` manually. In full-mode datacenters, the `KRaftMigrationJob` derives the cluster ID from the Kafka CR. Fetch the cluster ID from an existing Kafka broker and set it on the `KRaftController` before you deploy the `KRaftMigrationJob`: ```bash CLUSTER_ID=$(kubectl exec -0 -n -- \ grep cluster.id /mnt/data/data0/logs/meta.properties | cut -d= -f2) ``` Set the following fields based on your security configurations: * `spec.dependencies.migration.zookeeper.tls`: Required if ZooKeeper uses TLS. Set `tls.enabled: true` and use the TLS port, for example, 2182 instead of 2181. * `spec.dependencies.migration.zookeeper.authentication`: Required if ZooKeeper uses authentication, for example, digest authentication. * `spec.dependencies.mdsKafkaCluster`: Required if you use RBAC authorization. Point it to a Kafka cluster in a full-mode datacenter, because the 0.5DC has no local Kafka for Metadata Service (MDS) token validation. The `platform.confluent.io/kraft-migration-hold-krc-creation: "true"` annotation works the same as in full-mode datacenters. The lite-mode `KRaftMigrationJob` releases this hold. For a complete `KRaftController` example, see the [CFK examples repository](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/migration/KRaftMigration). Beyond a standard `KRaftController`, the 0.5DC adds the `spec.clusterID` and `spec.dependencies.migration.zookeeper` fields: ```yaml spec: clusterID: dependencies: migration: zookeeper: endpoint: :2182,:2182,:2182/ tls: enabled: true ``` ### Configure dynamic quorum (recommended) To migrate to dynamic quorum (`kraft.version=1`), apply the following configuration when you create the `KRaftController` CRs. If you are migrating to static quorum, skip this section. * Set `dynamicQuorumConfig.enabled: true` on the `KRaftController` in every region. * List every controller across all regions in `controllerQuorumVoters` on each `KRaftController`. The migration driver uses the full list to determine the expected quorum size. If the list is incomplete, the migration stays in the `PRE_MIGRATION` phase with a single-voter quorum. * Choose one region’s `KRaftController` as the bootstrap voter by setting `dynamicQuorumConfig.bootstrapPod: 0` on it. Apply the migration job in this region first. * Set `listeners.advertisedListenersEnabled: true` on the `KRaftController` in every region. This is required so that voters advertise their external DNS endpoints. Without it, voters advertise internal `svc.cluster.local` addresses and cross-cluster controller promotion fails. * Create the bootstrap `ConfigMap` and the associated Kubernetes RBAC role bindings in the bootstrap voter’s namespace, and set `podTemplate.serviceAccountName` on the bootstrap voter’s `KRaftController` so its pod can read and update the `ConfigMap`. Every non-bootstrap controller starts as an observer and joins the quorum as a voter later, during the `DUAL-WRITE` phase. For details, see [Step 4](#co-migrate-kraft-mrc-promote-voters-v2). For a complete dynamic quorum example, see the dynamic-quorum scenario in the [CFK examples repository](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/migration/KRaftMigration/multi-region-cluster/2.5dc/dynamic-quorum). ## Step 2: Migrate each region Apply the `KRaftMigrationJob` in every region. How you sequence the regions depends on the quorum mode you chose in [Choose a quorum mode](#co-migrate-kraft-mrc-quorum-mode-v2), your topology (full MRC or 2.5DC), and, for a static quorum, whether the cluster uses RBAC. ### Dynamic quorum (recommended) With a dynamic quorum, the bootstrap voter forms the quorum on its own, so you do not need a cross-region voter majority before brokers roll. Apply the `KRaftMigrationJob` one region at a time, starting with the bootstrap voter’s region. This sequence is the same for a full MRC and a 2.5DC, and for both secured and non-secured clusters. 1. Apply the `KRaftMigrationJob` CR in the bootstrap voter’s region. For details, see [Step 3 (Start migration)](co-migrate-kraft-procedure.md#co-migrate-kraft-deploy-migration-job-v2) in the single-cluster procedure. 2. Monitor the migration job until it reaches the `MIGRATE` phase with the subphase `SubPhaseMigrateMonitorMigrationProgress`. At this point, all broker rolls for this region are complete. For details, see [Step 4 (Monitor migration)](co-migrate-kraft-procedure.md#co-migrate-kraft-monitor-v2-4-1) in the single-cluster procedure. 3. Repeat the previous steps for each remaining region, one at a time. Wait for the same status before you move to the next region. Applying one region at a time sequences the broker rolls so that only one region rolls at a time. This eliminates the window where brokers that hold replicas of the same partition could restart in two regions at once. **2.5DC:** Migrate the 0.5DC with a lite-mode `KRaftMigrationJob`, as described in the following section. Because the 0.5DC has no brokers, you can apply its lite-mode job alongside the bootstrap voter’s region without affecting broker-roll sequencing. ### Static quorum With a static quorum, the quorum needs a majority of voters to elect a leader, so you must apply the `KRaftMigrationJob` in enough regions to form that majority. Whether you can keep broker rolls fully serialized depends on your topology and, for a full MRC, on whether the cluster uses RBAC. **2.5DC:** Use the 0.5DC as the tiebreaker. The 0.5DC has a controller voter but no brokers, so you can form the majority with the 0.5DC plus one full-mode region, and only that full-mode region rolls brokers. You can always apply these two regions together, on both secured and non-secured clusters. 1. Apply the `KRaftMigrationJob` in the 0.5DC (lite mode) and in the first full-mode region. Their voters form a majority, so the quorum forms and only the full-mode region rolls brokers. 2. Wait for the full-mode region to reach the `MIGRATE` phase with the subphase `SubPhaseMigrateMonitorMigrationProgress`. 3. Apply the `KRaftMigrationJob` in each remaining full-mode region, one at a time. Wait for the same status before you move to the next region. **Full MRC, non-secured:** Every region runs brokers, and there is no broker-less tiebreaker. On a non-secured cluster, a controller becomes ready on a listener check alone, so each region’s migration job advances to the `MIGRATE` phase on its own, without a cross-region quorum. Apply one region at a time, and wait for each region to reach the `MIGRATE` phase with the subphase `SubPhaseMigrateMonitorMigrationProgress` before you apply the next. Broker rolls stay fully serialized. **Full MRC, with RBAC:** The RBAC authorizer must read metadata from the quorum when a controller starts, so a region’s controller cannot become ready until a voter majority is up. The smallest majority spans two broker-bearing regions, which must come up together and roll brokers at the same time. You cannot fully serialize the rolls in this case. To minimize concurrency: 1. Apply the two regions that form the voter majority. Stagger the two applies by a few minutes to narrow the window in which both regions roll. The operator gates each roll on a cluster-wide under-replicated-partitions count of zero, which keeps at most one replica of any partition offline at a time. With an adequate replication factor, this remains a zero-downtime migration. 2. Wait for both regions to reach the `MIGRATE` phase with the subphase `SubPhaseMigrateMonitorMigrationProgress`. 3. Apply the `KRaftMigrationJob` in each remaining region, one at a time. Wait for each to reach the `MIGRATE` phase with the subphase `SubPhaseMigrateMonitorMigrationProgress` before you apply the next. To keep broker rolls fully serialized on a secured full MRC, use a dynamic quorum instead. ### Migrate the 0.5DC with a lite-mode KRaftMigrationJob Migrate the 0.5DC with a lite-mode `KRaftMigrationJob`. Create the `KRaftMigrationJob` without the `kafka` dependency. Omitting the `kafka` dependency triggers lite mode, which skips all Kafka-specific migration sub-phases, such as the broker rolling restart and the Kafka finalize RPC. ```yaml apiVersion: platform.confluent.io/v1beta1 kind: KRaftMigrationJob metadata: name: namespace: <0.5dc-namespace> spec: dependencies: # No kafka dependency. This omission triggers lite mode. zookeeper: name: namespace: <0.5dc-namespace> kRaftController: name: namespace: <0.5dc-namespace> ``` The lite-mode `KRaftMigrationJob` progresses through the same phases (`SETUP`, `MIGRATE`, `DUAL-WRITE`, and `COMPLETE`) but completes faster than full-mode jobs because it has no Kafka brokers to roll. When the final region completes its broker rolls, the `KRaftController` quorums across all regions detect that all voters have registered and the cluster transitions to the `DUAL-WRITE` phase. #### NOTE `DUAL-WRITE` is a cluster-wide state, not a per-region state. The last region to finish its broker rolls determines when the cluster reaches `DUAL-WRITE`. ## Step 3: Verify the cluster reaches DUAL-WRITE Confirm that every region reports the `DUAL-WRITE` phase before you finalize. For details, see [Step 5.1](co-migrate-kraft-procedure.md#co-migrate-kraft-apply-annotation-v2-5-1). Then validate cluster health across all regions. For details, see [Step 5.2](co-migrate-kraft-procedure.md#co-migrate-kraft-apply-annotation-v2-5-2). #### NOTE If validation fails, roll back instead of finalizing. For details, see [Step 6](#co-migrate-kraft-mrc-rollback-v2). Rollback is supported during the `SETUP`, `MIGRATE`, and `DUAL-WRITE` phases. ## Step 4: Promote observer controllers to voters (dynamic quorum) This step applies only to a dynamic quorum migration. If you are migrating to static quorum, proceed to [Step 5](#co-migrate-kraft-mrc-finalize-step-v2). With dynamic quorum, only the bootstrap voter starts as a voter. The `KRaftController` in every other region starts as an observer. While the cluster is in the `DUAL-WRITE` phase, promote each observer controller to a voter so that the final quorum spans all regions. Run `kafka-metadata-quorum ... add-controller` against each observer controller pod, one region at a time. The command promotes the controller pod it runs on, so run it on every observer pod and skip the bootstrap voter, which is already a voter. Point `--bootstrap-controller` at the bootstrap voter’s external DNS endpoint: ```bash BOOTSTRAP=:9074 kubectl --context exec -n -- \ kafka-metadata-quorum --bootstrap-controller $BOOTSTRAP add-controller ``` For a plaintext cluster, no `--command-config` is needed. For a secured cluster, on CFK 3.3.x and later, add `--command-config /opt/confluentinc/etc/kafka/kafka-client.properties` to the command. Confirm that the voter set includes a controller from every region before you finalize: ```bash kubectl --context exec -n -- \ kafka-metadata-quorum --bootstrap-controller localhost:9074 \ describe --replication ``` For the complete dynamic quorum example, including the bootstrap `ConfigMap` and RBAC, see the dynamic-quorum scenario in the [CFK examples repository](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/migration/KRaftMigration/multi-region-cluster/2.5dc/dynamic-quorum). ## Step 5: Finalize each region **Finalize the regions one at a time.** Finalization is always sequential, regardless of your topology, quorum mode, or security configuration. Wait for each region to reach the `COMPLETE` phase before you finalize the next. #### WARNING Finalization is irreversible per region. After ZooKeeper is removed from a region’s brokers, that region cannot roll back to ZooKeeper mode. 1. Trigger finalization in the first region. This removes the ZooKeeper dependency from the Kafka brokers and the migration configuration from the `KRaftController`, each of which requires a roll. For details, see [Step 5.3](co-migrate-kraft-procedure.md#co-migrate-kraft-apply-annotation-v2-5-3). 2. Wait for the region to reach the `COMPLETE` phase. 3. Repeat the previous steps for each remaining region. Wait for the `COMPLETE` phase before you move to the next region. After all regions report `COMPLETE`, complete the post-migration tasks. For details, see [Step 6 (Post-migration tasks)](co-migrate-kraft-procedure.md#co-migrate-kraft-post-migration-tasks-v2) in the single-cluster procedure. ## Step 6: Roll back each region (if needed) If you need to return to ZooKeeper, roll back the regions one at a time. Like finalization, rollback is always sequential, regardless of your topology, quorum mode, or security configuration. Rollback involves multiple Kafka broker rolls per region, plus a manual step to delete ZooKeeper znodes. For the full rollback procedure, see [Roll Back to ZooKeeper](co-migrate-kraft-rollback.md#co-migrate-kraft-rollback-v2). 1. Trigger rollback in the first region. Monitor its status until it reaches `SubPhaseRollbackToZkWaitForManualNodeRemovalFromZk`, which indicates that the first broker roll is complete and the job is waiting for manual intervention. For details, see [Step 1](co-migrate-kraft-rollback.md#co-migrate-kraft-rollback-v2). 2. Delete the `/controller` and `/migration` znodes from ZooKeeper as directed by the status condition, then apply the continue annotation to resume the rollback. For details, see [Step 2 and Step 3](co-migrate-kraft-rollback.md#co-migrate-kraft-rollback-step-3). Wait for the region to reach `SubPhaseRollbackToZkComplete`. 3. Repeat the previous steps for each remaining region. Wait for `SubPhaseRollbackToZkComplete` before you move to the next region. ## Migrate regions in parallel (optional) Migrating regions in parallel is a deliberate optimization in which brokers roll in more than one region at the same time. This differs from the 2.5DC tiebreaker pairing and the RBAC majority pairing in [Step 2](#co-migrate-kraft-mrc-migrate-step-v2), where only one broker-bearing region rolls at a time. If minimizing migration time is a priority and your cluster can handle multiple brokers restarting at the same time across regions, you can trigger all regions in parallel. This applies to both quorum modes, although a dynamic quorum migration already sequences the regions safely when you apply the jobs one at a time. Before you migrate in parallel, ensure that both of the following conditions are met: KRaft quorum availability : The `KRaftController` quorum must be large enough to keep a majority even when one controller per region restarts at the same time. A two-region deployment needs at least six KRaft controllers (three per region). A three-region deployment needs at least seven controllers. This keeps the quorum majority intact even with one controller down in every region at the same time. Topic availability : The replication factor for all topics must be greater than the number of regions. Set `min.insync.replicas` low enough that a partition keeps enough in-sync replicas to accept writes when one replica in each region is offline during its broker restart. For example, in a three-region deployment, `RF >= 4` with `min.insync.replicas <= RF - 3` ensures that one broker restarting per region does not cause topic unavailability. If either condition is not met, use the sequential, one-region-at-a-time guidelines for your topology and quorum mode in [Step 2](#co-migrate-kraft-mrc-migrate-step-v2). ## Related content * For the single-cluster migration procedure, see [Migrate a Single Cluster from ZooKeeper to KRaft](co-migrate-kraft-procedure.md#co-migrate-kraft-procedure-v2). * For the single-cluster dynamic quorum migration procedure, see [Migrate from ZooKeeper to KRaft with dynamic quorum](co-configure-kraft-dynamic-quorum.md#co-kraft-dynamic-quorum-zk-migration). * For multi-region cluster prerequisites, see [Multi-region cluster considerations](co-migrate-kraft-prerequisites.md#co-migrate-kraft-mrc-zookeeper-connect-v2). * To understand what happens at each migration phase, see [ZooKeeper to KRaft Migration Phases and Sub-phases](co-migrate-kraft-phases.md#co-migrate-kraft-phases-v2). * To roll back the migration, see [Roll Back to ZooKeeper](co-migrate-kraft-rollback.md#co-migrate-kraft-rollback-v2). * For current limitations and workarounds, see [Known Issues in ZooKeeper to KRaft Migration](co-migrate-kraft-known-issues.md#co-migrate-kraft-known-issues-v2).