ZooKeeper to KRaft Migration Phases and Sub-phases

This topic describes the phases and sub-phases of the ZooKeeper to KRaft migration process. Use this reference to monitor migration progress and troubleshoot issues by identifying the current phase in the CFK logs.

Overview

The ZooKeeper to KRaft migration process includes the following phases:

1. SETUP

2. MIGRATE

3. DUAL-WRITE

4. MoveToKRaft or RollbackToZk

5. COMPLETE

ZooKeeper to KRaft migration phase flow diagram

Migration phase: SETUP

When the migration job is in the SETUP phase, the log has the status, Migration current status SETUP/<sub-phase>.

CFK performs the following tasks at each sub-phase during the SETUP phase:

1. SubPhaseSetupDeriveIBPVersion: Derives the inter-broker protocol (IBP) version for migration. CFK automatically determines the IBP from the image tag for standard Confluent images. For custom images, CFK reads the platform.confluent.io/kraft-migration-ibp-version annotation from the Kafka CR. The migration job stores the derived IBP version in its status for use in subsequent phases.

2. SubPhaseSetupAddMigrationAnnotation: Adds the platform.confluent.io/kraft-migration-cr-lock annotation to the ZooKeeper, Kafka, and KRaft controller CRs.

3. SubPhaseSetupCheckHealthyKafka: Ensures Kafka is running healthy.

4. SubPhaseSetupCheckKafkaVersion: Ensures Kafka version is 7.6.0 or later.

5. SubPhaseSetupEnsureKRaftControllerExists: Ensure KRaftController is present in the hold state. The platform.confluent.io/kraft-migration-hold-krc-creation annotation prevents KRaftController from starting until CFK removes the annotation later, at SubPhaseSetupMutateKRaftController.

6. SubPhaseSetupTriggerIBPUpgrade: Upgrades inter.broker.protocol.version using the annotation-specified value for custom images or the operator-determined value for standard Confluent Platform images.

7. SubPhaseSetupEnsureIBPUpgradeComplete: Waits for Kafka roll to complete after CFK upgrades inter.broker.protocol.version in the previous sub-phase.

8. SubPhaseSetupFetchKafkaClusterId: Fetches the Kafka Cluster ID from Jolokia endpoint in Kafka cluster.

9. SubPhaseSetupFetchKafkaZookeeperEndpoint: Fetches the ZooKeeper endpoint from the Kafka CR Status.

10. SubPhaseSetupMutateKRaftController:

    • Removes the hold annotation, platform.confluent.io/kraft-migration-hold-krc-creation.

    • Supplies the Kafka Cluster ID from previous step, so that the bootstrapped cluster has the same cluster ID.

    • Adds migration configuration.

11. SubPhaseSetupKRaftControllerHealthy: Waits for KRaftController to become healthy.

12. SubPhaseSetupCheckKraftControllerVersion: Validates if KRaftController version is 7.6 or later.

13. SubPhaseSetupComplete: Marks the completion of the SETUP phase.

Migration phase: MIGRATE

When the migration job is in the MIGRATE phase, the log has the status, Migration current status MIGRATE/<sub-phase>.

CFK performs the following tasks at each sub-phase during the MIGRATE phase:

1. SubPhaseMigrateTriggerKafkaMigration: Adds the following configurations at this phase:

   • Migration configurations

   • KRaft reference

2. SubPhaseMigrateEnsureKafkaRollComplete: Waits for the Kafka roll to complete.

3. SubPhaseMigrateMonitorMigrationProgress: Monitors the migration progress, and checks if it reaches DUAL-WRITE mode.

4. SubPhaseMigrateDualWrite: Marks the completion of Migration phase.

Migration phase: DUAL-WRITE

The DUAL-WRITE phase does not have any sub-phases.

At this point, Confluent Platform is in DUAL-WRITE state (metadata copies to both ZooKeeper and KRaft controller), the migration job waits for your input to move to the KRaft only mode or to roll back to ZooKeeper.

When you apply the desired annotation, platform.confluent.io/kraft-migration-trigger-finalize-to-kraft or platform.confluent.io/kraft-migration-trigger-rollback-to-zk, the migration job moves to the MoveToKRaftControllerOnly or RollbackToZk phase.

Migration phase: MoveToKRaftControllerOnly

When the migration job is in the MoveToKRaftControllerOnly phase, the log has the status, Migration current status MoveToKRaftControllerOnly/<sub-phase>.

The migration job performs the following tasks at each sub-phase during the MoveToKRaftControllerOnly phase:

1. SubPhaseMoveToKRaftControllerTriggerZkRemoval: Removes ZooKeeper dependency from Kafka.

2. SubPhaseMoveToKRaftControllerEnsureKafkaRollIsComplete: Waits for Kafka roll to finish.

3. SubPhaseMoveToKRaftControllerTriggerKRaftControllerMigrationModeRemoval: Triggers migration config removal from KRaftControllers.

4. SubPhaseMoveToKRaftControllerEnsureKRaftControllerMigrationModeRemovalComplete: Waits for KRaftController roll to complete.

5. SubPhaseMoveToKRaftControllerComplete: Marks completion of the migration.

Migration phase: RollbackToZk

To roll back to ZooKeeper, the migration job performs the following tasks at each sub-phase during the RollbackToZK phase:

1. SubPhaseRollbackToZkMakeProcessRoleEmpty: Removes process.roles properties from Kafka.

2. SubPhaseRollbackToZkEnsureKafkaRollIsComplete: Ensures Kafka roll is complete.

3. SubPhaseRollbackToZkWaitForManualNodeRemovalFromZk: Waits for a manual removal for ZooKeeper controller node.

   When you remove the controller and migration nodes and trigger the node removal process as described in the second and third steps in Rollback to ZooKeeper, the migration moves to the next sub-phase.

4. SubPhaseRollbackToZkRemoveKRaftControllerDepsInKafka: Removes KRaft dependency from Kafka.

5. SubPhaseRollbackToZkEnsureKafkaRollIsComplete2: Ensures Kafka roll is complete.

6. SubPhaseRollbackToZkAddClusterMetadataCleanUpAnnotationInKafka: Adds platform.confluent.io/format-cluster-metadata-in-kafka annotation which directs the init container to remove cluster_metadata directory /mnt/data/data0/logs/__cluster_metadata in Kafka.

7. SubPhaseRollbackToZkEnsureKafkaRollIsComplete3: Ensures Kafka roll is complete.

8. SubPhaseRollbackToZkRemoveClusterMetadataCleanUpAnnotationInKafka: Removes platform.confluent.io/format-cluster-metadata-in-kafka.

9. SubPhaseRollbackToZkEnsureKafkaRollIsComplete4: Ensures Kafka roll is complete.

10. SubPhaseRollbackToZkComplete: Marks completion of rollback to ZooKeeper.

Migration phase: COMPLETE

The final phase of the ZooKeeper to KRaft migration has one of the following status:

• COMPLETE

  The final phase. No sub-phases.

• FAILURE

  The migration enters this phase when it encounters a non-recoverable error.

Related content

• For solutions to common problems, see Troubleshoot ZooKeeper to KRaft Migration Issues.

• For rollback procedures, see Rollback to ZooKeeper.

• For current limitations and workarounds, see Known Issues in ZooKeeper to KRaft Migration.