Known Issues in ZooKeeper to KRaft Migration

This topic lists the known issues in KRaft migration, including workarounds and information about fixed versions.

Tip

Before starting migration, review this page to check if your CFK version has known issues that may affect your migration.

Issue: KRaft rollback to ZooKeeper fails with incompatible IBP version in CFK 2.9.5 and later

Affects: CFK 2.9.5 and later

Symptoms:

When rolling back from KRaft to ZooKeeper, the controller may fail to register brokers with an error similar to the following:

[ERROR] 2025-11-24 06:20:42,779 [quorum-controller-9992-event-handler] org.apache.kafka.controller.ClusterControlManager throwUnsupportedVersionDueToIncompatibleFeature - [ControllerServer id=9992] Unable to register broker 2 because it does not support finalized version 114 of confluent.metadata.version. The broker wants a version between 108 and 108, inclusive.
[INFO] 2025-11-24 06:20:42,779 [quorum-controller-9992-event-handler] org.apache.kafka.controller.QuorumController handleEventException - [ControllerServer id=9992] registerBroker: event failed with UnsupportedVersionException in 246 microseconds. Exception message: Unable to register broker 2 because it does not support finalized version 114 of confluent.metadata.version. The broker wants a version between 108 and 108, inclusive.

Root cause:

The rollback process does not automatically add the platform.confluent.io/format-cluster-metadata-in-kafka annotation to the Kafka custom resource, so the controller does not reformat the cluster metadata with the correct IBP version. The platform.confluent.io/kraft-migration-ibp-version annotation controls the effective IBP used for migration. If the IBP value is incorrect, the migration or rollback fails with the error above.

Workaround:

Before rolling back from KRaft to ZooKeeper, manually add the platform.confluent.io/format-cluster-metadata-in-kafka annotation to the Kafka custom resource. Ensure that platform.confluent.io/kraft-migration-ibp-version is set to the correct IBP version for your target Confluent Platform version. Apply the updated custom resource and then retry the rollback to ZooKeeper.

Fixed version: This issue is fixed in CFK 2.9.8.

Recommendation: If you need rollback capability, upgrade to CFK 2.9.8 or later.

Issue: Migration fails in FIPS mode in CFK 2.8.x

Affects: CFK 2.8.x

Symptoms:

The migration job fails if Federal Information Processing Standards (FIPS) is enabled in the Confluent Platform deployment.

Workaround:

Disable FIPS, migrate to KRaft, and re-enable FIPS.

Fixed version: This issue is fixed in CFK 2.9.0.

Recommendation: If you use FIPS mode, upgrade to CFK 2.9.0 or later before starting the migration.

Issue: A Kafka pod is stuck during the rollback to ZooKeeper phase

Affects: CFK versions earlier than 2.9.0

Symptoms:

In the rollback to ZooKeeper process, a Kafka pod is stuck on the first restart with an error message that a topic was not found.

Workaround: None.

Fixed Version: This issue is fixed in CFK 2.9.0.

Recommendation: If you need rollback capability, ensure you are on CFK 2.9.0 or later.

Issue: During rollback to ZooKeeper, ZooKeeper nodes cannot be deleted

Affects: CFK versions earlier than 2.9.0

Symptoms:

During the rollback to ZooKeeper phase, the KRaftMigrationJob is stuck with the error message that the ZooKeeper nodes cannot be deleted.

Failed to delete some node(s) in the subtree!

Workaround:

Provide the required TLS certificates using the zookeeper-shell or zkcli tool. After you grant the required permission to ZooKeeper, you can complete the rollback process.

Fixed Version: This issue is fixed in CFK 2.9.0.

Recommendation: If you need rollback capability, ensure you are on CFK 2.9.0 or later.

Issue: Migration fails with an authorization error on an RBAC cluster

Affects: All versions

Symptoms:

When migrating an RBAC cluster, the migration fails with authorization errors in KRaftController logs.

Root Cause:

The principal for the KRaft controller needs to be a super user on the Kafka broker, and the Kafka broker’s principal needs to be a super user on the KRaft controller.

Solution:

Add the Kafka and KRaft super users to the Kafka and KRaft CRs as shown below:

kind: kafka
spec:
  authorization:
    superUsers:
    - User:kraft
    - User:kafka
    ...
    type: rbac

kind: KRaftController
spec:
  authorization:
    superUsers:
    - User:kraft
    - User:kafka
    ...
    type: rbac

Prevention:

Always configure super users correctly in both CRs before starting migration. For detailed RBAC configuration instructions, see ZooKeeper to KRaft Migration.

Issue: Kafka pods fail to restart in the initial rolling stage and the migration job gets stuck

Affects: CFK versions earlier than 2.9.5, 2.10.1, and 2.11.1

Symptoms:

If you have the inter-broker protocol version and the TRACE logger set as shown below, the Kafka pod cannot roll due to a known deadlock issue.

inter.broker.protocol.version=3.8

log4j.logger.org.apache.kafka.metadata.migration=TRACE

Workaround:

Remove one of the above configurations from the Kafka CR, and Kafka pods roll as part of the migration step.

Fixed Version: This issue is fixed in CFK 2.9.5, 2.10.1, and 2.11.1.

Recommendation:

  • If you need TRACE logging for debugging, use CFK 2.9.5 or later, 2.10.1 or later, or 2.11.1 or later.

  • Alternatively, remove TRACE logging during migration and enable it again after completion.

Issue: KRaft cannot authenticate to ZooKeeper during migration

Affects: All versions when ZooKeeper uses digest authentication

Symptoms:

When ZooKeeper uses digest authentication and the Kafka CR has zookeeper.set.acl=true in the spec.configOverrides.server section, KRaft cannot correctly authenticate with ZooKeeper. The migration gets stuck in PhaseMigration, not reaching the dual-write phase. And the KRaft log contains:

org.apache.kafka.metadata.migration.MigrationClientAuthException:
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
for /consumers.

Workaround:

Explicitly specify an override in the KRaftController CR to add the JAAS configuration to KRaft as shown below:

kind: KRaftController
spec:
  configOverrides:
    jvm:
      - -Djava.security.auth.login.config=/mnt/secrets/digest-jaas.conf

Prevention:

When using ZooKeeper with digest authentication, always add the JAAS configuration to KRaftController before starting migration.

Reporting new issues

If you encounter an issue not listed here:

  1. Check Troubleshoot ZooKeeper to KRaft Migration Issues for common problems

  2. Collect the following diagnostic information:

    • Migration job YAML

    • CFK operator logs

    • Kafka and KRaftController pod logs

    • Description of symptoms and timeline

  3. Contact Confluent Support <https://support.confluent.io/>`__ with collected information.

Related content