<a id="ansible-migrate-kraft"></a>

# Migrate Confluent Platform from ZooKeeper to KRaft using Ansible Playbooks

ZooKeeper has been deprecated and is no longer included in Confluent Platform 8.0 and later. This
topic describes the process to migrate a ZooKeeper-based Confluent Platform 7.9 deployment to
KRaft-based 7.9 deployment. After you migrate the Confluent Platform 7.9 to the
KRaft-based deployment, upgrade Confluent Platform to 8.3.0 as described in
[Upgrade KRaft-based Confluent Platform deployment](ansible-upgrade.md#ansible-upgrade-kraft).

To safely migrate your hosts and to achieve zero downtime, Confluent Ansible
performs rolling upgrades host by host, shutting down the component, upgrading
packages, restarting the service, and validating service health before moving
on to the next one.

## Requirements and considerations

* Confluent Ansible only supports migration over the same Confluent Platform version of 7.6 or
  later.

  Note that migrating Confluent Platform 7.6.0 clusters is not recommended for production
  environments.
* You need to upgrade Confluent Platform first before running the migration.

  You cannot upgrade of Confluent Platform version and migrate ZooKeeper to KRaft at the
  same time.
* You can upgrade from ZooKeeper to KRaft in isolated mode (the controller having
  `process.roles=controller` and broker having `process.roles=broker`)

  You cannot migrate to the combined mode where KRaft and brokers are on the
  same process (`role=controller, broker`).
* You can migrate co-located clusters where ZooKeeper and brokers are on the same
  node.
* You can migrate ZooKeeper cluster to KRaft running on the same node.

  Beware of port collisions if co-locating components on the same host. If ZooKeeper
  and KRaft Controller are co-located, use the variables
  `kafka_controller_jolokia_port` and `kafka_controller_jmxexporter_port` to
  define different ports for ZooKeeper and KRaft. For example,
  `kafka_controller_jolokia_port: 7777` and
  `kafka_controller_jmxexporter_port: 8081`.
* ACL is migrated from ZooKeeper to KRaft.
* Confluent Ansible supports one-to-many or many-to-one mapping of ZooKeeper to KRaft
  controllers where the number of ZooKeeper nodes differs from the number of
  controller nodes.
* Confluent Ansible supports migration with the same cluster configurations.

  Different security protocols on the ZooKeeper cluster and the KRaft cluster are
  not recommended in migration.

## Migrate to KRaft

To migrate a ZooKeeper-based Confluent Platform to KRaft:

1. Prepare the inventory file. Use the same inventory file you used for ZooKeeper
   cluster setup.
   1. Enable the migration flag:
      ```yaml
      all:
        vars:
          kraft_migration: true
      ```
   2. Add the `kafka_controller` host to your inventory file. For example:
      ```yaml
      kafka_controller:
        hosts:
          ip1.us-west-2.compute.amazonaws.com:

      zookeeper:
        hosts:
          ip2.us-west-2.compute.amazonaws.com:

      kafka_broker:
        hosts:
          ip3.us-west-2.compute.amazonaws.com:
      ```
   3. If migrating a cluster that is [configured with SASL/SCRAM](ansible-authenticate.md#ansible-sasl-scram-512), set the controller-to-controller authentication
      method using the `kafka_controller_sasl_protocol`
      variable in the `hosts.yml` inventory file.

      For example:
      ```yaml
      all:
        vars:
          sasl_protocol: scram

      kafka_controller:
        vars:
          kafka_controller_sasl_protocol: plain,scram
      ```

      For details and example usage of the variable, see
      [Configure SASL/SCRAM authentication](ansible-authenticate.md#ansible-kraft-scram).
   4. If migrating a cluster with Cluster Linking configured, you must set the
      `password.encoder.secret` and `password.encoder.old.secret` properties in
      the KRaft controller during migration. Use the same `password.encoder.secret`
      and `password.encoder.old.secret` (if set for Kafka) values in Kafka, and
      specify them in the KRaft [custom properties](ansible-configure.md#ansible-custom-properties)
      section (`kafka_controller_custom_properties:`) in the inventory file.
      ```yaml
      all:
        vars:
          kafka_controller_custom_properties:
            password.encoder.secret=<encoder-secret>
            password.encoder.old.secret=<encoder-old-secret>
      ```

      After migration, you can remove these properties.

      If not set, Cluster Linking will stop working after migrating to KRaft, and
      you will have to re-create the cluster links after migration.
2. Run the migration playbook.

   You have two options: migrate in two steps or migrate in one step
   * **Migrate in two steps with validation in between**

     This is the recommended way because rollback can only be done till the
     cluster is in the Dual Write mode. This workflow allows you to stop to
     ensure migration completion before moving to complete the KRaft state.

     To migrate in two steps:
     1. *Phase 1:* Migrate to the Dual Write mode.
        ```bash
        ansible-playbook -i <inventory-file> confluent.platform.ZKtoKraftMigration.yml \
          --tags migrate_to_dual_write
        ```
     2. *Validation phase:*
        1. Check and verify that all data has been migrated without any loss.
        2. Update the ACL authorizer.

           If using ACLs, change the authorizer from AclAuthorizer used for ZooKeeper to
           StandardAuthorizer used for KRaft.

           The authorizer must be updated in both the KRaft controller and broker.
           ```yaml
           kafka_broker_custom_properties:
             authorizer.class.name: org.apache.kafka.metadata.authorizer.StandardAuthorizer

           kafka_controller_custom_properties:
             authorizer.class.name: org.apache.kafka.metadata.authorizer.StandardAuthorizer
           ```
     3. *Phase 2:* complete the migration to the KRaft mode:
        ```bash
        ansible-playbook -i <inventory-file> confluent.platform.ZKtoKraftMigration.yml \
          --tags migrate_to_kraft
        ```
   * **Migrate in one step**

     If you want to migrate in one step without pausing at the Dual Write mode,
     run the following command without tags.
     ```bash
     ansible-playbook -i <inventory-file> confluent.platform.ZKtoKraftMigration.yml
     ```
3. Complete the post-migration cleanup.

   After the cluster is running in KRaft mode, the migration is complete and
   the cluster is now a pure KRaft deployment. Before you run any playbook
   against the cluster again, you must remove all migration-specific settings
   from the inventory file.

   #### IMPORTANT
   Treat any later playbook run as a pure KRaft configuration or update
   workflow, not as a continuation of the migration. If you leave
   `kraft_migration: true` in the inventory file, the standard
   `confluent.platform.all` playbook stops with an error to prevent
   migration logic from running against an already-migrated cluster. For
   details, see [Troubleshoot migration issues](#ansible-migrate-kraft-troubleshoot).

   Complete the following post-migration checklist before any subsequent
   playbook run:
   * Remove `kraft_migration: true` from the inventory file.
   * Remove the `zookeeper:` section from the inventory file. For example:
     ```yaml
     zookeeper:
       hosts:
         ip2.us-west-2.compute.amazonaws.com:
     ```
   * Remove any other ZooKeeper-related or migration-only variables from the
     inventory file that no longer apply to a pure KRaft deployment.
   * If ZooKeeper is not managing multiple Kafka clusters, stop your ZooKeeper ensemble.

## Roll back to ZooKeeper

If the migration fails, you can roll back to the ZooKeeper cluster in the migration
process prior to taking the KRaft controllers out of the migration mode. At
that point, the controller makes dual writes to KRaft and ZooKeeper. Since the data
in ZooKeeper is still consistent with that of the KRaft metadata log, it is still
possible to revert to ZooKeeper.

Once you take the controller out of the migration mode and restart in KRaft
mode, you can no longer roll back to ZooKeeper mode.

To roll back to ZooKeeper manually, follow the steps in [Reverting to ZooKeeper mode](https://docs.confluent.io/platform/current/installation/migrate-zk-kraft.html#reverting-to-zk-mode)
in the Confluent Platform documentation.

<a id="ansible-migrate-kraft-troubleshoot"></a>

## Troubleshoot migration issues

This section describes a few of the potential issues you might encounter while
migrating ZooKeeper to KRaft and presents the steps to troubleshoot the issues.

Migration failed with the error:
: ```text
  {"attempts": 10, "cache_control": "no-cache", "changed": false,
  "content_type": "text/plain; charset=utf-8", "cookies": {}, "cookies_string":
  "", "date": "Fri, 19 Jan 2024 11:12:14 GMT", "elapsed": 0, "expires": "Fri,
  19 Jan 2024 10:12:14 GMT", "json": {"request": {"mbean":
  "kafka.controller:name=ZkMigrationState,type=KafkaController", "type":
  "read"}, "status": 200, "timestamp": 1705662734, "value": {"Value": 2}},
  "msg": "OK (unknown bytes)", "pragma": "no-cache", "redirected": false,
  "status": 200, "transfer_encoding": "chunked", "url":
  "https://localhost:7770/jolokia/read/kafka.controller:type=KafkaController,name=ZkMigrationState"}
  ```
  <br/>
  **Solution:** Increase the `metadata_migration_retries` value. Due to the
  size of the cluster, it might be taking more time to migrate than expected.

Migration failed with the error:
: ```text
  {"msg": "The conditional check '( jolokia_output.content | from_json
  ).value.Value == 1' failed. The error was: Expecting value: line 1 column 1
  (char 0)"}
  ```
  <br/>
  One of the following can cause the error:
  <br/>
  * The KRaft controller has failed. For details of the failure, see
    `server.logs` of the KRaft controller.
  * Jolokia is disabled in the the KRaft controller.
  <br/>
  **Solution:** Enable Jolokia in the KRaft controller if needed. Or review
  and address the issue in `server.log`.

Migration failed with the error:
: ```text
  {"msg": "The conditional check '( jolokia_output.content | from_json
  ).value.Value == 1' failed. The error was: error while evaluating
  conditional (( jolokia_output.content | from_json ).value.Value == 1):
  'dict object' has no attribute 'value'"}
  ```
  <br/>
  Confluent Ansible playbooks are using an older version of Confluent Platform with
  `confluent_package_version` set to `7.5` or earlier.
  <br/>
  **Solution:** Use Confluent Platform 7.6 or later.

Migration failed with an authorization error on an RBAC cluster.
: When migrating an RBAC cluster, the KRaft controller and the Kafka broker
  need to share their principals.
  <br/>
  **Solution:** Add the Kafka and KRaft controller principals in the super user
  variables for both the Kafka broker and the KRaft controller.

Migration failed with the error when you have mTLS with RBAC and custom user:
: ```text
  ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
  java.nio.file.AccessDeniedException: /etc/controller/server.properties
  at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
  ```
  <br/>
  **Workaround:** Define `kafka_controller_user` and
  `kafka_controller_group` variables under `all` and `kafka_broker` in
  your inventory file if it is currently defined in the `kraft_controller`
  section only.
  <br/>
  ```yaml
  all:
    kafka_controller_user: <user>
    kafka_controller_group: <group>
  <br/>
  kafka_broker:
    kafka_controller_user: <user>
    kafka_controller_group: <group>
  ```

The `confluent.platform.all` playbook stops with the error:
: ```text
  kraft_migration flag is enabled. This flag should only be used with
  confluent.platform.ZKtoKraftMigration playbook. Set kraft_migration: false
  to run this playbook.
  ```
  <br/>
  The `kraft_migration: true` flag is still set in the inventory file while
  you run a standard playbook, such as `confluent.platform.all`. This guard
  prevents migration logic from running against a cluster that has already
  migrated to KRaft.
  <br/>
  **Solution:** Complete the post-migration checklist. Remove
  `kraft_migration: true` or set it to `false` and remove the
  `zookeeper:` section from the inventory file before you run the playbook
  again.

Migration stopped with the error:
: ```text
  Migration already completed. Cluster is in KRaft mode (ZkMigrationState=3).
  ```
  <br/>
  You ran the `confluent.platform.ZKtoKraftMigration.yml` playbook against a
  cluster that has already finished migrating and is running in pure KRaft
  mode (`ZkMigrationState=3`). The migration pre-flight check blocks the run
  to avoid re-applying migration steps to an already-migrated cluster.
  <br/>
  **Solution:** You do not need to take action, because the cluster has already
  migrated. To make configuration changes, complete the post-migration
  checklist, and then run a standard playbook, such as
  `confluent.platform.all`.