# Configure KRaft Dynamic Controller Quorum using Ansible Playbooks A KRaft dynamic controller quorum lets you add and remove controllers in a running cluster without rebuilding it. Ansible Playbooks for Confluent Platform (Confluent Ansible) supports dynamic quorum as an opt-in feature for Confluent Platform 8.4 and later. This topic describes how to: * Deploy a new Confluent Platform cluster with a dynamic quorum * Migrate an existing static quorum to a dynamic quorum * Add controllers to, and remove controllers from, a dynamic quorum ## Static quorum and dynamic quorum The following table compares the two quorum types. | Characteristic | Static quorum | Dynamic quorum | |---------------------------|------------------------------------------------------------|--------------------------------------------------------------| | Controller configuration | `controller.quorum.voters`, a fixed list of `id@host:port` | `controller.quorum.bootstrap.servers`, a list of `host:port` | | `kraft.version` | `0` | `1` | | Add or remove controllers | Not supported without rebuilding the cluster | Supported at runtime | ## Requirements and considerations * Confluent Platform 8.4.0 or later. * Dedicated controllers in the `kafka_controller` inventory group. * Confluent Ansible supports dynamic quorum for a single cluster. Multi-region deployments are not covered in this topic. * Every controller is reachable from the Ansible host during every playbook run. * Dynamic quorum is opt-in. Set `kraft_dynamic_quorum_enabled: true` in the inventory file to enable it. In the commands that follow, `` is the value of `kafka_controller_port` in your inventory file. The default is `9093`. ### Choose the number of controllers The quorum can elect a leader and accept metadata changes only while a majority of voters is available. For `N` voters, the majority is `floor(N / 2) + 1`. | Voters | Majority | Failures tolerated | |----------|------------|----------------------| | 3 | 2 | 1 | | 4 | 3 | 1 | | 5 | 3 | 2 | | 6 | 4 | 2 | ## Variables | Variable | Default | Description | |--------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `kraft_dynamic_quorum_enabled` | `false` | Set to `true` to use a dynamic quorum. | | `kafka_controller_initial_voter` | `""` | Required for a new deployment. The one controller that creates the new
quorum. The value must be a `kafka_controller` inventory hostname,
exactly as written in the inventory, not an IP address or a hostname
alias. Every controller must use the same value. Only used on the first
deployment. | | `kafka_controller_kraft_auto_join_enabled` | `true` | Controls whether a new controller is promoted from observer to voter
automatically. | Confluent Ansible builds `controller.quorum.bootstrap.servers` from the `kafka_controller` group, and follows `hostname_aliasing_enabled` the same way the controller listeners do. It also sets `controller.quorum.auto.join.enable=true` for you. Confluent Ansible does not set `controller.quorum.voters`, because only a static quorum uses that property. ## Deploy a new cluster with a dynamic quorum To deploy a new cluster with a dynamic quorum from the start: 1. Prepare the inventory file, for example a cluster with three controllers and three brokers. Set the following variables under `all.vars`: ```yaml all: vars: kraft_dynamic_quorum_enabled: true kafka_controller_initial_voter: kcontroller-1.example.com kafka_controller: hosts: kcontroller-1.example.com: kcontroller-2.example.com: kcontroller-3.example.com: kafka_broker: hosts: kbroker-1.example.com: kbroker-2.example.com: kbroker-3.example.com: ``` 2. Run the playbook: ```bash ansible-playbook -i hosts.yml confluent.platform.all ``` Confluent Ansible forms the quorum as follows: 1. Checks that Confluent Platform is 8.4.0 or later and KRaft is enabled. It then checks the data directory of every controller. A new quorum is created only if no controller is formatted yet. If Confluent Ansible cannot reach a controller, the run ends at this step without an error message. Make sure that every controller is reachable before you run the playbook. 2. Formats the initial voter with `--standalone`. It becomes the only voter of the new quorum. 3. Formats every other controller with `--no-initial-controllers` and the same cluster ID. They start as observers. 4. Promotes each observer to a voter through auto-join after it catches up with the leader. 5. Deploys the brokers with `controller.quorum.bootstrap.servers`. You can run the playbook again later. Confluent Ansible never formats a controller that is already formatted. A wiped controller rejoins the existing quorum as an observer and never creates a second quorum, even if it is the initial voter. ### Verify the quorum Run the following commands on any controller: ```bash # Every controller is a voter (Leader or Follower), with low Lag. kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --replication # The quorum has a leader, and CurrentVoters lists every controller. kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --status # kraft.version is 1. kafka-features --bootstrap-controller : \ --command-config /etc/controller/client.properties describe | grep kraft.version ``` In the `describe --replication` output, brokers always show as `Observer`. Controllers show as `Leader` or `Follower`. ### Troubleshoot a new deployment `requires every kafka_controller to set the same kafka_controller_initial_voter` : `kafka_controller_initial_voter` is empty, does not match a `kafka_controller` inventory hostname, or is set differently on some hosts. Set it once under `all.vars` to an inventory hostname. The run ends early with no error message : A controller was unreachable from the Ansible host when Confluent Ansible checked the data directories. The run ends with `failed=0` and no further tasks run. Check that the Ansible host can reach every host in the `kafka_controller` group, bring any unreachable controller online, and run the playbook again. A controller stays an observer : Check that the controller caught up with the leader (low `Lag` in the `describe --replication` output) and that `controller.quorum.auto.join.enable=true` is in its `server.properties` file. If you disabled auto-join, run the following command on that controller to promote it:
```bash kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/server.properties add-controller ``` `Detected a running static KRaft cluster` : The cluster already runs a static quorum. Use the migration procedure in the next section. ## Migrate a static quorum to a dynamic quorum You can migrate an existing static KRaft quorum (`kraft.version=0`) to a dynamic quorum (`kraft.version=1`) in place, without rebuilding the cluster. The migration is a one-time, one-way operation. The playbook rolls the controllers and brokers one at a time, so the cluster keeps serving during the migration. ### Requirements * Confluent Platform 8.4.0 or later, deployed by Confluent Ansible, running a static KRaft quorum. * A healthy controller quorum. The quorum has a leader and every controller is a voter. * The inventory file that you used to deploy the cluster. ### Why a migration playbook is needed You cannot set `kraft_dynamic_quorum_enabled: true` and run the normal playbooks. `controller.quorum.bootstrap.servers` works only after `kraft.version` is `1`, so the feature must be upgraded first while the controllers still use `controller.quorum.voters`. If you run the normal playbooks with `kraft_dynamic_quorum_enabled: true` on a running static cluster, they stop with an error that tells you to run `StaticToDynamicQuorumMigration`. ### Migration stages The migration changes the configuration in the following order. Controllers switch to `controller.quorum.bootstrap.servers` before brokers do. | Stage | Controller configuration | Broker configuration | `kraft.version` | |---------------------------------|---------------------------------------|---------------------------------------|-------------------| | Start | `controller.quorum.voters` | `controller.quorum.voters` | `0` | | After upgrading `kraft.version` | `controller.quorum.voters` | `controller.quorum.voters` | `1` | | After switching controllers | `controller.quorum.bootstrap.servers` | `controller.quorum.voters` | `1` | | After switching brokers | `controller.quorum.bootstrap.servers` | `controller.quorum.bootstrap.servers` | `1` | The playbook runs the following stages. Each stage has its own tag. | Stage | Tag | Action | |---------------------------------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| | Precheck | `static_to_dynamic_quorum_precheck` | Checks that `kraft_dynamic_quorum_enabled` is `true` and runs the
controller health check. | | Upgrade `kraft.version` | `static_to_dynamic_quorum_feature_upgrade` | Runs `kafka-features upgrade --feature kraft.version=1` if the version
is `0`, and then checks that it is `1`. No restart is needed. | | Switch controller configuration | `static_to_dynamic_quorum_controller_config` | Removes `controller.quorum.voters` and sets
`controller.quorum.bootstrap.servers` and
`controller.quorum.auto.join.enable` on every controller. | | Restart controllers | `static_to_dynamic_quorum_controller_restart` | Restarts the controllers one at a time, with a health check after each. | | Switch broker configuration | `static_to_dynamic_quorum_broker_config` | Removes `controller.quorum.voters` and sets
`controller.quorum.bootstrap.servers` on every broker. | | Restart brokers | `static_to_dynamic_quorum_broker_restart` | Restarts the brokers one at a time, with a health check after each. | ### Run the migration Before you run the migration, note that it is a one-way operation. #### WARNING The `kraft.version` upgrade cannot be undone. Kafka does not support downgrading `kraft.version` from `1` to `0`. Run the migration on a staging cluster first. To migrate: 1. Check the starting state on any controller. `kraft.version` must be `0`, and every controller must be a voter: ```bash kafka-features --bootstrap-controller : \ --command-config /etc/controller/client.properties describe | grep kraft.version kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --replication ``` 2. Add `kraft_dynamic_quorum_enabled: true` to the inventory file under `all.vars`. You don’t need `kafka_controller_initial_voter` because it is only used when a new cluster is created. 3. Run the migration playbook before any other playbook that would render the new configuration: ```bash ansible-playbook -i hosts.yml confluent.platform.StaticToDynamicQuorumMigration ``` 4. Verify the migration: ```bash # kraft.version is 1. kafka-features --bootstrap-controller : \ --command-config /etc/controller/client.properties describe | grep kraft.version # Controllers and brokers use bootstrap.servers and no longer use voters. # Run this command on each controller. grep -E '^controller\.quorum\.(voters|bootstrap\.servers)' /etc/controller/server.properties # Run this command on each broker. grep -E '^controller\.quorum\.(voters|bootstrap\.servers)' /etc/kafka/server.properties ``` After the migration, keep `kraft_dynamic_quorum_enabled: true` in the inventory file. The normal playbooks now render the dynamic quorum configuration. ### If the migration stops partway Fix the cause and run the same command again. Every stage is safe to repeat: * If `kraft.version` is already `1`, the upgrade is skipped. * Configuration lines that are already switched are left as they are. * The restart stages restart the controllers and brokers again, one at a time. This causes another rolling restart but no downtime. If the run stopped after the `kraft.version` upgrade, run it again soon. In that state, `kraft.version` is `1` but the controllers and brokers still use `controller.quorum.voters`. Don’t leave the cluster in this state for long, because disaster recovery is harder in this state. You can also run only the remaining stages with `--tags`. For example, `--tags static_to_dynamic_quorum_broker_config,static_to_dynamic_quorum_broker_restart`. ## Add controllers With a dynamic quorum, adding a controller is a normal playbook run. You add the host to the inventory and run the controller playbook. Confluent Ansible formats the new controller to join the existing cluster, and auto-join makes it a voter. You don’t need to run any `kafka-metadata-quorum` commands. Before you begin, make sure that: * The cluster runs a dynamic quorum, deployed as a new cluster or migrated. * The quorum is healthy. Every existing controller shows as `Leader` or `Follower` in the `describe --replication` output. * `kafka_controller_kraft_auto_join_enabled` is `true`, which is the default. * The new hosts are reachable from the Ansible host and from the existing controllers on the controller port, which defaults to `9093`. To add controllers: 1. Add the new hosts at the end of the `kafka_controller` group in the inventory file, with the same settings as the existing controllers. You don’t need to change anything else in the inventory. If `node_id` is not set, Confluent Ansible works out the node ID of each controller from its position in the group. Inserting a host earlier in the list changes the node IDs of the controllers after it, and they fail to start. 2. Run the controller playbook for the new controllers only: ```bash ansible-playbook -i hosts.yml confluent.platform.kafka_controller \ --limit kcontroller-4.example.com,kcontroller-5.example.com ``` * With `--limit`, Confluent Ansible installs and starts only the new controllers. It doesn’t restart the existing controllers. * Without `--limit`, Confluent Ansible also updates `controller.quorum.bootstrap.servers` on the existing controllers so that it lists the new hosts, and restarts them one at a time. This is not required, because controllers learn the voters from the metadata log. Confluent Ansible doesn’t restart the brokers in either case. A broker needs only one reachable controller in its `controller.quorum.bootstrap.servers` list. 3. Verify that every controller, old and new, is a voter: ```bash kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --replication kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --status ``` A new controller can show as `Observer` for a short time while it catches up. Check that every controller has the same `cluster.id`. Run the following command on each controller: ```bash grep cluster.id /var/lib/controller/data/meta.properties ``` ### Replace a controller To replace a controller whose host or disk is lost, remove the old controller from the quorum as described in [Remove controllers](#ansible-kraft-dynamic-quorum-remove), and then add the new host as described in [Add controllers](#ansible-kraft-dynamic-quorum-add). You can reuse the old inventory hostname. ### Troubleshoot added controllers A new controller stays an observer : Check that it caught up with the leader, that it can reach the other controllers on the controller port, and that `controller.quorum.auto.join.enable=true` is in its `server.properties` file. If you disabled auto-join, run the following command on the new controller:
```bash kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/server.properties add-controller ``` A new controller fails to start with `INCONSISTENT_CLUSTER_ID` : The storage of the controller was formatted for a different cluster, for example by a manual `kafka-storage format`. Stop the controller, empty its data directory, which defaults to `/var/lib/controller/data`, and run the controller playbook for that host again. ## Remove controllers Removing a controller is a manual procedure with `kafka-metadata-quorum remove-controller`. Confluent Ansible doesn’t automate it. Before you remove a controller, consider the following: * Remove one controller at a time and check the quorum after each one. * Keep enough voters running. Count the voters that are up, not only the voters in the voter set. Removing a running voter while another one is down can leave the quorum without a majority. * To replace a controller, remove the old controller first and then add the new one. Run the following steps for one controller at a time. The commands assume a package installation with Confluent Ansible defaults. Adapt the paths and service names for archive installations, custom users, or rootless deployments. 1. On any remaining controller, find the node ID and directory ID of the controller to remove: ```bash kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --replication ``` Note the `NodeId` and `DirectoryId` from the same row of the output. 2. If the host is lost, it is already down. Confirm that it is down, and then skip this step. Otherwise, on the controller you are removing, stop and disable KRaft: ```bash sudo systemctl disable --now confluent-kcontroller ``` If you changed the service name, use the value of `kafka_controller_service_name` instead of `confluent-kcontroller`. The controller must stay stopped. Auto-join adds any running, caught-up controller back to the voter set. If you remove a controller that is still running, or leave a removed host in the inventory so the next playbook run starts it again, auto-join adds it back. 3. On any remaining controller, remove the controller from the quorum: ```bash kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties \ remove-controller --controller-id --controller-directory-id ``` If the removed controller was the leader, the remaining voters elect a new leader. 4. Check the quorum. The removed controller must no longer be listed as `Leader` or `Follower`: ```bash kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --replication kafka-metadata-quorum --bootstrap-controller : \ --command-config /etc/controller/client.properties describe --status ``` 5. Update the inventory file: 1. Pin the node IDs first. If `node_id` is not set, Confluent Ansible works out the node ID of each controller from its position in the `kafka_controller` group, such as `9991` and `9992`. Removing a host from the middle of the list gives every controller after it a different node ID on the next playbook run, and those controllers fail to start. On every controller you keep, set `node_id` to its current node ID from the first step. 2. Delete the removed hosts from the `kafka_controller` group. 6. If the host is not lost, decommission it after the previous step confirms it is out of the quorum. A retired host still holds its KRaft data, including the cluster ID and its node ID. If the service starts on it again, for example after a reboot, a playbook run with an old inventory, or reuse of the machine, it tries to rejoin the cluster and auto-join adds it back as a voter. Delete the controller data so that the host has nothing to rejoin with: ```bash sudo rm -rf /var/lib/controller/data/* ``` The service is already stopped and disabled. ### Troubleshoot removed controllers The removed controller shows up as a voter again : The controller was still running or was started again, and auto-join added it back. Stop and disable it, remove it from the inventory, and run the `remove-controller` command again. `remove-controller` fails with an unknown voter or wrong directory ID : The `NodeId` and `DirectoryId` must come from the same row of the `describe --replication` output. Run the `describe --replication` command again and copy both values from the same line. Controllers fail to start after you remove a host from the inventory : `node_id` was not pinned, so removing a host from the middle of the `kafka_controller` group changed the node IDs of the controllers after it. Set `node_id` on each controller back to the node ID stored on its disk, which is `node.id` in `/var/lib/controller/data/meta.properties`, and then run the controller playbook again. A controller host is lost and cannot be stopped : A lost host is already stopped. From a remaining controller, find the node ID and directory ID, remove the controller, and check the quorum. The dead controller is listed with a `LastFetchTimestamp` that stops moving and a growing `Lag`. Lag alone does not mean a controller is gone, so confirm by other means that the host is down. ## Related content * [KIP-853: KRaft Controller Membership Changes](https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes) * [Migrate Confluent Platform from ZooKeeper to KRaft using Ansible Playbooks](ansible-migrate-kraft.md#ansible-migrate-kraft)