Configure KRaft Dynamic Controller Quorum using Ansible Playbooks

A KRaft dynamic controller quorum lets you add and remove controllers in a running cluster without rebuilding it. Ansible Playbooks for Confluent Platform (Confluent Ansible) supports dynamic quorum as an opt-in feature for Confluent Platform 8.4 and later.

This topic describes how to:

Deploy a new Confluent Platform cluster with a dynamic quorum
Migrate an existing static quorum to a dynamic quorum
Add controllers to, and remove controllers from, a dynamic quorum

Static quorum and dynamic quorum

The following table compares the two quorum types.

Characteristic	Static quorum	Dynamic quorum
Controller configuration	`controller.quorum.voters`, a fixed list of `id@host:port`	`controller.quorum.bootstrap.servers`, a list of `host:port`
`kraft.version`	`0`	`1`
Add or remove controllers	Not supported without rebuilding the cluster	Supported at runtime

Requirements and considerations

Confluent Platform 8.4.0 or later.
Dedicated controllers in the kafka_controller inventory group.
Confluent Ansible supports dynamic quorum for a single cluster. Multi-region deployments are not covered in this topic.
Every controller is reachable from the Ansible host during every playbook run.
Dynamic quorum is opt-in. Set kraft_dynamic_quorum_enabled: true in the inventory file to enable it.

In the commands that follow, <controller_port> is the value of kafka_controller_port in your inventory file. The default is 9093.

Choose the number of controllers

The quorum can elect a leader and accept metadata changes only while a majority of voters is available. For N voters, the majority is floor(N / 2) + 1.

Voters	Majority	Failures tolerated
3	2	1
4	3	1
5	3	2
6	4	2

Variables

Variable	Default	Description
`kraft_dynamic_quorum_enabled`	`false`	Set to `true` to use a dynamic quorum.
`kafka_controller_initial_voter`	`""`	Required for a new deployment. The one controller that creates the new quorum. The value must be a `kafka_controller` inventory hostname, exactly as written in the inventory, not an IP address or a hostname alias. Every controller must use the same value. Only used on the first deployment.
`kafka_controller_kraft_auto_join_enabled`	`true`	Controls whether a new controller is promoted from observer to voter automatically.

Confluent Ansible builds controller.quorum.bootstrap.servers from the kafka_controller group, and follows hostname_aliasing_enabled the same way the controller listeners do. It also sets controller.quorum.auto.join.enable=true for you. Confluent Ansible does not set controller.quorum.voters, because only a static quorum uses that property.

Deploy a new cluster with a dynamic quorum

To deploy a new cluster with a dynamic quorum from the start:

Prepare the inventory file, for example a cluster with three controllers and three brokers. Set the following variables under all.vars:

all:
  vars:
    kraft_dynamic_quorum_enabled: true
    kafka_controller_initial_voter: kcontroller-1.example.com

kafka_controller:
  hosts:
    kcontroller-1.example.com:
    kcontroller-2.example.com:
    kcontroller-3.example.com:

kafka_broker:
  hosts:
    kbroker-1.example.com:
    kbroker-2.example.com:
    kbroker-3.example.com:

Run the playbook:

ansible-playbook -i hosts.yml confluent.platform.all

Confluent Ansible forms the quorum as follows:

Checks that Confluent Platform is 8.4.0 or later and KRaft is enabled. It then checks the data directory of every controller. A new quorum is created only if no controller is formatted yet. If Confluent Ansible cannot reach a controller, the run ends at this step without an error message. Make sure that every controller is reachable before you run the playbook.
Formats the initial voter with --standalone. It becomes the only voter of the new quorum.
Formats every other controller with --no-initial-controllers and the same cluster ID. They start as observers.
Promotes each observer to a voter through auto-join after it catches up with the leader.
Deploys the brokers with controller.quorum.bootstrap.servers.

You can run the playbook again later. Confluent Ansible never formats a controller that is already formatted. A wiped controller rejoins the existing quorum as an observer and never creates a second quorum, even if it is the initial voter.

Verify the quorum

Run the following commands on any controller:

# Every controller is a voter (Leader or Follower), with low Lag.
kafka-metadata-quorum --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe --replication

# The quorum has a leader, and CurrentVoters lists every controller.
kafka-metadata-quorum --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe --status

# kraft.version is 1.
kafka-features --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe | grep kraft.version

In the describe --replication output, brokers always show as Observer. Controllers show as Leader or Follower.

Troubleshoot a new deployment

requires every kafka_controller to set the same kafka_controller_initial_voter

kafka_controller_initial_voter is empty, does not match a kafka_controller inventory hostname, or is set differently on some hosts. Set it once under all.vars to an inventory hostname.

The run ends early with no error message

A controller was unreachable from the Ansible host when Confluent Ansible checked the data directories. The run ends with failed=0 and no further tasks run. Check that the Ansible host can reach every host in the kafka_controller group, bring any unreachable controller online, and run the playbook again.

A controller stays an observer

Check that the controller caught up with the leader (low Lag in the describe --replication output) and that controller.quorum.auto.join.enable=true is in its server.properties file. If you disabled auto-join, run the following command on that controller to promote it:

kafka-metadata-quorum --bootstrap-controller <leader-host>:<controller_port> \
  --command-config /etc/controller/server.properties add-controller

Detected a running static KRaft cluster

The cluster already runs a static quorum. Use the migration procedure in the next section.

Migrate a static quorum to a dynamic quorum

You can migrate an existing static KRaft quorum (kraft.version=0) to a dynamic quorum (kraft.version=1) in place, without rebuilding the cluster. The migration is a one-time, one-way operation. The playbook rolls the controllers and brokers one at a time, so the cluster keeps serving during the migration.

Requirements

Confluent Platform 8.4.0 or later, deployed by Confluent Ansible, running a static KRaft quorum.
A healthy controller quorum. The quorum has a leader and every controller is a voter.
The inventory file that you used to deploy the cluster.

Why a migration playbook is needed

You cannot set kraft_dynamic_quorum_enabled: true and run the normal playbooks. controller.quorum.bootstrap.servers works only after kraft.version is 1, so the feature must be upgraded first while the controllers still use controller.quorum.voters.

If you run the normal playbooks with kraft_dynamic_quorum_enabled: true on a running static cluster, they stop with an error that tells you to run StaticToDynamicQuorumMigration.

Migration stages

The migration changes the configuration in the following order. Controllers switch to controller.quorum.bootstrap.servers before brokers do.

Stage	Controller configuration	Broker configuration	`kraft.version`
Start	`controller.quorum.voters`	`controller.quorum.voters`	`0`
After upgrading `kraft.version`	`controller.quorum.voters`	`controller.quorum.voters`	`1`
After switching controllers	`controller.quorum.bootstrap.servers`	`controller.quorum.voters`	`1`
After switching brokers	`controller.quorum.bootstrap.servers`	`controller.quorum.bootstrap.servers`	`1`

The playbook runs the following stages. Each stage has its own tag.

Stage	Tag	Action
Precheck	`static_to_dynamic_quorum_precheck`	Checks that `kraft_dynamic_quorum_enabled` is `true` and runs the controller health check.
Upgrade `kraft.version`	`static_to_dynamic_quorum_feature_upgrade`	Runs `kafka-features upgrade --feature kraft.version=1` if the version is `0`, and then checks that it is `1`. No restart is needed.
Switch controller configuration	`static_to_dynamic_quorum_controller_config`	Removes `controller.quorum.voters` and sets `controller.quorum.bootstrap.servers` and `controller.quorum.auto.join.enable` on every controller.
Restart controllers	`static_to_dynamic_quorum_controller_restart`	Restarts the controllers one at a time, with a health check after each.
Switch broker configuration	`static_to_dynamic_quorum_broker_config`	Removes `controller.quorum.voters` and sets `controller.quorum.bootstrap.servers` on every broker.
Restart brokers	`static_to_dynamic_quorum_broker_restart`	Restarts the brokers one at a time, with a health check after each.

Run the migration

Before you run the migration, note that it is a one-way operation.

Warning

The kraft.version upgrade cannot be undone. Kafka does not support downgrading kraft.version from 1 to 0. Run the migration on a staging cluster first.

To migrate:

Check the starting state on any controller. kraft.version must be 0, and every controller must be a voter:

kafka-features --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe | grep kraft.version

kafka-metadata-quorum --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe --replication

Add kraft_dynamic_quorum_enabled: true to the inventory file under all.vars. You don’t need kafka_controller_initial_voter because it is only used when a new cluster is created.
Run the migration playbook before any other playbook that would render the new configuration:
```
ansible-playbook -i hosts.yml confluent.platform.StaticToDynamicQuorumMigration
```

Verify the migration:

# kraft.version is 1.
kafka-features --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe | grep kraft.version

# Controllers and brokers use bootstrap.servers and no longer use voters.
# Run this command on each controller.
grep -E '^controller\.quorum\.(voters|bootstrap\.servers)' /etc/controller/server.properties

# Run this command on each broker.
grep -E '^controller\.quorum\.(voters|bootstrap\.servers)' /etc/kafka/server.properties

After the migration, keep kraft_dynamic_quorum_enabled: true in the inventory file. The normal playbooks now render the dynamic quorum configuration.

If the migration stops partway

Fix the cause and run the same command again. Every stage is safe to repeat:

If kraft.version is already 1, the upgrade is skipped.
Configuration lines that are already switched are left as they are.
The restart stages restart the controllers and brokers again, one at a time. This causes another rolling restart but no downtime.

If the run stopped after the kraft.version upgrade, run it again soon. In that state, kraft.version is 1 but the controllers and brokers still use controller.quorum.voters. Don’t leave the cluster in this state for long, because disaster recovery is harder in this state.

You can also run only the remaining stages with --tags. For example, --tags static_to_dynamic_quorum_broker_config,static_to_dynamic_quorum_broker_restart.

Add controllers

With a dynamic quorum, adding a controller is a normal playbook run. You add the host to the inventory and run the controller playbook. Confluent Ansible formats the new controller to join the existing cluster, and auto-join makes it a voter. You don’t need to run any kafka-metadata-quorum commands.

Before you begin, make sure that:

The cluster runs a dynamic quorum, deployed as a new cluster or migrated.
The quorum is healthy. Every existing controller shows as Leader or Follower in the describe --replication output.
kafka_controller_kraft_auto_join_enabled is true, which is the default.
The new hosts are reachable from the Ansible host and from the existing controllers on the controller port, which defaults to 9093.

To add controllers:

Add the new hosts at the end of the kafka_controller group in the inventory file, with the same settings as the existing controllers. You don’t need to change anything else in the inventory.

If node_id is not set, Confluent Ansible works out the node ID of each controller from its position in the group. Inserting a host earlier in the list changes the node IDs of the controllers after it, and they fail to start.
Run the controller playbook for the new controllers only:
```
ansible-playbook -i hosts.yml confluent.platform.kafka_controller \
  --limit kcontroller-4.example.com,kcontroller-5.example.com
```
- With --limit, Confluent Ansible installs and starts only the new controllers. It doesn’t restart the existing controllers.
- Without --limit, Confluent Ansible also updates controller.quorum.bootstrap.servers on the existing controllers so that it lists the new hosts, and restarts them one at a time. This is not required, because controllers learn the voters from the metadata log.
Confluent Ansible doesn’t restart the brokers in either case. A broker needs only one reachable controller in its controller.quorum.bootstrap.servers list.

Verify that every controller, old and new, is a voter:

kafka-metadata-quorum --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe --replication

kafka-metadata-quorum --bootstrap-controller <controller-host>:<controller_port> \
  --command-config /etc/controller/client.properties describe --status

A new controller can show as Observer for a short time while it catches up.

Check that every controller has the same cluster.id. Run the following command on each controller:

grep cluster.id /var/lib/controller/data/meta.properties

Replace a controller

To replace a controller whose host or disk is lost, remove the old controller from the quorum as described in Remove controllers, and then add the new host as described in Add controllers. You can reuse the old inventory hostname.

Troubleshoot added controllers

A new controller stays an observer

Check that it caught up with the leader, that it can reach the other controllers on the controller port, and that controller.quorum.auto.join.enable=true is in its server.properties file. If you disabled auto-join, run the following command on the new controller:

kafka-metadata-quorum --bootstrap-controller <leader-host>:<controller_port> \
  --command-config /etc/controller/server.properties add-controller

A new controller fails to start with INCONSISTENT_CLUSTER_ID

The storage of the controller was formatted for a different cluster, for example by a manual kafka-storage format. Stop the controller, empty its data directory, which defaults to /var/lib/controller/data, and run the controller playbook for that host again.

Remove controllers

Removing a controller is a manual procedure with kafka-metadata-quorum remove-controller. Confluent Ansible doesn’t automate it.

Before you remove a controller, consider the following:

Remove one controller at a time and check the quorum after each one.
Keep enough voters running. Count the voters that are up, not only the voters in the voter set. Removing a running voter while another one is down can leave the quorum without a majority.
To replace a controller, remove the old controller first and then add the new one.

Run the following steps for one controller at a time. The commands assume a package installation with Confluent Ansible defaults. Adapt the paths and service names for archive installations, custom users, or rootless deployments.

On any remaining controller, find the node ID and directory ID of the controller to remove:

kafka-metadata-quorum --bootstrap-controller <remaining-controller>:<controller_port> \
  --command-config /etc/controller/client.properties describe --replication

Note the NodeId and DirectoryId from the same row of the output.

If the host is lost, it is already down. Confirm that it is down, and then skip this step. Otherwise, on the controller you are removing, stop and disable KRaft:
```
sudo systemctl disable --now confluent-kcontroller
```
If you changed the service name, use the value of kafka_controller_service_name instead of confluent-kcontroller.

The controller must stay stopped. Auto-join adds any running, caught-up controller back to the voter set. If you remove a controller that is still running, or leave a removed host in the inventory so the next playbook run starts it again, auto-join adds it back.

On any remaining controller, remove the controller from the quorum:

kafka-metadata-quorum --bootstrap-controller <remaining-controller>:<controller_port> \
  --command-config /etc/controller/client.properties \
  remove-controller --controller-id <node-id> --controller-directory-id <directory-id>

If the removed controller was the leader, the remaining voters elect a new leader.

Check the quorum. The removed controller must no longer be listed as Leader or Follower:

kafka-metadata-quorum --bootstrap-controller <remaining-controller>:<controller_port> \
  --command-config /etc/controller/client.properties describe --replication

kafka-metadata-quorum --bootstrap-controller <remaining-controller>:<controller_port> \
  --command-config /etc/controller/client.properties describe --status

Update the inventory file:
1. Pin the node IDs first. If node_id is not set, Confluent Ansible works out the node ID of each controller from its position in the kafka_controller group, such as 9991 and 9992. Removing a host from the middle of the list gives every controller after it a different node ID on the next playbook run, and those controllers fail to start. On every controller you keep, set node_id to its current node ID from the first step.
2. Delete the removed hosts from the kafka_controller group.
If the host is not lost, decommission it after the previous step confirms it is out of the quorum. A retired host still holds its KRaft data, including the cluster ID and its node ID. If the service starts on it again, for example after a reboot, a playbook run with an old inventory, or reuse of the machine, it tries to rejoin the cluster and auto-join adds it back as a voter. Delete the controller data so that the host has nothing to rejoin with:
```
sudo rm -rf /var/lib/controller/data/*
```
The service is already stopped and disabled.

Troubleshoot removed controllers

The removed controller shows up as a voter again: The controller was still running or was started again, and auto-join added it back. Stop and disable it, remove it from the inventory, and run the remove-controller command again.
remove-controller fails with an unknown voter or wrong directory ID: The NodeId and DirectoryId must come from the same row of the describe --replication output. Run the describe --replication command again and copy both values from the same line.
Controllers fail to start after you remove a host from the inventory: node_id was not pinned, so removing a host from the middle of the kafka_controller group changed the node IDs of the controllers after it. Set node_id on each controller back to the node ID stored on its disk, which is node.id in /var/lib/controller/data/meta.properties, and then run the controller playbook again.
A controller host is lost and cannot be stopped: A lost host is already stopped. From a remaining controller, find the node ID and directory ID, remove the controller, and check the quorum. The dead controller is listed with a LastFetchTimestamp that stops moving and a growing Lag. Lag alone does not mean a controller is gone, so confirm by other means that the host is down.