Command and Configuration Options for the Auto Data Balancer

The following command flags and configuration options are specific to the rebalancer tool. For an example, see Execute the rebalancer. For any configuration options related to metrics reporting, see Confluent Metrics Reporter configuration.

The configuration file path for the rebalancer is passed using the --config-file option. Connectivity and communication to Kafka using the Admin Client can be provided in a properties file with the --command-config option. These along with other configuration and connection options are described in the sections below.

confluent-rebalancer command flags

The following options are available on the confluent-rebalancer command. In $CONFLUENT_HOME, if you run ./bin/confluent-rebalancer from with no arguments, you will get a list of supported commands along with a description. For an example of running this command with some of the flags, see Execute the rebalancer in the tutorial.

--zookeeper

Note: This option is deprecated on Confluent Platform 5.5.0 and later. Use only --bootstrap-server instead. For an example, see Execute the rebalancer.

The connection string for the ZooKeeper connection in the form host:port. You can specify multiple URLs, separated by commas, to allow for failover should a ZooKeeper node go down.

  • Type: string
  • Default: empty string
  • Importance: high
--bootstrap-server

The connection string for the cluster’s broker(s) in the form of host:port. You can specify multiple URLs, separated by commas, to allow for failover should a broker node go down. This is required.

  • Type: string
  • Default: empty string
  • Importance: high
--exclude-internal-topics

Exclude internal topics like _consumer_offsets while rebalancing. This option is ignored when - -topics is used to specify the topics to rebalance directly.

  • Type: boolean
  • Default: false
  • Importance: low
--force

Suppress prompts if true.

  • Type: boolean
  • Default: false
  • Importance: low
--incremental

Indicates that partitions should be moved incrementally. The rebalancer will reassign partitions in batches so that max.concurrent.moves.per.leader is not exceeded. The command will not return until all reassignments have completed. This option is only available for brokers with Confluent Platform 5.4 (Apache Kafka® 2.4) or higher. For an example, see Incremental reassignment in the Quick Start.

  • Type: boolean
  • Default: false
  • Importance: low
--metrics-bootstrap-server

The connection string for the Kafka cluster where the metrics reporter publishes cluster level metrics (that is, where the metrics topic is running). Specify this in the form of host:port. To learn more about this setting, see confluent.rebalancer.metrics.topic and the example in Execute the rebalancer.

  • Type: string
  • Default: empty string
  • Importance: low
--remove-broker-ids

Partitions will be moved away from the specified brokers (a comma-separated list of IDs). This option can be used to decommission brokers.

  • Type: string
  • Default: empty string
  • Importance: low
--replica-placement-only

Limits a rebalance plan to topics that have changed their replica placement constraints, and are not satisfying this constraint under their current location.

  • Type: boolean
  • Default: false
  • Importance: medium
--throttle

Specifies the maximum bandwidth, in bytes per second, allocated to moving replicas. This is required. See Execute the rebalancer for an example, especially Limiting bandwidth usage during data migration.

  • Type: int
  • Default: empty string
  • Importance: high
--topics

Limits the rebalance plan to a specified topic or comma-separated list of topics. This overrides any settings specified for - -exclude-internal-topics

  • Type: string
  • Default: empty string
  • Importance: medium
--verbose

Also output before and after rebalance per broker information on disk space usage, leader count, replica count, and topic partitions.

  • Type: boolean
  • Default: false
  • Importance: low
--command-config

Specifies a property file containing configurations to be passed to the AdminClient Auto Data Balancer and used to connect and communicate with the Kafka cluster. The property file is a text file with no requirements for file naming or any particular extension. Available properties are detailed in Kafka AdminClient Configurations for Confluent Platform.

This option is used only with the --bootstrap-server option, and is required for some use cases. For example, if the server is secured with SSL (security.protocol=SSL), you must specify this in the --command-config to the Admin Client and the --config-file for your metrics consumer, which gets data from confluent.rebalancer.metrics.topic.

To learn more about this setting, see Using a command-config file to talk to Kafka

  • Type: string
  • Default: empty string
  • Importance: low
--config-file

Specifies a property file containing configurations for Auto Data Balancer. You can use this approach in place of specifying rebalancer configurations directly from the command line. The property file is a text file with no requirements for file naming or any particular extension. Available properties are described in the section Specifying Auto Data Balancer properties in a config-file.

  • Type: string
  • Default: empty string
  • Importance: low

Using a command-config file to talk to Kafka

You can create a text file to specify properties for how the rebalancer will connect and communicate with the Kafka cluster. This must be used along with the the --bootstrap-server command option, and is required in some cases, such as in the context of secure connections.

Available properties are described in the section Kafka AdminClient Configurations for Confluent Platform.

To learn more, see - -command-config in the command flags reference above.

Specifying Auto Data Balancer properties in a config-file

The following settings must be specified in a properties file that is passed to Auto Data Balancer as an argument to the --config-file option on the confluent-rebalancer command. See - -config-file in the command flags reference above.

To test the rebalancer tool, you must specify --bootstrap-server to connect to the Kafka cluster. The confluent.license is required after 30 days, however it’s still required that the license client is configured to communicate with the cluster. The following configurations are supported.

confluent.license

Confluent will issue a license key to each subscriber. The license key will be a short snippet of text that you can copy and paste. Without the license key, you can use the Auto Data Balancer for a 30-day trial period. If you are a subscriber and don’t have a license key, contact Confluent Support at support@confluent.io. This must be specified in the --config-file.

  • Type: string
  • Valid Values: Confluent Platform license
  • Importance: high

You must configure the rebalancer client to register or validate the license with an Kafka cluster by prefixing the configuration names with confluent.license.. For example, to configure the license client to use SASL_SSL, specify the following:

confluent.license.security.protocol=SASL_SSL
confluent.license.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.license.ssl.truststore.password=<ssl-truststore-password>
confluent.license.ssl.keystore.location=/var/private/ssl/keystore.jks
confluent.license.ssl.keystore.password=<ssl-keystore-password>
confluent.license.ssl.key.password=<ssl-key-password>
confluent.license.sasl.mechanism=GSSAPI
confluent.license.sasl.kerberos.service.name=kafka
confluent.license.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true \
        keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

Tip

For complete license information for Confluent Platform, see Confluent Platform Licenses.

confluent.rebalancer.metrics.collection.timeout.ms

The maximum amount of time the rebalancer will collect metrics for. If the collection does not succeed before the timeout elapses, then the rebalance command will fail.

  • Type: int
  • Default: 60000
  • Importance: low
confluent.rebalancer.metrics.topic

The topic in which the metrics reporter publishes its metrics.

  • Type: string
  • Default: “_confluent-metrics”
  • Importance: low
confluent.rebalancer.min.free.volume.space.percentage

The log.dir volume will have at least the specified percentage of free space during and after the rebalance. For example, if the total volume space is 100 GB and this configuration is defined as 20, then the rebalancer will use up to 80 GB during the rebalance. This is supported in Confluent 3.2 only (for both rebalancer and brokers), and only if every broker in the cluster has a single log.dir. In versions where this attribute supported, it is enabled by default; otherwise, it is disabled.

  • Type: double
  • Default: 20.0
  • Valid Values: [0,…,100]
  • Importance: low

You can configure the consumer used to retrieve metrics data by prefixing the consumer configuration name with confluent.rebalancer.metrics.. For example, to configure the consumer to use SASL_SSL, specify the following:

confluent.rebalancer.metrics.security.protocol=SASL_SSL
confluent.rebalancer.metrics.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.rebalancer.metrics.ssl.truststore.password=<ssl-truststore-password>
confluent.rebalancer.metrics.ssl.keystore.location=/var/private/ssl/keystore.jks
confluent.rebalancer.metrics.ssl.keystore.password=<ssl-keystore-password>
confluent.rebalancer.metrics.ssl.key.password=<ssl-key-password>
confluent.rebalancer.metrics.sasl.mechanism=GSSAPI
confluent.rebalancer.metrics.sasl.kerberos.service.name=kafka
confluent.rebalancer.metrics.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true \
        keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

Example: Run the rebalancer with Security, Metrics, and License Configurations

This section shows an example configuration for Auto Data Balancer to connect to a secured Kafka cluster. Auto Data Balancer uses a consumer to retrieve metrics data, a license client to check for license, and an Admin Client to rebalance the cluster.

The example shows how to connect to a Kafka cluster with SASL_SSL enabled. The SASL mechanism is GSSAPI.

  1. Create a configuration file called adb.properties which will be passed with flags --config-file and --command-config when running the rebalancer. Add the following parameters to the adb.properties file.

    # Metrics client configuration
confluent.rebalancer.metrics.security.protocol=SASL_SSL
confluent.rebalancer.metrics.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.rebalancer.metrics.ssl.truststore.password=xxxxxxx
confluent.rebalancer.metrics.sasl.mechanism=GSSAPI
confluent.rebalancer.metrics.sasl.kerberos.service.name=kafka
confluent.rebalancer.metrics.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
 useKeyTab=true storeKey=true keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

# License client configuration
confluent.license=<enter-your-license>
confluent.license.security.protocol=SASL_SSL
confluent.license.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.license.ssl.truststore.password=xxxxxxx
confluent.license.sasl.mechanism=GSSAPI
confluent.license.sasl.kerberos.service.name=kafka
confluent.license.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
  useKeyTab=true storeKey=true keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

# Admin client configuration
security.protocol=SASL_SSL
ssl.truststore.location=/var/private/ssl/truststore.jks
ssl.truststore.password=xxxxxxx
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
  useKeyTab=true storeKey=true keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

    Tip

    Alternatively, you can remove all *.sasl.jaas.config from the config file and instead pass the information in a JAAS file. After the JAAS file is created you would need export this file prior to running rebalancer. To learn more, see JAAS Configurations.

    export REBALANCER_OPTS="-
Djava.security.auth.login.config=</path/to/jaas.conf>"

  2. Now, run Auto Data Balancer with the properties file you just created, using it as an argument for both --config-file (reads metrics and license information) and --command-config (reads Admin Client properties):

    confluent-rebalancer execute \
--metrics-bootstrap-server <broker>:<port> --config-file adb.properties \
--bootstrap-server <broker>:<port> --command-config adb.properties

Example: Run the rebalancer with a Separate Metrics Cluster

Confluent Metrics Reporter is required to compute the rebalance plan for Auto Data Balancer, This example shows how to point to a dedicated metrics cluster. In the example, both the metrics cluster and the rebalancer cluster are secured with two-way SSL (mTLS).

  1. Provide information about the metrics and license clusters.

    To calculate metrics on a separate metrics cluster, Auto Data Balancer needs the connection information for the metrics cluster in a --config-file. This configuration should also contain connection information for the license cluster (bootstrap-server). For example, you can put this information in a file called metrics-and-license.properties:

    confluent.rebalancer.metrics.security.protocol=SSL
confluent.rebalancer.metrics.ssl.truststore.location=/etc/kafka/secrets/metrics-truststore.jks
confluent.rebalancer.metrics.ssl.truststore.password=metrics-password
confluent.rebalancer.metrics.ssl.keystore.location=/etc/kafka/secrets/metrics-keystore.jks
confluent.rebalancer.metrics.ssl.keystore.password=metrics-password

confluent.license=<your-confluent-license>
confluent.license.security.protocol=SSL
confluent.license.ssl.truststore.location=/etc/kafka/secrets/truststore.jks
confluent.license.ssl.truststore.password=password
confluent.license.ssl.keystore.location=etc/kafka/secrets/keystore.jks
confluent.license.ssl.keystore.password=password

  2. Provide security settings for the Admin Client.

    The --command-config file is used to include the security settings for the Admin client connecting to the bootstrap-server cluster for rebalancing. For example, you could put these settings in a file called admin.properties:

    security.protocol=SSL
ssl.truststore.location=/etc/kafka/secrets/truststore.jks
ssl.truststore.password=password
ssl.keystore.location=/etc/kafka/secrets/keystore.jks
ssl.keystore.password=password
ssl.enabled.protocols=TLSv1.2
request.timeout.ms=60000

  3. Run Auto Data Balancer with the properties file you just created:

    confluent-rebalancer execute \
  --metrics-bootstrap-server <metrics-broker>:<port> --config-file metrics-and-license.properties \
  --bootstrap-server <broker-to-rebalance>:<port> --command-config admin.properties

ACLs for Auto Data Balancing

In deployments where ACLs are enabled, the following ACLs are needed for the given resources. For details on creating ACLs, see Authorization using ACLs. For a full list of associated operations, resources, and APIs, see the subtopic, Operations.

Cluster-level ACLs

The following ACLs are needed on the clusters.

Operation Resource API
CREATE Cluster CreateTopics, Metadata
DESCRIBE Cluster DescribeAcls
ALTER Cluster CreateAcls, DeleteAcls
ALTERCONFIGS Cluster AlterConfigs

Topic-level ACLs

Topic ACLs are needed to manage licensing and monitor metrics using system topics, as shown below.

ACLs for License Management

Topic-level ACLs are needed for the _confluent-license and _confluent-command system topics.

Task (System Topics) Operation Resource API
_confluent-command All Topic All

The _confluent-command topic contains the license that corresponds to the license key supplied through the confluent.license property. It is created by default. Access to this topic requires the following ACLs configured:

  • CREATE and DESCRIBE on the cluster to have permissions to create the topic.

  • DESCRIBE, READ, and WRITE on the _confluent-command topic.

    Important

    You may also use DESCRIBE and READ without WRITE to restrict access to read-only for license topic ACLs. If a topic exists, the LicenseManager will not try to create the topic.

You can provide access either individually for each principal that will use the license or use a Wildcard principals to allow all clients. The following examples show commands that you can use to configure ACLs for the resource cluster and _confluent-command topic.

  1. Set a CREATE and DESCRIBE ACL on the resource cluster:

    kafka-acls --bootstrap-server localhost:9092 --command-config adminclient-configs.conf \
--add --allow-principal User:<principal> \
--operation CREATE --operation DESCRIBE --cluster

  2. Set a DESCRIBE, READ, and WRITE ACL on the _confluent-command topic:

    kafka-acls --bootstrap-server localhost:9092 --command-config adminclient-configs.conf \
--add --allow-principal User:<principal> \
--operation DESCRIBE --operation READ --operation WRITE --topic _confluent-command

For complete license information for Confluent Platform, see Confluent Platform Licenses.

ACLs for Metrics and Monitoring Metrics

Topic-level ACLs are needed to read metrics with the _confluent-metrics system topic.

Task (System Topics) Operation Resource API
_confluent-metrics READ Topic Fetch