Command and Configuration Options for the Auto Data Balancer

The following command flags and configuration options are specific to the rebalancer tool. For an example, see Execute the rebalancer. For any configuration options related to metrics reporting, see Confluent Metrics Reporter configuration.

The configuration file path for the rebalancer is passed using the --config-file option. Connectivity and communication to Kafka using the AdminClient can be provided in a properties file with the --command-config option. These along with other configuration and connection options are described in the sections below.

confluent-rebalancer command flags

The following options are available on the confluent-rebalancer command. In $CONFLUENT_HOME, if you run ./bin/confluent-rebalancer from with no arguments, you will get a list of supported commands along with a description. For an example of running this command with some of the flags, see Execute the rebalancer in the tutorial.

--zookeeper

Note: This option is deprecated on Confluent Platform 5.5.0 and later. Use only --bootstrap-server instead. For an example, see Execute the rebalancer.

The connection string for the ZooKeeper connection in the form host:port. You can specify multiple URLs, separated by commas, to allow for failover should a ZooKeeper node go down.

  • Type: string
  • Default: empty string
  • Importance: high
--bootstrap-server

The connection string for the cluster’s broker(s) in the form of host:port. You can specify multiple URLs, separated by commas, to allow for failover should a broker node go down. This is required.

  • Type: string
  • Default: empty string
  • Importance: high
--exclude-internal-topics

Exclude internal topics like _consumer_offsets while rebalancing. This option is ignored when - -topics is used to specify the topics to rebalance directly.

  • Type: boolean
  • Default: false
  • Importance: low
--force

Suppress prompts if true.

  • Type: boolean
  • Default: false
  • Importance: low
--incremental

Indicates that partitions should be moved incrementally. The rebalancer will reassign partitions in batches so that max.concurrent.moves.per.leader is not exceeded. The command will not return until all reassignments have completed. This option is only available for brokers with Confluent Platform 5.4 (Apache Kafka® 2.4) or higher. For an example, see Incremental reassignment in the Quick Start.

  • Type: boolean
  • Default: false
  • Importance: low
--metrics-bootstrap-server

The connection string for the Kafka cluster where the metrics reporter publishes cluster level metrics (that is, where the metrics topic is running). Specify this in the form of host:port. See also, confluent.rebalancer.metrics.topic and the example in Execute the rebalancer.

  • Type: string
  • Default: empty string
  • Importance: low
--remove-broker-ids

Partitions will be moved away from the specified brokers (comma-separated). This option can be used to decommission brokers.

  • Type: boolean
  • Default: false
  • Importance: low
--replica-placement-only

Limits a rebalance plan to topics that have changed their replica placement constraints, and are not satisfying this constraint under their current location.

  • Type: boolean
  • Default: false
  • Importance: medium
--throttle

Specifies the maximum bandwidth, in bytes per second, allocated to moving replicas. This is required. See Execute the rebalancer for an example, especially Limiting bandwidth usage during data migration.

  • Type: int
  • Default: empty string
  • Importance: high
--topics

Limits the rebalance plan to a specified topic or comma-separated list of topics. This overrides any settings specified for - -exclude-internal-topics

  • Type: string
  • Default: empty string
  • Importance: medium
--verbose

Also output before and after rebalance per broker information on disk space usage, leader count, replica count, and topic partitions.

  • Type: boolean
  • Default: false
  • Importance: low
--command-config

Specifies a property file containing configurations to be passed to the AdminClient Auto Data Balancer and used to connect and communicate with the Kafka cluster. The property file is a text file with no requirements for file naming or any particular extension. Available properties are detailed in AdminClient Configurations.

This option is used only with the --bootstrap-server option, and is required for some use cases. For example, if the server is secured with SSL (security.protocol=SSL), you must specify this in the --command-config to the Admin Client and the --config-file for your metrics consumer, which gets data from confluent.rebalancer.metrics.topic.

See also, Using a command-config file to talk to Kafka

  • Type: string
  • Default: empty string
  • Importance: low
--config-file

Specifies a property file containing configurations for Auto Data Balancer. You can use this approach in place of specifying rebalancer configurations directly from the command line. The property file is a text file with no requirements for file naming or any particular extension. Available properties are described in the section Specifying Auto Data Balancer properties in a config-file.

  • Type: string
  • Default: empty string
  • Importance: low

Using a command-config file to talk to Kafka

You can create a text file to specify properties for how the rebalancer will connect and communicate with the Kafka cluster. This must be used along with the the --bootstrap-server command option, and is required in some cases, such as in the context of secure connections.

Available properties are described in the section AdminClient Configurations.

See also, - -command-config in the command flags reference above.

Specifying Auto Data Balancer properties in a config-file

The following settings must be specified in a properties file that is passed to Auto Data Balancer as an argument to the --config-file option on the confluent-rebalancer command. See - -config-file in the command flags reference above.

To test the rebalancer tool, you must specify --bootstrap-server to connect to the Kafka cluster. The confluent.license is required after 30 days, however it’s still required that the license client is configured to communicate with the cluster. The following configurations are supported.

confluent.license

Confluent will issue a license key to each subscriber. The license key will be a short snippet of text that you can copy and paste. Without the license key, you can use the Auto Data Balancer for a 30-day trial period. If you are a subscriber and don’t have a license key, contact Confluent Support at support@confluent.io. This must be specified in the --config-file.

  • Type: string
  • Valid Values: Confluent Platform license
  • Importance: high

You must configure the rebalancer client to register or validate the license with an Kafka cluster by prefixing the configuration names with confluent.license.. For example, to configure the license client to use SASL_SSL, specify the following:

confluent.license.security.protocol=SASL_SSL
confluent.license.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.license.ssl.truststore.password=<ssl-truststore-password>
confluent.license.ssl.keystore.location=/var/private/ssl/keystore.jks
confluent.license.ssl.keystore.password=<ssl-keystore-password>
confluent.license.ssl.key.password=<ssl-key-password>
confluent.license.sasl.mechanism=GSSAPI
confluent.license.sasl.kerberos.service.name=kafka
confluent.license.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true \
        keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

Tip

For complete license information for Confluent Platform, see Confluent Platform Licenses.

confluent.rebalancer.metrics.collection.timeout.ms

The maximum amount of time the rebalancer will collect metrics for. If the collection does not succeed before the timeout elapses, then the rebalance command will fail.

  • Type: int
  • Default: 60000
  • Importance: low
confluent.rebalancer.metrics.topic

The topic in which the metrics reporter publishes its metrics.

  • Type: string
  • Default: “_confluent-metrics”
  • Importance: low
confluent.rebalancer.min.free.volume.space.percentage

The log.dir volume will have at least the specified percentage of free space during and after the rebalance. For example, if the total volume space is 100 GB and this configuration is defined as 20, then the rebalancer will use up to 80 GB during the rebalance. This is supported in Confluent 3.2 only (for both rebalancer and brokers), and only if every broker in the cluster has a single log.dir. In versions where this attribute supported, it is enabled by default; otherwise, it is disabled.

  • Type: double
  • Default: 20.0
  • Valid Values: [0,…,100]
  • Importance: low

You can configure the consumer used to retrieve metrics data by prefixing the consumer configuration name with confluent.rebalancer.metrics.. For example, to configure the consumer to use SASL_SSL, specify the following:

confluent.rebalancer.metrics.security.protocol=SASL_SSL
confluent.rebalancer.metrics.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.rebalancer.metrics.ssl.truststore.password=<ssl-truststore-password>
confluent.rebalancer.metrics.ssl.keystore.location=/var/private/ssl/keystore.jks
confluent.rebalancer.metrics.ssl.keystore.password=<ssl-keystore-password>
confluent.rebalancer.metrics.ssl.key.password=<ssl-key-password>
confluent.rebalancer.metrics.sasl.mechanism=GSSAPI
confluent.rebalancer.metrics.sasl.kerberos.service.name=kafka
confluent.rebalancer.metrics.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true \
        keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";