<a id="rebalancer-config-options"></a>

# Auto Data Balancer Configuration and Command Reference for Confluent Platform

The following command flags and configuration options are specific to the rebalancer tool.
For an example, see [Execute the rebalancer](quickstart.md#execute-rebalancer). For any configuration options related to metrics reporting,
see [Confluent Metrics Reporter configuration](../../monitor/metrics-reporter.md#metrics-reporter-configuration).

The configuration file path for the rebalancer is passed using the `--config-file` option. Connectivity and
communication to Kafka using the Admin Client can be provided in a properties file with
the `--command-config` option. These along with other configuration and connection options are described in the sections below.

## confluent-rebalancer command flags

The following options are available on the `confluent-rebalancer` command. In `$CONFLUENT_HOME`,
if you run `./bin/confluent-rebalancer` from with no arguments, you will get a list of supported
commands along with a description. For an example of running this command with some of the flags,
see [Execute the rebalancer](quickstart.md#execute-rebalancer) in the [tutorial](quickstart.md#rebalancer).

`--bootstrap-server`
: The connection string for the cluster’s broker(s) in the form of `host:port`.
  You can specify multiple URLs, separated by commas, to allow for failover
  should a broker node go down. This is required.
  <br/>
  * Type: string
  * Default: empty string
  * Importance: high

<a id="rebalancer-exclude-internal-topics"></a>

`--exclude-internal-topics`
: Exclude internal topics like `_consumer_offsets` while rebalancing. This option is ignored when
  [- -topics](#rebalancer-topics) is used to specify the topics to rebalance directly.
  <br/>
  * Type: boolean
  * Default: false
  * Importance: low

`--force`
: Suppress prompts if true.
  <br/>
  * Type: boolean
  * Default: false
  * Importance: low

`--incremental`
: Indicates that partitions should be moved incrementally. The rebalancer will reassign partitions in batches so that
  `max.concurrent.moves.per.leader` is not exceeded. The command will not return until all reassignments have completed.
  This option is only available for brokers with Confluent Platform 5.4 (Apache Kafka® 2.4) or higher. For an example, see [Incremental reassignment](quickstart.md#adb-incremental-example) in the Quick Start.
  <br/>
  * Type: boolean
  * Default: false
  * Importance: low

`--metrics-bootstrap-server`
: The connection string for the Kafka cluster where the metrics reporter publishes cluster level metrics
  (that is, where the metrics topic is running). Specify this in the form of `host:port`.
  To learn more about this setting, see [confluent.rebalancer.metrics.topic](#rebalancer-metrics-topic) and the example in [Execute the rebalancer](quickstart.md#execute-rebalancer).
  <br/>
  * Type: string
  * Default: empty string
  * Importance: low

`--remove-broker-ids`
: Partitions will be moved away from the specified brokers (a comma-separated list of IDs). This option can be used to decommission brokers.
  <br/>
  * Type: string
  * Default: empty string
  * Importance: low

`--replica-placement-only`
: Limits a rebalance plan to topics that have changed their replica placement constraints, and are not satisfying this constraint under their current location.
  <br/>
  * Type: boolean
  * Default: false
  * Importance: medium

`--throttle`
: Specifies the maximum bandwidth, in bytes per second, allocated to moving replicas. This is required.
  See [Execute the rebalancer](quickstart.md#execute-rebalancer) for an example, especially [Limiting bandwidth usage during data migration](quickstart.md#adb-throttle).
  <br/>
  * Type: int
  * Default: empty string
  * Importance: high

<a id="rebalancer-topics"></a>

`--topics`
: Limits the rebalance plan to a specified topic or comma-separated list of topics. This overrides any settings
  specified for [- -exclude-internal-topics](#rebalancer-exclude-internal-topics)
  <br/>
  * Type: string
  * Default: empty string
  * Importance: medium

`--verbose`
: Also output before and after rebalance per broker information on disk space usage, leader count, replica count, and topic partitions.
  <br/>
  * Type: boolean
  * Default: false
  * Importance: low

<a id="rebalancer-command-file-flag"></a>

`--command-config`
: Specifies a property file containing configurations to be passed to the AdminClient Auto Data Balancer and used to connect
  and communicate with the Kafka cluster. The property file is a text file with no requirements for file naming or any
  particular extension. Available properties are detailed in [Kafka AdminClient Configurations for Confluent Platform](../../installation/configuration/admin-configs.md#cp-config-admin).
  <br/>
  This option is used only with the `--bootstrap-server` option, and is required for some use cases. For example, if the
  server is secured with TLS/SSL (`security.protocol=SSL`), you must specify this in the `--command-config` to the Admin Client
  and the `--config-file` for your metrics consumer, which gets data from `confluent.rebalancer.metrics.topic`.
  <br/>
  To learn more about this setting, see [Using a command-config file to talk to Kafka](#rebalancer-command-file)
  <br/>
  * Type: string
  * Default: empty string
  * Importance: low

<a id="rebalancer-config-file-flag"></a>

`--config-file`
: Specifies a property file containing configurations for Auto Data Balancer. You can use this approach in place of specifying rebalancer
  configurations directly from the command line. The property file is a text file with no requirements for file naming or any
  particular extension. Available properties are described in the section [Specifying Auto Data Balancer properties in a config-file](#rebalancer-config-file).
  <br/>
  * Type: string
  * Default: empty string
  * Importance: low

<a id="rebalancer-command-file"></a>

## Using a command-config file to talk to Kafka

You can create a text file to specify properties for how the rebalancer will
connect and communicate with the Kafka cluster. This must be used along with the
the `--bootstrap-server` command option, and is required in some cases, such
as in the context of secure connections.

Available properties are described in the section [Kafka AdminClient Configurations for Confluent Platform](../../installation/configuration/admin-configs.md#cp-config-admin).

To learn more, see [- -command-config](#rebalancer-command-file-flag) in the command flags reference above.

<a id="rebalancer-config-file"></a>

## Specifying Auto Data Balancer properties in a config-file

The following settings must be specified in a properties file that is passed to Auto Data Balancer as
an argument to the `--config-file` option on the `confluent-rebalancer` command.
See [- -config-file](#rebalancer-config-file-flag) in the command flags reference above.

To test the rebalancer tool, you must specify `--bootstrap-server`
to connect to the Kafka cluster. The `confluent.license` is required after
30 days, however it’s still required that the license client is configured to
communicate with the cluster. The following configurations are supported.

`confluent.license`
: Confluent will issue a license key to each subscriber. The license key will be
  a short snippet of text that you can copy and paste. Without the license key,
  you can use the Auto Data Balancer for a 30-day trial period. If you are a
  subscriber and don’t have a license key, contact [Confluent Support](https://support.confluent.io/)
  for more information. This must be specified in the `--config-file`.
  <br/>
  * Type: string
  * Valid Values: Confluent Platform license
  * Importance: high

You must configure the rebalancer client to register or validate the license with an
Kafka cluster by prefixing the configuration names with `confluent.license.`. For
example, to configure the license client to use SASL_SSL, specify the following:

```bash
confluent.license.security.protocol=SASL_SSL
confluent.license.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.license.ssl.truststore.password=<ssl-truststore-password>
confluent.license.ssl.keystore.location=/var/private/ssl/keystore.jks
confluent.license.ssl.keystore.password=<ssl-keystore-password>
confluent.license.ssl.key.password=<ssl-key-password>
confluent.license.sasl.mechanism=GSSAPI
confluent.license.sasl.kerberos.service.name=kafka
confluent.license.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true \
        keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";
```

`confluent.rebalancer.max.concurrent.moves.per.leader`
: The maximum number of partitions to reassign concurrently for each preferred leader.
  This is only relevant when the `--incremental` flag is passed. Note that reassignments per active leader may
  exceed this value when the active leader is not the preferred leader.
  <br/>
  * Type: int
  * Default: 4
  * Valid Values: [1,…,100]
  * Importance: low

`confluent.rebalancer.metrics.collection.timeout.ms`
: The maximum amount of time the rebalancer will collect metrics for. If the
  collection does not succeed before the timeout elapses, then the rebalance
  command will fail.
  <br/>
  * Type: int
  * Default: 60000
  * Importance: low

<a id="rebalancer-metrics-topic"></a>

`confluent.rebalancer.metrics.topic`
: The topic in which the metrics reporter publishes its metrics.
  <br/>
  * Type: string
  * Default: “_confluent-metrics”
  * Importance: low

`confluent.rebalancer.min.free.volume.space.percentage`
: The `log.dir` volume will have at least the specified percentage of free space
  during and after the rebalance. For example, if the total volume space is 100 GB
  and this configuration is defined as 20, then the rebalancer will use up to
  80 GB during the rebalance. This is supported in Confluent 3.2 only (for both
  rebalancer and brokers), and only if every broker in the cluster has a single
  log.dir. In versions where this attribute supported, it is enabled by default;
  otherwise, it is disabled.
  <br/>
  * Type: double
  * Default: 20.0
  * Valid Values: [0,…,100]
  * Importance: low

You can configure the consumer used to retrieve metrics data by prefixing the
consumer configuration name with `confluent.rebalancer.metrics.`. For example,
to configure the consumer to use SASL_SSL, specify the following:

```bash
confluent.rebalancer.metrics.security.protocol=SASL_SSL
confluent.rebalancer.metrics.ssl.truststore.location=/var/private/ssl/truststore.jks
confluent.rebalancer.metrics.ssl.truststore.password=<ssl-truststore-password>
confluent.rebalancer.metrics.ssl.keystore.location=/var/private/ssl/keystore.jks
confluent.rebalancer.metrics.ssl.keystore.password=<ssl-keystore-password>
confluent.rebalancer.metrics.ssl.key.password=<ssl-key-password>
confluent.rebalancer.metrics.sasl.mechanism=GSSAPI
confluent.rebalancer.metrics.sasl.kerberos.service.name=kafka
confluent.rebalancer.metrics.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true \
        keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";
```

## Example: Run the rebalancer with Security, Metrics, and License Configurations

This section shows an example configuration for Auto Data Balancer to connect to a secured Kafka cluster.
Auto Data Balancer uses a consumer to retrieve metrics data, a license client to check for license,
and an Admin Client to rebalance the cluster.

The example shows how to connect to a Kafka cluster with SASL_SSL enabled.
The SASL mechanism is [GSSAPI](../../security/authentication/sasl/gssapi/overview.md#rebalancer-sasl-config).

1. Create a configuration file called `adb.properties` which will be passed with flags `--config-file` and `--command-config`
   when running the rebalancer.  Add the following parameters to the `adb.properties` file.
   ```properties
   # Metrics client configuration
   confluent.rebalancer.metrics.security.protocol=SASL_SSL
   confluent.rebalancer.metrics.ssl.truststore.location=/var/private/ssl/truststore.jks
   confluent.rebalancer.metrics.ssl.truststore.password=xxxxxxx
   confluent.rebalancer.metrics.sasl.mechanism=GSSAPI
   confluent.rebalancer.metrics.sasl.kerberos.service.name=kafka
   confluent.rebalancer.metrics.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
    useKeyTab=true storeKey=true keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

   # License client configuration
   confluent.license=<enter-your-license>
   confluent.license.security.protocol=SASL_SSL
   confluent.license.ssl.truststore.location=/var/private/ssl/truststore.jks
   confluent.license.ssl.truststore.password=xxxxxxx
   confluent.license.sasl.mechanism=GSSAPI
   confluent.license.sasl.kerberos.service.name=kafka
   confluent.license.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
     useKeyTab=true storeKey=true keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";

   # Admin client configuration
   security.protocol=SASL_SSL
   ssl.truststore.location=/var/private/ssl/truststore.jks
   ssl.truststore.password=xxxxxxx
   sasl.mechanism=GSSAPI
   sasl.kerberos.service.name=kafka
   sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
     useKeyTab=true storeKey=true keyTab="/path/to/kerberos/kafka.user.keytab" principal="kafka@KAFKA.SECURE";
   ```
2. Now, run Auto Data Balancer with the properties file you just created, using it as an argument for
   both  `--config-file` (reads metrics and license information) and `--command-config` (reads Admin Client properties):
   ```bash
   confluent-rebalancer execute \
   --metrics-bootstrap-server <broker>:<port> --config-file adb.properties \
   --bootstrap-server <broker>:<port> --command-config adb.properties
   ```

## Example: Run the rebalancer with a Separate Metrics Cluster

[Monitor Kafka with Metrics Reporter in Confluent Platform](../../monitor/metrics-reporter.md#metrics-reporter) is required to compute the rebalance plan for Auto Data Balancer,
This example shows how to point to a dedicated metrics cluster.  In the example,
both the metrics cluster and the rebalancer cluster are secured with mutual TLS (mTLS).

1. Provide information about the metrics and license clusters.

   To calculate metrics on a separate metrics cluster, Auto Data Balancer needs the connection
   information for the metrics cluster in a `--config-file`. This configuration
   should also contain connection information for the license cluster (bootstrap-server).
   For example, you can put this information in a file called `metrics-and-license.properties`:
   ```properties
   confluent.rebalancer.metrics.security.protocol=SSL
   confluent.rebalancer.metrics.ssl.truststore.location=/etc/kafka/secrets/metrics-truststore.jks
   confluent.rebalancer.metrics.ssl.truststore.password=metrics-password
   confluent.rebalancer.metrics.ssl.keystore.location=/etc/kafka/secrets/metrics-keystore.jks
   confluent.rebalancer.metrics.ssl.keystore.password=metrics-password

   confluent.license=<your-confluent-license>
   confluent.license.security.protocol=SSL
   confluent.license.ssl.truststore.location=/etc/kafka/secrets/truststore.jks
   confluent.license.ssl.truststore.password=password
   confluent.license.ssl.keystore.location=etc/kafka/secrets/keystore.jks
   confluent.license.ssl.keystore.password=password
   ```
2. Provide security settings for the Admin Client.

   The `--command-config` file is used to include the security settings for the Admin
   client connecting to the bootstrap-server cluster for rebalancing.  For example,
   you could put these settings in a file called `admin.properties`:
   ```properties
   security.protocol=SSL
   ssl.truststore.location=/etc/kafka/secrets/truststore.jks
   ssl.truststore.password=password
   ssl.keystore.location=/etc/kafka/secrets/keystore.jks
   ssl.keystore.password=password
   ssl.enabled.protocols=TLSv1.2
   request.timeout.ms=60000
   ```
3. Run Auto Data Balancer with the properties file you just created:
   ```bash
   confluent-rebalancer execute \
     --metrics-bootstrap-server <metrics-broker>:<port> --config-file metrics-and-license.properties \
     --bootstrap-server <broker-to-rebalance>:<port> --command-config admin.properties
   ```

<a id="rebalancer-acls"></a>

## ACLs for Auto Data Balancing

In deployments where ACLs are enabled, the following ACLs are needed for the given resources.
For details on creating ACLs, see [Use Access Control Lists (ACLs) for Authorization in Confluent Platform](../../security/authorization/acls/overview.md#kafka-authorization). For a full list of associated operations, resources, and APIs, see the subtopic, [ACL operations and allowed APIs](../../security/authorization/acls/overview.md#acl-operations).

### Cluster-level ACLs

The following ACLs are needed on the clusters.

| Operation    | Resource   | API                    |
|--------------|------------|------------------------|
| CREATE       | Cluster    | CreateTopics, Metadata |
| DESCRIBE     | Cluster    | DescribeAcls           |
| ALTER        | Cluster    | CreateAcls, DeleteAcls |
| ALTERCONFIGS | Cluster    | AlterConfigs           |

### Topic-level ACLs

Topic ACLs are needed to [manage licensing](quickstart.md#licensing-for-rebalancer)  and monitor metrics using system topics, as shown below.

#### ACLs for License Management

Topic-level ACLs are needed for the `_confluent-license` and `_confluent-command` system topics.

| Task (System Topics)   | Operation   | Resource   | API   |
|------------------------|-------------|------------|-------|
| `_confluent-command`   | All         | Topic      | All   |

The `_confluent-command` topic contains the license that corresponds to the
license key supplied through the `confluent.license` property. It is created
by default. Access to this topic requires the following ACLs configured:

* CREATE and DESCRIBE on the cluster to have permissions to create the topic.
* DESCRIBE, READ, and WRITE on the `_confluent-command` topic.

  #### IMPORTANT
  You may also use DESCRIBE and READ without WRITE to restrict access to
  read-only for license topic ACLs. If a topic exists, the LicenseManager
  will not try to create the topic.

You can provide access either individually for each principal that will
use the license or use a [Wildcard principals](../../security/authorization/acls/overview.md#acl-wildcard-principals) to
allow all clients. The following examples show commands that you can use to
configure ACLs for the resource cluster and `_confluent-command` topic.

1. Set a CREATE and DESCRIBE ACL on the resource cluster:
   ```none
   kafka-acls --bootstrap-server localhost:9092 --command-config adminclient-configs.conf \
   --add --allow-principal User:<principal> \
   --operation CREATE --operation DESCRIBE --cluster
   ```
2. Set a DESCRIBE, READ, and WRITE ACL on the `_confluent-command` topic:
   ```none
   kafka-acls --bootstrap-server localhost:9092 --command-config adminclient-configs.conf \
   --add --allow-principal User:<principal> \
   --operation DESCRIBE --operation READ --operation WRITE --topic _confluent-command
   ```

For complete license information for Confluent Platform, see [Manage Confluent Platform Licenses](../../installation/license.md#cp-license-overview).

#### ACLs for Metrics and Monitoring Metrics

Topic-level ACLs are needed to read metrics with the `_confluent-metrics` system topic.

| Task (System Topics)   | Operation   | Resource   | API   |
|------------------------|-------------|------------|-------|
| `_confluent-metrics`   | READ        | Topic      | Fetch |