Dynamic Configs --------------- Some of the broker and topic configs can be updated without restarting the broker. This section talks about updating these configuration options dynamically and securing password configs by storing them in encrypted form in |zk|. Changing Broker Configs Dynamically ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some of the broker configs can be updated without restarting the broker. See the ``Dynamic Update Mode`` option in :ref:`Broker Configurations ` for the update mode of each broker config. * ``read-only``: Requires a broker restart for update * ``per-broker``: May be updated dynamically for each broker * ``cluster-wide``: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing. To alter the current broker configs for broker ID 0 (for example, the number of log cleaner threads): .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 To describe the current dynamic broker configs for broker ID 0: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe To delete a config override and revert to the statically configured or default value for broker ID 0 (for example, the number of log cleaner threads): .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads Some configs may be configured as a cluster-wide default to maintain consistent values across the whole cluster. All brokers in the cluster will process the cluster default update. For example, to update log cleaner threads on all brokers: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2 To describe the currently configured dynamic cluster-wide default configs: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe All configs that are configurable at cluster level may also be configured at per-broker level (e.g. for testing). If a config value is defined at different levels, the following order of precedence is used: * Dynamic per-broker config stored in |zk| * Dynamic cluster-wide default config stored in |zk| * Static broker config from ``server.properties`` * |ak-tm| default, see `broker configs <#brokerconfigs>`__ Updating Password Configs Dynamically ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Password config values that are dynamically updated are encrypted before storing in |zk|. The broker config ``password.encoder.secret`` must be configured in ``server.properties`` to enable dynamic update of password configs. The secret may be different on different brokers. The secret used for password encoding may be rotated with a rolling restart of brokers. The old secret used for encoding passwords currently in |zk| must be provided in the static broker config ``password.encoder.old.secret`` and the new secret must be provided in ``password.encoder.secret``. All dynamic password configs stored in |zk| will be re-encoded with the new secret when the broker starts up. In the current version, all dynamically updated password configs must be provided in every alter request when updating configs using ``kafka-configs`` even if the password config is not being altered. This constraint will be removed in a future release. Updating Password Configs in |zk| Before Starting Brokers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``kafka-configs`` enables dynamic broker configs to be updated using |zk| before starting brokers for bootstrapping. This enables all password configs to be stored in encrypted form, avoiding the need for clear passwords in ``server.properties``. The broker config ``password.encoder.secret`` must also be specified if any password configs are included in the alter command. Additional encryption parameters may also be specified. Password encoder configs will not be persisted in |zk|. For example, to store SSL key password for listener ``INTERNAL`` on broker 0: .. codewithvars:: bash bin/kafka-configs --zookeeper localhost:2181 --entity-type brokers --entity-name 0 --alter --add-config \ 'listener.name.internal.ssl.key.password=key-password,password.encoder.secret=secret,password.encoder.iterations=8192' The configuration ``listener.name.internal.ssl.key.password`` will be persisted in |zk| in encrypted form using the provided encoder configs. The encoder secret and iterations are not persisted in |zk|. Updating SSL Keystore of an Existing Listener ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Brokers may be configured with SSL keystores with short validity periods to reduce the risk of compromised certificates. Keystores may be updated dynamically without restarting the broker. The config name must be prefixed with the listener prefix ``listener.name.{listenerName}.`` so that only the keystore config of a specific listener is updated. The following configs may be updated in a single alter request at per-broker level: * ``ssl.keystore.type`` * ``ssl.keystore.location`` * ``ssl.keystore.password`` * ``ssl.key.password`` If the listener is the inter-broker listener, the update is allowed only if the new keystore is trusted by the truststore configured for that listener. For other listeners, no trust validation is performed on the keystore by the broker. Certificates must be signed by the same certificate authority that signed the old certificate to avoid any client authentication failures. Updating SSL Truststore of an Existing Listener ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Broker truststores may be updated dynamically without restarting the broker to add or remove certificates. Updated truststore will be used to authenticate new client connections. The config name must be prefixed with the listener prefix ``listener.name.{listenerName}.`` so that only the truststore config of a specific listener is updated. The following configs may be updated in a single alter request at per-broker level: * ``ssl.truststore.type`` * ``ssl.truststore.location`` * ``ssl.truststore.password`` If the listener is the inter-broker listener, the update is allowed only if the existing keystore for that listener is trusted by the new truststore. For other listeners, no trust validation is performed by the broker before the update. Removal of CA certificates used to sign client certificates from the new truststore can lead to client authentication failures. Changing Topic Configs Dynamically ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Many config settings in |ak| are static and are wired through the properties file. However, there are several settings that you can change per topic. These settings can be changed dynamically using the ``/bin/kafka-topics`` tool without having to restart the brokers. When changed using the ``kafka-topics`` tool, each change is persistent and remains through broker restarts. Here are some per-topic configs that you can change. ~~~~~~~~~~~~~~ cleanup.policy ~~~~~~~~~~~~~~ Indicates whether old segments in the log are deleted or deduplicated. - Type: string - Default ``delete`` - Importance: Medium ~~~~~~~~~~~~~~ flush.messages ~~~~~~~~~~~~~~ The number of messages that can be written to the log before a flush is forced. - Type: long - Default: ``Long.MAX_VALUE`` - Importance: Medium ~~~~~~~~ flush.ms ~~~~~~~~ The amount of time the log can have dirty data before a flush is forced. - Type: long - Default: ``Long.MAX_VALUE`` - Importance: Medium ~~~~~~~~~~~~~~~~~ max.message.bytes ~~~~~~~~~~~~~~~~~ The maximum size of a message. - Type: int - Default: ``Integer.MAX_VALUE`` - Importance: Medium ~~~~~~~~~~~~~~~~~~~ min.insync.replicas ~~~~~~~~~~~~~~~~~~~ If the number of insync replicas drops below this number, |ak| stops accepting writes with -1 (or all) ``request.required.acks``. - Type: int - Default: ``1`` - Importance: High ~~~~~~~~~~~~~~~ retention.bytes ~~~~~~~~~~~~~~~ The maximum size a log partition can grow to before old log segments are discarded to free up space. This only applies if the "delete" retention policy is used. - Type: long - Default: ``Long.MAX_VALUE`` - Importance: Low ~~~~~~~~~~~~ retention.ms ~~~~~~~~~~~~ The maximum time a log is retained before old log segments are discarded. This only applies if the "delete" retention policy is used. - Type: long - Default: ``Long.MAX_VALUE`` - Importance: Low ~~~~~~~~~~~~~ segment.bytes ~~~~~~~~~~~~~ The hard maximum for the size of a segment file in the log. - Type: int - Default: ``1073741824`` - Importance: Low | ~~~~~~~~~~~~~~~~~ segment.jitter.ms ~~~~~~~~~~~~~~~~~ The maximum random jitter subtracted from the scheduled segment roll time to avoid segment rolling. - Type: long - Default: ``0L`` - Importance: Low ~~~~~~~~~~ segment.ms ~~~~~~~~~~ The soft maximum on the amount of time before a new log segment is rolled. - Type: long - Default: ``Long.MAX_VALUE`` - Importance: Low ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ unclean.leader.election.enable ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Indicates whether unclean leader election is enabled. If it is, then a leader may be moved to a replica that is not insync with the leader when all insync replicas are not available, leading to possible data loss. - Type: boolean - Default: ``false`` - Importance: High