.. _kafka-dynamic-configurations: Change |ak| Configurations Without Restart for |cp| --------------------------------------------------- Some of the |ak-tm| broker and topic configuration property values can be updated without restarting the |ak| broker. This topic discusses and provides examples for how to update these configuration options dynamically and how to secure password configurations by storing them in encrypted form in |zk|. .. _kafka_dyn_broker_config: Dynamically change broker settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some of the broker configuration settings can be updated without restarting the broker. See the ``Update Mode`` option in :ref:`Broker Configurations ` for the update mode of each broker configuration. * ``read-only``: Requires a broker restart for update. * ``per-broker``: May be updated dynamically for each broker. * ``cluster-wide``: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing. To alter the current broker settings for broker ID 0 (for example, the number of log cleaner threads): .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 To alter several configuration settings at the same time, or to alter configurations with complex values (for example, the :ref:`Router config for Audit Logs `), create a file with the new values in the ``.properties`` format and use ``--add-config-file`` parameter. For example, if you have the new settings in a file named ``new.properties``, the command might look like the following: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config-file new.properties The command to to describe the current dynamic broker settings for broker ID 0 is: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe To delete a configuration override and revert to the statically configured or default value for broker ID 0 (for example, the number of log cleaner threads), use the following command: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads Some settings can be configured as cluster-wide defaults to maintain consistent values across the whole cluster. All brokers in the cluster will process the cluster default update. For example, to update log cleaner threads on all brokers, you could use the following command: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2 To describe the currently configured dynamic cluster-wide default configurations, use the following command: .. codewithvars:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe All settings that are configurable at cluster level may also be configured at per-broker level, for testing, for example. If a configuration value is defined at different levels, the following order of precedence is used: * Dynamic per-broker configuration * Dynamic cluster-wide default configuration * Static broker configuration from the ``server.properties`` file .. _dynamic-config-passwords-upgrade: Update password configurations dynamically ****************************************** Password configuration values that are dynamically updated are encrypted before storing in |zk|. The broker config ``password.encoder.secret`` must be configured in ``server.properties`` to enable dynamic update of password settings. The secret may be different on different brokers. The secret used for password encoding may be rotated with a rolling restart of brokers. The old secret used for encoding passwords currently in |zk| must be provided in the static broker configuration ``password.encoder.old.secret`` and the new secret must be provided in ``password.encoder.secret``. All dynamic password settings stored in |zk| will be re-encoded with the new secret when the broker starts up. All dynamically updated password settings must be provided in every alter request when updating configurations using ``kafka-configs``, even if the password configuration is not being altered. If performing an upgrade, add the broker passwords to |zk| and remove them from ``server.properties``, and then restart the broker in the newly-upgraded version. If passwords remain in ``server.properties``, they are ignored, and the value from |zk| takes higher precedence. If you wish to roll back to an older version without dynamic configuration support, then you must add the passwords back into ``server.properties`` before restarting the broker. The following command shows how you can dynamically update the password configuration for a broker while it is running: .. code-block:: bash bin/kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config 'listener.name.internal.ssl.key.password=key-password' Update password settings in |zk| before starting brokers ******************************************************** ``kafka-configs`` enables dynamic broker settings to be updated using |zk| before starting brokers for bootstrapping. This enables all password configurations to be stored in encrypted form, avoiding the need for clear passwords in ``server.properties``. The broker configuration ``password.encoder.secret`` must also be specified if any password configurations are included in the alter command. Additional encryption parameters may also be specified. Password encoder configurations are not be persisted in |zk|. For example, to store a TLS/SSL key password for listener ``INTERNAL`` on broker 0: .. codewithvars:: bash bin/kafka-configs --zookeeper localhost:2181 --entity-type brokers --entity-name 0 --alter --add-config \ 'listener.name.internal.ssl.key.password=key-password,password.encoder.secret=secret,password.encoder.iterations=8192' The configuration ``listener.name.internal.ssl.key.password`` is persisted in |zk| in encrypted form using the provided encoder configurations. The encoder secret and iterations are not persisted in |zk|. Update the TLS/SSL keystore of an existing listener *************************************************** Brokers may be configured with TLS/SSL key stores with short validity periods to reduce the risk of compromised certificates. Key stores may be updated dynamically without restarting the broker. The configuration name must be prefixed with the listener prefix ``listener.name.{listenerName}.`` so that only the key store configuration of a specific listener is updated. The following configurations may be updated in a single alter request at per-broker level: * ``ssl.keystore.type`` * ``ssl.keystore.location`` * ``ssl.keystore.password`` * ``ssl.key.password`` If the listener is the inter-broker listener, then the update is allowed only if the new key store is trusted by the trust store configured for that listener. For other listeners, no trust validation is performed on the key store by the broker. Certificates must be signed by the same certificate authority that signed the old certificate to avoid any client authentication failures. Update the TLS/SSL trust store of an existing listener ****************************************************** Broker trust stores may be updated dynamically without restarting the broker to add or remove certificates. Updated trust store will be used to authenticate new client connections. The configuration name must be prefixed with the listener prefix ``listener.name.{listenerName}.`` so that only the trust store configuration of a specific listener is updated. The following configurations may be updated in a single alter request at per-broker level: * ``ssl.truststore.type`` * ``ssl.truststore.location`` * ``ssl.truststore.password`` If the listener is the inter-broker listener, then the update is allowed only if the existing key store for that listener is trusted by the new trust store. For other listeners, no trust validation is performed by the broker before the update. Removal of CA certificates used to sign client certificates from the new trust store can lead to client authentication failures. To dynamically update TLS/SSL listeners so that connections use a new keystore (and expired certificates are updated on brokers without a rolling restart): :: kafka-configs --command-config /etc/kafka/client.properties --bootstrap-server hostname:port --entity-type brokers --entity-name --alter --add-config listener.name..ssl.keystore.location= .. _kafka_dyn-topic_config: Dynamically change topic settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some config settings in |ak| are static, meaning they only be changed in a properties file and require a broker restart. However, there are some settings that you can dynamically change on a per-topic basis using the ``kafka-configs`` tool. When changed using the ``kafka-configs`` tool, each change is persistent and remains through broker restarts. Following are some per-topic configurations that you can change: - :platform:`cleanup.policy|installation/configuration/topic-configs.html#cleanup-policy` - :platform:`flush.messages|installation/configuration/topic-configs.html#flush-messages` - :platform:`flush.ms|installation/configuration/topic-configs.html#flush-ms` - :platform:`max.message.bytes|installation/configuration/topic-configs.html#max-message-bytes` - :platform:`min.insync.replicas|installation/configuration/topic-configs.html#min-insync-replicas` - :platform:`retention.bytes|installation/configuration/topic-configs.html#retention-bytes` - :platform:`retention.ms|installation/configuration/topic-configs.html#retention-ms` - :platform:`segment.bytes|installation/configuration/topic-configs.html#segment-bytes` - :platform:`segment.jitter.ms|installation/configuration/topic-configs.html#segment-jitter-ms` - :platform:`segment.ms|installation/configuration/topic-configs.html#segment-ms` - :platform:`unclean.leader.electon.enable|installation/configuration/topic-configs.html#unclean-leader-election-enable` For a full list of topic-level configurations, see :ref:`cp-config-topics`.