Manage Kafka Cluster Configuration Settings in Confluent Cloud¶
This topic describes the default Apache Kafka® cluster configuration settings in Confluent Cloud. For a complete description of all Kafka configurations, see Confluent Platform Configuration Reference.
When editing cluster settings, remember the following:
- You cannot edit cluster settings on Confluent Cloud on Basic, Standard, and Enterprise clusters, but many configuration settings are available at the topic level instead. For more information, see Work With Topics in Confluent Cloud.
- You can change some configuration settings on Dedicated clusters using the Kafka CLI or REST API. See Change cluster settings for Dedicated clusters.
Access cluster settings in the Confluent Cloud Console¶
You can access settings for your clusters with the Cloud Console.
To do so:
Sign in to your Confluent account.
Select an environment and choose a cluster.
In the navigation menu, select Cluster Overview > Cluster settings and the Cluster settings page displays.
Note
You may also see Networking and Security tabs depending on the type of cluster and how it is configured.
Change cluster settings for Dedicated clusters¶
The following table lists editable cluster settings for Dedicated clusters and their default parameter values.
Parameter Name | Default | Editable | More Info |
---|---|---|---|
auto.create.topics.enable | false | Yes | |
ssl.cipher.suites | “” | Yes | |
num.partitions | 6 | Yes | Limits vary, see: Kafka Cluster Types in Confluent Cloud |
log.cleaner.max.compaction.lag.ms | 9223372036854775807 | Yes | Min: 604800000 |
log.retention.ms | 604800000 | Yes | Set to -1 for infinite storage |
To modify these settings, you can use the kafka-configs
script that
is a part of the
Kafka command line tools (installed as a part of Confluent Platform).
To use this script, you will need the bootstrap server for your cluster. See Access cluster settings in the Confluent Cloud Console for how to
retrieve the bootstrap server in the Confluent Cloud Console.
Alternatively, you can use the Kafka REST APIs to change these settings. You will need the REST endpoint and the cluster ID for your cluster to make Kafka REST calls. If you don’t know where to find these, see Find the REST endpoint address and cluster ID to access them in the Cloud Console. For more on how to use the REST APIs, see Kafka REST API Quick Start for Confluent Cloud Developers.
Changes to the settings are applied to your Confluent Cloud cluster without additional action on your part and are persistent until the setting is explicitly changed again.
Important
These settings apply only to Dedicated clusters and cannot be modified on Basic, Standard, and Enterprise clusters.
Enable automatic topic creation¶
Automatic topic creation (auto.create.topics.enable
) is disabled (false
) by default to help prevent unexpected costs. The following commands show
how to enable it. For more on this property, see broker configurations.
bin/kafka-configs --bootstrap-server <bootstrap> --command-config <config-properties> --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true
curl --location --request PUT 'https://<REST endpoint>/kafka/v3/clusters/<cluster-id>/broker-configs/auto.create.topics.enable' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'Content-Type: application/json' \
--data-raw '{
"value": "true"
}'
Restrict cipher suites¶
The following commands show how to restrict the allowed TLS/SSL cipher suites (ssl.cipher.suites
).
For more on this property, see broker configurations.
bin/kafka-configs --bootstrap-server <bootstrap> --command-config config.properties --entity-type brokers --entity-default --alter --add-config ssl.cipher.suites=["list-item-1,list-item-2"]
curl --location --request PUT 'https://<REST endpoint>/kafka/v3/clusters/<cluster-id>/broker-configs/ssl.cipher.suites' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'Content-Type: application/json' \
--data-raw '{
"value": "<int>"
}'
The valid list of ciphers includes:
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
Change the default number of partitions for new topics¶
The following commands show how to set the default number of partitions (num.partitions
) for newly
created topics. On clusters, num.partitions
allows you set a default for newly created topics. On
topics, num.partitions
allows you to specify the number of partitions for a particular topic. For
more information, see broker configurations
and topic configurations.
bin/kafka-configs --bootstrap-server <bootstrap> --command-config <auth-configs> --entity-type brokers --entity-default --alter --add-config num.partitions=<int>
curl --location --request PUT 'https://<REST endpoint>/kafka/v3/clusters/<cluster-id>/broker-configs/num.partitions' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'Content-Type: application/json' \
--data-raw '{
"value": "<int>"
}'
Change maximum compaction lag time¶
The following commands show how to set the default maximum lag compaction time (log.cleaner.max.compaction.lag.ms
) for new topics.
For more on this property, see log.cleaner.max.compaction.lag.ms
and max.compaction.lag.ms.
bin/kafka-configs --bootstrap-server <bootstrap> --command-config <auth-configs> --entity-type brokers --entity-default --alter --add-config log.cleaner.max.compaction.lag.ms=<int>
curl --location --request PUT 'https://<REST endpoint>/kafka/v3/clusters/<cluster-id>/broker-configs/log.cleaner.max.compaction.lag.ms' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'Content-Type: application/json' \
--data-raw '{
"value": "<int>"
}'
Change log retention time¶
The following commands show how to set the default log retention time (log.retention.ms
) for new topics.
For more on this property, see log.retention.ms
and retention.ms.
bin/kafka-configs --bootstrap-server <bootstrap> --command-config <auth-configs> --entity-type brokers --entity-default --alter --add-config log.retention.ms=<int>
curl --location --request PUT 'https://<REST endpoint>/kafka/v3/clusters/<cluster-id>/broker-configs/log.retention.ms' \
--header 'Authorization: Basic <base64-encoded-key-and-secret>' \
--header 'Content-Type: application/json' \
--data-raw '{
"value": "<int>"
}'