Manage Kafka Clusters Using Control Center for Confluent Platform

The Clusters page enables you to view tiles for healthy and unhealthy clusters at a glance or search for a cluster being managed by Control Center. What you see on the Clusters page varies depending on whether you are running Control Center in Normal or Reduced infrastructure mode.

The following image shows a example of the Clusters page in Normal mode. Click on a cluster tile for a view of metrics for that cluster.

Clusters - Normal mode

Cluster overview page

The overview page for a single Apache Kafka® cluster provides a summary view of the cluster and its connected services.

The following table describes the panels found on the Clusters page by mode. All of the panels are clickable and navigate you directly to the relevant sections.

Section	Normal mode	Reduced infrastructure mode
Brokers overview	Total brokers with production and consumption throughput.	Not visible in Reduced infrastructure mode.
Topics overview	Total topics, total partitions, under replicated partitions, out of sync replicas.	Total topics and total partitions.
Connect overview	Number of Connect clusters and connector status.	Same as Normal mode.
ksqlDB overview	Number of ksqlDB clusters and persistent queries.	Same as Normal mode.

You can view and edit cluster properties and broker configurations in the Cluster settings pages. When you click the Cluster settings sub-menu for a cluster, the General tab appears by default.

Edit a cluster name

You can edit the cluster name and view the host details. The host field lock icon indicates it cannot be changed.

To change the cluster name:

1. Click the cluster from the cluster navigation bar.

2. Click the Cluster settings menu.

3. Edit the cluster name in Cluster Name field. You should choose a human-friendly name for the cluster that follows these rules:

   • Cluster names can only include characters from the English alphabet, numbers, underscores (_), and dashes (-).
   • Cluster names are case-sensitive.

   The new cluster name is saved to the Cluster Registry in Confluent Platform, if it is enabled.

4. Click Save changes to apply your changes.

   

View-only cluster configuration

View the cluster configuration. If broker configuration editing is disabled, the view is read-only and the Edit Settings feature is not available.

Important

If RBAC is enabled for Control Center, it takes precedence over Access Control settings. The privileges associated with your assigned roles determine what you can view and access.

To view a cluster configuration:

1. Click the cluster from the cluster navigation bar.

2. Click the Cluster settings menu.

3. Click the Cluster defaults tab.

   

Each setting section indicates if there are overrrides applied to that setting.

Edit dynamic cluster configuration settings

The dynamic cluster configuration option is enabled by default. The feature allows editing cluster configurations within Control Center that do not require a restart. Any configuration option that has a per-broker or cluster-wide Dynamic Update Mode can be edited. Any options designated as read-only for the dynamic update mode requires a restart and cannot be edited dynamically.

For information about available broker configuration options, including the dynamic update mode per option, see Kafka Broker and Controller Configuration Reference for Confluent Platform.

To edit the cluster configuration:

1. Click the cluster from the cluster navigation bar.

2. Click the Cluster settings menu.

3. Click the Cluster defaults tab.

4. Find the section for the setting you want to edit, and click Edit settings.

   1. Hover over the info icon for a configuration option to view its description.
   2. Make your configuration changes.
   3. Notice a blue undo icon indicates the setting was modified from the default value and you can set it back to the default by clicking the arrow. A gray undo icon indicates the settings is set to its default.
   

5. Click Save to save your changes, or Cancel to cancel edits to settings.

To see examples of changing broker configurations dynamically using the Kafka command line kafka-configs tool, see Dynamically change broker settings.

Enable and disable dynamic editing of broker configurations

The Dynamic Cluster Configuration feature that allows editing settings without a restart is enabled by default. The feature can be disabled in Control Center if an organization does not want any users to access the feature. After disabling the feature, the Edit Settings button in the Brokers tab in the Control Center UI is no longer visible.

Important

If RBAC is enabled for Control Center, it takes precedence over Access Control settings.

To disable the edit broker config settings feature in Control Center:

1. Set the confluent.controlcenter.broker.config.edit.enable option in your control-center.properties file to false.

   ...
   confluent.controlcenter.broker.config.edit.enable=false
   ...
   

   Note

   Make the change in the appropriate Control Center properties files configured for your environments, including control-center-dev.properties or control-center-production.properties. The properties files are located in /path-to-confluent/etc/confluent-control-center/.

2. Restart Control Center and pass in the properties file for the configuration to take effect. For more information, see Control Center Configuration Examples for Confluent Platform.

   • Local (dev) environment:

     Important

     Do not use confluent local for production use.

     confluent local services control-center stop
     confluent local services control-center start --config ../etc/confluent-control-center/control-center.properties
     

   • Production environment:

     ./bin/control-center-stop
     ./bin/control-center-start ../etc/confluent-control-center/control-center.properties
     

To enable the feature again, set the option back to true and restart Control Center with the updated properties file.

Work with Self-Balancing Clusters

For a complete guide to Self-Balancing Clusters, see Manage Self-Balancing Kafka Clusters in Confluent Platform, including a primer on Configuration and monitoring and comprehensive hands-on tutorial.

Monitor and configure Self-Balancing

1. Click the cluster from the cluster navigation bar.

2. Click the Cluster settings menu.

3. Click the Self-balancing tab.

   

4. To change dynamic configurations, click Edit Settings.

   Self-Balancing has several dynamic properties which can be configured while the cluster is running, including options to:

   • Turn Self-Balancing On or Off. In the example server.properties file that ships with Confluent Platform, confluent.balancer.enable is set to true, which means Self-Balancing is on.
   • Override the default throttle value (10485760 or 10MB/sec), which determines the maximum network bandwidth available for Self-Balancing. (confluent.balancer.throttle.bytes.per.second)
   • Toggle the trigger condition for rebalance to either Only when brokers are added or removed (the default) or Anytime. (confluent.balancer.heal.uneven.load.trigger)

5. Make changes and click Save.

When Self-Balancing is on and set to trigger Only when brokers are added or removed (the default), the card on the Brokers overview shows “Workload optimizer” as “Disabled”.

When Self-Balancing is on, and set to trigger Anytime, the card on the Brokers overview shows “Workload optimizer” as “Balanced” if rebalancing is not in progress, or one of the work-in-progress statuses during a rebalance.

Remove a broker

If you remove a broker from the cluster, Self-Balancing will redistribute topic data to the other brokers. A summary of this procedure is shown below. For a more detailed walkthrough, see Remove a broker in the context of the Tutorial: Add and Remove Brokers with Self-Balancing in Confluent Platform.

Important

• In practice, you can remove a lead broker. It may cause a short delay in cluster balancing, which is why we suggest not doing so for this example. To learn more, What happens if the lead broker (controller) is removed or lost?.
• If the broker you attempt to remove contains the only replica for a topic, the broker removal will fail. To learn more, see Limitations.
• Starting with Confluent Platform 7.3.0, Self-Balancing Clusters supports Apache Kafka® Raft Metadata mode (KRaft), see KRaft: Apache Kafka without ZooKeeper. If you are using Self-balancing with Confluent for Kubernetes, and using CFK to shut down the broker, there is no further action needed. However, if you are running Confluent Platform outside of CFK on a KRaft supported version of Confluent Platform, and separately calling kafka-remove-brokers, you must take an additional action to call the UnregisterBroker API through kafka-cluster.sh after the broker removal is complete. If you call kafka-remove-brokers with shouldShutdown=false, you must manually remove the broker; if you set shouldShutdown=true, SBC will shut down the broker for you. In either case, on KRaft mode Confluent Platform deployments, you must call UnregisterBroker after broker shutdown is complete. This will cleanly remove all traces of the broker on the cluster. In non-KRaft Confluent Platform, this step is not required; but in KRaft mode, simply calling kafka-remove-brokers is not enough to remove all lingering metadata internally within the controllers. The UnregisterBroker API does this cleanup.

1. Remove a broker using the Control Center option on the Brokers overview page.

   • Select Brokers, scroll to the bottom of the Overview page to view the list of brokers currently online.

   • Select a broker to remove, to drill down to its monitoring details.

     

   • At the bottom of this page, click Remove broker, then type REMOVE in the input field to verify that you want to take this action.

   When the broker removal is completed, click the confirm option to acknowledge.

   Note

   If you get an error message that broker removal failed due to insufficient metrics, Self-Balancing is still initializing, which can take up to 30 minutes. If this happens, retry broker removal after several minutes, and it should succeed.

2. Use the Control Center to monitor the rebalance.

   On Control Center click Cluster settings > Self-balancing to track the progress.

   When the rebalance is complete, both the Brokers overview page and Cluster settings > Broker defaults will show only the remaining brokers, and not the one you removed.

Suggested Resources

• To learn more, see Manage Self-Balancing Kafka Clusters in Confluent Platform.
• For troubleshooting related to Self-Balancing and Control Center, see Self-Balancing options do not show up on Control Center, Broker metrics are not displayed on Control Center, and Consumer lag reflected on Control Center in the Self-balancing Clusters troubleshooting section.

Configure Tiered Storage

For a complete guide to setting up and working with Tiered Storage, see Tiered Storage in Confluent Platform.

To configure and work with Tiered Storage starting from Control Center:

1. Click the cluster from the cluster navigation bar.

2. Click the Cluster settings menu.

3. Click the Tiered storage tab.

   

   You can hide or show the on-screen setup instructions, which walk through cloud provider setup as fully described in Tiered Storage in Confluent Platform.

4. To view and edit dynamic settings, click Edit settings.

   

   View or change settings and click Cancel or Save changes as appropriate.

5. To set up storage, choose a cloud provider (click the GCS or S3 tab).

   The S3 configuration options are shown here as an example.

   

6. Specify property values and paths to your credentials, then click Generate configurations.

   

7. Copy the generated configurations block and paste it into the properties files for your brokers (for example, $CONFLUENT_HOME/etc/kafka/server.properties).

   

   Important

   • The same bucket must be used across all brokers within a Tiered Storage enabled cluster. This applies to all supported platforms.

   • The Tiered Storage internal topic defaults to a replication factor of 3. If you use confluent local services start to run a single broker cluster such as that described in Quick Start for Confluent Platform,’ you must add an additional line to the broker file, $CONFLUENT_HOME/etc/kafka/server.properties:

     confluent.tier.metadata.replication.factor=1

   • As a recommended best practice, do not set a retention policy on the cloud storage (such as an AWS S3 bucket) because this may conflict with the Kafka topic retention policy.

8. After you update these configurations to enable Tiered Storage, restart the brokers. This can be done in a rolling fashion.

9. View cluster-wide metrics for Tiered Storage are shown on the Tiered Storage card on the Brokers overview page for the cluster.

   

   Click into these initial stats to view a metrics chart for Tiered Storage.

   

   Hover and slide the cursor over a chart to get details on data at any particular point in time.

   

10. To get storage metrics on a specific topic, navigate to the topic (choose Cluster > Topics, select a topic from the list).

    The Storage card is shown on the Overview page for the topic.

    

To learn more, see Tiered Storage in Confluent Platform.