.. :title: Monitor broker metrics in Confluent Control Center .. meta:: :description: This document describes monitoring brokers in Kafka clusters using the Confluent Control Center user interface. .. _controlcenter_userguide_brokers: ############################################# Manage |ak| Brokers Using |c3-short| for |cp| ############################################# The Brokers overview page enables you to easily assess the health of brokers (nodes) in a |ak| cluster. You can quickly drill into Broker metrics or view the configuration settings for a specific broker. .. _c3_brokers_overview_metrics: Brokers and |c3-short| mode *************************** The panels that are visible on the Brokers overview page, and whether they are clickable, depends on if you are running |c3-short| in :ref:`Normal mode ` or :ref:`Reduced infrastructure mode `. The following table summarizes the view by mode: .. list-table:: :widths: 25 25 50 :header-rows: 1 * - Panel - Normal - Reduced infrastructure mode * - **Partitioning and replication** - Visible and clickable - Visible * - **Active controller** - Visible and clickable - Visible * - **Self-balancing** - Visible and clickable - Visible and clickable * - **Tiered Storage** - Visible and clickable - Visible and clickable * - **Production** and **Consumption** - Visible and clickable - Not visible * - **Zookeeper** - Visible and clickable - Not visible * - **Disk** - Visible and clickable - Not visible * - **System** - Visible and clickable - Not visible *********************** Access the Brokers page *********************** To access the Brokers overview page, select a cluster and then select **Brokers** from the navigation menu in the left panel of the page. .. figure:: /images/c3-brokers-overview.png :scale: 80% :alt: Brokers overview page View of Brokers page in Normal mode ************* Brokers table ************* On the bottom of the Brokers overview page, is table that contains a list of brokers for the cluster. The Brokers table lists all brokers by ID. .. figure:: /images/c3-brokers-table.png :scale: 80% :alt: Brokers table Use this table to: - Search for a broker by its ID. - Sort a column by clicking in the column title cell. - View throughput (bytes in and out per second) and latency percentage metrics. - Access and download broker configuration settings .. _download_broker_config_settings: Download broker configuration settings ====================================== #. From the brokers table, select a broker. #. Click the **Configuration** tab, and then **Overview**. #. Click **Download**. A JSON file that contains the broker configuration settings is downloaded into your downloads directory. The filename is based on the broker name. The file contains a ``KafkaBrokerConfigList`` that in Turn contains a list of ``KafkaBrokerConfig`` objects. Example filename: ``broker_config_0.json`` Example KafkaBrokerConfig: .. code:: json { "kind":"KafkaBrokerConfig", "metadata":{ "self":"http://somehost:9021/api/kafka-rest/12345-qTpmvu9WlO3g6wA/kafka/v3/clusters/12345-qTpmvu9WlO3g6wA/brokers/0/configs/advertised.listeners", "resource_name":"crn:///kafka=12345-qTpmvu9WlO3g6wA/broker=0/config=advertised.listeners" }, "cluster_id":"12345-qTpmvu9WlO3g6wA", "name":"advertised.listeners", "value":null, "is_read_only":false, "is_sensitive":false, "source":"DEFAULT_CONFIG", "synonyms":[ ], "broker_id":0, "is_default":true } .. _c3-brokers-metrics: ************** Broker metrics ************** All of the metrics for your broker are summarized on one page. To view metrics, click any of the following panels on the Brokers overview page: - :ref:`Production ` - :ref:`Consumption ` - :ref:`Partitioning and replication ` - :ref:`Zookeeper and Broker uptime ` - :ref:`Disk ` - :ref:`System ` .. figure:: /images/c3-broker-metrix.png :scale: 80% :alt: Broker Metrics page .. tip:: You can set alerts on many of these metrics, such as: production request latency, :ref:`under-replicated partitions `, out of sync replica count, |zk| status, and more. Send alerts action notifications though email, Slack, or PagerDuty. For details, see :ref:`controlcenter_userguide_alerts`. .. _c3_brokers_metrics_selector: Filter brokers ============== In a multiple broker environment, you can view the metrics charts for multiple brokers. Click **Select all** or **Deselect all**, or individually select each broker for which you want to view metrics. If all brokers are deselected, each panel displays a No Brokers Selected message. .. figure:: /images/c3-filter-brokers.png :scale: 60% :alt: Broker selection panel .. _c3_customize_brokers_metrics_dash: Customize the dashboard for broker metrics ========================================== Drag and drop to rearrange the order of the brokers metrics dashboard panels. .. note:: The settings only persist across the same browser and computer. #. Click **Customize Dashboard** on the Brokers Overview > Metrics page. .. figure:: /images/c3-brokers-metrics-cust-dash.png :scale: 60% :alt: Customize dashboard broker metrics #. Drag the panels into the order you want. #. Click **Save**. .. _c3_brokers_request_latency_selector: Viewing latency by percentage ============================= To view production or consumption request latency by a specific percentage, select a percentage from the menu over the **Request latency** graph. .. figure:: /images/c3-broker-request-latency-menu.png :scale: 60% :alt: Request latency percentage menu .. tip:: Click a point on the graph in the Request latency panels to view details on Production or Consumption request latency. .. _c3_brokers_inspect_cursor: Hover to view details ===================== Hover on any point in a metrics chart to view details for a specific point in time. .. figure:: /images/c3-broker-metrics-hover-details.png :scale: 60% :alt: Inspection cursor broker metrics .. _c3_brokers_production_metrics: Production metrics panel ======================== To view Production metrics, click the **Production** panel from the Brokers overview page. The chart shows throughput, request latency, and any failed production requests. .. figure:: /images/c3-brokers-metrics-prod-panel.png :scale: 60% :alt: Production metrics panel brokers metrics Click a point on the graph in the Request latency panel to view details on Production request latency. .. figure:: /images/c3-production-request-latency.png :scale: 50% :alt: Production request latency Production request latency .. _c3_brokers_consumption_metrics: Consumption metrics panel ========================= .. important:: Production and consumption metrics are visible in :ref:`Normal mode ` only. The charts show throughput, request latency, and any failed consumption requests. .. figure:: /images/c3-brokers-metrics-consump-panel.png :scale: 60% :alt: Consumption metrics panel brokers metrics Click a point on the graph in the Request latency panel to view details on Consumption request latency. .. figure:: /images/c3-consumption-request-latency.png :scale: 50% :alt: Consumption request latency diagram Consumption request latency .. _c3_partition_replicas_metrics: Partition and replication metrics ==================================== To view partition and replication metrics, click the **Partitioning and replication** panel. The charts show the total number of partitions, leaders, under-replicated partitions, and under min in-sync replicas. .. figure:: /images/c3-brokers-partition-replicas-panel.png :scale: 50% :alt: Partition replicas panel broker metrics .. _c3_broker_zookeper_metric: Broker uptime and |zk| metrics ============================== To access the broker uptime and |zk| panel, click the **ZooKeeper** panel on the :ref:`c3_brokers_overview_metrics`. .. Important:: Broker uptime and zookeeper metrics are visible in :ref:`Normal mode ` only. The charts show uptime for available brokers and |zk|. .. figure:: /images/c3-brokers-metrics-uptime-panel.png :scale: 50% :alt: Broker uptime metrics panel .. note:: Broker uptime metrics are cluster-wide, and do not apply to individual brokers. The :ref:`Filter broker controls ` are not applicable to the Broker uptime panel. .. _c3_brokers_disk_metrics: Disk usage panel ================ To view disk usage metrics, click the **Disk** panel from the Brokers overview page. This panel will show a red sidebar if disk usage is skewed and should be rebalanced. .. Important:: The Disk usage panel is visible in :ref:`Normal mode ` only. The Disk panel shows maximum usage, minimum usage, and distribution. .. figure:: /images/c3-brokers-metrics-disk-panel.png :scale: 50% :alt: Disk usage panel .. _c3_brokers_system_metrics: System usage panel ================== To access the System panel, click the **System** summary card from the :ref:`c3_brokers_overview_metrics`. .. Important:: The System panel is visible in :ref:`Normal mode ` only. The System panel shows network and request pool usage. Network pool usage is the average network pool capacity usage across all brokers; that is, the percentage of time that the network processor threads are not idle. Request pool usage is the average request handler capacity usage across all brokers; that is, the percentage of time that the request handler threads are not idle. .. figure:: /images/c3-brokers-metrics-system-panel.png :scale: 50% :alt: System usage panel network and request pool usage .. _c3_active_controller_metric: ***************** Active controller ***************** The **Active controller** panel identifies which broker in your cluster is reporting as the Active controller. In a |ak| cluster, one of the brokers serves as the controller, which is responsible for managing the states of partitions and replicas and for performing administrative tasks like reassigning partitions. At any given time there is only one controller broker in your cluster. This identifier panel enables operators to know which broker is working as the controller. .. figure:: /images/c3-active-controller-panel.png :scale: 50% :alt: Active controller panel .. _c3_self_balancing: ********** |sbc-long| ********** Starting with |cp| 6.1, when the |sbc-long| feature is enabled, clicking the **Self-balancing** card displays broker task statuses. .. figure:: /images/sbc-c3-brokers-task-remove-in-progress.png :scale: 50% :alt: self balancing broker task statuses If |sbc| is disabled, the **Self-balancing** card takes you to **Cluster Settings -> Self-balancing**, which includes a tab where to edit settings and enable |sbc|. The Self-Balancing panel on the **Brokers** page itself indicates the status of |sbc|. .. figure:: /images/c3-self-balancing-panel.png :scale: 50% :alt: self balancing panel To learn more, see the following topics: - :ref:`c3-sbc` in |c3-short| - :ref:`sbc` in the |ak| Operations documentation .. _c3_tiered storage: ************** Tiered storage ************** The **Tiered storage** panel takes you to **Cluster Settings -> Storage**, which includes a tab where you can :ref:`configure Tiered Storage `. .. figure:: /images/c3-tiered-storage-panel.png :scale: 80% :alt: Tiered Storage panel disabled Once **Tiered storage** is up and running, this panel shows storage data statistics. .. figure:: /images/c3-tiered-storage-panel-enabled.png :scale: 80% :alt: Tiered Storage panel enabled Click into these initial stats to view a metrics chart for Tiered Storage. .. figure:: /images/c3-tiered-storage-metrics.png :scale: 80% :alt: Tiered Storage metrics chart To learn more, see the following topics: - :ref:`c3-tiered-storage` in |c3-short| - :ref:`tiered-storage` in the |ak| Operations documentation