USM Agent: Metrics and Metadata Reference
The Unified Stream Manager (USM) Agent collects metrics and metadata from your Confluent Platform deployment and sends them to Confluent Cloud to power USM capabilities in the Confluent Cloud UI. The collected data is organized by source component:
Metadata collected: Metadata records, such as topic configurations, connector configurations, and plugin configuration definitions
Metrics collected: Metrics, such as counters, gauges, and deltas
Metadata collected
The USM Agent collects the following metadata records:
Kafka topic metadata, including topic configurations, from Kafka KRaft-based clusters
Kafka Connect (Connect) connector metadata, including connector configurations and connector and task status, from Connect clusters
Connect plugin metadata, including plugin configuration definitions, from Connect clusters
Topic and connector metadata is emitted when resources are created, updated, or deleted. Plugin metadata is emitted as a snapshot of the plugins installed on the Connect worker.
Kafka topic metadata
Topic metadata fields collected from Kafka include the following.
Name |
Description |
Example |
|---|---|---|
|
Confluent Platform version |
|
|
Event subject |
|
|
Epoch value |
|
|
Partition key |
|
|
Whether the broker role is enabled |
|
|
Kafka node ID |
|
|
Java version of the emitting process |
|
|
Data schema when present, or |
— |
|
Whether the controller role is enabled |
|
|
Kafka cluster ID |
|
|
Kafka commit ID |
|
|
Data media type |
|
|
Event timestamp (ISO 8601) |
|
|
Host name of the emitting node |
|
|
Route |
|
|
Broker or controller node ID |
|
|
Unique topic identifier |
|
|
Topic name |
|
|
Number of partitions |
|
|
Replication factor |
|
|
Retention period in milliseconds, or -1 if unset |
|
|
Retention limit in bytes, or -1 if unset |
|
|
Cleanup policy |
|
|
Compression type |
|
|
Minimum in-sync replicas |
|
|
Maximum message size in bytes |
|
|
Segment size in bytes |
|
|
Segment roll time in milliseconds |
|
|
Timestamp type |
|
|
Delay before deleting segment files |
|
|
Number of messages before flush |
|
|
Time before flush in milliseconds |
|
|
Index interval in bytes |
|
|
Maximum compaction lag in milliseconds |
|
|
Minimum cleanable dirty ratio |
|
|
Segment index size in bytes |
|
|
Delete retention in milliseconds |
|
|
Topic creation time, in seconds since epoch |
|
|
Topic creation time, in nanoseconds |
|
|
Topic last update time, in seconds since epoch |
|
|
Topic last update time, in nanoseconds |
|
|
Whether to preallocate the file on disk when creating a new log segment |
|
|
Whether replicas that are not in the in-sync replica set can be elected as leader |
|
|
Whether schema ID validation is enabled for record keys |
|
|
Whether schema ID validation is enabled for record values |
|
|
Minimum time a message stays uncompacted, in milliseconds |
|
|
Maximum random jitter subtracted from the segment roll time, in milliseconds |
|
|
Replicas throttled on the leader side |
|
|
Replicas throttled on the follower side |
|
|
Whether down-conversion of message formats is enabled for consumer requests, when supported by the broker |
|
|
Message format version, when supported by the broker |
|
|
Maximum difference allowed between the record timestamp and the broker timestamp, when supported by the broker |
|
Connect connector metadata
Connector metadata is emitted from Connect when connectors or their configurations change. The following table lists the connector metadata fields collected from Connect.
Name |
Description |
Example |
|---|---|---|
|
Connector name |
|
|
Connect cluster group ID |
|
|
Kafka cluster ID the Connect cluster uses |
|
|
Connector class |
|
|
Topics used by the connector |
|
|
Connector configuration entries, as key-value pairs |
|
|
Maximum number of tasks |
|
|
Value converter class |
|
|
Connector type |
|
|
Kafka cluster ID the connector reads from or writes to |
|
|
Current state of the connector |
|
|
Error stack trace of the connector, or empty when there is no error |
— |
|
ID of a connector task |
|
|
Current state of the task |
|
|
Error stack trace of the task, or empty when there is no error |
— |
|
Time of the connector metadata event |
|
Connect plugin metadata
To support connector management from Confluent Cloud, the USM Agent also collects metadata about the plugins installed on each Connect worker. Plugins include connectors (source and sink), converters, header converters, transformations, and predicates. The Connect worker emits one plugin metadata event for each installed plugin as part of a snapshot of its plugin inventory. Confluent Cloud uses the configuration definitions in this metadata to help you configure connectors.
The following table lists the plugin metadata fields collected from Connect.
Name |
Description |
Example |
|---|---|---|
|
Connect cluster group ID |
|
|
Kafka cluster ID the Connect cluster uses |
|
|
Fully qualified class name of the plugin |
|
|
Plugin version |
|
|
Plugin type. One of |
|
|
Location of the plugin on the Connect worker |
|
|
URL of the plugin documentation, when available |
— |
|
List of configuration definitions for the plugin. For the fields in each definition, see the following table. |
— |
|
Time the event was emitted |
|
Each entry in plugin.config_definitions describes one configuration
property that the plugin accepts. The USM Agent collects the definition
of the property, not the value that you set for it.
Name |
Description |
Example |
|---|---|---|
|
Configuration property name |
|
|
Value type. One of |
|
|
Whether the property is required |
|
|
Default value of the property |
|
|
Importance of the property. One of |
|
|
Description of the property |
|
|
Group that the property is displayed in |
|
|
Position of the property within its group |
|
|
Display width of the property. One of |
|
|
Display name of the property |
|
|
Names of other properties that depend on this property |
— |
Metrics collected
The USM Agent collects metrics from Kafka and Connect. When you enable client monitoring, it also collects metrics from client applications, including Kafka Streams. Metrics are sampled periodically, typically every 60 seconds, and sent to Confluent Cloud.
Kafka cluster metrics
Kafka cluster metrics collected by the USM Agent include the following.
Metric name |
Description |
Additional applicable tags |
|---|---|---|
|
The current number of bytes retained by the cluster, summed across all partitions. Sampled every 60 seconds. |
|
|
The cumulative size, in bytes, of a topic partition’s data on tiered (object) storage, summed across all partitions at the cluster level. Each partition is counted once, so the value is not multiplied by the replication factor. Sampled every 60 seconds. Requires Confluent Platform 7.9.10, 8.1.6, 8.2.4, or 8.3.2, or a later patch in the same release line. Brokers on earlier patches do not send this metric. |
|
|
The number of partitions whose tiered storage tasks are in an error state, summed across brokers. A healthy cluster reports 0. A non-zero value means a broker hit a non-retriable error on a partition, such as a corrupted segment or index. The count stays non-zero until the affected broker restarts, even after the cause is fixed. Check the broker logs for tier archiver errors. Sampled every 60 seconds. |
|
|
The offset of the first message in a partition. Sampled every 60 seconds. |
|
|
The offset of the last message in a partition. Sampled every 60 seconds. |
|
|
The number of bytes received from the network since the previous sample. Sampled every 60 seconds. |
|
|
The number of bytes sent over the network since the previous sample. Sampled every 60 seconds. |
|
|
The number of records received since the previous sample. Sampled every 60 seconds. |
|
|
The number of records sent since the previous sample, including unsuccessful sends. Sampled every 60 seconds. |
|
|
The number of active controllers in the cluster. Alert if the aggregated sum across all brokers is anything other than one, as there must be exactly one controller per cluster. |
|
|
The number of voters in the KRaft quorum. Sampled every 60 seconds. The implicit time and space aggregation for this metric is |
|
|
The number of partitions without an active leader. These partitions are neither writable nor readable. Alert if the value is greater than 0. |
|
|
The rate of unclean leader elections. |
|
|
The number of under-replicated partitions. In a healthy cluster, the number of in-sync replicas (ISRs) equals the total number of replicas. Under-replicated partitions occur when a broker is down or cannot replicate fast enough from the leader. |
|
|
The number of partitions where the in-sync replica (ISR) count is lower than the configured minimum. |
|
|
The average fraction of time that network processor threads are idle. Values range from 0, meaning all resources are in use, to 1, meaning all resources are available. |
|
|
The average fraction of time that request handler threads are idle. |
|
|
The number of leaders on the broker. |
|
|
The number of partitions on the broker. |
|
|
The count of active connections. |
|
|
The total capacity of the storage volume, in bytes. |
|
|
The amount of free space on the storage volume, in bytes. |
|
|
The number of failed produce requests since the previous sample. Sampled every 60 seconds. |
|
|
The number of failed fetch requests since the previous sample. Sampled every 60 seconds. |
|
|
The total duration of inbound requests within the sampling window. To calculate the average time per request, divide this value by |
|
|
The number of requests received by the broker within the sampling window. Sampled every 60 seconds. |
|
|
Lag between a member’s committed offset and the partition high watermark. |
|
|
The maximum pending rebalance time, in milliseconds, among all members of a consumer group |
|
|
The number of bytes received by the client since the previous sample. Sampled every 60 seconds. |
|
|
The number of bytes sent by the client since the previous sample. Sampled every 60 seconds. |
|
|
The p50 (median) duration of inbound requests. The value is the maximum p50 latency across requests of each type within the sampling window. |
|
|
The p95 duration of inbound requests. The value is the maximum p95 latency across requests of each type within the sampling window. |
|
|
The p99 duration of inbound requests. The value is the maximum p99 latency across requests of each type within the sampling window. |
|
|
The p99.9 duration of inbound requests. The value is the maximum p99.9 latency across requests of each type within the sampling window. |
|
|
The current size of the log, in bytes. Sampled every 60 seconds. |
|
|
The number of replicas for a partition. Sampled every 60 seconds. |
|
|
Indicates whether a partition is under-replicated (1) or not (0). A partition is under-replicated when its in-sync replica (ISR) count is lower than its total replica count. Sampled every 60 seconds. |
|
|
The state of a consumer group, as an integer: 0 = UNKNOWN, 1 = PREPARING_REBALANCE, 2 = COMPLETING_REBALANCE, 3 = STABLE, 4 = EMPTY, 5 = ASSIGNING, 6 = RECONCILING. |
|
|
The current count of active nodes for the Confluent Platform resource. Sampled every 60 seconds. The implicit time aggregation for this metric is MAX. |
Only the tags listed under Node count metric tags |
Connect cluster metrics
Connect cluster metrics collected by the USM Agent include the following.
Metric name |
Description |
Additional applicable tags |
|---|---|---|
|
The total number of tasks from all workers for a connector. Sampled every 60 seconds. The implicit time aggregation for this metric is MAX. |
|
|
The number of failed tasks from all workers for a connector. Ideally, this value is 0. Sampled every 60 seconds. The implicit time aggregation for this metric is MAX. |
|
|
The number of records sent from transformations and written to Kafka by the source connector since the previous sample. Sampled every 60 seconds. |
|
|
The number of records received by the sink connector since the previous sample. Sampled every 60 seconds. |
|
|
The number of bytes sent from transformations and written to Kafka by the source connector since the previous sample. Sampled every 60 seconds. |
|
|
The number of bytes received by the sink connector since the previous sample. Sampled every 60 seconds. |
|
|
The number of dead letter queue records written to Kafka by the sink connector since the previous sample. Sampled every 60 seconds. |
— |
|
The average size of all client requests to the Connect cluster within a 60-second window. |
— |
|
The current count of active nodes for the Confluent Platform resource. Sampled every 60 seconds. The implicit time aggregation for this metric is MAX. |
Only the tags listed under Node count metric tags |
Client metrics
The USM Agent collects these metrics only when client monitoring is enabled. Client monitoring is disabled by default. For setup, see Configure client monitoring for Confluent Platform with USM and Configure Kafka Streams monitoring for Confluent Platform with USM.
Metric name |
Description |
Additional applicable tags |
|---|---|---|
|
The number of new connections established by the producer since the previous sample. |
|
|
The number of requests sent by the producer since the previous sample. |
|
|
The average request latency of the producer, in milliseconds. |
|
|
The rate of bytes sent by the producer per topic, in bytes per second. |
|
|
The number of new connections established by the consumer since the previous sample. |
|
|
The number of requests sent by the consumer since the previous sample. |
|
|
The average time taken for a fetch request, in milliseconds. |
|
|
The rate of bytes consumed by the consumer per topic, in bytes per second. |
|
|
The current state of the Kafka Streams client. |
|
|
The current state of a Kafka Streams thread. |
|
|
The fraction of time the thread spent polling records. |
|
|
The fraction of time the thread spent committing records. |
|
|
The fraction of time the thread spent processing records. |
|
|
The fraction of time the thread spent executing punctuators. |
|
|
The average number of records processed per poll by the Kafka Streams thread. |
|
|
The average end-to-end latency of a processor node. |
|
|
The maximum end-to-end latency of a processor node. |
|
|
The minimum end-to-end latency of a processor node. |
|
|
The percentage of RocksDB block cache usage. |
|
|
The estimated number of keys. |
|
|
The size of all RocksDB memtables, in bytes. |
|