Important
You are viewing documentation for an older version of Confluent Platform. For the latest, click here.
Confluent Metrics Reporter¶
Confluent Metrics Reporter collects various metrics from the Kafka cluster necessary for Confluent Control Center system health monitoring and Confluent Auto Data Balancer to operate.
This data is published to a configurable topic (_confluent-metrics
by default) in a configurable Kafka cluster.
You may choose to publish metrics to the cluster the broker is part of – as described in the following section –
or to a different Kafka cluster (e.g. a dedicated metrics cluster).
Installation¶
To enable Confluent Metrics Reporter, install either the confluent-rebalancer
package (installed automatically if
either the confluent-control-center
or the confluent
package is installed) in each broker and add the following
settings in each server.properties
(updated to match your environment):
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=localhost:9092
confluent.metrics.reporter.zookeeper.connect=localhost:2181
# Uncomment the following if the metrics cluster has < 3 brokers
#confluent.metrics.reporter.topic.replicas=1
For convenience, the server.properties
shipped with the Confluent platform includes the above configs commented out
under the Confluent Metrics Reporter
section.
Note that you can choose to publish the metrics to the cluster the broker is part of (like in the config above) or to a different cluster (e.g. a dedicated metrics cluster).
A rolling bounce of the brokers is required for the config changes to be picked up.
In order for this to work correctly, we require the following:
- The broker’s principal must have permission to create the metrics topic in the configured Kafka cluster. This requires write access to the corresponding Zookeeper.
- The broker’s principal must have permission to produce to the metrics topic.
- The tool’s principal must have permission to consume from the metrics topic.
Configuration Options¶
These configs should be added to the server.properties
of each broker that is part of the Kafka cluster to be rebalanced. Only the first two configs below
are required although confluent.metrics.reporter.topic.replicas
should be changed if there are less than 3 brokers in the Kafka metrics cluster. The
other configs allow one to tune the publisher for additional performance and reliability.
confluent.metrics.reporter.bootstrap.servers
Bootstrap servers for the Kafka cluster where metrics will be published to.
- Type: string
- Importance: high
confluent.metrics.reporter.zookeeper.connect
The Zookeeper URL for the Kafka cluster used for publishing metrics. The metrics cluster may be different from the cluster(s) whose metrics are being collected. Several production Kafka clusters can publish to a single metrics cluster, for example.
- Type: string
- Importance: high
confluent.metrics.reporter.publish.ms
The metrics reporter will publish new metrics to the metrics topic in intervals defined by this setting. This means that control center system health data lags by this duration, or that rebalancer may compute a plan based on broker data that is stale by this duration. The default of 15 seconds is a reasonable value for production environments and it typically does not need to be changed.
- Type: long
- Default: 15000
- Importance: medium
confluent.metrics.reporter.topic
Topic on which metrics data will be written.
- Type: string
- Default: “_confluent-metrics”
- Importance: medium
confluent.metrics.reporter.topic.partitions
Number of partitions in the metric topic.
- Type: int
- Default: 12
- Importance: medium
confluent.metrics.reporter.topic.replicas
Number of replicas in the metric topic. It should not be higher than the number of brokers in the Kafka cluster.
- Type: int
- Default: 3
- Importance: medium
confluent.metrics.reporter.topic.retention.bytes
Retention bytes for the metrics topic.
- Type: long
- Default: -1
- Importance: medium
confluent.metrics.reporter.topic.retention.ms
Retention time for the metrics topic.
- Type: long
- Default: 259200000 (3 days)
- Importance: medium
confluent.metrics.reporter.topic.roll.ms
Log rolling time for the metrics topic.
- Type: long
- Default: 14400000 (4 hours)
- Importance: medium
confluent.metrics.reporter.whitelist
Regex matching the yammer metric mbean name or Kafka metric name to be published to the metrics topic.
By default this includes all the metrics required by Confluent Control Center and Confluent Auto Data Balancer. This should typically never be modified unless requested by Confluent.
- Type: string
- Default: includes all the metrics necessary for Confluent Control Center and Confluent Auto Data Balancer
- Importance: low