Confluent Developer Marketplace Get started
  1. Home
  2. Deploy

Recording Rules for an External Prometheus

Control Center depends on a set of Prometheus recording rules to display metrics. If you bring your own Prometheus, or BYOP, you must load these rules into your external Prometheus instance.

What the recording rules are

A Prometheus recording rule computes an expression ahead of time and saves the result as a new time series. Control Center provides its recording rules in the recording_rules-generated.yml file, which currently contains 22 recording rules.

If the recording rules are missing or the Prometheus version is earlier than 3.0, Control Center shows an error in its UI. For the Prometheus version requirement, see Prometheus version for an external Prometheus.

Get the recording rules file

Copy the file from /etc/confluent-control-center/ on the Control Center host or container, or from etc/confluent-control-center/ in the installation directory for an archive installation. You can also save the following Control Center 2.7 rules as recording_rules-generated.yml:

groups:
  - name: partition_level_heavy_queries
    rules:
      - record: code:io_confluent_kafka_server_log_size_by_broker:total
        expr: sum by (kafka_broker_id, kafka_cluster_id) (io_confluent_kafka_server_log_size{topic!="__cluster_metadata"})
      - record: code:io_confluent_kafka_server_log_size_by_topic:total
        expr: sum by (topic, tenant, tenant_topic, kafka_cluster_id) (io_confluent_kafka_server_log_size{topic!="__cluster_metadata"})
      - record: code:io_confluent_kafka_server_partition_under_min_isr_by_broker:total
        expr: sum by (kafka_broker_id, kafka_cluster_id) (io_confluent_kafka_server_partition_under_min_isr)
      - record: code:io_confluent_kafka_server_log_tier_size_by_broker:total
        expr: sum by (kafka_cluster_id, kafka_broker_id) (io_confluent_kafka_server_log_tier_size{topic!="__cluster_metadata"})
      - record: code:io_confluent_kafka_server_partition_replicas_count_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_partition_replicas_count)"
      - record: code:io_confluent_kafka_server_partition_in_sync_replicas_count_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_partition_in_sync_replicas_count)"
      - record: code:io_confluent_kafka_server_broker_topic_total_fetch_requests_rate_1_min_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_broker_topic_total_fetch_requests_rate_1_min)"
      - record: code:io_confluent_kafka_server_broker_topic_total_fetch_requests_rate_1_min_by_broker:total
        expr: "sum by (kafka_cluster_id, kafka_broker_id) (io_confluent_kafka_server_broker_topic_total_fetch_requests_rate_1_min)"
      - record: code:io_confluent_kafka_server_partition_under_replicated_by_broker:total
        expr: "sum by (kafka_cluster_id, kafka_broker_id) (io_confluent_kafka_server_partition_under_replicated)"
      - record: code:io_confluent_kafka_server_partition_observer_replicas_count_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_partition_observer_replicas_count)"
      - record: code:io_confluent_kafka_server_partition_under_replicated_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_partition_under_replicated)"
      - record: code:io_confluent_kafka_server_partition_under_replicated_by_tenant:total
        expr: "sum by (kafka_cluster_id, tenant) (io_confluent_kafka_server_partition_under_replicated)"
      - record: code:io_confluent_kafka_server_log_tier_size_by_cluster:total
        expr: "sum by (kafka_cluster_id) (max by (partition, topic, kafka_cluster_id) (io_confluent_kafka_server_log_tier_size))"
      - record: code:io_confluent_kafka_server_partition_count_by_topic:total
        expr: "count by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_partition_replicas_count > 0)"
      - record: code:io_confluent_kafka_server_partition_caught_up_replicas_count_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (io_confluent_kafka_server_partition_caught_up_replicas_count)"
      - record: code:io_confluent_kafka_server_log_tier_size_by_topic:total
        expr: "sum by (kafka_cluster_id, tenant, tenant_topic, topic) (max by (topic, tenant, tenant_topic, partition, kafka_cluster_id) (io_confluent_kafka_server_log_tier_size))"
      - record: code:active_consumer_clients_1w:list
        expr: group by(client_external_kafka_cluster_id, client_external_client_id, client_external_principal, client_software_version, client_software_name, tenant, user) (present_over_time(org_apache_kafka_consumer_connection_creation_total[604800000ms]))
      - record: code:active_producer_clients_1w:list
        expr: group by(client_external_kafka_cluster_id, client_external_client_id, client_external_principal, client_software_version, client_software_name, tenant, user) (present_over_time(org_apache_kafka_producer_connection_creation_total[604800000ms]))
      - record: code:active_consumer_clients_1d:list
        expr: group by(client_external_kafka_cluster_id, client_external_client_id, client_external_principal, client_software_version, client_software_name, tenant, user) (present_over_time(org_apache_kafka_consumer_connection_creation_total[86400000ms]))
      - record: code:active_producer_clients_1d:list
        expr: group by(client_external_kafka_cluster_id, client_external_client_id, client_external_principal, client_software_version, client_software_name, tenant, user) (present_over_time(org_apache_kafka_producer_connection_creation_total[86400000ms]))
      - record: code:active_consumer_clients_1h:list
        expr: group by(client_external_kafka_cluster_id, client_external_client_id, client_external_principal, client_software_version, client_software_name, tenant, user) (present_over_time(org_apache_kafka_consumer_connection_creation_total[3600000ms]))
      - record: code:active_producer_clients_1h:list
        expr: group by(client_external_kafka_cluster_id, client_external_client_id, client_external_principal, client_software_version, client_software_name, tenant, user) (present_over_time(org_apache_kafka_producer_connection_creation_total[3600000ms]))

The rules can change in later Control Center versions. Use the rules that match your Control Center version.

Load the recording rules into Prometheus

To load the recording rules into your external Prometheus instance:

  1. Copy the recording_rules-generated.yml file to a location that your Prometheus instance can read. Do not modify the file.

  2. Add the file to the rule_files setting in your Prometheus configuration:

    rule_files:
  - /etc/prometheus/rules/recording_rules-generated.yml

  3. Reload the Prometheus configuration, or restart Prometheus.

  4. Verify that Prometheus loaded the rules. In the Prometheus UI, open the Rule health page (/rules), and confirm that the recording rules from recording_rules-generated.yml are listed.

When you upgrade Control Center, get the recording rules file again and reload it, because the rules can change between versions.

Related content