Monitor the Confluent for Kubernetes operator

Confluent for Kubernetes (CFK) exposes its own Prometheus metrics—signals about how well the operator itself is managing your Confluent Platform resources, such as reconciling custom resources (CRs), keeping up with work, and staying healthy. These metrics describe the operator process, not the Kafka/Confluent Platform data it manages.

Note

This is a different endpoint from the JMX Prometheus exporter described in JMX Metrics. That exporter runs on each Confluent Platform component pod and exposes metrics about Confluent Platform itself. The endpoint on this page runs on the CFK operator pod and exposes metrics about the operator process. Both use port 7778, but on different pods.

The endpoint

• Where: http://<operator-pod>:7778/metrics. The Helm chart exposes this port through a ClusterIP Service named confluent-operator.

• Format: plain-text Prometheus/OpenMetrics exposition format, a standard format that most monitoring tools can read. Nothing Kubernetes-specific is required to consume it, and collecting it with a scraper is optional.

• Access: cluster-internal only. The endpoint is unauthenticated and has no TLS, so only the Kubernetes network can reach it. Do not expose it publicly.

Metric types

Each metric is one of the following types, which tells you how to read it:

• counter: A cumulative value that only increases, such as total reconciles. Read it as a rate.

• gauge: A current value that can go up or down, such as queue depth. Read it as-is.

• histogram: A distribution of measurements, for calculating percentiles. On this page, histograms measure durations, so their names end in _seconds.

Global (operator-wide) metrics

These metrics are emitted once for the operator process.

Metric	Type	Meaning
io_confluent_cfk_confluent_operator_replica_count	gauge	Number of operator replicas running.
io_confluent_cfk_confluent_operator_liveness_probe_count_total	counter	Liveness probe (/healthz) hits.
io_confluent_cfk_confluent_operator_readiness_probe_count_total	counter	Readiness probe (/readyz) hits.
rest_client_requests_total	counter	The operator’s calls to the Kubernetes API, labeled by code, method, and host. A rise in 4xx/5xx responses indicates RBAC or API-server trouble.
certwatcher_read_certificate_total / certwatcher_read_certificate_errors_total	counter	Webhook TLS certificate reloads and reload failures.
controller_runtime_webhook_panics_total	counter	Admission webhook panics. Should be 0.
go_*, process_*	gauge/counter	Go runtime and process metrics: memory, goroutines, garbage collection, CPU, and file descriptors. Use these for leak or resource-pressure detection.

Controller (per-resource) metrics

A controller is the reconcile loop that manages one CR type. The operator runs one controller per CR type, and every controller emits the same set of metrics, distinguished by a controller (or cfk_controller_name) label.

This covers both Day-1 and Day-2 resources. Every CR type has a controller and emits these metrics, even if no CR of that type currently exists in the cluster (the series reads 0):

• Day-1 (workloads): kafka, zookeeper, kraftcontroller, connect, schemaregistry, ksqldb, controlcenter, kafkarestproxy, gateway, flinkapplication, flinkenvironment, usmagent

• Day-2 (REST-driven): kafkatopic, schema, connector, confluentrolebinding, clusterlink, schemaexporter, schemaimporter

• Configuration/migration: kafkarestclass, cmfrestclass, kraftmigrationjob

For example, a KafkaTopic (Day-2) controller emits the same reconcile_total, workqueue_depth, and latency metrics as a Kafka (Day-1) controller. Only the label value differs.

Reconcile

Metric	Type	Meaning
controller_runtime_reconcile_total	counter	Reconcile outcomes, by result: success, requeue, requeue_after, or error. The primary health signal for a controller.
controller_runtime_reconcile_errors_total	counter	Reconciles that returned an error.
controller_runtime_terminal_reconcile_errors_total	counter	Errors the operator gave up retrying. These need human action.
controller_runtime_reconcile_panics_total	counter	Panics during reconcile. Should be 0.
controller_runtime_reconcile_time_seconds	histogram	Reconcile duration.
controller_runtime_active_workers	gauge	Workers currently reconciling.
controller_runtime_max_concurrent_reconciles	gauge	The worker cap for this controller. CFK runs one worker per controller.

Work queue

These metrics describe the items waiting to be reconciled by a controller.

Metric	Type	Meaning
workqueue_depth	gauge	Items waiting to be reconciled. 0 is healthy. A sustained non-zero value means the operator is falling behind.
workqueue_adds_total	counter	Items enqueued.
workqueue_retries_total	counter	Re-enqueues. A high value relative to workqueue_adds_total indicates churn.
workqueue_queue_duration_seconds	histogram	Time an item waits in the queue before pickup.
workqueue_work_duration_seconds	histogram	Time to process one item.
workqueue_unfinished_work_seconds	gauge	In-progress work. A growing value indicates a stuck reconcile.
workqueue_longest_running_processor_seconds	gauge	Duration of the longest single in-flight reconcile.

CFK-specific

These metrics come from CFK’s own instrumentation and supplement the controller-runtime and work queue metrics.

Metric	Type	Meaning
io_confluent_cfk_confluent_operator_reconcile_errors_total	counter	The operator’s own per-controller error count, labeled by cfk_controller_name.
io_confluent_cfk_confluent_operator_controller_reconcile_latency_time_seconds	histogram	The operator’s own reconcile latency.
io_confluent_cfk_confluent_operator_secret_ref_errors_total	counter	Failures resolving a secretRef in a CR. Appears only after it first fires.
io_confluent_cfk_confluent_operator_confluent_operator_configure_requeue_count_total	counter	Requeues in the Kafka configure workflow, caused by an internal-topic produce issue, a stuck file watcher, or a secret that has not synced yet. Appears only after it first fires.