# Monitor the Confluent for Kubernetes operator Confluent for Kubernetes (CFK) exposes its own Prometheus metrics—signals about how well the operator itself is managing your Confluent Platform resources, such as reconciling custom resources (CRs), keeping up with work, and staying healthy. These metrics describe the operator process, not the Kafka/Confluent Platform data it manages. #### NOTE This is a different endpoint from the JMX Prometheus exporter described in [JMX Metrics](co-monitor-cp.md#co-monitor-jmx). That exporter runs on each Confluent Platform component pod and exposes metrics about Confluent Platform itself. The endpoint on this page runs on the CFK operator pod and exposes metrics about the operator process. Both use port 7778, but on different pods. ## The endpoint * **Where:** `http://:7778/metrics`. The Helm chart exposes this port through a `ClusterIP` Service named `confluent-operator`. * **Format:** plain-text Prometheus/OpenMetrics exposition format, a standard format that most monitoring tools can read. Nothing Kubernetes-specific is required to consume it, and collecting it with a scraper is optional. * **Access:** cluster-internal only. The endpoint is unauthenticated and has no TLS, so only the Kubernetes network can reach it. Do not expose it publicly. ## Metric types Each metric is one of the following types, which tells you how to read it: * `counter`: A cumulative value that only increases, such as total reconciles. Read it as a rate. * `gauge`: A current value that can go up or down, such as queue depth. Read it as-is. * `histogram`: A distribution of measurements, for calculating percentiles. On this page, histograms measure durations, so their names end in `_seconds`. ## Global (operator-wide) metrics These metrics are emitted once for the operator process. | Metric | Type | Meaning | |----------------------------------------------------------------------------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `io_confluent_cfk_confluent_operator_replica_count` | gauge | Number of operator replicas running. | | `io_confluent_cfk_confluent_operator_liveness_probe_count_total` | counter | Liveness probe (`/healthz`) hits. | | `io_confluent_cfk_confluent_operator_readiness_probe_count_total` | counter | Readiness probe (`/readyz`) hits. | | `rest_client_requests_total` | counter | The operator’s calls to the Kubernetes API, labeled by `code`,
`method`, and `host`. A rise in `4xx`/`5xx` responses
indicates RBAC or API-server trouble. | | `certwatcher_read_certificate_total` /
`certwatcher_read_certificate_errors_total` | counter | Webhook TLS certificate reloads and reload failures. | | `controller_runtime_webhook_panics_total` | counter | Admission webhook panics. Should be `0`. | | `go_*`, `process_*` | gauge/counter | Go runtime and process metrics: memory, goroutines, garbage
collection, CPU, and file descriptors. Use these for leak or
resource-pressure detection. | ## Controller (per-resource) metrics A controller is the reconcile loop that manages one CR type. The operator runs one controller per CR type, and every controller emits the same set of metrics, distinguished by a `controller` (or `cfk_controller_name`) label. This covers both Day-1 and Day-2 resources. Every CR type has a controller and emits these metrics, even if no CR of that type currently exists in the cluster (the series reads `0`): * **Day-1 (workloads):** `kafka`, `zookeeper`, `kraftcontroller`, `connect`, `schemaregistry`, `ksqldb`, `controlcenter`, `kafkarestproxy`, `gateway`, `flinkapplication`, `flinkenvironment`, `usmagent` * **Day-2 (REST-driven):** `kafkatopic`, `schema`, `connector`, `confluentrolebinding`, `clusterlink`, `schemaexporter`, `schemaimporter` * **Configuration/migration:** `kafkarestclass`, `cmfrestclass`, `kraftmigrationjob` For example, a `KafkaTopic` (Day-2) controller emits the same `reconcile_total`, `workqueue_depth`, and latency metrics as a `Kafka` (Day-1) controller. Only the label value differs. ### Reconcile | Metric | Type | Meaning | |------------------------------------------------------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------| | `controller_runtime_reconcile_total` | counter | Reconcile outcomes, by `result`: `success`, `requeue`,
`requeue_after`, or `error`. The primary health signal for a
controller. | | `controller_runtime_reconcile_errors_total` | counter | Reconciles that returned an error. | | `controller_runtime_terminal_reconcile_errors_total` | counter | Errors the operator gave up retrying. These need human action. | | `controller_runtime_reconcile_panics_total` | counter | Panics during reconcile. Should be `0`. | | `controller_runtime_reconcile_time_seconds` | histogram | Reconcile duration. | | `controller_runtime_active_workers` | gauge | Workers currently reconciling. | | `controller_runtime_max_concurrent_reconciles` | gauge | The worker cap for this controller. CFK runs one worker per
controller. | ### Work queue These metrics describe the items waiting to be reconciled by a controller. | Metric | Type | Meaning | |-----------------------------------------------|-----------|----------------------------------------------------------------------------------------------------------------------| | `workqueue_depth` | gauge | Items waiting to be reconciled. `0` is healthy. A sustained
non-zero value means the operator is falling behind. | | `workqueue_adds_total` | counter | Items enqueued. | | `workqueue_retries_total` | counter | Re-enqueues. A high value relative to `workqueue_adds_total`
indicates churn. | | `workqueue_queue_duration_seconds` | histogram | Time an item waits in the queue before pickup. | | `workqueue_work_duration_seconds` | histogram | Time to process one item. | | `workqueue_unfinished_work_seconds` | gauge | In-progress work. A growing value indicates a stuck reconcile. | | `workqueue_longest_running_processor_seconds` | gauge | Duration of the longest single in-flight reconcile. | ### CFK-specific These metrics come from CFK’s own instrumentation and supplement the controller-runtime and work queue metrics. | Metric | Type | Meaning | |----------------------------------------------------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `io_confluent_cfk_confluent_operator_reconcile_errors_total` | counter | The operator’s own per-controller error count, labeled by
`cfk_controller_name`. | | `io_confluent_cfk_confluent_operator_controller_reconcile_latency_time_seconds` | histogram | The operator’s own reconcile latency. | | `io_confluent_cfk_confluent_operator_secret_ref_errors_total` | counter | Failures resolving a `secretRef` in a CR. Appears only after it
first fires. | | `io_confluent_cfk_confluent_operator_confluent_operator_configure_requeue_count_total` | counter | Requeues in the Kafka configure workflow, caused by an internal-topic
produce issue, a stuck file watcher, or a secret that has not
synced yet. Appears only after it first fires. |