Monitor Confluent Platform with Confluent for Kubernetes

Monitor your Confluent for Kubernetes (CFK) environment using the following tools and resources:

Confluent Control Center - Recommended option
Confluent Control Center (Legacy) - Available with Confluent Platform versions 7.9 or earlier
Confluent Health+ with Telemetry
JMX metrics monitoring integrations

The preceding tools monitor Confluent Platform components as deployed by CFK. To monitor the CFK operator process itself, see Monitor the Confluent for Kubernetes operator.

Confluent Control Center

Control Center is a web-based tool for managing and monitoring Confluent Platform. In Confluent for Kubernetes (CFK) version 2.11.1 and later, you can configure and deploy Confluent Control Center to monitor Confluent Platform. Control Center provides a user interface that enables developers and operators to:

Get a quick overview of cluster health
Observe and control messages, topics, and Schema Registry
Develop and run ksqlDB queries

For details about Control Center, see Confluent Control Center Overview.

Note

You cannot install Control Center 2.1 or earlier versions on a host on an Internet Protocol version 6 (IPv6) network.

To deploy and use Control Center in CFK, you need to configure the following components:

Control Center

Control Center ships with a Prometheus version that has the specific configurations to work with Control Center. By default, you cannot use any other Prometheus version besides that particular version, unless you configure Control Center to use your own external Prometheus instead of the bundled one. For more information, see Bring your own Prometheus (BYOP).

Control Center, Prometheus, and Alertmanager must have the same image tags.

Note that the Prometheus bundled with Control Center can not be used to ingest data to Grafana or any other tool in your environment.
Kafka
KRaft

The preceding components connect to one another as follows:

Apache Kafka® and KRaft connect to Control Center’s Prometheus service.
Control Center connects to Control Center’s Prometheus service.
Control Center connects to Control Center’s Alertmanager.
Prometheus connects to Alertmanager.

See sample snippets of custom resources (CRs) for Control Center, Kafka, and KRaft.

See an end-to-end sample configurations in the CFK example repository in GitHub.

Configure Control Center

Control Center shares the same custom resource definition (CRD) as Confluent Control Center (Legacy).

For additional configuration details, see the following Confluent Platform configuration topics, such as Authentication, RBAC, TLS, Networking, and Storage.

For the Prometheus and Alertmanager services, Control Center supports basic and mutual TLS (mTLS) authentication authentication methods and TLS encryption method.

Some of the TLS-related features, such as secrets in vaults, auto-generated certificates, or Java KeyStore format certificates, are not currently supported for Control Center.

Note

You cannot enable JSON logging for the bundled Control Center Prometheus and Alertmanager services. The services.prometheus and services.alertmanager specifications do not expose a log-format setting, and these services do not use Log4j 2.

Server-side configuration for Prometheus and Alertmanager

Configure the server-side configuration for Prometheus and Alertmanager:

kind: ControlCenter
spec:
  services:
    alertmanager:            --- [1]
      authentication:
        basic:               --- [2]
        type:                --- [3]
      image:                 --- [4]
      tls:                   --- [5]
    prometheus:              --- [6]
      authentication:
        basic:               --- [7]
        type:                --- [8]
      externalAccess:        --- [9]
      image:                 --- [10]
      tls:                   --- [11]
      pvc:
        dataVolumeCapacity:  --- [12]
      containerTemplate:
        envVars:             --- [13]
        - name: PROMETHEUS_HOST
          value:             --- [14]

[1] The server-side settings for the Alertmanager service.
[2] The configuration for basic authentication with the Alertmanager service. See Basic authentication for the available settings and credential configurations.
[3] The authentication type for the Alertmanager service. Valid options are basic and mtls.
[4] The image and the image tag of the Alertmanager service, confluentinc/cp-enterprise-alertmanager:<tag>.

Use the same image tag specified in spec.image.application, which is the Control Center release number you are installing.
[5] TLS settings for the Alertmanager service. For the available settings, see Configure Network Encryption.

Some of the TLS-related features, such as secrets in vaults, auto-generated certificates, or Java KeyStore format certificates, are not supported.
[6] The server-side settings for the Prometheus service.
[7] The configuration for basic authentication with the Prometheus service. See Basic authentication for the available settings and credential configurations.
[8] The authentication type for the Prometheus service. Valid options are basic and mtls.
[9] The external access method to use for Prometheus. See Configure Networking for the available options.
[10] The image and the image tag of the Prometheus service, confluentinc/cp-enterprise-prometheus:<tag>.

Use the same image tag specified in spec.image.application, which is the Control Center release number you are installing.
[11] TLS settings for the Prometheus service. For the available settings, see Configure Network Encryption.

Some of the TLS-related features, such as secrets in vaults, auto-generated certificates, or Java KeyStore formats certificates, are not supported.
[12] Required. The Prometheus data size.
[13] The environment variables for the Prometheus service.
[14] The host for the Prometheus service.

For Prometheus deployments in an IPv6 Kubernetes environment, if the host is a literal IPv6 address, it must be enclosed in square brackets, as in "[2001:db8::1]" or "[fe80::1%zone]".

For example:
```
containerTemplate:
  envVars:
  - name: PROMETHEUS_HOST
    value: "[::]"
  - name: PROMETHEUS_PORT
    value: "9090"
```

Client-side configuration for Prometheus and Alertmanager

Configure the client-side configuration to access Prometheus and Alertmanager.

Note

Prometheus and AlertManager clients do not inherit spec.tls configuration. You must explicitly set tls.enabled and tls.secretRef under the dependencies.prometheusClient and dependencies.alertManagerClient sections when TLS is required.

kind: ControlCenter
spec:
  dependencies:
    alertManagerClient:      --- [1]
      authentication:        --- [2]
      tls:                   --- [3]
        enabled:             --- [3a]
        secretRef:           --- [3b]
      url:                   --- [4]
    prometheusClient:        --- [5]
      authentication:        --- [6]
      tls:                   --- [7]
        enabled:             --- [7a]
        secretRef:           --- [7b]
      url:                   --- [8]

[1] The client-side settings to connect to the Alertmanager. The settings must match the server-side settings you configured in services.alertmanager.
[2] The client-side authentication settings for the Alertmanager. See Confluent Platform authentication for available settings and credential configurations.
[3] Required when using TLS. The client-side TLS configuration for the Alertmanager. This configuration does not inherit from spec.tls.
[3a] Required when using TLS. Set to true to enable TLS for the Alertmanager client connection.
[3b] Required when using TLS. The name of the Kubernetes Secret containing the TLS certificate for the Alertmanager client.
[4] Required. The URL endpoint of the Alertmanager service in the <control-center-instance-name>.<control-center-cluster-name>.<namespace>.svc.cluster.local:9093 pattern.
[5] The client-side settings to connect to the Prometheus service. The settings must match the server-side settings you configured in services.prometheus ([6]).
[6] The client-side authentication settings for the Prometheus service. See Confluent Platform authentication for available settings and credential configurations.
[7] Required when using TLS. The client-side TLS configuration for the Prometheus service. This configuration does not inherit from spec.tls.
[7a] Required when using TLS. Set to true to enable TLS for the Prometheus client connection.
[7b] Required when using TLS. The name of the Kubernetes Secret containing the TLS certificate for the Prometheus client.
[8] Required. The URL endpoint of the Prometheus service in the <control-center-instance-name>.<control-center-cluster-name>.<namespace>.svc.cluster.local:9090 pattern.

Reduced infrastructure mode

Control Center supports a reduced infrastructure mode, also called management mode, in which the bundled Prometheus monitoring stack does not run. Use this mode to run Control Center only for management tasks, such as managing topics, schemas, and connectors, without the metrics and alerting features.

To enable this mode, set the following Control Center server configurations in the ControlCenter CR using configOverrides.server:

kind: ControlCenter
spec:
  configOverrides:
    server:
      - confluent.controlcenter.mode.enable=management
      - confluent.controlcenter.prometheus.enable=false

Important

Although the Prometheus and Alertmanager services do not run in this mode, their configuration blocks are still required in the ControlCenter CR. Continue to define them under spec.services for the server side and spec.dependencies for the client-side.

Bring your own Prometheus (BYOP)

Instead of the Prometheus and Alertmanager services bundled with Control Center, you can configure Control Center to use your own externally managed Prometheus instance.

Supported versions and compatibility

BYOP requires Control Center 2.7 and CFK 3.4, or later. It also requires Prometheus 3.0 or later. Earlier Control Center versions do not support BYOP, and setting the BYOP properties with them does not work. For Control Center 2.7 information, see the Control Center documentation.

BYOP supports only the following:

Metrics that Kafka brokers and KRaft controllers push to Prometheus over the OpenTelemetry Protocol (OTLP). Pull-based (scrape) ingestion of these metrics is not supported.
Native, vanilla Prometheus. Customized or vendor-specific Prometheus distributions are not tested and are not supported.
A single Prometheus URL. For high availability, see Configure high availability.

Note

Alerting is not supported when using BYOP. Only Prometheus metrics ingestion is supported. The bundled Alertmanager, and the alerting features it powers, are unavailable in this mode.

External Prometheus prerequisites

Before you configure Control Center to use an external Prometheus instance, prepare that Prometheus instance as follows:

Enable the Prometheus OTLP receiver by starting Prometheus with the --web.enable-otlp-receiver flag. Kafka brokers and KRaft controllers push their metrics to the external Prometheus instance over OTLP.

Configure resource-attribute promotion so the labels Control Center relies on are available on the ingested metrics. Prometheus 3.0 or later is required because Control Center depends on OTLP resource-attribute promotion. Add the following to your Prometheus configuration:

otlp:
  promote_resource_attributes:
    - host.hostname
    - java.version
    - kafka.broker.id
    - kafka.cluster.id
    - kafka.version
    - type
    - kafka.process.roles
    - kafka.process.roles.broker
    - kafka.process.roles.controller
    - kafka.kafka.node.id
    - kafka.broker.rack

Configure an out-of-order time window, to accommodate metrics that arrive out of order. Add the following to your Prometheus configuration:
```
storage:
  tsdb:
    out_of_order_time_window: 10m
```
Load the Control Center Prometheus recording rules into your external Prometheus instance. The rules are in the recording_rules-generated.yml file. Copy the file from /etc/confluent-control-center/ in the Control Center container, or copy its contents from Recording Rules for an External Prometheus.

If the recording rules are missing or the Prometheus version is earlier than 3.0, Control Center shows an error in its UI. Apply the recording rules file as provided, without modification. The number of rules can change in later Control Center versions, so reload the file when you upgrade Control Center.

The promoted labels apply to the whole Prometheus instance. If you share the instance with other tools, the labels also affect their metrics.

Connectivity and security

Control Center, and the Kafka brokers and KRaft controllers that report metrics to Prometheus, must be able to reach your external Prometheus endpoint over the network. The endpoint must be reachable from the Control Center pod and from every broker and controller.

Three kinds of components connect to the external Prometheus. Control Center reads metrics with PromQL. Every Kafka broker and every KRaft controller pushes metrics over OTLP. Each one is a separate TLS client.

You can secure these connections in one of the following ways. Use at most one authentication mode. Do not combine Basic authentication with mTLS.

TLS only. Prometheus presents a server certificate, and every client verifies it with the certificate authority (CA) that signed it. Without Basic authentication, any client that can reach Prometheus can read and push metrics.
Basic authentication over TLS. In addition to TLS, every client sends a user name and password.
mTLS. In addition to TLS, every client presents its own certificate, which Prometheus verifies.

In every mode, Control Center, every broker, and every controller must trust the CA that signed the Prometheus server certificate. With mTLS, Prometheus must also trust the CA of every client certificate it receives.

The Prometheus server certificate must have a Subject Alternative Name (SAN) that matches the host name in the Prometheus URL. For example, for https://prometheus.example.com:9090, the SAN must include prometheus.example.com. If you use an IP address in the URL, the SAN must include that IP address. Otherwise, the TLS handshake fails.

With mTLS, if a client certificate has an extended key usage, it must include clientAuth. Prometheus rejects a certificate that allows only serverAuth, with the error x509: certificate specifies an incompatible key usage. A certificate with no extended key usage is accepted.

Configure TLS and authentication on Prometheus

Prometheus reads its TLS and authentication settings from the file that you pass with the --web.config.file flag.

For TLS, configure the server certificate:

tls_server_config:
  cert_file: /etc/prometheus/certs/server.crt
  key_file: /etc/prometheus/certs/server.key

For Basic authentication, also add a user with a bcrypt-hashed password:

basic_auth_users:
  c3: <bcrypt_hash_of_password>

For mTLS, set client_ca_file and require a verified client certificate:

tls_server_config:
  cert_file: /etc/prometheus/certs/server.crt
  key_file: /etc/prometheus/certs/server.key
  client_ca_file: /etc/prometheus/certs/client-ca-bundle.crt
  client_auth_type: RequireAndVerifyClientCert

The client_ca_file must contain every CA that signed a client certificate. If the Control Center client certificate and the node certificates are signed by different CAs, put both CA certificates in this file. If the file has only the Control Center CA, Prometheus rejects the broker and controller pushes, and dashboards are missing broker and controller data.

Configure Secrets for TLS and authentication

You configure the Control Center read connection on the ControlCenter CR, under spec.dependencies.prometheusClient. You configure the broker and controller push connection on each Kafka and KRaftController CR, under spec.dependencies.metricsClient.

Pass certificates to Kubernetes in Secrets, using one of the following formats:

PEM files named fullchain.pem, privkey.pem, and cacerts.pem.
A Kubernetes TLS Secret with the keys tls.crt, tls.key, and ca.crt.

For TLS, the Secret needs the CA that signed the Prometheus server certificate. For mTLS, the Secret also needs the client certificate and its private key.

Always set secretRef under tls. If tls.enabled is true and there is no secretRef, CFK accepts the resource but cannot reconcile it. The ControlCenter status shows FAILED with the message missing .spec.tls. in spec, and Control Center does not start. This happens even when Control Center has its own spec.tls.

For Basic authentication, the Secret holds a file named basic.txt with the following lines:

username=<user>
password=<password>

Configure Control Center for BYOP

To use BYOP, disable both the bundled Prometheus and the bundled Alertmanager, and point Control Center at your external Prometheus endpoint:

kind: ControlCenter
spec:
  services:
    prometheus:
      enabled: false          --- [1]
    alertmanager:
      enabled: false          --- [2]
  dependencies:
    prometheusClient:
      url:                    --- [3]
      authentication:         --- [4]
        type:
        basic:
      tls:                    --- [5]
        enabled:
        secretRef:
      tlsClientAliasName:     --- [6]

[1] Required. Set to false to skip deploying the bundled Prometheus. Keep the services.prometheus block in the CR even when you disable it. The image and pvc settings are optional when it is disabled.
[2] Required. Set to false together with [1]. Control Center does not allow the bundled Prometheus and the bundled Alertmanager to be enabled independently of each other.
[3] Required. The URL endpoint of your external Prometheus instance. spec.dependencies.prometheusClient.url is required in every mode, not just BYOP.
[4] The authentication settings for connecting to the external Prometheus instance. Valid types are basic and mtls. For the available Basic authentication settings and credential configurations, see Basic authentication.
[5] Required when using TLS or mTLS. The client-side TLS configuration for the external Prometheus connection. For the available settings, see Configure Network Encryption.
[6] Optional. Used only when authentication.type is mtls. The alias of the client certificate in the keystore from the Secret referenced by [5]. When the Secret contains PEM files, set this property to "1".

If Control Center uses TLS for its own endpoints (spec.tls), add the CA that signed the Control Center server certificate to the Prometheus client_ca_file. The Control Center server certificate must also be valid for client authentication. If it has an extended key usage, that must include clientAuth.

Configure brokers and KRaft controllers for BYOP

Brokers and controllers push metrics to the external Prometheus. Configure this on the Kafka and KRaftController CRs, under spec.dependencies.metricsClient. Set url to the base URL of your external Prometheus. CFK adds the /api/v1/otlp path. The authentication settings must match the external Prometheus, not the ControlCenter CR.

For TLS:

spec:
  dependencies:
    metricsClient:
      url: https://prometheus.example.com:9090
      tls:
        enabled: true
        secretRef: prometheus-client-tls

For Basic authentication over TLS, add an authentication block to the prometheusClient section of the ControlCenter CR and to every metricsClient section:

authentication:
  type: basic
  basic:
    secretRef: prometheus-client-creds

For mTLS, reference a Secret with the node’s client certificate, its private key, and the Prometheus CA:

spec:
  dependencies:
    metricsClient:
      url: https://prometheus.example.com:9090
      tls:
        enabled: true
        secretRef: <node_client_tls_secret>
      authentication:
        type: mtls

Prometheus must trust the CA that signed the node certificates. If the nodes use the cluster certificate, add the cluster CA to the Prometheus client_ca_file. You can update the Kafka and KRaftController CRs in place.

Sample ControlCenter CRs

Replace <c3_image_tag> with a Control Center image tag of 2.7.0 or later, and <init_container_tag> with your init container image tag.

The following ControlCenter CR uses Basic authentication over TLS:

apiVersion: platform.confluent.io/v1beta1
kind: ControlCenter
metadata:
  name: controlcenter
  namespace: confluent
spec:
  replicas: 1
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:<c3_image_tag>
    init: confluentinc/confluent-init-container:<init_container_tag>
  dataVolumeCapacity: 10Gi
  services:
    prometheus:
      enabled: false
    alertmanager:
      enabled: false
  dependencies:
    kafka:
      bootstrapEndpoint: kafka.confluent.svc.cluster.local:9071
    prometheusClient:
      url: https://prometheus.example.com:9090
      tls:
        enabled: true
        secretRef: prometheus-client-tls
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds

For mTLS, use the same CR with the following prometheusClient section:

prometheusClient:
  url: https://prometheus.example.com:9090
  tls:
    enabled: true
    secretRef: c3-prometheus-client-tls
  authentication:
    type: mtls
  tlsClientAliasName: "1"

Validation errors

CFK validates the BYOP configuration and rejects the ControlCenter resource in the following cases:

spec.dependencies.prometheusClient.url is not set: spec.dependencies.prometheusClient.url is required.
The bundled Prometheus is disabled but the bundled Alertmanager is left enabled: the bundled Alertmanager cannot run against an external Prometheus: set services.alertmanager.enabled: false.
The bundled Prometheus is enabled but the bundled Alertmanager is disabled: the bundled Alertmanager must be enabled together with the bundled Prometheus: set services.alertmanager.enabled: true.
The services.prometheus block is missing: services.prometheus must be configured.
A bundled component is enabled but its image is not set, for example the bundled Prometheus is enabled but services.prometheus.image is not set.
BYOP is enabled and the Control Center image is 2.6.0 or earlier: BYOP requires a C3 image newer than 2.6.0 (the latest released version) ... set services.prometheus.enabled: true. This check does not apply to digest-pinned images or custom tags, because CFK cannot determine their version.

Configure high availability

BYOP uses a single Prometheus URL. For high availability, run your own OpenTelemetry Collector in front of your Prometheus replicas. The Collector receives the OTLP pushes from the brokers and controllers and forwards them to the OTLP endpoint of each replica, which is https://<replica>:<port>/api/v1/otlp. Configure the Collector to accept pushes on the path /api/v1/otlp/v1/metrics.

Point metricsClient on the Kafka and KRaftController resources at the Collector. Point prometheusClient on the ControlCenter resource at one replica, because Control Center reads metrics from it.

Control Center reads from one replica. If that replica is unavailable, Control Center cannot show metrics until it can reach that replica again, or until you point it at another replica.

Migrate to BYOP

For new deployments, configure BYOP as described earlier from the start. No bundled Prometheus exists, so no migration is needed.

For existing deployments, move from the bundled Prometheus to BYOP as follows:

Prepare your external Prometheus instance as described earlier in this section, under External Prometheus prerequisites.
Update the Kafka and KRaftController CRs to add spec.dependencies.metricsClient, as shown earlier, and apply the changes. You can update these CRs in place.
To keep the metric history of the bundled Prometheus, set the reclaim policy of its persistent volume to Retain before you delete the ControlCenter CR:
```
PV=$(kubectl -n <namespace> get pvc prometheus-data-<controlcenter_name>-0 \
  -o jsonpath='{.spec.volumeName}')
kubectl patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
```
The persistent volume and its data remain after the claim is deleted. Control Center does not read this data after the migration. Delete the persistent volume and its underlying storage when you no longer need the data.
Delete the ControlCenter CR, and then create it again with services.prometheus.enabled: false, services.alertmanager.enabled: false, and dependencies.prometheusClient pointing at the external Prometheus.

Warning

Deleting the ControlCenter CR also deletes its persistent volume claims, including the bundled Prometheus claim and the Control Center data claim. If the reclaim policy of a volume is Delete, the default for dynamically provisioned volumes, the volume and its data are deleted too, including the bundled Prometheus metric history. Complete the previous step first to keep the history.

Do not update the existing ControlCenter CR in place. Disabling the bundled Prometheus removes its volume claim template, and Kubernetes rejects that change on the existing StatefulSet.

The metric history in the bundled Prometheus is not migrated to the external Prometheus.

If the external Prometheus endpoint is unreachable or misconfigured, Control Center cannot retrieve metrics and its own monitoring views are unavailable until connectivity is restored. This does not affect Control Center’s other management functions (managing topics, schemas, and connectors).

Roll back to the bundled Prometheus

To switch back to the bundled Prometheus:

Delete the ControlCenter CR, and then create it again with the bundled services enabled and their settings in place:
- services.prometheus.enabled: true
- services.alertmanager.enabled: true
- services.prometheus.image, services.prometheus.pvc, and services.alertmanager.image set
Set the URLs of the bundled services:
- dependencies.prometheusClient.url: http://<controlcenter_name>.<namespace>.svc.cluster.local:9090
- dependencies.alertManagerClient.url: http://<controlcenter_name>.<namespace>.svc.cluster.local:9093
Set the metricsClient URL on the Kafka and KRaftController resources back to the bundled Prometheus.

Configure Kafka for Control Center

Configure Kafka to send data to the Prometheus service in Control Center.

kind: Kafka
spec:
  dependencies:
    metricsClient:
      authentication:  --- [1]
      tls:             --- [2]
      url:             --- [3]

[1] The authentication settings for the Prometheus service. The settings must match the server-side authentication settings you configured in the ControlCenter CR (see Configure Control Center).
[2] The client-side TLS settings for the Prometheus service.
[3] Required. The URL endpoint of the Prometheus service.

Configure KRaft for Control Center

Configure KRaft to send data to the Prometheus service in Control Center.

kind: KRaftController
spec:
  dependencies:
    metricsClient:
      authentication:  --- [1]
      tls:             --- [2]
      url:             --- [3]

[1] The authentication settings for the Prometheus service. The settings must match the server-side authentication settings you configured in the ControlCenter CR (see Configure Control Center).
[2] The client-side TLS settings for the Prometheus service.
[3] Required. The URL endpoint of the Prometheus service.

Sample CR snippets of Control Center, Kafka, and KRaft

This section contains snippets of sample CRs to configure Control Center with supported authentication.

mTLS authentication to Prometheus

kind: KRaftController
spec:
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: ControlCenter
spec:
  image:
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      authentication:
        type: mtls
      tls:
        secretRef: prometheus-tls
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0
      authentication:
        type: mtls
      tls:
        secretRef: alertmanager-tls
---

Basic authentication and TLS to Prometheus

kind: KRaftController
spec:
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: ControlCenter
spec:
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:2.0.0
    init: confluentinc/confluent-init-container:2.11.0
    pullSecretRef:
      - confluent-registry
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      authentication:
        type: basic
        basic:
          secretRef: alertmanager-client-creds
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      authentication:
        type: basic
        basic:
          secretRef: prometheus-credentials
      tls:
        secretRef: prometheus-tls
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0
      authentication:
        type: basic
        basic:
         secretRef: alertmanager-credentials
      tls:
        secretRef: alertmanager-tls
---

TLS-enabled Control Center

The following is a sample Control Center CR with TLS enabled and deployed with Confluent Platform 7.9.0:

kind: KRaftController
metadata:
  name: kraftcontroller
  namespace: operator
spec:
  configOverrides:
    jvm:
      - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
  mountedVolumes:
    volumes:
      - name: customtruststore
        secret:
          secretName: mycustomtruststore
    volumeMounts:
      - name: customtruststore
        mountPath: /mnt/jvmtruststore
  dataVolumeCapacity: 10G
  image:
    application: confluentinc/cp-server:7.9.0
    init: confluentinc/confluent-init-container:2.11.0
  replicas: 3
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
---
kind: Kafka
metadata:
  name: kafka
  namespace: operator
spec:
  configOverrides:
    jvm:
      - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
  mountedVolumes:
    volumes:
      - name: customtruststore
        secret:
          secretName: mycustomtruststore
    volumeMounts:
      - name: customtruststore
        mountPath: /mnt/jvmtruststore
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
  dataVolumeCapacity: 10G
  image:
    application: confluentinc/cp-server:7.9.0
    init: confluentinc/confluent-init-container:2.11.0
  replicas: 3
---
kind: ControlCenter
metadata:
  name: controlcenter-next-gen
  namespace: operator
spec:
  replicas: 1
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:2.0.0
    init: confluentinc/confluent-init-container:2.11.0
    pullSecretRef:
      - confluent-registry
  dataVolumeCapacity: 10Gi
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0

Confluent Control Center (Legacy)

Important

Starting with Confluent Platform version 8.0, Confluent Control Center (Legacy) is no longer supported with Confluent Platform. Use Control Center with Confluent Platform 8.0 and later.

Confluent Control Center (Legacy) is a web-based tool for managing and monitoring Confluent Platform. Control Center (Legacy) provides a user interface that enables developers and operators to:

Get a quick overview of cluster health
Observe and control messages, topics, and Schema Registry
Develop and run ksqlDB queries

For the metrics available for monitoring, see Metrics available in Control Center.

Configure Control Center (Legacy) to monitor Kafka clusters

The Confluent Metrics Reporter collects various metrics from an Apache Kafka® cluster. Control Center (Legacy) then uses those metrics to provide a detailed monitoring view of the Kafka cluster.

By default, the Confluent Metrics Reporter is enabled and configured to send Kafka metrics to a set of topics on the same Kafka cluster.

To send metrics to a different cluster, or to configure specific authentication settings, configure the Kafka custom resource (CR):

kind: Kafka
spec:
  metricReporter:
    enabled:                       --- [1]
    authentication:
      type:                        --- [2]
      jaasConfigPassThrough:
        secretRef:                 --- [3]
        directoryPathInContainer:  --- [4]
    tls:
      enabled:                     --- [5]

[1] Set to true or false to enable or disable the metrics reporting.
[2] Set to the authentication type to use for Kafka. See Configure authentication to access Kafka and KRaft for details.
[3] Set to the Kubernetes Secret name used to authenticate to Kafka.
[4] Set to the directory path in the Kafka container where the Kafka authentication credentials are injected by Vault.

See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
[5] Set to true if the Kafka cluster has TLS network encryption enabled.

Once Confluent Metrics Reporter is set up for a Kafka cluster, configure Control Center (Legacy) to monitor the cluster.

By default, Control Center (Legacy) is set up to monitor the Kafka cluster it is using to store its own state. This Kafka cluster is defined using spec.dependencies.kafka in the Confluent Control Center (Legacy) CR.

If there is another Kafka cluster to monitor, you can configure that in the Control Center (Legacy) CR as below:

kind: ControlCenter
spec:
  monitoringKafkaClusters:
  - name:                         --- [1]
    bootstrapEndpoint:            --- [2]
    authentication:
      type:                       —-- [3]
      jaasConfig:                 —-- [4]
      jaasConfigPassThrough:      —-- [5]
      oauthbearer:                —-- [6]
        secretRef:                —-- [7]
        directoryPathInContainer: --- [8]
    tls:
      enabled:                    —-- [9]

[1] Set to Kafka cluster name.
[2] Set to the Kafka bootstrap endpoint.
[3] Set to the Kafka client authentication type.

When RBAC is not enabled, valid options are plain and mtls.

When RBAC is enabled, the only valid option is oauthbearer.
[4] [5] For authenticating to a Kafka cluster using SASL/PLAIN, see Client-side SASL/PLAIN authentication for Kafka and KRaft.
[6] When Confluent Control Center (Legacy) authorization type is set to RBAC (spec.authorization.type: rbac) and the authentication type is set to oauthbearer in [3], use the OAuth method to authenticate to the Kafka cluster.
[7] The username and password are loaded through secretRef. The expected key is bearer.txt, and the value for the key is:
```
username=<username>
password=<password>
```
An example command to create a secret to use for this property:
```
kubectl create secret generic oauth-client \
  --from-file=bearer.txt=/some/path/bearer.txt \
  --namespace <namespace>
```
[8] The directory in the Confluent Control Center (Legacy) container where the expected Bearer credentials are injected by vault. See above ([7]) for the expected format.

See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
[9] For authenticating to a Kafka cluster using mTLS, see Client-side mTLS authentication for Kafka and KRaft.

Configure Control Center (Legacy) to monitor remote Kafka clusters

To monitor a Kafka cluster in a different Kubernetes cluster:

Configure the Control Center (Legacy) CR as described in monitoring additional Kafka clusters.

When RBAC is enabled, you must set authentication.type to oauthbearer, and provide oauthbearer credentials in the Control Center (Legacy) CR.
Configure the replication listener with external access on the remote Kafka as described in Configure Kafka in MRC with external access URLs.

If RBAC is enabled on the remote Kafka cluster, the Kafka token listener piggy backs on the replication listener.
Configure DNS where Control Center (Legacy) runs to be able to resolve the Kafka replication listener endpoint.

Configure Control Center (Legacy) to monitor ksqlDB, Connect, and Schema Registry clusters

You can configure Control Center (Legacy) to provide a detailed monitoring or management view of ksqlDB, Connect, and Schema Registry clusters.

The following is an example of the dependencies section in a Control Center (Legacy) CR. The example connects two Schema Registry clusters, two ksqlDB clusters, and two Connect clusters to Control Center (Legacy):

kind: ControlCenter
spec:
 dependencies:
   schemaRegistry:
     url: https://schemaregistry.confluent.svc.cluster.local:8081
     tls:
       enabled: true
     authentication:
       type: mtls
     clusters:
     - name: schemaregistry-dev
       url: https://schemaregistry-dev.confluent.svc.cluster.local:8081
       tls:
         enabled: true
       authentication:
        type: mtls
   ksqldb:
   - name: ksql-dev
     url: https://ksqldb.confluent.svc.cluster.local:8088
     tls:
       enabled: true
     authentication:
       type: mtls
   - name: ksql-dev1
     url: https://ksqldb-dev.confluent.svc.cluster.local:8088
     tls:
       enabled: true
     authentication:
       type: mtls
   connect:
   - name: connect-dev
     url: https://connect.confluent.svc.cluster.local:8083
     tls:
       enabled: true
     authentication:
       type: mtls
   - name: connect-dev2
     url: https://connect-dev.confluent.svc.cluster.local:8083
     tls:
       enabled: true
     authentication:
       type: mtls

For an example scenario to configure Confluent Control Center (Legacy) to monitor multiple ksqlDB, Connect, and Schema Registry clusters, see Connect Control Center to Multiple Connect, ksqlDB, and Schema Registry Clusters.

Confluent Health+

Confluent Health+ provides ongoing, real-time analysis of performance and configuration data for your Confluent Platform deployment. From this analysis, Health+ sends out notifications to alert users to potential environmental issues before they become critical problems.

For more information, see Confluent Health+.

Telemetry Reporter

The Confluent Telemetry Reporter is a plugin that runs inside each Confluent Platform service to push metadata about the service to Confluent. Telemetry Reporter enables product features based on the metadata, like Health+. Data is sent over HTTP using an encrypted connection.

Telemetry is disabled by default in CFK. You can enable and configure it globally at the CFK level.

When you globally enable Telemetry, you have an option to disable for specific Confluent Platform components.

Each Confluent Platform component CR provides the status condition, in status.conditions, whether Telemetry is enabled or disabled.

For more information and supported settings for Telemetry Reporter, see Confluent Telemetry Reporter.

For a list of the metrics that are collected for Health+, see Telemetry Reporter Metrics.

Globally configure Telemetry

To globally enable Telemetry Reporter for CFK and all Confluent Platform components:

Set the followings in the CFK values file.
```
telemetry:
  operator:
     enabled: true  --- [1]
  enabled: true     --- [2]
```
- [1] Enable Telemetry Reporter for CFK.
- [2] Enable Telemetry Reporter for all Confluent Platform components.

Apply the change with the following command:

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  --namespace <namespace>

To globally configure the Telemetry Reporter settings for all Confluent Platform components:

Set the following in the CFK values file:
```
telemetry:
  secretRef:                 --- [1]
  directoryPathInContainer:  --- [2]
```
- [1] [2] CFK supports the secretRef and directoryPathInContainer methods to load Telemetry configuration through Helm. Specify only one method.
  
  The Telemetry configuration should be specified in telemetry.txt, and the file must contain api.key and api.secret.
  
  If using a proxy, additional properties are required as shown below.
```
api.key=<cloud_key>
api.secret=<cloud_secret>
proxy.url=<proxy_url>           # Only required if proxy is enabled
proxy.username=<proxy_username> # Only required if proxy requires credential
proxy.password=<proxy_password> # Only required if proxy requires credential
```
- [1] secretRef takes the precedence over directoryPathInContainer if both are configured.
  
  The expected key is telemetry.txt.
  
  If the referenced secretRef is not read or data is not in the expected format, CFK fails to start.
- [2] Provide the directory path where telemetry.txt is present.
  
  If telemetry.txt is not in the expected format, CFK fails to start.
See Provide secrets for Confluent Platform application CR for providing the keys/values and required annotations when using Vault.
To apply changes in Telemetry settings, in the referenced Secret, or in the telemetry.txt file, manually restart CFK and Confluent Platform:
- Restart CFK:
```
kubectl rollout restart deployment/confluent-operator
```
- Restart Confluent Platform components as described in Restart Confluent Platform Using Confluent for Kubernetes.

Disable Telemetry for a Confluent Platform component

To disable Telemetry for a specific Confluent Platform component, set the following in the component CR and apply the change with the kubectl apply command:

telemetry:
  global: false

JMX metrics

CFK can deploy Confluent components with Java Management Extensions (JMX) metrics.

Note

For enhanced security, JMX metrics are disabled by default starting with CFK 3.2.1. In earlier versions, JMX metrics were enabled by default.

When enabled, these JMX metrics are made available on all pods at the following endpoints:

JMX metrics are available on port 7203 of each pod.
Jolokia (a REST interface for JMX metrics) is available on port 7777 of each pod.
JMX Prometheus exporter is available on port 7778.

CFK does not support authentication or encryption for the JMX Prometheus exporter, and it does not provide an option to disable the JMX Prometheus exporter.

JMX remote access backward compatibility

The default behavior for JMX remote access has changed:

Previous behavior: The JMX port was accessible remotely without authentication. Users could read and write through the exposed JMX port.
Current behavior: If you do not configure JMX authentication in the CR specification, remote JMX access is disabled by default. The JMX port is only accessible locally within the pod.

Impact on existing deployments

If you currently use remote JMX access without authentication, you must configure JMX authentication to continue using remote JMX. Internal applications like Jolokia metrics and JMX Prometheus exporter continue to work because they use the in-process JMX API and are not affected by remote connector settings.

Note

JMX authentication files do not support hot-reloading because the JVM only loads JMX credentials at startup. Changes to password or access control files require a pod restart to take effect.
The JMX password and access control secretRef are not owned by the CR and are not watched. JMX credential updates require a manual pod restart.

Configure JMX authentication

You can configure password-based authentication and access control for JMX connections to secure JMX endpoints.

JMX authentication requires explicit configuration. You must provide either a secretRef or directoryPathInContainer to enable JMX authentication. The operator does not support auto-generated passwords.

Note

Use JMX authentication with TLS/SSL encryption in production environments. See Configure Network Encryption for TLS configuration.

Configure password-based authentication

To enable password-based authentication for JMX:

Create a password file named jmxremote.password with the following format:
```
<username> <password>
<username> <password>
```
For example:
```
admin secretpassword
readonly readonlypassword
```
Set the password file permissions to 0600 (read/write for owner only).

Note

When you use secretRef, the operator automatically sets file permissions by specifying items in the Kubernetes SecretVolumeSource. The Kubernetes Kubelet applies these permissions when mounting the secrets into pods, ensuring the JMX password file is readable only by the Kafka process (UID 1001) without requiring manual user intervention.

When you use directoryPathInContainer with Vault, you must set the correct file permissions before the Vault sidecar injects the password file into the container.

Create a Kubernetes Secret with the password file:

kubectl -n <namespace> create secret generic my-jmx-password \
  --from-file=jmxremote.password=<path-to-jmxremote.password>

Configure the component CR:

kind: Kafka
spec:
  metrics:
    jmx:
      authentication:
        secretRef: my-jmx-password

Alternatively, you can use directoryPathInContainer to provide the path to a JMX password file (jmxremote.password) injected by a sidecar (for example, Vault).

Configure access control

Use JMX access control to restrict MBean operations with read-only or read-write permissions.

To configure access control:

Create an access control file named jmxremote.access with the following format:
```
<username> <access-level>
```
Valid access levels are readonly and readwrite.

For example:
```
admin readwrite
readonly readonly
```
Set the access control file permissions to 0644 (read for all, write for owner).

Note

When you use secretRef, the operator automatically sets file permissions. When you use directoryPathInContainer with Vault, you must set the correct file permissions before the Vault sidecar injects the access control file into the container.

Create a Kubernetes Secret with the access control file:

kubectl -n <namespace> create secret generic my-jmx-access \
  --from-file=jmxremote.access=<path-to-jmxremote.access>

Configure the component CR:

kind: Kafka
spec:
  metrics:
    jmx:
      authentication:
        secretRef: my-jmx-password
      accessControl:
        enabled: true
        secretRef: my-jmx-access

If you set accessControl.enabled to true without providing secretRef or directoryPathInContainer, the operator creates a default configuration with read-only access. When using the default access control file, you must add at least one of the following roles in the password file: monitorRole or controlRole. The default access control file only defines these roles with read-only access levels.

If you set accessControl.enabled to false, the system does not apply access control, and all users can read and write.

accessControl has no effect unless authentication is also configured on the same jmx block. If authentication is not configured, CFK skips accessControl entirely.

Configure with Vault injection

As an alternative to Kubernetes Secrets, you can use directoryPathInContainer to provide JMX password and access control files injected by Vault.

To configure JMX authentication and access control with Vault:

Configure the component CR with Vault annotations:

apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
spec:
  metrics:
    jmx:
      authentication:
        directoryPathInContainer: /vault/secrets/jmx
      accessControl:
        enabled: true
        directoryPathInContainer: /vault/secrets/jmx
  podTemplate:
    annotations:
      # Vault Agent Injection
      vault.hashicorp.com/agent-inject: "true"
      vault.hashicorp.com/agent-inject-status: "update"
      vault.hashicorp.com/preserve-secret-case: "true"
      vault.hashicorp.com/role: "confluent-operator"
      vault.hashicorp.com/agent-run-as-user: "1001"
      vault.hashicorp.com/agent-run-as-group: "1001"

      # Inject JMX password file
      vault.hashicorp.com/agent-inject-secret-jmx-password: "secret/kafka/jmx/password"
      vault.hashicorp.com/secret-volume-path-jmx-password: "/vault/secrets/jmx"
      vault.hashicorp.com/agent-inject-file-jmx-password: "jmxremote.password"
      vault.hashicorp.com/agent-inject-template-jmx-password: |
        {{- with secret "secret/kafka/jmx/password" -}}
        {{ .Data.data.content }}
        {{- end }}
      vault.hashicorp.com/agent-inject-perms-jmx-password: "0600"

      # Inject JMX access control file
      vault.hashicorp.com/agent-inject-secret-jmx-access: "secret/kafka/jmx/access"
      vault.hashicorp.com/secret-volume-path-jmx-access: "/vault/secrets/jmx"
      vault.hashicorp.com/agent-inject-file-jmx-access: "jmxremote.access"
      vault.hashicorp.com/agent-inject-template-jmx-access: |
        {{- with secret "secret/kafka/jmx/access" -}}
        {{ .Data.data.content }}
        {{- end }}
      vault.hashicorp.com/agent-inject-perms-jmx-access: "0644"

Important

When using directoryPathInContainer, you must set the correct file permissions before the Vault sidecar injects the files into the container:

Password file (jmxremote.password): 0600 (read/write for owner only)
Access control file (jmxremote.access): 0644 (read for all, write for owner)

As shown in the preceding example, use the vault.hashicorp.com/agent-inject-perms-<file> annotation to set these permissions.

When using DPIC (Dynamic Pod Identity Credentials), you must also include the following annotations:

vault.hashicorp.com/agent-run-as-user: "1001"
vault.hashicorp.com/agent-run-as-group: "1001"

The Vault sidecar injects the JMX password file (jmxremote.password) and access control file (jmxremote.access) into the specified directory path with the required file permissions.

Configure Jolokia access control

Jolokia provides JMX-over-HTTP access on port 7777 of each pod. You can apply a Jolokia access control policy to restrict the HTTP methods, commands, and MBeans that clients can access through this interface.

Jolokia access control is separate from the JMX access control covered in Configure JMX authentication. JMX access control governs JMX RMI connections, while Jolokia access control governs the Jolokia HTTP interface.

To configure Jolokia access control, use the spec.metrics.jolokia.accessControl settings on the component CR:

spec:
  metrics:
    jolokia:
      accessControl:
        enabled:                    --- [1]
        secretRef:                  --- [2]
        directoryPathInContainer:   --- [3]

[1] Set to true to enable Jolokia access control. When set to false or omitted, CFK does not apply access control, and all clients have full access. This is the default behavior.
[2] The name of a Kubernetes Secret that contains your custom access control policy. The Secret must contain a key named jolokia-access-control.xml. Do not configure secretRef together with directoryPathInContainer.
[3] The path to a directory in the container where the access control policy file is mounted, typically through Vault. Do not configure directoryPathInContainer together with secretRef.

If you set enabled to true without providing secretRef or directoryPathInContainer, CFK applies a default access control policy.

Enable the default access control policy

To apply the default Jolokia access control policy, set enabled to true without specifying a policy file:

kind: Kafka
spec:
  metrics:
    jolokia:
      accessControl:
        enabled: true

Customize the access control policy

To restrict access to specific HTTP methods, commands, or MBeans, provide a custom Jolokia access control policy in an XML file named jolokia-access-control.xml. The file uses the standard Jolokia policy format.

The following example allows only read and list commands over HTTP GET, and limits the accessible MBeans:

<?xml version="1.0" encoding="UTF-8"?>
<restrict>
  <http>
    <method>get</method>
  </http>
  <commands>
    <command>read</command>
    <command>list</command>
  </commands>
  <allow>
    <mbean>
      <name>kafka.server:*</name>
    </mbean>
    <mbean>
      <name>java.lang:type=Memory</name>
      <attribute>HeapMemoryUsage</attribute>
    </mbean>
  </allow>
</restrict>

For the full policy syntax, see the Jolokia security documentation.

Provide a custom policy with a Secret

Create a Kubernetes Secret from your access control file. The Secret key must be jolokia-access-control.xml:

kubectl -n <namespace> create secret generic my-jolokia-access-control \
  --from-file=jolokia-access-control.xml=<path-to-access-control-file>

Reference the Secret in the component CR:

kind: Kafka
spec:
  metrics:
    jolokia:
      accessControl:
        enabled: true
        secretRef: my-jolokia-access-control

Provide a custom policy with Vault injection

As an alternative to a Kubernetes Secret, you can use directoryPathInContainer to provide the access control policy file injected by Vault. Set directoryPathInContainer to the directory where the Vault sidecar mounts the jolokia-access-control.xml file:

apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
spec:
  metrics:
    jolokia:
      accessControl:
        enabled: true
        directoryPathInContainer: /vault/secrets/jolokia
  podTemplate:
    annotations:
      # Vault Agent Injection
      vault.hashicorp.com/agent-inject: "true"
      vault.hashicorp.com/role: "confluent-operator"
      vault.hashicorp.com/agent-run-as-user: "1001"
      vault.hashicorp.com/agent-run-as-group: "1001"

      # Inject Jolokia access control file
      vault.hashicorp.com/agent-inject-secret-jolokia-access-control.xml: "secret/kafka/jolokia/access"
      vault.hashicorp.com/secret-volume-path-jolokia-access-control.xml: "/vault/secrets/jolokia"
      vault.hashicorp.com/agent-inject-file-jolokia-access-control.xml: "jolokia-access-control.xml"
      vault.hashicorp.com/agent-inject-template-jolokia-access-control.xml: |
        {{- with secret "secret/kafka/jolokia/access" -}}
        {{ .Data.data.content }}
        {{- end }}

Configure security on JMX metrics endpoints

By default, authentication and encryption are not enabled on JMX or Prometheus metric endpoints, but you have options to configure authentication and TLS for JMX or Prometheus metric endpoints in the component CR:

spec:
  metrics:
    authentication:
      type:                    --- [1]
    prometheus:                --- [2]
      rules:                   --- [3]
        - attrNameSnakeCase:
          cache:
          help:
          labels:
          name:
          pattern:
          type:
          value:
          valueFactor:
      blacklist:               --- [4]
      whitelist:               --- [5]
    tls:
      enabled:                 --- [6]

[1] Set to mtls for mTLS authentication.

If you set this to mtls, you must set tls.enabled: true ([6]).

Note that CFK does not currently support mTLS authentication for Prometheus even with this set to mtls.
[2] Specify Prometheus configurations to override the default settings.

For more information about the rules, blacklist, and whitelist properties, see Prometheus.

[3] A list of rules to apply.

For example:

spec:
  metrics:
    prometheus:
      rules:
        - pattern: "org.apache.kafka.metrics<type=(\w+), name=(\w+)><>Value: (\d+)"
          name: "kafka_$1_$2"
          value: "$3"
          valueFactor: "0.000001"
          labels:
            "$1": "$4"
            "$2": "$3"
          help: "Kafka metric $1 $2"
          cache: false
          type: "GAUGE"
          attrNameSnakeCase: false

[4] An array of patterns in string format to identify what not to query.

For example:

spec:
  metrics:
    prometheus:
      blacklist:
      - "org.apache.kafka.metrics:*"

[5] An array of patterns (in the string format) to identify what to query.

For example:

spec:
  metrics:
    prometheus:
      whitelist:
      - "org.apache.kafka.metrics:type=ColumnFamily,*"

[6] If set to true, metrics are configured with global or component TLS as described in Configure Network Encryption.

Note

When using directoryPathInContainer for metrics TLS configuration (for example, with Vault-injected secrets), JKS passwords are resolved at pod startup time. If you rotate or change the JKS password in your secret store, you must perform a manual rolling restart of the component pods for the new password to take effect.

To trigger a rolling restart, see Restart Confluent Platform Using Confluent for Kubernetes.

This setting is ignored for Prometheus. CFK currently does not support TLS for Prometheus.

Configure external access on JMX metrics

By default, CFK does not configure external access on JMX metric endpoints.

As an advanced configuration option, CFK supports enabling external access on JMX metrics endpoint for Kafka as an advanced configuration option. You need this setup only if the Jolokia HTTP-based solution is not enough.

Note that JMX metrics external access is currently only supported for Kafka, and not for other Confluent Platform components.

To enable external access for Kafka at the component CR level, use the annotation:

Set up a Kubernetes external access service that allows incoming requests to an external FQDN to be routed to the pod’s JMX/Prometheus metric endpoint port. See Networking Service.

Apply the annotation on the Kafka CR using a comma-separated list of the DNS (or IP) and the port you got from the previous step when you created a service.

"platform.confluent.io/jmx-rmi-server-urls": "pod1.external-address:<port1>,pod2.external-address:<port2>, ... ,podn.external-address:<portn>"

For example, using the external IP address of the node, 35.35.35.35 and nodePorts 30001 to 30003, the following CR configures a node port external access on the Kafka nodes:

kind: Kafka
metadata:
  name: kafka
  namespace: operator
  annotations:
    "platform.confluent.io/jmx-rmi-server-urls": "35.35.35.35:30001,35.35.35.35:30002,35.35.35.35:30003"
spec:
  replicas: 3

Configure Prometheus and Grafana

You can configure Prometheus to capture and aggregate JMX metrics from Confluent components. Then you configure Grafana to visualize those metrics in a dashboard.

Note

On a Confluent Private Cloud multi-tenant Physical Kafka Cluster (PKC), metrics carry three tags: tenant=<lkc-id>, which identifies the Logical Kafka Cluster (LKC), topic=<lkc-id>_<name>, the topic name with the LKC prefix, and tenant_topic=<name>, the name without the prefix. You can filter or group metrics per LKC in Prometheus and Control Center using these tags.

For more information, see Monitor and Manage Per-LKC Resource Quotas.

For example configuration scenarios, see an example of monitoring with Prometheus and Grafana in jmx-monitoring-stacks and an example of monitoring with Grafana metrics dashboard.