<a id="co-monitor-cp"></a>

# Monitor Confluent Platform with Confluent for Kubernetes

Monitor your Confluent for Kubernetes (CFK) environment using the following tools and
resources:

* [Confluent Control Center](#co-c3) - Recommended option
* [Confluent Control Center (Legacy)](#co-monitor-c3)- Available with the Confluent Platform versions 7.9 or earlier
* [Confluent Health+ with Telemetry](#co-monitor-health)
* [JMX metrics monitoring integrations](#co-monitor-jmx)

<a id="co-c3"></a>

## Confluent Control Center

In Confluent for Kubernetes (CFK) version 2.11.1 and later, you can configure and deploy
Confluent Control Center to monitor Confluent Platform. Control Center is a web-based tool for managing and
monitoring Confluent Platform. Control Center provides a user interface that enables developers and
operators to:

* Get a quick overview of cluster health
* Observe and control messages, topics, and Schema Registry
* Develop and run ksqlDB queries

For details about Control Center, see [Confluent Control Center Overview](http://docs.confluent.io/control-center/current/overview.html).

#### NOTE
You cannot install Control Center 2.1 or earlier versions on a host on an Interne Protocol version 6 (IPv6) network.

To deploy and use Control Center in CFK, you need to configure the following
components:

* [Control Center](#co-configure-c3)

  Control Center ships with a Prometheus version that has the specific configurations to
  work with Control Center. Currently, you cannot use any other Prometheus version
  besides that particular version.

  Control Center, Prometheus, and Alertmanager must have the same image tags.

  Note that the Prometheus bundled with Control Center can not be used to ingest data to
  Grafana or any other tool in your environment.
* [Kafka](#co-configure-kafka-for-c3)
* [KRaft](#co-configure-kraft-for-c3)

The above components connect to one another as follows:

* Kafka and KRaft connects to Control Center’s Prometheus service.
* Control Center connects to Control Center’s Prometheus service.
* Control Center connects to Control Center’s Alertmanager.
* Prometheus connects to Alertmanager.

See [sample snippets](#co-configure-c3-samples) of custom resources (CRs)
for Control Center, Kafka, and KRaft.

See an [end-to-end sample configurations](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/control-center-next-gen)
in the CFK example repository in GitHub.

<a id="co-configure-c3"></a>

### Configure Control Center

Control Center shares the same custom resource definition (CRD) as Confluent Control Center (Legacy).

For additional
configuration details, see the Confluent Control Center (Legacy) sections, such as in [Authentication](co-authenticate-cp.md#co-authenticate-cp),
[RBAC](co-rbac.md#co-rbac), [TLS](co-network-encryption.md#co-network-encryption), [Networking](co-networking-overview.md#co-networking-overview), and [Storage](co-storage.md#co-storage).

For the Prometheus and Alertmanager services, Control Center supports basic and mTLS
authentication methods and TLS encryption method.

Some of the TLS-related features, such as  secrets in vaults, auto-generated
certificates, or Java KeyStore format certificates, are not currently supported
for Control Center.

#### NOTE
You cannot enable JSON logging for the bundled Control Center Prometheus and
Alertmanager services. The `services.prometheus` and
`services.alertmanager` specifications do not expose a log-format setting,
and these
services do not use Log4j 2.

#### Server-side configuration for Prometheus and Alertmanager

Configure the server-side configuration for Prometheus and Alertmanager:

```yaml
kind: ControlCenter
spec:
  services:
    alertmanager:            --- [1]
      authentication:
        basic:               --- [2]
        type:                --- [3]
      image:                 --- [4]
      tls:                   --- [5]
    prometheus:              --- [6]
      authentication:
        basic:               --- [7]
        type:                --- [8]
      externalAccess:        --- [9]
      image:                 --- [10]
      tls:                   --- [11]
      pvc:
        dataVolumeCapacity:  --- [12]
      containerTemplate:
        envVars:             --- [13]
        - name: PROMETHEUS_HOST
          value:             --- [14]
```

* [1] The server-side settings for the Alertmanager service.
* [2] The configuration for basic authentication with the Alertmanager service.
  See [Basic authentication](co-authenticate-cp.md#co-authenticate-cp-basic) for the available settings and credential
  configurations.
* [3] The authentication type for the Alertmanager service. Valid options are
  `basic` and `mtls`.
* [4] The image and the image tag of the Alertmanager service,
  `confluentinc/cp-enterprise-alertmanager:<tag>`.

  Use the same image tag specified in `spec.image.application`, which is the
  Control Center release number you are installing.
* [5] TLS settings for the Alertmanager service. For the available settings,
  see [Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes](co-network-encryption.md#co-network-encryption).

  Some of the TLS-related features, such as  secrets in vaults, auto-generated
  certificates, or Java KeyStore format certificates, are not supported.
* [6] The server-side settings for the Prometheus service.
* [7] The configuration for basic authentication with the Prometheus service.
  See [Basic authentication](co-authenticate-cp.md#co-authenticate-cp-basic) for the available settings and credential
  configurations.
* [8] The authentication type for the Prometheus service.
  Valid options are `basic` and `mtls`.
* [9] The external access method to use for Prometheus. See
  [Configure Networking for Confluent Platform in Confluent for Kubernetes](co-networking-overview.md#co-networking-overview) for the available options.
* [10] The image and the image tag of the Prometheus service,
  `confluentinc/cp-enterprise-prometheus:<tag>`.

  Use the same image tag specified in `spec.image.application`, which is the
  Control Center release number you are installing.
* [11] TLS settings for the Prometheus service. For the available settings, see
  [Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes](co-network-encryption.md#co-network-encryption).

  Some of the TLS-related features, such as  secrets in vaults, auto-generated
  certificates, or Java KeyStore formats certificates, are not supported.
* [12] Required. The Prometheus data size.
* [13] The environment variables for the Prometheus service.
* [14] The host for the Prometheus service.

  For Prometheus deployments in an IPv6 Kubernetes environment, if the host is a
  literal IPv6 address, it must be enclosed in square brackets, as in
  `"[2001:db8::1]"` or `"[fe80::1%zone]"`.

  For example:
  ```yaml
  containerTemplate:
    envVars:
    - name: PROMETHEUS_HOST
      value: "[::]"
    - name: PROMETHEUS_PORT
      value: "9090"
  ```

#### Client-side configuration for Prometheus and Alertmanager

Configure the client-side configuration to access Prometheus and Alertmanager.

#### NOTE
Prometheus and AlertManager clients do not inherit `spec.tls` configuration.
You must explicitly set `tls.enabled` and `tls.secretRef` under the
`dependencies.prometheusClient` and `dependencies.alertManagerClient`
sections when TLS is required.

```yaml
kind: ControlCenter
spec:
  dependencies:
    alertManagerClient:      --- [1]
      authentication:        --- [2]
      tls:                   --- [3]
        enabled:             --- [3a]
        secretRef:           --- [3b]
      url:                   --- [4]
    prometheusClient:        --- [5]
      authentication:        --- [6]
      tls:                   --- [7]
        enabled:             --- [7a]
        secretRef:           --- [7b]
      url:                   --- [8]
```

* [1] The client-side settings to connect to the Alertmanager. The settings
  must match the server-side settings you configured in
  `services.alertmanager`.
* [2] The client-side authentication settings for the Alertmanager. See
  [Confluent Platform authentication](co-authenticate-cp.md#co-authenticate-cp) for available
  settings and credential configurations.
* [3] Required when using TLS. The client-side TLS configuration for the
  Alertmanager. This configuration does not inherit from `spec.tls`.
* [3a] Required when using TLS. Set to `true` to enable TLS for the
  Alertmanager client connection.
* [3b] Required when using TLS. The name of the Kubernetes Secret containing
  the TLS certificate for the Alertmanager client.
* [4] Required. The URL endpoint of the Alertmanager service in the
  `<control-center-instance-name>.<control-center-cluster-name>.<namespace>.svc.cluster.local:9093`
  pattern.
* [5] The client-side settings to connect to the Prometheus service. The settings
  must match the server-side settings you configured in `services.prometheus`
  ([6]).
* [6] The client-side authentication settings for the Prometheus service. See
  [Confluent Platform authentication](co-authenticate-cp.md#co-authenticate-cp) for available
  settings and credential configurations.
* [7] Required when using TLS. The client-side TLS configuration for the
  Prometheus service. This configuration does not inherit from `spec.tls`.
* [7a] Required when using TLS. Set to `true` to enable TLS for the
  Prometheus client connection.
* [7b] Required when using TLS. The name of the Kubernetes Secret containing
  the TLS certificate for the Prometheus client.
* [8] Required. The URL endpoint of the Prometheus service in the
  `<control-center-instance-name>.<control-center-cluster-name>.<namespace>.svc.cluster.local:9090`
  pattern.

#### Reduced infrastructure mode

Control Center supports a reduced infrastructure mode, also called management mode,
in which the bundled Prometheus monitoring stack does not run. Use this
mode to run Control Center only for management tasks, such as managing topics,
schemas, and connectors, without the metrics and alerting features.

To enable this mode, set the following Control Center server configurations in the
`ControlCenter` CR using `configOverrides.server`:

```yaml
kind: ControlCenter
spec:
  configOverrides:
    server:
      - confluent.controlcenter.mode.enable=management
      - confluent.controlcenter.prometheus.enable=false
```

#### IMPORTANT
Although the Prometheus and Alertmanager services do not run in this
mode, their configuration blocks are still required in the
`ControlCenter` CR. Continue to define them under `spec.services`
for the server side and `spec.dependencies` for the client-side.

<a id="co-configure-kafka-for-c3"></a>

### Configure Kafka for Control Center

Configure Kafka to send data to the Prometheus service in Control Center.

```yaml
kind: Kafka
spec:
  dependencies:
    metricsClient:
      authentication:  --- [1]
      tls:             --- [2]
      url:             --- [3]
```

* [1] The authentication settings for the Prometheus service. The settings must
  match the server-side authentication settings you configured in the
  ControlCenter CR in the previous section.
* [2] The client-side TLS settings for the Prometheus service.
* [3] Required. The URL endpoint of the Prometheus service.

<a id="co-configure-kraft-for-c3"></a>

### Configure KRaft for Control Center

Configure KRaft to send data to the Prometheus service in Control Center.

```yaml
kind: KRaftController
spec:
  dependencies:
    metricsClient:
      authentication:  --- [1]
      tls:             --- [2]
      url:             --- [3]
```

* [1] The authentication settings for the Prometheus service. The settings must
  match the server-side authentication settings you configured in the
  ControlCenter CR in the previous section.
* [2] The client-side TLS settings for the Prometheus service.
* [3] Required. The URL endpoint of the Prometheus service.

<a id="co-configure-c3-samples"></a>

### Sample CR snippets of Control Center, Kafka, and KRaft

This section contains snippets of sample CRs to configure Control Center with supported
authentication.

#### mTLS authentication to Prometheus

```yaml
kind: KRaftController
spec:
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: ControlCenter
spec:
  image:
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      authentication:
        type: mtls
      tls:
        secretRef: prometheus-tls
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0
      authentication:
        type: mtls
      tls:
        secretRef: alertmanager-tls
---
```

#### Basic authentication and TLS to Prometheus

```yaml
kind: KRaftController
spec:
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: ControlCenter
spec:
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:2.0.0
    init: confluentinc/confluent-init-container:v0.1208.3
    pullSecretRef:
      - confluent-registry
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      authentication:
        type: basic
        basic:
          secretRef: alertmanager-client-creds
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      authentication:
        type: basic
        basic:
          secretRef: prometheus-credentials
      tls:
        secretRef: prometheus-tls
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0
      authentication:
        type: basic
        basic:
         secretRef: alertmanager-credentials
      tls:
        secretRef: alertmanager-tls
---
```

#### TLS-enabled Control Center

The following is a sample Control Center CR with TLS enabled and deployed with Confluent Platform 7.9.0:

```yaml
kind: KRaftController
metadata:
  name: kraftcontroller
  namespace: operator
spec:
  configOverrides:
    jvm:
      - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
  mountedVolumes:
    volumes:
      - name: customtruststore
        secret:
          secretName: mycustomtruststore
    volumeMounts:
      - name: customtruststore
        mountPath: /mnt/jvmtruststore
  dataVolumeCapacity: 10G
  image:
    application: confluentinc/cp-server:7.9.0
    init: confluentinc/confluent-init-container:2.11.0
  replicas: 3
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
---
kind: Kafka
metadata:
  name: kafka
  namespace: operator
spec:
  configOverrides:
    jvm:
      - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
  mountedVolumes:
    volumes:
      - name: customtruststore
        secret:
          secretName: mycustomtruststore
    volumeMounts:
      - name: customtruststore
        mountPath: /mnt/jvmtruststore
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
  dataVolumeCapacity: 10G
  image:
    application: confluentinc/cp-server:7.9.0
    init: confluentinc/confluent-init-container:2.11.0
  replicas: 3
---
kind: ControlCenter
metadata:
  name: controlcenter-next-gen
  namespace: operator
spec:
  replicas: 1
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:2.0.0
    init: confluentinc/confluent-init-container:2.11.0
    pullSecretRef:
      - confluent-registry
  dataVolumeCapacity: 10Gi
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    pvc:
      dataVolumeCapacity: 10Gi
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
```

<a id="co-monitor-c3"></a>

## Confluent Control Center (Legacy)

#### IMPORTANT
Starting with Confluent Platform version 8.0, Confluent Control Center (Legacy) is no longer supported with Confluent Platform.
Use Control Center with Confluent Platform 8.0 and later.

Confluent Control Center (Legacy) is a web-based tool for managing and monitoring Confluent Platform. Control Center (Legacy) provides a
user interface that enables developers and operators to:

* Get a quick overview of cluster health
* Observe and control messages, topics, and Schema Registry
* Develop and run ksqlDB queries

For the metrics available for monitoring, see [Metrics available in Control
Center](https://docs.confluent.io/control-center/current/brokers.html#broker-metrics).

### Configure Control Center (Legacy) to monitor Kafka clusters

The Confluent Metrics Reporter collects various metrics from an Apache Kafka® cluster.
Control Center (Legacy) then uses those metrics to provide a detailed monitoring view of the
Kafka cluster.

By default, the Confluent Metrics Reporter is enabled and configured to send
Kafka metrics to a set of topics on the same Kafka cluster.

To send metrics to a different cluster, or to configure specific authentication
settings, configure the Kafka custom resource (CR):

```yaml
kind: Kafka
spec:
  metricReporter:
    enabled:                       --- [1]
    authentication:
      type:                        --- [2]
      jaasConfigPassThrough:
        secretRef:                 --- [3]
        directoryPathInContainer:  --- [4]
    tls:
      enabled:                     --- [5]
```

* [1] Set to `true` or `false` to enable or disable the metrics reporting.
* [2] Set to the authentication type to use for Kafka. See
  [Configure authentication to access Kafka and KRaft](co-authenticate-kafka.md#co-authenticate-kafka) for details.
* [3] Set to the Kubernetes Secret name used to authenticate to Kafka.
* [4] Set to the directory path in the Kafka container where the Kafka
  authentication credentials are injected by Vault.

  See  [Provide secrets for Confluent Platform component CR](co-credentials.md#co-vault-category-1) for providing the credential and
  required annotations when using Vault.
* [5] Set to `true` if the Kafka cluster has TLS network encryption enabled.

Once Confluent Metrics Reporter is set up for a Kafka cluster, configure
Control Center (Legacy) to monitor the cluster.

By default, Control Center (Legacy) is set up to monitor the Kafka cluster it is using to
store its own state. This Kafka cluster is defined using
`spec.dependencies.kafka` in the Confluent Control Center (Legacy) CR.

<a id="co-monitor-remote-ak"></a>

If there is another Kafka cluster to monitor, you can configure that in the
Control Center (Legacy) CR as below:

```yaml
kind: ControlCenter
spec:
  monitoringKafkaClusters:
  - name:                         --- [1]
    bootstrapEndpoint:            --- [2]
    authentication:
      type:                       —-- [3]
      jaasConfig:                 —-- [4]
      jaasConfigPassThrough:      —-- [5]
      oauthbearer:                —-- [6]
        secretRef:                —-- [7]
        directoryPathInContainer: --- [8]
    tls:
      enabled:                    —-- [9]
```

* [1] Set to Kafka cluster name.
* [2] Set to the Kafka bootstrap endpoint.
* [3] Set to the Kafka client authentication type.

  When RBAC is not enabled, valid options are `plain` and `mtls`.

  When RBAC is enabled, the only valid option is `oauthbearer`.
* [4] [5] For authenticating to a Kafka cluster using SASL/PLAIN, see
  [Client-side SASL/PLAIN authentication for Kafka and KRaft](co-authenticate-kafka.md#co-authenticate-kafka-client-plain).
* [6] When Confluent Control Center (Legacy) authorization type is set to RBAC
  (`spec.authorization.type: rbac`) and the authentication type is set to
  `oauthbearer` in [3], use the OAuth method to authenticate to the
  Kafka cluster.
* [7] The username and password are loaded through secretRef. The expected key
  is `bearer.txt`, and the value for the key is:
  ```text
  username=<username>
  password=<password>
  ```

  An example command to create a secret to use for this property:
  ```bash
  kubectl create secret generic oauth-client \
    --from-file=bearer.txt=/some/path/bearer.txt \
    --namespace confluent
  ```
* [8] The directory in the Confluent Control Center (Legacy) container where the expected Bearer credentials
  are injected by vault. See above ([7]) for the expected format.

  See  [Provide secrets for Confluent Platform component CR](co-credentials.md#co-vault-category-1) for providing the credential and
  required annotations when using Vault.
* [9] For authenticating to a Kafka cluster using mTLS, see
  [Client-side mTLS authentication for Kafka and KRaft](co-authenticate-kafka.md#co-authenticate-kafka-client-mtls).

#### Configure Control Center (Legacy) to monitor remote Kafka clusters

To monitor a Kafka cluster in a different Kubernetes cluster:

1. Configure the Control Center (Legacy) CR as described in [monitoring additional Kafka
   clusters](#co-monitor-remote-ak).

   When RBAC is enabled, you must set `authentication.type` to
   `oauthbearer`, and provide `oauthbearer` credentials in the Control Center (Legacy)
   CR.
2. Configure the replication listener with external access on the remote Kafka as
   described in [Configure Kafka in MRC with external access URLs](co-multi-region.md#co-mrc-ak-external-access).

   If RBAC is enabled on the remote Kafka cluster, the Kafka token listener piggy
   backs on the replication listener.
3. Configure DNS where Control Center (Legacy) runs to be able to resolve the Kafka
   replication listener endpoint.

### Configure Control Center (Legacy) to monitor ksqlDB, Connect and Schema Registry clusters

You can configure Control Center (Legacy) to provide a detailed monitoring or management view
of ksqlDB, Connect, and Schema Registry clusters.

The following is an example of the `dependencies` section in a Control Center (Legacy) CR.
The example connects two Schema Registry clusters, two ksqlDB clusters, and two
Connect clusters to Control Center (Legacy):

```yaml
kind: ControlCenter
spec:
 dependencies:
   schemaRegistry:
     url: https://schemaregistry.confluent.svc.cluster.local:8081
     tls:
       enabled: true
     authentication:
       type: mtls
     clusters:
     - name: schemaregistry-dev
       url: https://schemaregistry-dev.confluent.svc.cluster.local:8081
       tls:
         enabled: true
       authentication:
        type: mtls
   ksqldb:
   - name: ksql-dev
     url: https://ksqldb.confluent.svc.cluster.local:8088
     tls:
       enabled: true
     authentication:
       type: mtls
   - name: ksql-dev1
     url: https://ksqldb-dev.confluent.svc.cluster.local:8088
     tls:
       enabled: true
     authentication:
       type: mtls
   connect:
   - name: connect-dev
     url: https://connect.confluent.svc.cluster.local:8083
     tls:
       enabled: true
     authentication:
       type: mtls
   - name: connect-dev2
     url: https://connect-dev.confluent.svc.cluster.local:8083
     tls:
       enabled: true
     authentication:
       type: mtls
```

For an example scenario to configure Confluent Control Center (Legacy) to monitor multiple ksqlDB,
Connect, and Schema Registry clusters, see [Connect Control Center to Multiple Connect,
ksqlDB, and Schema Registry Clusters](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/monitoring/controlcenter-multi-ksql-connect).

<a id="co-monitor-health"></a>

## Confluent Health+

Confluent Health+ provides ongoing, real-time analysis of performance and
configuration data for your Confluent Platform deployment. From this analysis, Health+ sends
out notifications to alert users to potential environmental issues before they
become critical problems.

For more information, see [Confluent Health+](https://docs.confluent.io/platform/current/health-plus/index.html).

### Telemetry Reporter

The Confluent Telemetry Reporter is a plugin that runs inside each Confluent Platform service
to push metadata about the service to Confluent. Telemetry Reporter enables
product features based on the metadata, like Health+. Data is sent over HTTP
using an encrypted connection.

Telemetry is disabled by default in CFK. You can enable and configure it
globally at the CFK level.

When you globally enable Telemetry, you have an option to disable for specific
Confluent Platform components.

Each Confluent Platform component CR provides the status condition, in `status.conditions`,
whether Telemetry is enabled or disabled.

For more information and supported settings for Telemetry Reporter, see
[Confluent Telemetry Reporter](https://docs.confluent.io/platform/current/monitor/telemetry.html).

For a list of the metrics that are collected for Health+, see [Telemetry
Reporter Metrics](https://docs.confluent.io/platform/current/monitor/telemetry-reporter-metrics.html).

### Globally configure Telemetry

To globally enable Telemetry Reporter for CFK and all Confluent Platform components:

1. Set the followings in the [CFK values file](co-deploy-custom-values.md#co-values-file).
   ```yaml
   telemetry:
     operator:
        enabled: true  --- [1]
     enabled: true     --- [2]
   ```

   * [1] Enable Telemetry Report for CFK.
   * [2] Enable Telemetry Report for all Confluent Platform components.
2. Apply the change with the following command:
   ```bash
   helm upgrade --install confluent-operator \
     confluentinc/confluent-for-kubernetes \
     --values <path-to-values-file> \
     --namespace <namespace>
   ```

To globally configure the Telemetry Reporter settings for all Confluent Platform components:

1. Set the following in the [CFK values file](co-deploy-custom-values.md#co-values-file):
   ```yaml
   telemetry:
     secretRef:                 --- [1]
     directoryPathInContainer:  --- [2]
   ```

   * [1] [2] CFK supports the `secretRef` and `directoryPathInContainer`
     methods to load Telemetry configuration through Helm. Specify only one
     method.

     The Telemetry configuration should be specified in `telemetry.txt`, and
     the file must contain `api.key` and `api.secret`.

     If using a proxy, additional properties are required as shown below.
     ```text
     api.key=<cloud_key>
     api.secret=<cloud_secret>
     proxy.url=<proxy_url>           # Only required if proxy is enabled
     proxy.username=<proxy_username> # Only required if proxy requires credential
     proxy.password=<proxy_password> # Only required if proxy requires credential
     ```
   * [1] `secretRef` takes the precedence over `directoryPathInContainer` if
     both are configured.

     The expected key is `telemetry.txt`.

     If the referenced `secretRef` is not read or data is not in the expected
     format, CFK will fail to start.
   * [2] Provide the directory path where `telemetry.txt` is
     present.

     If `telemetry.txt` is not in the expected format, CFK will fail to
     start.

   See  [Provide secrets for Confluent Platform application CR](co-credentials.md#co-vault-category-2) for providing the keys/values and required
   annotations when using Vault.
2. To apply changes in Telemetry settings, in the referenced Secret, or in the
   `telemetry.txt` file, manually restart CFK and Confluent Platform:
   * Restart CFK:
     ```bash
     kubectl rollout restart deployment/confluent-operator
     ```
   * Restart Confluent Platform components as described in [Restart Confluent Platform Using Confluent for Kubernetes](co-roll-cluster.md#co-rolling-restart).

### Disable Telemetry for a Confluent Platform component

To disable Telemetry for a specific Confluent Platform component, set the following in the
component CR and apply the change with the `kubectl apply` command:

```yaml
telemetry:
  global: false
```

<a id="co-monitor-jmx"></a>

## JMX Metrics

CFK can deploy Confluent components with JMX metrics.

#### NOTE
For enhanced security, JMX metrics are disabled by default starting with CFK 3.2.1. In earlier versions, JMX metrics were enabled by default.

When enabled, these JMX metrics are made available on all pods at the following endpoints:

* JMX metrics are available on port 7203 of each pod.
* Jolokia (a REST interface for JMX metrics) is available on port 7777 of each
  pod.
* JMX Prometheus exporter is available on port 7778.

  CFK does not support authentication or encryption for the JMX Prometheus
  exporter, and it does not provide an option to disable the JMX Prometheus
  exporter.

### Backward compatibility

The default behavior for JMX remote access has changed:

* **Previous behavior**: The JMX port was accessible remotely without
  authentication. Users could read and write through the exposed JMX port.
* **Current behavior**: If you do not configure JMX authentication in the CR
  specification, remote JMX access is disabled by default. The JMX port is only
  accessible locally within the pod.

**Impact on existing deployments:**

If you currently use remote JMX access without authentication, you must
configure JMX authentication to continue using remote JMX. Internal
applications like Jolokia metrics and JMX Prometheus exporter continue to work
because they use the in-process JMX API and are not affected by remote
connector settings.

#### NOTE
- JMX authentication files do not support hot-reloading because the JVM only
  loads JMX credentials at startup. Changes to password or access control
  files require a pod restart to take effect.
- The JMX password and access control `secretRef` are not owned by the CR
  and are not watched. JMX credential updates require a manual pod restart.

### Configure JMX authentication

You can configure password-based authentication and access control for JMX
connections to secure JMX endpoints.

JMX authentication requires explicit configuration. You must provide either a
`secretRef` or `directoryPathInContainer` to enable JMX authentication.
The operator does not support auto-generated passwords.

#### NOTE
Use JMX authentication with TLS/SSL encryption in production environments.
See [Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes](co-network-encryption.md#co-network-encryption) for TLS configuration.

#### Configure password-based authentication

To enable password-based authentication for JMX:

1. Create a password file named `jmxremote.password` with the following
   format:
   ```text
   <username> <password>
   <username> <password>
   ```

   For example:
   ```text
   admin secretpassword
   readonly readonlypassword
   ```

   Set the password file permissions to `0600` (read/write for owner only).

   #### NOTE
   When you use `secretRef`, the operator automatically sets file
   permissions by specifying items in the Kubernetes `SecretVolumeSource`.
   The Kubernetes Kubelet applies these permissions when mounting the secrets
   into pods, ensuring the JMX password file is readable only by the Kafka
   process (UID 1001) without requiring manual user intervention.

   When you use `directoryPathInContainer` with Vault, you must set the
   correct file permissions before the Vault sidecar injects the password
   file into the container.
2. Create a Kubernetes Secret with the password file:
   ```bash
   kubectl -n <namespace> create secret generic my-jmx-password \
     --from-file=jmxremote.password=<path-to-jmxremote.password>
   ```
3. Configure the component CR:
   ```yaml
   kind: Kafka
   spec:
     metrics:
       jmx:
         authentication:
           secretRef: my-jmx-password
   ```

Alternatively, you can use `directoryPathInContainer` to provide the path to
a JMX password file (`jmxremote.password`) injected by a sidecar (for
example, Vault).

#### Configure access control

Use JMX access control to restrict MBean operations with read-only or
read-write permissions.

To configure access control:

1. Create an access control file named `jmxremote.access` with the
   following format:
   ```text
   <username> <access-level>
   ```

   Valid access levels are `readonly` and `readwrite`.

   For example:
   ```text
   admin readwrite
   readonly readonly
   ```

   Set the access control file permissions to `0644` (read for all, write
   for owner).

   #### NOTE
   When you use `secretRef`, the operator automatically sets file
   permissions. When you use `directoryPathInContainer` with Vault, you
   must set the correct file permissions before the Vault sidecar injects
   the access control file into the container.
2. Create a Kubernetes Secret with the access control file:
   ```bash
   kubectl -n <namespace> create secret generic my-jmx-access \
     --from-file=jmxremote.access=<path-to-jmxremote.access>
   ```
3. Configure the component CR:
   ```yaml
   kind: Kafka
   spec:
     metrics:
       jmx:
         authentication:
           secretRef: my-jmx-password
         accessControl:
           enabled: true
           secretRef: my-jmx-access
   ```

If you set `accessControl.enabled` to `true` without providing
`secretRef` or `directoryPathInContainer`, the operator creates a default
configuration with read-only access. When using the default access control
file, you must add at least one of the following roles in the password file:
`monitorRole` or `controlRole`. The default access control file only
defines these roles with read-only access levels.

If you set `accessControl.enabled` to `false`, the system does not apply
access control, and all users can read and write.

#### Configure with Vault injection

As an alternative to Kubernetes Secrets, you can use
`directoryPathInContainer` to provide JMX password and access control files
injected by Vault.

To configure JMX authentication and access control with Vault:

1. Configure the component CR with Vault annotations:
   ```yaml
   apiVersion: platform.confluent.io/v1beta1
   kind: Kafka
   metadata:
     name: kafka
   spec:
     metrics:
       jmx:
         authentication:
           directoryPathInContainer: /vault/secrets/jmx
         accessControl:
           enabled: true
           directoryPathInContainer: /vault/secrets/jmx
     podTemplate:
       annotations:
         # Vault Agent Injection
         vault.hashicorp.com/agent-inject: "true"
         vault.hashicorp.com/agent-inject-status: "update"
         vault.hashicorp.com/preserve-secret-case: "true"
         vault.hashicorp.com/role: "confluent-operator"
         vault.hashicorp.com/agent-run-as-user: "1001"
         vault.hashicorp.com/agent-run-as-group: "1001"

         # Inject JMX password file
         vault.hashicorp.com/agent-inject-secret-jmx-password: "secret/kafka/jmx/password"
         vault.hashicorp.com/secret-volume-path-jmx-password: "/vault/secrets/jmx"
         vault.hashicorp.com/agent-inject-file-jmx-password: "jmxremote.password"
         vault.hashicorp.com/agent-inject-template-jmx-password: |
           {{- with secret "secret/kafka/jmx/password" -}}
           {{ .Data.data.content }}
           {{- end }}
         vault.hashicorp.com/agent-inject-perms-jmx-password: "0600"

         # Inject JMX access control file
         vault.hashicorp.com/agent-inject-secret-jmx-access: "secret/kafka/jmx/access"
         vault.hashicorp.com/secret-volume-path-jmx-access: "/vault/secrets/jmx"
         vault.hashicorp.com/agent-inject-file-jmx-access: "jmxremote.access"
         vault.hashicorp.com/agent-inject-template-jmx-access: |
           {{- with secret "secret/kafka/jmx/access" -}}
           {{ .Data.data.content }}
           {{- end }}
         vault.hashicorp.com/agent-inject-perms-jmx-access: "0644"
   ```

   #### IMPORTANT
   When using `directoryPathInContainer`, you must set the correct file
   permissions before the Vault sidecar injects the files into the
   container:
   - Password file (`jmxremote.password`): `0600` (read/write for
     owner only)
   - Access control file (`jmxremote.access`): `0644` (read for all,
     write for owner)

   As shown in the example above, use the
   `vault.hashicorp.com/agent-inject-perms-<file>` annotation to set
   these permissions.

   When using DPIC (Dynamic Pod Identity Credentials), you must also include
   the following annotations:
   - `vault.hashicorp.com/agent-run-as-user: "1001"`
   - `vault.hashicorp.com/agent-run-as-group: "1001"`

   The Vault sidecar injects the JMX password file (`jmxremote.password`)
   and access control file (`jmxremote.access`) into the specified directory
   path with the required file permissions.

<a id="co-monitor-jolokia-access-control"></a>

### Configure Jolokia access control

Jolokia provides JMX-over-HTTP access on port 7777 of each pod. You can apply
a Jolokia access control policy to restrict the HTTP methods, commands, and
MBeans that clients can access through this interface.

Jolokia access control is separate from the JMX access control covered in the
previous section. JMX access control governs JMX RMI connections, while Jolokia
access control governs the Jolokia HTTP interface.

To configure Jolokia access control, use the
`spec.metrics.jolokia.accessControl` settings on the component CR:

```yaml
spec:
  metrics:
    jolokia:
      accessControl:
        enabled:                    --- [1]
        secretRef:                  --- [2]
        directoryPathInContainer:   --- [3]
```

* [1] Set to `true` to enable Jolokia access control. When set to `false`
  or omitted, CFK does not apply access control, and all clients have full
  access. This is the default behavior.
* [2] The name of a Kubernetes Secret that contains your custom access control
  policy. The Secret must contain a key named `jolokia-access-control.xml`.
  Do not configure `secretRef` together with `directoryPathInContainer`.
* [3] The path to a directory in the container where the access control policy
  file is mounted, typically through Vault. Do not configure
  `directoryPathInContainer` together with `secretRef`.

If you set `enabled` to `true` without providing `secretRef` or
`directoryPathInContainer`, CFK applies a default access control policy.

#### Enable the default access control policy

To apply the default Jolokia access control policy, set `enabled` to `true`
without specifying a policy file:

```yaml
kind: Kafka
spec:
  metrics:
    jolokia:
      accessControl:
        enabled: true
```

#### Customize the access control policy

To restrict access to specific HTTP methods, commands, or MBeans, provide a
custom Jolokia access control policy in an XML file named
`jolokia-access-control.xml`. The file uses the standard Jolokia policy
format.

The following example allows only read and list commands over HTTP GET, and
limits the accessible MBeans:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<restrict>
  <http>
    <method>get</method>
  </http>
  <commands>
    <command>read</command>
    <command>list</command>
  </commands>
  <allow>
    <mbean>
      <name>kafka.server:*</name>
    </mbean>
    <mbean>
      <name>java.lang:type=Memory</name>
      <attribute>HeapMemoryUsage</attribute>
    </mbean>
  </allow>
</restrict>
```

For the full policy syntax, see the [Jolokia security documentation](https://jolokia.org/reference/html/manual/security.html).

#### Provide a custom policy with a Secret

1. Create a Kubernetes Secret from your access control file. The Secret key
   must be `jolokia-access-control.xml`:
   ```bash
   kubectl -n <namespace> create secret generic my-jolokia-access-control \
     --from-file=jolokia-access-control.xml=<path-to-access-control-file>
   ```
2. Reference the Secret in the component CR:
   ```yaml
   kind: Kafka
   spec:
     metrics:
       jolokia:
         accessControl:
           enabled: true
           secretRef: my-jolokia-access-control
   ```

#### Provide a custom policy with Vault injection

As an alternative to a Kubernetes Secret, you can use
`directoryPathInContainer` to provide the access control policy file injected
by Vault. Set `directoryPathInContainer` to the directory where the Vault
sidecar mounts the `jolokia-access-control.xml` file:

```yaml
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
spec:
  metrics:
    jolokia:
      accessControl:
        enabled: true
        directoryPathInContainer: /vault/secrets/jolokia
  podTemplate:
    annotations:
      # Vault Agent Injection
      vault.hashicorp.com/agent-inject: "true"
      vault.hashicorp.com/role: "confluent-operator"
      vault.hashicorp.com/agent-run-as-user: "1001"
      vault.hashicorp.com/agent-run-as-group: "1001"

      # Inject Jolokia access control file
      vault.hashicorp.com/agent-inject-secret-jolokia-access-control.xml: "secret/kafka/jolokia/access"
      vault.hashicorp.com/secret-volume-path-jolokia-access-control.xml: "/vault/secrets/jolokia"
      vault.hashicorp.com/agent-inject-file-jolokia-access-control.xml: "jolokia-access-control.xml"
      vault.hashicorp.com/agent-inject-template-jolokia-access-control.xml: |
        {{- with secret "secret/kafka/jolokia/access" -}}
        {{ .Data.data.content }}
        {{- end }}
```

### Configure security on JMX metrics endpoints

By default, authentication and encryption are not enabled on JMX or Prometheus
metric endpoints, but you have options to configure authentication and TLS for
JMX or Prometheus metric endpoints in the component CR:

```yaml
spec:
  metrics:
    authentication:
      type:                    --- [1]
    prometheus:                --- [2]
      rules:                   --- [3]
        - attrNameSnakeCase:
          cache:
          help:
          labels:
          name:
          pattern:
          type:
          value:
          valueFactor:
      blacklist:               --- [4]
      whitelist:               --- [5]
    tls:
      enabled:                 --- [6]
```

* [1] Set to `mtls` for mTLS authentication.

  If you set this to `mtls`, you must set `tls.enabled: true` ([6]).

  Note that CFK does not currently support mTLS authentication for Prometheus
  even with this set to `mtls`.
* [2] Specify Prometheus configurations to override the default settings.

  See [Prometheus](https://github.com/prometheus/jmx_exporter) for more
  information about the `rules`, `blacklist`, and `whitelist` properties.
* [3] A list of rules to apply.

  For example:
  ```yaml
  spec:
    metrics:
      prometheus:
        rules:
          - pattern: "org.apache.kafka.metrics<type=(\w+), name=(\w+)><>Value: (\d+)"
            name: "kafka_$1_$2"
            value: "$3"
            valueFactor: "0.000001"
            labels:
              "$1": "$4"
              "$2": "$3"
            help: "Kafka metric $1 $2"
            cache: false
            type: "GAUGE"
            attrNameSnakeCase: false
  ```
* [4] An array of patterns (in the string format) to identify what not to query.

  For example:
  ```yaml
  spec:
    metrics:
      prometheus:
        blacklist:
        - "org.apache.kafka.metrics:*"
  ```
* [5] An array of patterns (in the string format) to identify what to query.

  For example:
  ```yaml
  spec:
    metrics:
      prometheus:
        whitelist:
        - "org.apache.kafka.metrics:type=ColumnFamily,*"
  ```
* [6] If set to `true`, metrics are configured with global or component TLS as
  described in [Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes](co-network-encryption.md#co-network-encryption).

  #### NOTE
  When using `directoryPathInContainer` for metrics TLS configuration (for example, with Vault-injected secrets),
  JKS passwords are resolved at pod startup time. If you rotate or change the JKS password in your secret
  store, you must perform a manual rolling restart of the component pods for the new password to take effect.

  To trigger a rolling restart, see [Restart Confluent Platform Using Confluent for Kubernetes](co-roll-cluster.md#co-rolling-restart).

  This setting is ignored for Prometheus. CFK currently does not support TLS
  for Prometheus.

### Configure external access on JMX metrics

By default, CFK does not configure external access on JMX metric endpoints.

As an advanced configuration option, CFK supports enabling external access on
JMX metrics endpoint for Kafka as an advanced configuration option. You need this
setup only if the Jolokia HTTP-based solution is not enough.

Note that JMX metrics external access is currently only supported for Kafka, and
not for other Confluent Platform components.

To enable external access for Kafka at the component CR level, use
the annotation:

1. Set up an Kubernetes external access service that allows incoming requests to
   an external FQDN to be routed to the pod’s JMX/Prometheus metric endpoint
   port. See [Networking Service](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types).
2. Apply the annotation on the Kafka CR using a comma-separated list of
   the DNS (or IP) and the port you got from the previous step when you created
   a service.
   ```none
   "platform.confluent.io/jmx-rmi-server-urls": "pod1.external-address:<port1>,pod2.external-address:<port2>, ... ,podn.external-address:<portn>"
   ```

   For example, using the external IP address of the node, `35.35.35.35` and
   nodePorts `30001 to 30003`, the following CR configures a node port external
   access on the Kafka nodes:
   ```yaml
   kind: Kafka
   metadata:
     name: kafka
     namespace: operator
     annotations:
       "platform.confluent.io/jmx-rmi-server-urls": "35.35.35.35:30001,35.35.35.35:30002,35.35.35.35:30003"
   spec:
     replica: 3
   ```

### Configure Prometheus and Grafana

You can configure Prometheus to capture and aggregate JMX metrics from Confluent
components. Then you configure Grafana to visualize those metrics in a
dashboard.

For example configuration scenarios, see [an example of monitoring with
Prometheus and Grafana in jmx-monitoring-stacks](https://github.com/confluentinc/jmx-monitoring-stacks/tree/main/jmxexporter-prometheus-grafana/cfk)
and [an example of monitoring with Grafana metrics dashboard](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/monitoring/grafana-dashboard).