<a id="co-configure-misc"></a>

# Advanced Configuration Options in Confluent for Kubernetes

This topic describes a few of the advanced configuration options Confluent for Kubernetes
(CFK) provides for Confluent components.

## Configure Kafka Connect & ksqlDB using Confluent Cloud

Confluent for Kubernetes supports deploying and managing Connect, ksqlDB, and Control Center to
connect to a Confluent Cloud Kafka and Schema Registry.

For an illustrative walkthrough on configuring this, see the [tutorial for connecting to
Confluent Cloud](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/hybrid/ccloud-integration).

<a id="co-service-account"></a>

## Provide custom service account

Each Confluent component pod deployed by Confluent for Kubernetes has an associated [Kubernetes
Service Account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
These service accounts serve several functions in Confluent for Kubernetes. For example, to
deploy Confluent Docker images from a private Docker image registry, you can
associate Docker registry credentials to the service account associated with
your Confluent components.

If you do not specify a service account, the `default` service account is
assigned to the component pod in the same namespace.

To provide a custom service account, set `serviceAccountName` in the component
custom resource (CR) configuration:

```yaml
spec:
  podTemplate:
    serviceAccountName:
```

<a id="co-default-jvm-settings"></a>

## Default JVM settings

CFK applies a common set of JVM options to all Confluent Platform components, including
Kafka, KRaft, Connect, ksqlDB, Schema Registry, Control Center, REST Proxy, and ZooKeeper. The
defaults are applied at pod startup.

The following JVM flags are set by default:

```text
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-XX:+ExplicitGCInvokesConcurrent
-XX:G1HeapRegionSize=16M
-XX:MinMetaspaceFreeRatio=50
-XX:MaxMetaspaceFreeRatio=80
-XX:MetaspaceSize=96m
-XX:ParallelGCThreads=1
-XX:ConcGCThreads=1
-XX:+PrintFlagsFinal
-XX:+UnlockDiagnosticVMOptions
```

To change a default JVM flag, add a new flag, or remove a default flag, set
`spec.configOverrides.jvm` in the component CR:

* To add a flag or override the default value of an existing flag, see
  [Configuration overrides](#co-config-overrides).

  For example, to override the default `-XX:ConcGCThreads=1` and
  `-XX:ParallelGCThreads=1` values:
  ```yaml
  spec:
    configOverrides:
      jvm:
        - "-XX:ConcGCThreads=3"
        - "-XX:ParallelGCThreads=12"
  ```
* To remove a default flag, prefix the flag with `--`. For details, see
  [Remove default configuration](#co-configoverrides-remove).

  For example, to remove the default `-XX:ParallelGCThreads` and
  `-XX:ConcGCThreads` settings:
  ```yaml
  spec:
    configOverrides:
      jvm:
        - "---XX:ParallelGCThreads"
        - "---XX:ConcGCThreads"
  ```
* To configure the JVM heap size (`-Xmx`, `-Xms`,
  `-XX:MaxRAMPercentage`, `-XX:+UseContainerSupport`), see
  [Define Java heap size](co-resources.md#co-define-jvm-heap-size).

Any change to JVM settings triggers a rolling restart of the component pods.

<a id="co-config-overrides"></a>

## Configuration overrides

You can override or remove default configuration parameters of a Confluent Platform components
in the component custom resource (CR) as below:

```yaml
spec:
  configOverrides:
    server: []
    jvm: []
    log4j: []
    log4j2: []
```

Refer to [Configuration Reference](https://docs.confluent.io/platform/current/installation/configuration) for
configuration parameters used in Confluent Platform components.

For setting `log4j2`, see [Configure Log4j 2](#co-config-log4j2).

### Override default configuration

Under the `configOverrides` property, specify the  `key=value` of the
configuration to override the default setting for the `server` or `log4j`.
In the case of `jvm`, specify the parameter you want to change inside double
quotes.

The example below for Kafka has the following effects:

* Enables automatic Topic creation (disabled by default).
* Enables the Cluster Linking feature (disabled by default).
* Sets a few JVM flags related to memory management.

  For example, default values for `-XX:ConcGCThreads=1` and `-XX:ParallelGCThreads=1` are overridden to `-XX:ConcGCThreads=3` and `-XX:ParallelGCThreads=12` respectively.
* Changes the log-level to `DEBUG` from the default of `INFO`.

```yaml
kind: Kafka
spec:
  configOverrides:
    server:
      - auto.create.topics.enable=true
      - confluent.cluster.link.enable=true
    jvm:
      - "-Xmx6g"
      - "-XX:MetaspaceSize=96m"
      - "-XX:+UseG1GC"
      - "-Djavax.net.ssl.trustStore=/mnt/sslcerts/kafka-tls/truststore.p12"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
      - "-XX:ConcGCThreads=3"
      - "-XX:ParallelGCThreads=12"
    log4j:
      - log4j.rootLogger=DEBUG, stdout
```

To set a property to an empty value, omit the `=` sign and the value
entirely and specify only the key. Entries in the `key=` format, with no
value after the `=` sign, are rejected with an error like
`missing value for key <property_name>`.

For example, to set `ssl.endpoint.identification.algorithm` to `""`
(empty string):

```yaml
spec:
  configOverrides:
    server:
      - ssl.endpoint.identification.algorithm
```

For more information about disabling endpoint identification, see [How to
Disable Endpoint Identification](https://support.confluent.io/hc/en-us/articles/360001221083-How-to-Disable-Endpoint-Identification)
(requires login).

<a id="co-configoverrides-remove"></a>

### Remove default configuration

To remove a default setting, under the `configOverrides` property, add
`--` in front of the `key=value` of the configuration setting.

For example, to remove `autopurge.purgeInterval=1` property in ZooKeeper:

```yaml
kind: Zookeeper
spec:
  configOverrides:
    server:
      - --autopurge.purgeInterval=1
```

<a id="co-config-log4j2"></a>

## Configure Log4j 2

Starting in CFK 3.0, Log4j 2 is supported and enabled by default. Confluent Platform 8.0
only supports Log4j 2. To learn about using Log4j 2 for Confluent Platform, see [Confluent
Platform Logging](https://docs.confluent.io/platform/current/monitor/cp-logging.html).

The `configOverrides.log4j2` setting does not apply to the bundled Control Center
Prometheus and Alertmanager services, which do not use Log4j 2 and do not
support JSON logging. For details, see [Configure Control Center](co-monitor-cp.md#co-configure-c3).

By default, CFK does internally generate a `log4j2.yaml` file. Optionally, to
customize Log4j 2, specify the content of the Log4j 2 property file (the
`Configuration:` section) in the `configOverrides.log4j2` section of the
component CR files.

For the supported log properties, see [Configure Confluent Platform Logs](https://docs.confluent.io/platform/current/monitor/cp-logging.html).

Following is a sample snippet of Log4j 2 configurations. The highlighted section
is the content of the Log4j2 property file in Confluent Platform.

```yaml
kind: <component>
spec:
  configOverrides:
    log4j2:
      Configuration:
        Loggers:
          Logger:
            - name: io.confluent.rest-utils
              level: DEBUG
              additivity: false
              AppenderRef:
                - ref: stdout
            - name: io.confluent.rbacapi
              level: DEBUG
              additivity: false
              AppenderRef:
                - ref: stdout
```

<a id="co-annotations"></a>

## Annotate Confluent custom resources

Confluent for Kubernetes (CFK) provides a set of public annotations that you can use to modify
a certain workflow or a state of Confluent Platform components. The annotations are applied to
Confluent Platform custom resources (CRs).

platform.confluent.io/force-reconcile
: Triggers a reconcile cycle of cluster. Once the reconcile cycle is complete,
  the annotation value gets reset to `false`.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: All CRs

platform.confluent.io/block-reconcile
: Blocks the reconcile even when internal resources or the CR
  spec is changed. This is used primarily to allow users to perform manual
  workflows. When this is enabled, CFK discards any changes done out of band to
  the CR.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: All CRs

platform.confluent.io/roll-precheck
: When set to `disable`, CFK does not perform the pre-check for
  under-replicated partitions.
  <br/>
  * Supported values: `disable`, `enable`
  * Default value: `enable`
  * CR types applied to: Kafka

platform.confluent.io/roll-pause
: When set to `true`, the current pod roll will be paused.
  <br/>
  * Supported values: `false`, `true`
  * Default value: `false`
  * CR types applied to: Kafka

platform.confluent.io/disable-garbage-collection
: Disables CFK from garbage collecting Kubernetes resources that CFK
  internally manages.
  <br/>
  * Supported values: `false`, `true`
  * Default value: `true`
  * CR types applied to: Control Center, Connect, Kafka, REST Proxy, ksqlDB, Schema Registry,
    ZooKeeper

platform.confluent.io/enable-shrink
: Enables the shrink workflow for the Kafka CR. This should only be enabled when
  the Kafka image of the version 7.0 or higher.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `true`
  * CR types applied to: Kafka

platform.confluent.io/disable-internal-rolebindings-creation
: Defines whether to disable internal rolebinding creation in RBAC security
  settings.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: Control Center, Connect, REST Proxy, ksqlDB, Schema Registry

platform.confluent.io/soft-delete-versions
: A list of versions to trigger a soft delete workflow for the Schema CR.
  <br/>
  * Supported values: A JSON formatted array, for example, `[1,2,3]`
  * Default value: None
  * CR types applied to: Schema

platform.confluent.io/delete-versions
: A list of versions to trigger a hard delete workflow for the Schema CR.
  <br/>
  * Supported values: A JSON formatted array, for example, `[1,2,3]`
  * Default value: None
  * CR types applied to: Schema

platform.confluent.io/restart-connector
: Triggers a restart of the Connector.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: Connector

platform.confluent.io/pause-connector
: Pauses the connector.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: Connector

platform.confluent.io/resume-connector
: Resumes the connector.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: Connector

platform.confluent.io/restart-task
: Triggers a restart of the specified Connector task.
  <br/>
  * Supported values: A `int32` type number
  * Default value: None
  * CR types applied to: Connector

platform.confluent.io/http-timeout-in-seconds
: Specifies the HTTP client timeout in seconds for the CR workflows.
  <br/>
  * Supported values: A `int32` type number
  * Default value: None
  * CR types applied to: Control Center, Connect, Kafka, KafkaTopic, ClusterLink,
    Schema

platform.confluent.io/confluent-hub-install-extra-args
: Additional arguments for the Connect CR. The extra arguments will be used
  when the Connect starts up and downloads plugins from the Confluent Hub.
  <br/>
  * Supported values:  A string of flags, for example,
    `--worker-configs /dev/null --component-dir /mnt/plugins`
  * Default value: None
  * CR types applied to: Connect

platform.confluent.io/pod-overlay-configmap-name
: Configures additional Kubernetes features that are not supported in the CFK
  API.
  <br/>
  * Supported values: A ConfigMap name. For details on the Pod Overlay feature
    and the associated ConfigMap, see [Customize Confluent Platform pods with Pod Overlay](#co-pod-overlay).
  * Default value: None
  * CR types applied to: Control Center, Connect, Kafka, REST Proxy, ksqlDB, Schema Registry,
    ZooKeeper, KRaft

platform.confluent.io/enable-dynamic-configs
: Enables dynamic TLS certificates rotation for Kafka listeners and Kafka
  REST class service so that the Kafka cluster does not roll when certificates
  change.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: Kafka
  <br/>
  #### NOTE
  When the Kafka internal listener uses OAuth or bearer token authentication,
  CFK does not support dynamic configurations. CFK applies dynamic configurations through
  a combination of Confluent REST Proxy and the Kafka internal listener, and CFK is
  not wired to communicate with an Kafka internal listener that uses OAuth or
  bearer token authentication. To use dynamic configurations, configure the
  internal listener with a different authentication mechanism, such as
  SASL/PLAIN or mTLS.

platform.confluent.io/pvc-access-mode
: Sets the Persistent Volume Claim (PVC) access mode which specifies how CFK
  pods can interact with the underlying storage provided by a Persistent Volume.
  <br/>
  * Supported values: `ReadWriteOnce`, `ReadWriteMany`
  * Default value: `ReadWriteOnce`
  * CR types applied to: Kafka, ZooKeeper (Confluent Platform 7.9 or earlier only), KRaft, ksqlDB, Control Center, REST Proxy
  <br/>
  For details, see [Configure Storage for Confluent Platform Using Confluent for Kubernetes](co-storage.md#co-storage).

platform.confluent.io/disable-hard-delete-schema
: Disables hard delete of a schema. A hard delete removes all metadata, including
  schema IDs.
  <br/>
  * Supported values: `true`, `false`
  * Default value: `false`
  * CR types applied to: Schema

To add an annotation, run the following command:

```bash
kubectl annotate <CR type> <CR name> -n <namespace> <annotation>="<annotation value>"
```

To delete an annotation, run the following command:

```bash
kubectl annotate <CR type> <CR name> -n <namespace> <annotation>-
```

<a id="co-pod-annotations"></a>

## Annotate Confluent pods

An annotation in Kubernetes defines an unstructured key value map that can be
set by external tools to store and retrieve metadata. Annotations are stored
with pods.

You can define custom annotations for Confluent Platform components, and those annotations are
applied to the Kubernetes pods for the components.

Annotation values must pass Kubernetes annotations validation. See [the
Kubernetes documentation on Annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set)
for details.

Define annotations in the component custom resource (CR) as below:

```yaml
spec:
  podTemplate:
    annotations:
      key1: value1
      key2: value2
      ...
```

`annotations` must be a map of string keys and string values.

The following are example annotations in the Kafka pod for HashiCorp vault:

```yaml
spec:
  podTemplate:
    annotations:
      vault.hashicorp.com/agent-inject: true
      vault.hashicorp.com/agent-inject-status: update
      vault.hashicorp.com/preserve-secret-case: true
      vault.hashicorp.com/agent-inject-secret-jksPassword.txt: secret/jksPassword.txt
```

<a id="co-pod-overlay"></a>

## Customize Confluent Platform pods with Pod Overlay

Confluent for Kubernetes (CFK) supports a subset of Kubernetes PodTemplateSpec in the CFK API
(`spec.podTemplate` in the component custom resource) where you configure
StatefulSet PodTemplate for the Confluent Platform components. .

To set and use additional Kubernetes features that are not supported by the CFK
API, you can use the Pod Overlay feature.

Example use cases that you can use the Pod Overlay feature include:

* To deploy a Confluent Platform cluster with a custom init container that runs alongside the
  CFK init container.
* To deploy a Confluent Platform cluster with a custom sidecar container that runs alongside
  the main Confluent Platform container, for example, to add command-line tools for in-pod
  debugging or connectivity checks.
* To set Confluent Platform configuration properties that do not yet have first-class
  support in the CFK API and cannot be added through configuration
  overrides.

Make sure that you do not have conflict values between what’s set in the CFK
podTemplate API and in Pod Overlay. For example, if you specify
`podSecurityContext` in the `kafka.spec.podTemplate`, you cannot use Pod
Overlay to specify different values in `spec.template.spec.securityContext`.

To use Pod Overlay:

1. Create a template file (`<template-file>`) with the settings you want to
   add:
   ```yaml
   spec:
     template:
   ```

   The template file has to start with `spec: template: <xxx>`, and it has to
   follow the [Kubernetes StatefulSetSpec API](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#statefulsetspec-v1-apps).

   You can configure fields only inside `spec.template`.

   Fields specified outside of `spec.template` will be considered as invalid.

   The following example is for a custom init container:
   ```yaml
   spec:
     template:
       spec:
         initContainers:
         - name: busybox
           image: busybox:1.28
           command: ["echo","I am a custom init-conatiner"]
           imagePullPolicy: IfNotPresent
         hostNetwork: true
         dnsPolicy: ClusterFirstWithHostNet
   ```

   Note that when `hostNetwork:` is set to `true`, `dnsPolicy:` must be
   set to `ClusterFirstWithHostNet`.

   When a tool needs exclusive access to the data directory before the main
   container starts, use an init container. For example, tools like
   `kafka-metadata-recovery` require an exclusive file lock on the metadata
   log directory, which is only available if the main container has not
   started yet.

   The following example is for a custom sidecar container that runs
   alongside the main container:
   ```yaml
   spec:
     template:
       spec:
         containers:
         - name: debug-sidecar
           image: <image-with-curl>
           imagePullPolicy: IfNotPresent
           command: ["sh", "-c", "sleep infinity"]
           resources:
             requests:
               cpu: 50m
               memory: 128Mi
   ```

   When a tool needs network or process access while the main container is
   running, use a sidecar container. For example, if your Confluent Platform image does not
   include command-line tools such as `curl`, you can add a sidecar
   container with its own image to perform in-pod debugging or connectivity
   checks without modifying the Confluent Platform image.

   #### NOTE
   Starting with Confluent Platform 8.3, images built on the `cp-base-java-micro` base
   image (built on `ubi9-micro`) no longer include `curl` and other
   debugging tools. If you use one of these images, use a sidecar
   container as described here to perform in-pod debugging or
   connectivity checks.

   Init containers and sidecar containers are not interchangeable. You cannot
   use a sidecar container for tools that require an exclusive lock on the
   data directory while the main container is running, because the main
   container already holds that lock. If you need exclusive access to the
   data directory before the main container starts, choose an init
   container. If you need network or process access while the main
   container is running, choose a sidecar container.
2. Create a ConfigMap (`<configmap>`) using the file created in the previous step
   (`<template-file>`). You must use `pod-template.yaml` as the key with
   `--from-file` option.
   ```bash
   kubectl create configmap <configmap> --from-file=pod-template.yaml=<template-file> -n <namespace>
   ```
3. Add the `platform.confluent.io/pod-overlay-configmap-name` annotation on
   the Confluent Platform component resource CR.

   For example:
   ```yaml
   kind: Kafka
   metadata:
     name: kafka
     namespace: operator
     annotations:
       platform.confluent.io/pod-overlay-configmap-name: <configmap>
   ```

   According to the Kubernetes convention, a ConfigMap can only be referenced by
   the pods residing in the same namespace. So CFK will look for
   `<configmap>` within the same namespace as the component CR object.

For configuration examples, see [the tutorial for Pod Overlay](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/advanced-configuration/pod-overlay).

<a id="co-hpa"></a>

## Configure for Kubernetes Horizontal Pod Autoscaler

In Kubernetes, the [Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
feature automatically scales the number of pod replicas.

Starting in Confluent for Kubernetes (CFK) 2.1.0, you can configure Confluent Platform components to use
HPA based on CPU and memory utilization of Confluent Platform pods.

HPA is not supported for ZooKeeper and Control Center.

To use HPA with a Confluent Platform component, create an HPA resource for the component
custom resource (CR) out of band to integrate with CFK.

The following example is to create an HPA resource for Connect based on CPU
utilization and memory usage:

```yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: connect-cluster-hpa
  namespace: confluent
spec:
  scaleTargetRef:                             --- [1]
    apiVersion: platform.confluent.io/v1beta1 --- [2]
    kind: Connect                             --- [3]
    name: connect                             --- [4]
  minReplicas: 2                              --- [5]
  maxReplicas: 4                              --- [6]
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50              --- [7]
    - type: Resource
      resource:
        name: memory
        targetAverageValue: 1000Mi            --- [8]
```

* [1] Required. Specify the Confluent component-specific information in this
  section.
* [2] Required. CFK API version.
* [3] Required. The CR kind of the object to scale.
* [4] Required. The CR name of the object to scale.
* [5] The minimum number of replicas when scaling down.

  If your Kafka default replication factor is N, the `minReplicas`
  on your HPA for your Kafka cluster must be >= N.

  If you want Schema Registry, Connect, ksqlDB to be HA, set `minReplicas` >= 2
* [6] The maximum number of replicas when scaling up.
* [7] The target average CPU utilization of 50%.
* [8] The target average memory usage value of 1000 Mi.

Take the following into further consideration when setting up HPA for Confluent Platform:

* If you have `oneReplicaPerNode` set to `true` for Kafka (which is the
  default), your upper bound for Kafka brokers is the number of
  available Kubernetes worker nodes you have.
* If you have affinity or taint/toleration rules set for Kafka, that
  further constrains the available nodes.
* If your underlying Kubernetes cluster doesn’t itself support autoscaling of
  the Kubernetes worker nodes, make sure there is enough Kubernetes worker nodes
  to allow HPA is successful.

You can check the current status of HPA by running:

```bash
kubectl get hpa
```

<a id="co-custom-volume-mounts"></a>

## Mount custom volumes

Kubernetes provides storage abstraction through [Kubernetes Volumes](https://kubernetes.io/docs/concepts/storage/volumes/). These volumes can be
ephemeral, where the volumes get destroyed when the pod dies, or persistent,
where the lifetime of the volumes goes beyond the pod lifetime.

In CFK, you can have a Confluent Platform pod configured to have multiple volumes of various
types attached to the pod simultaneously and mounted at desired paths in the
pod.

#### NOTE
Mounting custom volumes does not support multiple PersistentVolumes for
ZooKeeper and Kafka data. CFK configures and manages one PersistentVolume for
ZooKeeper and Kafka data.

The following are a few of the common use cases for custom volume mounts:

* Third-party secrets providers

  As an alternative to using Kubernetes secrets to secure sensitive information,
  you can use a vault product like HashiCorp Vault, AWS Secrets Manager, and
  Azure KeyVault.

  You integrate a third-party secrets provider by configuring an ephemeral
  volume mount for the Confluent component pod that takes the credentials from
  the secrets provider.
* Kafka connectors

  Some Kafka connectors require JARs, that are outside of the Connect plugin
  but need to be available to the Connect pods. You can create persistent
  volumes with the connector JARs and mount them on the Connect worker pods.
* Multiple custom partitions

  For example, you could write logs to a separate persistent volume of your
  choice.

<br/>

In CFK, you mount custom volumes to Confluent component pods by defining custom
volume mounts in the component custom resources (CRs), such as for Kafka, ZooKeeper,
Control Center, Schema Registry, ksqlDB, Connect, and Kafka REST Proxy. The same volume
will be mounted on all the pods in the component cluster in the specified paths.

To mount custom volumes to a Confluent Platform component:

1. Configure the volumes according to the driver specification.
2. Add the following to the Confluent Platform component CR:
   ```yaml
   spec:
     mountedVolumes:         --- [1]
       volumes:              --- [2]
       volumeMounts:         --- [3]
   ```

   * [1] `mountedVolumes` is an array of the `volumes` and `volumeMounts`
     that are requested for this component.
   * [2] Required. `volumes` is an array of named volumes in a pod that
     may be accessed by any container in the pod.

     For the supported volume types and the specific configuration properties
     required for each volume type, see [Kubernetes Volume Types](https://kubernetes.io/docs/concepts/storage/volumes/#volume-types) for
     the supported volume types.
   * [3] Required. Describes mounting paths of the `volumes` within this
     container.

     For the configuration properties for volume mount, see [Kubernetes Pod
     volumeMounts](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#volumes-1).
3. Apply the CR using the `kubectl apply` command.

Before the volumes and volume mounts are added to the component pod template,
CFK performs a validation to ensure that there is no conflict with internal
volume mounts. Reconcile will fail, and the error will be added to the CFK logs
in the following cases:

* A custom volume’s mount path conflicts with an internal mount path.

  These are the internal mounts used by Confluent Platform components:
  * `/mnt/config`
  * `/mnt/config/init`
  * `/mnt/config/shared`
  * `/mnt/data/data0`
  * `/mnt/plugins`
  * `/opt/confluentinc`
* A custom volume’s mount path conflicts with a custom-mounted secret.
* There is a conflict between the custom volume names or custom volume mount
  paths.

The below example is to mount an Azure file volume and HashiCorp vault with
SecretProviderClass and a CSI driver:

```yaml
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
spec:
  mountedVolumes:
    volumes:
    - name: azure
      azureFile:
        secretName: azure-secret
        shareName: aksshare
        readOnly: true
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-database"
    volumeMounts:
    - name: azure
      mountPath: /mnt/azurePath1
    - name: azure
      mountPath: /mnt/azurePath2
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
```

<a id="co-pdb"></a>

## Configure Pod Disruption Budget

CFK provides the first-class CFK API to customize the [Pod Disruption Budget
(PDB)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets)
for all Confluent Platform components.

The PDB setting is typically used when upgrading a Kubernetes node. The pods are
moved to different nodes for the node upgrade and then are moved back to the
node. Or, when you want to reduce the size of the node pool, you would drain
that node by moving the pods out of that node.

PDB is also used when upgrading and reconfiguring components pods where pods
need to restart to provide a disruption budget and help with the availability of
the service.

You can disable the PDB for a specific Confluent Platform component.

By default, a PDB is configured based on a pre-determined formula:

* For Kafka: `maxUnavailable` := `1`
* For other Confluent Platform components, `MaxUnavailable` is based on the number of
  nodes: `maxUnavailable` := (`replicas` - `1` ) / `2`

PDB value is rounded down to an integer. For example, if you have 2 replicas,
`MaxUnavailable` will be `0` and not `0.5`.

To set a PDB, configure the settings in the component PR:

```yaml
kind: <Component>
spec:
  pdb:
    enabled:              --- [1]
    maxUnavailable:       --- [2]
```

* [1] Required. Set to `false` to disable PDB for this Confluent Platform component.
* [2] The maximum number of pods that can be unavailable during a disruption.

<a id="co-maintenance-mode"></a>

## Configure maintenance mode

The maintenance mode feature allows you to put specific Confluent Platform pods into
maintenance mode, preventing them from starting their main processes while
keeping them available for manual maintenance operations. This is particularly
useful for performing maintenance tasks, such as persistence file system
changes, log analysis, or troubleshooting without affecting the entire cluster.

The feature is available in CFK 3.1 or higher. It is also available in the
patch versions, 3.0.1, 2.11.3, and 2.10.3.

For configuration details, examples, and troubleshooting tips, see [the tutorial
for Maintenance Mode](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/advanced-configuration/maintenance-mode).

The maintenance mode feature is enabled in the following order:

1. User adds annotation to CR.
2. CFK generates maintenance mode configuration.
3. CFK mounts the configuration to the pod at `/mnt/config/init/maintenance_mode.config`
4. When a pod starts, CFK checks whether the current pod is in maintenance mode.
   * If the pod is in maintenance mode, the main container handles graceful
     signals and pauses indefinitely right before the component process starts.
   * If the pod is not in maintenance mode, it continues with normal startup.

#### NOTE
Starting in CFK 3.3.0, a pod in maintenance mode pauses inside its main
container immediately before the component process starts, rather than
pausing inside the CFK init container. Because the pause now happens within
the main container, you gain direct access to all command-line tools included
with the component image, such as `kafka-metadata-quorum`. You can use these
tools to troubleshoot and perform disaster recovery tasks that are not
possible from the init container.

The following Confluent Platform components support the maintenance mode feature: Kafka, Schema Registry,
ksqlDB, Connect, Control Center, REST Proxy, ZooKeeper, and KRaft.

#### NOTE
Maintenance mode is not supported for USMAgent or Confluent Gateway. Although CFK
deploys these components, the maintenance mode annotation has no effect on
them.

### Considerations and best practices

* Put pods into maintenance mode one at a time to maintain service availability.
* Back up intended files before making changes.
* Test changes in a non-production environment first.
* Monitor cluster health during maintenance operations.
* Remove maintenance mode annotations when done.
* Prefer using maintenance mode during off-peak hours to minimize impact.
* The maintenance mode pods remain accessible for manual intervention.
* Ensure proper RBAC permissions for maintenance operations.
* Log all maintenance activities for audit purposes.
* Avoid exposing maintenance mode pods to untrusted networks.
* Use secure methods (for example, `kubectl exec` over TLS) for accessing maintenance mode pods.
* Avoid putting multiple pods into maintenance mode simultaneously in production environments, as this may lead to service disruption.
* Pods in maintenance mode consume cluster resources but don’t serve traffic.
* The maintenance mode feature only affects the specific pods listed in the annotation.
* Manual intervention (deleting the pod) is required to enter and exit maintenance mode.
* When pods are in maintenance mode, the cluster roll operation is blocked until all pods exit maintenance mode.

### Put pods into maintenance mode

To put specific pods into maintenance mode:

1. Add the `platform.confluent.io/maintenance-mode` annotation to the
   component CR.
   ```bash
   kubectl annotate <component> <name> \
     platform.confluent.io/maintenance-mode=<comma-separated-pod-names> \
     --overwrite
   ```

   Pod names must match the actual pod names in the cluster, such as
   `kafka-0`, `schemaregistry-1`.

   For example, to put Schema Registry pods 1 and 2 into maintenance mode:
   ```bash
   kubectl -n confluent annotate schemaregistry schemaregistry \
     platform.confluent.io/maintenance-mode=schemaregistry-1,schemaregistry-2 \
     --overwrite
   ```
2. Wait for a few seconds for CFK to generate the configuration.

   CFK generates a JSON configuration file at `/mnt/config/init/maintenance_mode.config`:

   An example configuration file:
   ```json
   {
     "kafka-0": {
       "mode": "MAINTENANCE"
     },
     "kafka-1": {
       "mode": "MAINTENANCE"
     }
   }
   ```
3. Delete the pods to trigger the maintenance mode behavior.
   ```bash
   kubectl delete pod <list of maintenance mode pods>
   ```
4. Verify the maintenance mode status.
   ```bash
   # Check if annotation exists
   kubectl get <component> <name> \
     -o jsonpath='{.metadata.annotations.platform\.confluent\.io/maintenance-mode}'

   # List all pods in maintenance mode
   kubectl get pods -o json | jq -r '.items[] | \
     select(.metadata.annotations."platform.confluent.io/maintenance-mode") | .metadata.name'
   ```

### Signal handling

Pods in maintenance mode handle the following signals:

* SIGTERM: Clean termination, such as `kubectl delete pod`.
* SIGINT: Clean termination, such as Ctrl+C.
* Other signals: Ignored to maintain the maintenance state.

### Perform maintenance

Once pods are in maintenance mode, you can execute commands in the paused pod to
perform maintenance tasks.

For example:

* To execute commands in the Schema Registry pod 1:
  ```bash
  kubectl exec -it schemaregistry-1 -n confluent -- /bin/bash
  ```
* To change persistence file system:
  ```bash
  kubectl exec -it schemaregistry-1 -n confluent
  ```

### Exit maintenance mode

To return pods to normal operation:

1. Remove the maintenance mode annotation:
   ```bash
   kubectl -n confluent annotate <component> <name> \
     platform.confluent.io/maintenance-mode- \
     --overwrite
   ```
2. Delete the pods to restart them normally:
   ```bash
   kubectl delete pod <list of maintenance mode pods>
   ```