<a id="co-cfk-settings"></a>

# Configure CFK-Level Settings

Configure operator-level settings for Confluent for Kubernetes (CFK), including service
accounts, webhooks, KRaft data recovery, replicas, and custom environment
variables.

<a id="co-deployment-custom-service-account"></a>

## Deploy CFK with custom service account

To provide a custom service account to manage CFK, add `--set
serviceAccount.create=false --set serviceAccount.name=<name>` to the install
command:

```bash
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --set serviceAccount.create=false \
  --set serviceAccount.name=<service-account-name> \
  --namespace <namespace>
```

Expected: `Release "confluent-operator" has been upgraded. Happy Helming!`,
followed by `STATUS: deployed`.

You can also update the `values.yaml` file as described in
[Deploy CFK with custom values](co-deploy-custom-values.md#co-values-file), and set the following
property:

```yaml
serviceAccount:
  create: false
  name: <service-account-name>
```

If you use a custom service account and set `rbac=false`, meaning that
the roles and role bindings were pre-created by your Kubernetes cluster
administrator, then you must ensure that your `<service-account-name>`
matches the subject name in the pre-created role binding.

## Automount service account tokens for CFK

CFK requires access to the Kubernetes API to manage Confluent Platform resources. This
access is provided through a `ServiceAccount` token that Kubernetes automatically
mounts into pods.

* CFK operator

  The CFK operator must have `automountServiceAccountToken` set to `true`,
  which is the Kubernetes default. This applies whether you use the default
  service account or create a custom service account.
* Confluent Platform components

  Confluent Platform components deployed by CFK do not directly interact with the Kubernetes
  API server. For enhanced security, you can disable
  `automountServiceAccountToken` at the pod level for specific Confluent Platform
  components. For more details, see [Opt out of API credential automounting](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting).

<a id="co-deployment-webhooks"></a>

## Deploy CFK with cluster object deletion protection

CFK provides validating admission webhooks for deletion events of
the Confluent Platform clusters. CFK webhooks are disabled by default in this release of CFK.

CFK provides the following webhooks:

* **Webhook to prevent component deletion when its persistent volume (PV)
  reclaim policy is set to Delete**

  This webhook (`cfk-resources.webhooks.platform.confluent.io`) blocks
  deletion requests on CRs with PVs in `ReclaimPolicy: Delete`. Without this
  prevention, a CR deletion results in the deletion of those PVs and data
  loss.

  This webhook only applies to the components that have persistent volumes,
  namely, ZooKeeper (Confluent Platform 7.9 or earlier), Apache Kafka®, ksqlDB, and Control Center (Legacy).

  This webhook also blocks deletion requests on those CRs, as described
  above. It blocks updates to prevent manual modifications during ZooKeeper to
  KRaft migration when the CR has the
  `platform.confluent.io/kraft-migration-cr-lock: "true"` annotation set.

  Outside of KRaft migration, this webhook does not block normal CR
  updates.
* **Webhook to prevent CFK StatefulSet deletion**

  The proper way to delete Confluent Platform resources is to delete the component custom
  resource (CR) as CFK watches those deletion events and properly cleans
  everything up. Deletion of StatefulSets can result in unintended PV deletion
  and data loss.

  This webhook (`core-resources.webhooks.platform.confluent.io`) blocks delete
  requests on CFK StatefulSets.
* **Webhook to prevent unsafe Kafka pod deletion**

  This webhook (`kafka-pods.webhooks.platform.confluent.io`) blocks Kafka pod
  deletion when the removal of a broker results in fewer in-sync replicas
  than configured in the `min.insync.replicas` Kafka property. Dropping below
  that value can result in data loss. Pod deletion can happen during
  Kubernetes maintenance without warning, such as during node replacement, and
  this webhook is an additional safeguard for your Kafka data.

  Review the following when using this webhook:
  * This webhook is only supported on clusters with fewer than 140,000
    partitions.
  * This webhook does not take the Kafka setting, [minimum in-sync replicas](https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#brokerconfigs_min.insync.replicas)
    (`min.insync.replicas`), into consideration.

    The minimum in-sync replicas setting on all topics is assumed to be `2`
    for Kafka with 3 or more replicas. Do not create topics with minimum in-sync
    replicas set to 1.
  * To avoid having an internal ksqlDB topic with min in-sync replicas set
    to 1, set the `internalTopicReplicationFactor` property to `3` in the
    ksqlDB CR:
    ```yaml
    spec:
      internalTopicReplicationFactor: 3
    ```
* **Webhook to prevent unsafe pod eviction**

  This webhook (`evictions.webhooks.platform.confluent.io`) follows the same
  logic as the pod deletion webhook described above and prevents the creation of
  the pod eviction object which results in the draining of the pod nodes.

### Requirements

Before you enable CFK webhooks, complete the following prerequisites.

<a id="co-webhooks-tls-certificates"></a>

#### TLS certificates

Before you deploy CFK with webhooks enabled, you must provide TLS certificates
to be used for secure communication between the webhook server and the
Kubernetes API server:

1. Create signed TLS keys and certificates in the format as described in
   [Provide TLS keys and certificates in PEM format](co-network-encryption.md#co-certs-pem) or [Provide TLS keys and certificates in Java KeyStore format](co-network-encryption.md#co-certs-jks).

   The certificate must have the Subject Alternative Name (SAN) of the form,
   `confluent-operator.<namespace>.svc`, which is the cluster-internal DNS
   name for the service in the namespace the CFK pod is getting deployed into.
2. Provide the above certificates to the CFK pod using one of the following:
   * Add the certificates to a [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/) and put them
     in the namespace that the CFK pod is getting deployed in.
   * [Add the certificates in Vault](co-credentials.md#co-directory-path-in-container). All
     certificates must be in the same directory.

#### Kubernetes metadata label

If on the Kubernetes version lower than 1.21, before you deploy CFK with
webhooks enabled, add the `kubernetes.io/metadata.name` label on all
namespaces where the webhooks should validate requests on.

This label is automatically set in Kubernetes 1.21 or later. For reference,
see [Automatic labeling](https://kubernetes.io/docs/concepts/overview/_print/#automatic-labelling).

1. Check labels on the namespace Confluent Platform is getting deployed in:
   ```bash
   kubectl get namespace <namespace> --show-labels
   ```

   An example output:
   ```bash
   NAME       STATUS   AGE   LABELS
   operator   Active   48d   <none>
   ```
2. Set the `kubernetes.io/metadata.name` label to the name of the namespace
   itself:
   ```bash
   kubectl label namespace <namespace> kubernetes.io/metadata.name=<namespace>
   ```

   Expected: `namespace/<namespace> labeled`
3. Validate that the label was applied.
   ```bash
   kubectl get namespace <namespace> --show-labels
   ```

   Expected:
   ```bash
   NAME       STATUS   AGE   LABELS
   operator   Active   48d   kubernetes.io/metadata.name=operator
   ```

<a id="co-enable-webhooks"></a>

### Enable webhooks

Use the Helm value to enable or disable the validating webhooks.

Update the `values.yaml` file as described in
[Deploy CFK with custom values](co-deploy-custom-values.md#co-values-file), and set the following
properties:

```yaml
webhooks:
  enabled: true                  --- [1]
  tls:                           --- [2]
    secretRef:
    directoryPathInContainer:
```

* [1] Required to enable CFK webhooks.
* [2] Specify `secretRef` or `directoryPathInContainer` value that you
  created in [TLS certificates](#co-webhooks-tls-certificates).

When enabling the webhook to prevent unsafe Kafka pod deletion for clusters with
100,000 or more partitions, increase the memory limit in `values.yaml`:

```yaml
resources:
 limits:
   memory: 1024Mi
```

#### Enable the webhook for RBAC-enabled Kafka

After you deploy CFK with the Kafka deletion webhook enabled, if you are
deploying an RBAC-enabled Kafka, you might need to give the RBAC user read
access to all Kafka topics if that user is not a super user.

In the Kafka custom resource (CR), if the user configured in
`spec.dependencies.kafkaRest.authentication.bearer.secretRef` is not included
in the `spec.authorization.superUsers` list, create a rolebinding CR for that
user as in the following example:

```yaml
apiVersion: platform.confluent.io/v1beta1
kind: ConfluentRolebinding
metadata:
  name: clusterread
  namespace: <namespace>
spec:
  principal:
    type: user
    name: <principal>
  role: DeveloperRead
  resourcePatterns:
    - resourceType: Topic
      name: '*'
  kafkaRestClassRef:
    name: primary
```

<a id="co-disable-webhooks"></a>

### Disable webhooks

After you deploy CFK with webhooks enabled, you can disable CFK webhooks at
the namespace or component level by applying the following labels for a
namespace or for a component CR.

* Disable all CFK validation webhooks:
  ```yaml
  confluent-operator.webhooks.platform.confluent.io/disable: "true"
  ```
* Disable the webhook that validates StatefulSet deletion:
  ```yaml
  confluent-operator.webhooks.platform.confluent.io/allow-statefulset-deletion: "true"
  ```

  This label is only applied at the CR level.
* Disable the webhook that validates CR deletion for PV reclaim policy:
  ```yaml
  confluent-operator.webhooks.platform.confluent.io/allow-pv-deletion: "true"
  ```

  For example, to allow Kafka CR deletion when PVs are in the `Delete` mode,
  apply the following label:
  ```bash
  kubectl -n <namespace> label kafka kafka \
    confluent-operator.webhooks.platform.confluent.io/allow-pv-deletion="true"
  ```

  Expected: `kafka.platform.confluent.io/kafka labeled`
* Disable the webhook that validates Kafka pod deletion:
  ```yaml
  confluent-operator.webhooks.platform.confluent.io/allow-kafka-pod-deletion: "true"
  ```

<a id="co-deployment-kraft"></a>

## Deploy CFK with KRaft data recovery option

The KRaft data recovery feature supports cluster recovery after accidental
deletion of a cluster CR. When the feature is enabled, CFK stores the
KRaft cluster ID as an annotation on the persistent volume that
corresponds to the persistent volume claim. This feature requires
persistent volumes to use the `retain` reclaim policy.

Starting CFK 2.8.0, this feature is optional and is disabled by default.

To enable the KRaft data recovery feature, deploy CFK with the `--set
kRaftEnabled=true` flag in the `helm upgrade` command as shown in the
following example command:

```bash
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --set kRaftEnabled=true \
  --namespace <namespace>
```

Expected: `Release "confluent-operator" has been upgraded. Happy Helming!`,
followed by `STATUS: deployed`.

You cannot enable the data recovery feature in a namespace-scoped deployment
because CFK needs to create the cluster-wide permissions on persistent
volumes, ClusterRole, and ClusterRolebinding, required for the feature.

If ClusterRole and ClusterRolebindings are created out-of-band, it should have
the following rule, alongside the namespace-scoped rules described in
[Prepare Kubernetes Cluster for Confluent Platform and Confluent for Kubernetes](co-prepare.md#co-prepare), to grant CFK the cluster-wide access it needs to deploy
KRaft controllers:

```yaml
kind: ClusterRole
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - persistentvolumeclaims
  - persistentvolumes
  - secrets
  - secrets/finalizers
  - pods
  - services
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
```

<a id="co-deployment-multiple-replicas"></a>

## Deploy multiple replicas of CFK

When you deploy CFK with multiple replicas, one replica is active while the
others are passive and standby. CFK logs are only generated by the active
replica.

If the active replica fails, one of the standby replicas takes over and becomes
active.

To deploy multiple replicas of CFK, in the CFK install command, set the
`replicas` configuration property to the number of replicas you want.

For example, to deploy two CFK replicas:

```bash
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --set replicas=2 \
  --namespace <namespace>
```

Expected: `Release "confluent-operator" has been upgraded. Happy Helming!`,
followed by `STATUS: deployed`.

You can also update the `values.yaml` file as described in
[Deploy CFK with custom values](co-deploy-custom-values.md#co-values-file) to set the following
property:

```yaml
replicas: <number of CFK replicas>
```

<a id="co-deployment-custom-vars"></a>

## Deploy CFK with custom environment variables

You can add custom variables in `values.yaml` to be used by Helm during CFK
installation.

As a use case, you can set `HTTP_PROXY`, `HTTPS_PROXY`, or `NO_PROXY`, and these
environment variables get picked up by `http.ProxyFromEnvironment` set in the
`http.transport`.

The following snippet of a `values.yaml` file specifies the address of an HTTP
and HTTPS proxy servers:

```YAML
customEnvVars:
  - name: HTTP_PROXY
    value: "http://proxy.example.com:3128"
  - name: HTTPS_PROXY
    value: "http://proxy.example.com:3128"
  - name: NO_PROXY
    value: "localhost,127.0.0.1,.newdomain.com,.svc.cluster.local"
```