Configure CFK-Level Settings

Configure operator-level settings for Confluent for Kubernetes (CFK), including service accounts, webhooks, KRaft data recovery, replicas, and custom environment variables.

Deploy CFK with custom service account

To provide a custom service account to manage CFK, add --set serviceAccount.create=false --set serviceAccount.name=<name> to the install command:

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --set serviceAccount.create=false \
  --set serviceAccount.name=<service-account-name> \
  --namespace <namespace>

Expected: Release "confluent-operator" has been upgraded. Happy Helming!, followed by STATUS: deployed.

You can also update the values.yaml file as described in Deploy CFK with custom values, and set the following property:

serviceAccount:
  create: false
  name: <service-account-name>

If you use a custom service account and set rbac=false, meaning that the roles and role bindings were pre-created by your Kubernetes cluster administrator, then you must ensure that your <service-account-name> matches the subject name in the pre-created role binding.

Automount service account tokens for CFK

CFK requires access to the Kubernetes API to manage Confluent Platform resources. This access is provided through a ServiceAccount token that Kubernetes automatically mounts into pods.

CFK operator

The CFK operator must have automountServiceAccountToken set to true, which is the Kubernetes default. This applies whether you use the default service account or create a custom service account.
Confluent Platform components

Confluent Platform components deployed by CFK do not directly interact with the Kubernetes API server. For enhanced security, you can disable automountServiceAccountToken at the pod level for specific Confluent Platform components. For more details, see Opt out of API credential automounting.

Deploy CFK with cluster object deletion protection

CFK provides validating admission webhooks for deletion events of the Confluent Platform clusters. CFK webhooks are disabled by default in this release of CFK.

CFK provides the following webhooks:

Webhook to prevent component deletion when its persistent volume (PV) reclaim policy is set to Delete

This webhook (cfk-resources.webhooks.platform.confluent.io) blocks deletion requests on CRs with PVs in ReclaimPolicy: Delete. Without this prevention, a CR deletion results in the deletion of those PVs and data loss.

This webhook only applies to the components that have persistent volumes, namely, ZooKeeper (Confluent Platform 7.9 or earlier), Apache Kafka®, ksqlDB, and Control Center (Legacy).

This webhook also blocks deletion requests on those CRs, as described above. It blocks updates to prevent manual modifications during ZooKeeper to KRaft migration when the CR has the platform.confluent.io/kraft-migration-cr-lock: "true" annotation set.

Outside of KRaft migration, this webhook does not block normal CR updates.
Webhook to prevent CFK StatefulSet deletion

The proper way to delete Confluent Platform resources is to delete the component custom resource (CR) as CFK watches those deletion events and properly cleans everything up. Deletion of StatefulSets can result in unintended PV deletion and data loss.

This webhook (core-resources.webhooks.platform.confluent.io) blocks delete requests on CFK StatefulSets.
Webhook to prevent unsafe Kafka pod deletion

This webhook (kafka-pods.webhooks.platform.confluent.io) blocks Kafka pod deletion when the removal of a broker results in fewer in-sync replicas than configured in the min.insync.replicas Kafka property. Dropping below that value can result in data loss. Pod deletion can happen during Kubernetes maintenance without warning, such as during node replacement, and this webhook is an additional safeguard for your Kafka data.

Review the following when using this webhook:
- This webhook is only supported on clusters with fewer than 140,000 partitions.
- This webhook does not take the Kafka setting, minimum in-sync replicas (min.insync.replicas), into consideration.
  
  The minimum in-sync replicas setting on all topics is assumed to be 2 for Kafka with 3 or more replicas. Do not create topics with minimum in-sync replicas set to 1.
- To avoid having an internal ksqlDB topic with min in-sync replicas set to 1, set the internalTopicReplicationFactor property to 3 in the ksqlDB CR:
```
spec:
  internalTopicReplicationFactor: 3
```
Webhook to prevent unsafe pod eviction

This webhook (evictions.webhooks.platform.confluent.io) follows the same logic as the pod deletion webhook described above and prevents the creation of the pod eviction object which results in the draining of the pod nodes.

Requirements

Before you enable CFK webhooks, complete the following prerequisites.

TLS certificates

Before you deploy CFK with webhooks enabled, you must provide TLS certificates to be used for secure communication between the webhook server and the Kubernetes API server:

Create signed TLS keys and certificates in the format as described in Provide TLS keys and certificates in PEM format or Provide TLS keys and certificates in Java KeyStore format.

The certificate must have the Subject Alternative Name (SAN) of the form, confluent-operator.<namespace>.svc, which is the cluster-internal DNS name for the service in the namespace the CFK pod is getting deployed into.
Provide the above certificates to the CFK pod using one of the following:
- Add the certificates to a Kubernetes secret and put them in the namespace that the CFK pod is getting deployed in.
- Add the certificates in Vault. All certificates must be in the same directory.

Kubernetes metadata label

If on the Kubernetes version lower than 1.21, before you deploy CFK with webhooks enabled, add the kubernetes.io/metadata.name label on all namespaces where the webhooks should validate requests on.

This label is automatically set in Kubernetes 1.21 or later. For reference, see Automatic labeling.

Check labels on the namespace Confluent Platform is getting deployed in:

kubectl get namespace <namespace> --show-labels

An example output:

NAME       STATUS   AGE   LABELS
operator   Active   48d   <none>

Set the kubernetes.io/metadata.name label to the name of the namespace itself:
```
kubectl label namespace <namespace> kubernetes.io/metadata.name=<namespace>
```
Expected: namespace/<namespace> labeled

Validate that the label was applied.

kubectl get namespace <namespace> --show-labels

Expected:

NAME       STATUS   AGE   LABELS
operator   Active   48d   kubernetes.io/metadata.name=operator

Enable webhooks

Use the Helm value to enable or disable the validating webhooks.

Update the values.yaml file as described in Deploy CFK with custom values, and set the following properties:

webhooks:
  enabled: true                  --- [1]
  tls:                           --- [2]
    secretRef:
    directoryPathInContainer:

[1] Required to enable CFK webhooks.
[2] Specify secretRef or directoryPathInContainer value that you created in TLS certificates.

When enabling the webhook to prevent unsafe Kafka pod deletion for clusters with 100,000 or more partitions, increase the memory limit in values.yaml:

resources:
 limits:
   memory: 1024Mi

Enable the webhook for RBAC-enabled Kafka

After you deploy CFK with the Kafka deletion webhook enabled, if you are deploying an RBAC-enabled Kafka, you might need to give the RBAC user read access to all Kafka topics if that user is not a super user.

In the Kafka custom resource (CR), if the user configured in spec.dependencies.kafkaRest.authentication.bearer.secretRef is not included in the spec.authorization.superUsers list, create a rolebinding CR for that user as in the following example:

apiVersion: platform.confluent.io/v1beta1
kind: ConfluentRolebinding
metadata:
  name: clusterread
  namespace: <namespace>
spec:
  principal:
    type: user
    name: <principal>
  role: DeveloperRead
  resourcePatterns:
    - resourceType: Topic
      name: '*'
  kafkaRestClassRef:
    name: primary

Disable webhooks

After you deploy CFK with webhooks enabled, you can disable CFK webhooks at the namespace or component level by applying the following labels for a namespace or for a component CR.

Disable all CFK validation webhooks:

confluent-operator.webhooks.platform.confluent.io/disable: "true"

Disable the webhook that validates StatefulSet deletion:
```
confluent-operator.webhooks.platform.confluent.io/allow-statefulset-deletion: "true"
```
This label is only applied at the CR level.
Disable the webhook that validates CR deletion for PV reclaim policy:
```
confluent-operator.webhooks.platform.confluent.io/allow-pv-deletion: "true"
```
For example, to allow Kafka CR deletion when PVs are in the Delete mode, apply the following label:
```
kubectl -n <namespace> label kafka kafka \
  confluent-operator.webhooks.platform.confluent.io/allow-pv-deletion="true"
```
Expected: kafka.platform.confluent.io/kafka labeled

Disable the webhook that validates Kafka pod deletion:

confluent-operator.webhooks.platform.confluent.io/allow-kafka-pod-deletion: "true"

Deploy CFK with KRaft data recovery option

The KRaft data recovery feature supports cluster recovery after accidental deletion of a cluster CR. When the feature is enabled, CFK stores the KRaft cluster ID as an annotation on the persistent volume that corresponds to the persistent volume claim. This feature requires persistent volumes to use the retain reclaim policy.

Starting CFK 2.8.0, this feature is optional and is disabled by default.

To enable the KRaft data recovery feature, deploy CFK with the --set kRaftEnabled=true flag in the helm upgrade command as shown in the following example command:

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --set kRaftEnabled=true \
  --namespace <namespace>

Expected: Release "confluent-operator" has been upgraded. Happy Helming!, followed by STATUS: deployed.

You cannot enable the data recovery feature in a namespace-scoped deployment because CFK needs to create the cluster-wide permissions on persistent volumes, ClusterRole, and ClusterRolebinding, required for the feature.

If ClusterRole and ClusterRolebindings are created out-of-band, it should have the following rule, alongside the namespace-scoped rules described in Prepare Kubernetes Cluster for Confluent Platform and Confluent for Kubernetes, to grant CFK the cluster-wide access it needs to deploy KRaft controllers:

kind: ClusterRole
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - persistentvolumeclaims
  - persistentvolumes
  - secrets
  - secrets/finalizers
  - pods
  - services
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete

Deploy multiple replicas of CFK

When you deploy CFK with multiple replicas, one replica is active while the others are passive and standby. CFK logs are only generated by the active replica.

If the active replica fails, one of the standby replicas takes over and becomes active.

To deploy multiple replicas of CFK, in the CFK install command, set the replicas configuration property to the number of replicas you want.

For example, to deploy two CFK replicas:

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --set replicas=2 \
  --namespace <namespace>

Expected: Release "confluent-operator" has been upgraded. Happy Helming!, followed by STATUS: deployed.

You can also update the values.yaml file as described in Deploy CFK with custom values to set the following property:

replicas: <number of CFK replicas>

Deploy CFK with custom environment variables

You can add custom variables in values.yaml to be used by Helm during CFK installation.

As a use case, you can set HTTP_PROXY, HTTPS_PROXY, or NO_PROXY, and these environment variables get picked up by http.ProxyFromEnvironment set in the http.transport.

The following snippet of a values.yaml file specifies the address of an HTTP and HTTPS proxy servers:

customEnvVars:
  - name: HTTP_PROXY
    value: "http://proxy.example.com:3128"
  - name: HTTPS_PROXY
    value: "http://proxy.example.com:3128"
  - name: NO_PROXY
    value: "localhost,127.0.0.1,.newdomain.com,.svc.cluster.local"