Advanced Configuration Options in Confluent for Kubernetes Blueprints¶

This topic describes a few of the advanced configuration options Confluent for Kubernetes (CFK) Blueprints provides for Confluent Platform.

Mount custom volumes¶

Kubernetes provides storage abstraction through Kubernetes Volumes. These volumes can be ephemeral, where the volumes get destroyed when the pod dies, or persistent, where the lifetime of the volumes goes beyond the pod’s lifetime.

In CFK Blueprints, you can have a Confluent Platform pod configured to have multiple volumes of various types attached to the pod simultaneously and mounted at desired paths in the pod.

Note

Mounting custom volumes does not support multiple PersistentVolumes for ZooKeeper and Kafka data. CFK Blueprints configures and manages one PersistentVolume for ZooKeeper and Kafka data.

The following are a few of the common use cases for custom volume mounts:

Third-party secrets providers

As an alternative to using Kubernetes secrets to secure sensitive information, you can use a vault product like Hashicorp Vault, AWS Secrets Manager, and Azure KeyVault.

You integrate a third-party secrets provider by configuring an ephemeral volume mount for the Confluent component pod that takes the credentials from the secrets provider.
Kafka connectors

Some Kafka connectors require JARs, that are outside of the Connect plugin but need to be available to the Connect pods. You can create persistent volumes with the connector JARs and mount them on the Connect worker pods.
Multiple custom partitions

For example, you could write logs to a separate persistent volume of your choice.

In CFK Blueprints, you mount custom volumes to Confluent component pods by defining custom volume mounts in the component class custom resources (CRs). The same volume will be mounted on all the pods in the component cluster in the specified paths.

To mount custom volumes to a Confluent Platform component:

Configure the volumes according to the driver specification.
Add the following to the Confluent Platform component CR:
```
kind: <component>ClusterClass
spec:
  provisioner:
    cfk:
      k8s:
        mountedVolumes:
          - volumes:            --- [1]
          - volumeMounts:       --- [2]
```
- [1] Required. An array of named volumes in a pod that may be accessed by any container in the pod.
  
  For the supported volume types and the specific configuration properties required for each volume type, see Kubernetes Volume Types for the supported volume types.
- [2] Required. An array of mounting paths of the volumes within this container.
  
  For the configuration properties for volume mount, see Kubernetes Pod volumeMounts.

Before the volumes and volume mounts are added to the component pod template, CFK Blueprints performs a validation to ensure that there is no conflict with internal volume mounts. Reconcile will fail, and the error will be added to the CFK Blueprints logs in the following cases:

A custom volume’s mount path conflicts with an internal mount path.

These are the internal mounts used by Confluent Platform components:
- /mnt/config
- /mnt/config/init
- /mnt/config/shared
- /mnt/data/data0
- /opt/confluentinc
A custom volume’s mount path conflicts with a custom-mounted secret.
There is a conflict between the custom volume names or custom volume mount paths.

The below example is to mount an Azure file volume and HashiCorp vault with SecretProviderClass and a CSI driver:

apiVersion: platform.confluent.io/v1beta1
kind: Kafka
spec:
  provisioner:
    cfk:
      k8s:
        mountedVolumes:
          volumes:
            - name: azure
              azureFile:
                secretName: azure-secret
                shareName: aksshare
                readOnly: true
            - name: secrets-store-inline
              csi:
                driver: secrets-store.csi.k8s.io
                readOnly: true
                volumeAttributes:
                  secretProviderClass: "vault-database"
          volumeMounts:
            - name: azure
              mountPath: /mnt/azurePath1
            - name: azure
              mountPath: /mnt/azurePath2
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true

Customize Confluent pods with Pod Overlay¶

Confluent for Kubernetes (CFK) Blueprints supports a subset of Kubernetes PodTemplateSpec in its API. When you configure the stateful sets for Confluent Platform components, you can use the Pod Overlay feature to set and use additional Kubernetes features that the CFK Blueprints API does not natively support.

Example use cases for the Pod Overlay feature include:

Deploy a Confluent Platform cluster with a custom init container that runs alongside the CFK init container.

In this case, the custom init container runs before the CFK init container.
Use a newly introduced Kubernetes feature that has not been added to the CFK Blueprints API.

To use Pod Overlay:

Create a template file (<template-file>) with the settings you want to add:
```
spec:
  template:
```
The template file must start with spec.template as above, and it has to follow the Kubernetes StatefulSetSpec API.

You can configure fields only inside spec.template.

Fields specified outside of spec.template will be considered as invalid.

The following example is for a custom init container:
```
spec:
  template:
    spec:
      initContainers:
      - name: busybox
        image: busybox:1.28
        command: ["echo","I am a custom init-conatiner"]
        imagePullPolicy: IfNotPresent
```
Create a ConfigMap (<configmap>) using the file created in the previous step (<template-file>). You must use pod-template.yaml as the key with the --from-file option.
```
kubectl create configmap <configmap> --from-file=pod-template.yaml=<template-file> -n <namespace>
```
Reference the above config map in the Confluent Platform component class custom resource (<component>ClusterClass CR):
```
kind: <component>ClusterClass
spec:
  provisioner:
    cfk:
      podOverlayConfigMapRef:
        name: <configmap>
```
<configmap>> is the name of the config map that you created in the previous step to provide Pod Overlay configurations.

According to the Kubernetes convention, a ConfigMap can only be referenced by the pods residing in the same namespace. So CFK Blueprints will look for <configmap> within the same namespace as the component ClusterClass CR object.

A ConfigMap can’t be shared across the Blueprints. But it can be shared within the Blueprint among multiple Confluent component classes.

Provide custom service account¶

Each Confluent component pod deployed by CFK Blueprints has an associated Kubernetes Service Account. These service accounts serve several functions in CFK Blueprints. For example, to deploy Confluent Docker images from a private Docker image registry, you can associate Docker registry credentials to the service account associated with your Confluent components.

If you do not specify a service account, the default service account is assigned to the component pod in the same namespace.

To provide a custom service account, set serviceAccountName in the component class custom resource (CR):

kind: <component>ClusterClass
spec:
  provisioner:
    cfk:
      k8s:
        podTemplate
          serviceAccountName: