Advanced Configuration Options for Confluent Platform¶

This topic describes a few of the advanced configuration options Confluent for Kubernetes (CFK) provides for Confluent components.

Configure Kafka Connect & ksqlDB using Confluent Cloud¶

Confluent for Kubernetes supports deploying and managing Connect, ksqlDB, and Confluent Control Center to connect to a Confluent Cloud Kafka and Schema Registry.

For a an illustrative walkthrough on configuring this, see the tutorial for connecting to Confluent Cloud.

Provide custom service account¶

Each Confluent component pod deployed by Confluent for Kubernetes has an associated Kubernetes Service Account. These service accounts serve several functions in Confluent for Kubernetes. For example, to deploy Confluent Docker images from a private Docker image registry, you can associate Docker registry credentials to the service account associated with your Confluent components.

If you do not specify a service account, the default service account is assigned to the component pod in the same namespace.

To provide a custom service account, set serviceAccountName in the component custom resource (CR) configuration:

spec:
  podTemplate:
    serviceAccountName:

Configuration overrides¶

You can override default configuration parameters of a Confluent Platform components in the component custom resource (CR) as below:

spec:
  configOverrides:
    server: []
    jvm: []
    log4j: []

Refer to Configuration Reference for configuration parameters used in Confluent Platform components.

The example below for Kafka has the following effects:

Enables automatic Topic creation (disabled by default).
Enables the Cluster Linking feature (disabled by default).
Sets a few JVM flags related to memory management.
Changes the log-level to DEBUG from the default of INFO.

spec:
  configOverrides:
    server:
      - auto.create.topics.enable=true
      - confluent.cluster.link.enable=true
    jvm:
      - "-Xmx6g"
      - "-XX:MetaspaceSize=96m"
      - "-XX:+UseG1GC"
      - "-Djavax.net.ssl.trustStore=/mnt/sslcerts/kafka-tls/truststore.p12"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
    log4j:
      - log4j.rootLogger=DEBUG, stdout

Apply the changes with the kubectl apply command.

Provide mounted secrets¶

When installing Confluent Platform components, Confluent for Kubernetes (CFK) securely stores sensitive credential configuration values you provide as Kubernetes Secrets. However, there are cases where you may wish to provide your own pre-existing secrets to be used in the configuration of Confluent Platform, rather than having CFK create the secret object for you. The most common scenario for this is when providing custom configuration overrides, which CFK treats as passthrough configuration and does not consider whether that data belongs in a secret object or not.

To utilize your own secret objects, specify a list of secret objects to mount in-memory into the containers of a Confluent Platform component. Combined with the Externalizing Secrets capability of Confluent Platform and custom configuration overrides, you can configure Confluent Platform with your secret data so that you store the information in an external file and pass the content of the a key file without exposing the sensitive information.

A change in mounted secrets will roll the cluster.

To provide and use mounted secrets, add the following in the component CR:

spec:
  mountedSecrets:
  - secretRef:                                          --- [1]
    keyItems:                                           --- [2]
    - key:                                              --- [3]
      path:                                             --- [4]

  configOverrides:
    server:
      - property=${file:/mnt/secrets/<secretRef>:<key>} --- [5]

[1] Set to the name of your secret. You must use the secret which are referenced in the other part of the component custom resource (CR).
[2] Optional. If keyItems are not specified, the secret is stored under /mnt/secrets/<secretRef> where <secretRef> is the value set in [1].

See Projection of Secret keys to specific paths for more information.
[3] Set to the secret key.
[4] Set to the relative path where the secret key ([3]) is stored. It may not be an absolute path and may not contain the string ...

For example, if secretRef is set to my-secret-aws, key is set to credentials, and path is set to aws/creds, the credentials are stored in /mnt/secrets/my-secret-aws/aws/creds/credentials.
[5] Set the externalized file in the ${file:/mnt/secrets/<secretRef>:<key>} format.

The following is an example workflow of configuring mounted secrets.

Create a file, my_file.txt, that contains your sensitive data:
```
username: user1
password: $ecret_passw0rd
```
Create a secret, example-secret, which stores the content of the above file in the my_credentials key:
```
kubectl create secret generic example-secret \
  --from-file=my_credentials=path/to/my_file.txt
```
The contents of my_file.txt will be Base64-encoded and stored under the my_credentials key of example-secret.

Set the configOverrides and mountedSecrets in the Kafka CR:

spec:
  mountedSecrets:
    - secretRef: example-secret
  configOverrides:
    server:
      - kafka.user=${file:/mnt/secrets/example-secret/my_credentials:username}
      - kafka.password=${file:/mnt/secrets/example-secret/my_credentials:password}

Deploy Kafka:
```
kubectl apply -f <Kafka CR>
```

Define pod annotations¶

An annotation in Kubernetes defines an unstructured key value map that can be set by external tools to store and retrieve metadata. Annotations are stored with pods.

You can define custom annotations for Confluent Platform components, and those annotations are applied to the Kubernetes pods for the components.

Annotation values must pass Kubernetes annotations validation. See the Kubernetes documentation on Annotations for details.

Define annotations in the component custom resource (CR) as below:

spec:
  podTemplate:
    annotations:
      key1: value1
      key2: value2
      ...

annotations must be a map of string keys and string values.

The following are example annotations in the Kafka pod for Hashcorp vault:

spec:
  podTemplate:
    annotations:
      vault.hashicorp.com/agent-inject: true
      vault.hashicorp.com/agent-inject-status: update
      vault.hashicorp.com/preserve-secret-case: true
      vault.hashicorp.com/agent-inject-secret-jksPassword.txt: secret/jksPassword.txt

Configure for Kubernetes Horizontal Pod Autoscaler¶

In Kubernetes, the Horizontal Pod Autoscaler (HPA) feature automatically scales the number of pod replicas.

Starting in Confluent for Kubernetes (CFK) 2.1.0, you can configure Confluent Platform components to use HPA based on CPU and memory utilization of Confluent Platform pods.

HPA is not supported for ZooKeeper and Confluent Control Center.

To use HPA with a Confluent Platform component, create an HPA resource for the component custom resource (CR) out of band to integrate with CFK.

The following example is to create an HPA resource for Connect based on CPU utilization and memory usage:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: connect-cluster-hpa
spec:
  scaleTargetRef:                             --- [1]
    apiVersion: platform.confluent.io/v1beta1 --- [2]
    kind: Connect                             --- [3]
    name: connect                             --- [4]
  minReplicas: 2                              --- [5]
  maxReplicas: 4                              --- [6]
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50              --- [7]
    - type: Resource
      resource:
        name: memory
        targetAverageValue: 1000Mi            --- [8]

[1] Required. Specify the Confluent component-specific information in this section.
[2] Required. CFK API version.
[3] Required. The CR kind of the object to scale.
[4] Required. The CR name of the object to scale.
[5] The minimum number of replicas when scaling down.

If your Kafka default replication factor is N, the minReplicas on your HPA for your Kafka cluster must be >= N.

If you want Schema Registry, Connect, ksqlDB to be HA, set minReplicas >= 2
[6] The maximum number of replicas when scaling up.
[7] The target average CPU utilization of 50%.
[8] The target average memory usage value of 1000 Mi.

Take the following into further consideration when setting up HPA for Confluent Platform:

If you have oneReplicaPerNode set to true for Kafka (which is the default), your upper bound for Kafka brokers is the number of available Kubernetes worker nodes you have.
If you have affinity or taint/toleration rules set for Kafka, that further constrains the available nodes.
If your underlying Kubernetes cluster doesn’t itself support autoscaling of the Kubernetes worker nodes, make sure there is enough Kubernetes worker nodes to allow HPA is successful.

You can check the current status of HPA by running:

kubectl get hpa