Configure Confluent Operator and Confluent Platform¶

The following is the process of configuring Confluent Operator and Confluent Platform:

Create the global configuration file.
Configure the global infrastructure settings.
Optionally configure the Confluent Platform component level settings.

The examples in this guide use the following assumptions:

$VALUES_FILE refers to the configuration file you set up in Create the global configuration file.
To present simple and clear examples in the Operator documentation, all the configuration parameters are specified in the config file ($VALUES_FILE). However, in your production deployments, use the --set or --set-file option when applying sensitive data with Helm. For example:
```
helm upgrade --install kafka \
 --set kafka.services.mds.ldap.authentication.simple.principal="cn=mds\,dc=test\,dc=com" \
 --set kafka.services.mds.ldap.authentication.simple.credentials=”Developer!” \
 --set kafka.enabled=true
```
operator is the namespace that Confluent Platform is deployed in.
All commands are executed in the helm directory under the directory Confluent Operator was downloaded to.

Configuration overview¶

During installation, Confluent Operator and Confluent Platform components are created based on parameters stored in multiple Helm Chart values.yaml files (one for Operator and one for each Confluent Platform component) and the global configuration file.

Do not modify parameters in the individual component values.yaml files. If you need to adjust capacity, add a parameter, or change a parameter for a component, you modify the component section in the global configuration file. You can also adjust configuration parameters after installation using the helm upgrade command.

The global configuration file is layered over the values.yaml files at installation and contains values that are specific to environments and that can be modified by you prior to installation.

After you download the Helm bundle you’ll see that:

The values.yaml file for each Confluent Platform component is stored in helm/confluent-operator/charts/<component>/.
The values.yaml file for Confluent Operator is stored in helm/confluent-operator/.
The <provider>.yaml file for each provider is stored in helm/providers/.

At installation, Helm reads the values files in the following layered order:

The values.yaml for the Confluent Platform component is read.
The values.yaml for Operator is read.
The global configuration file is read.

Create the global configuration file¶

To customize the default configuration file:

Go to the helm/providers directory under the directory where you downloaded the Confluent Operator bundle.
Make a copy of the provider file corresponding to your provider environment. For example, copy gcp.yaml to my-values.yaml if your provider is Google Cloud.
Set an environment variable pointing to your copy of the configuration file. For example:
```
export VALUES_FILE="/path/to/my-values.yaml"
```
The remainder of this topic uses $VALUES_FILE to refer to the global configuration file.

Configure the global settings¶

The following are the configuration changes necessary for the initial deployment of Confluent Operator and Confluent Platform. You specify the settings in the configuration file ($VALUES_FILE) you created above.

You can manually update the the configuration file after deployment if necessary. See Updating configuration for detail.

Validate or change your region, and zone or zones (if your Kubernetes cluster spans multiple availability zones).
Specify the storage class.
Validate or change the Docker image registry endpoint.
Configure DNS and load balancers for access to Confluent Platform from outside Kubernetes.
Validate or configure network encryption, authentication, and authorization.
Validate or configure your license key.

Add a license key¶

You can use Confluent Operator, Kafka, and Confluent Control Center for a 30-day trial period without a license key. After 30 days, Operator, Kafka, and Control Center require license keys.

Tip

For complete license information for Confluent Platform, see Confluent Platform Licenses.

To configure Confluent Platform with RBAC, Schema Registry requires a license key.

Add the license key to your configuration file ($VALUES_FILE) and run the helm upgrade command to activate the license. The examples below assume you are using your configuration file ($VALUES_FILE) to update your configuration.

Operator license¶

Add the following section to the Operator block in your configuration file ($VALUES_FILE) file and specify a license key:

operator:
  licenseKey:

Run the following command to activate the license:

helm upgrade --install <operator-release-name> \
  --values $VALUES_FILE \
  --set operator.enabled=true \
  ./confluent-operator

Kafka license¶

To add or update the Kafka license:

Add the following section to the Kafka block in your configuration file ($VALUES_FILE) file and specify a license key:
```
kafka:
  license:
```

Activate the license with the following command:

helm upgrade --install <kafka-release-name> \
  --values $VALUES_FILE \
  --set kafka.enabled=true \
  ./confluent-operator

Restart the Kafka cluster.

To add or update the Kafka license, you need to restart the Kafka cluster by either restarting the Kafka pods or rolling the Kafka cluster.

Confluent Control Center license¶

Add the following section to the Confluent Control Center block in your configuration file ($VALUES_FILE) file and specify a license key:

controlcenter:
  license:

Run the following command to activate the license:

helm upgrade --install <controlcenter-release-name> \
  --values $VALUES_FILE \
  --set controlcenter.enabled=true \
  ./confluent-operator

Schema Registry license¶

Add the following section to the Schema Registry block in your configuration file ($VALUES_FILE) file and specify a license key:

schemaregistry:
  license:

Run the following command to activate the license:

helm upgrade --install <schema-registry-release-name> \
  --values $VALUES_FILE \
  --set schemaregistry.enabled=true \
  ./confluent-operator

Specify Confluent Platform version¶

Confluent Platform ships as Docker images, with one image per Confluent Platform component. Each Confluent Platform release has a version number, and the Docker images for that release are put on Confluent’s Docker Hub with the version as the tag.

Confluent Operator allows you to specify the Confluent Platform version to deploy. Each Operator release supports specific Confluent Platform versions as described in Supported Versions.

When configuring for deployment, for each component, in the configuration file ($VALUES_FILE), specify the Confluent Platform version to deploy:

global:
  initContainer:
    image:
      tag:

<component>:
  image:
    tag:

For example, to deploy Kafka 6.0.1:

global:
  initContainer:
    image:
      tag: 6.0.1.0

kafka:
  image:
    tag: 6.0.1.0

Custom Docker registry¶

By default, Confluent Operator deploys publicly-available Docker images hosted on Docker Hub from the confluentinc repositories.

To have Operator deploy images from a different registry, or from your own set of repositories on Docker Hub, follow the steps in this section.

Pull the desired images from the Confluent repositories and push them to your target registry and/or repositories.

The default list of Docker image URIs are as follows with the <tag> replaced with the corresponding tags from the IMAGES file in the Operator download directory.

docker.io/confluentinc/cp-server-connect-operator:<tag>
docker.io/confluentinc/cp-enterprise-control-center-operator:<tag>
docker.io/confluentinc/cp-server-operator:<tag>
docker.io/confluentinc/cp-operator-service:<tag>
docker.io/confluentinc/cp-ksqldb-server-operator:<tag>
docker.io/confluentinc/cp-enterprise-replicator-operator:<tag>
docker.io/confluentinc/cp-schema-registry-operator:<tag>
docker.io/confluentinc/cp-zookeeper-operator:<tag>
docker.io/confluentinc/cp-init-container-operator:<tag>

Configure the following settings in your configuration file ($VALUES_FILE):
```
global:
  provider:
    registry:
      fqdn:                 ----- [1]
      credential:
        required:           ----- [2]
        username:           ----- [3]
        password:           ----- [4]
```
- [1] This is the FQDN (fully-qualified domain name) of your Docker registry, e.g. docker.io.
- [2] Set this to true if authentication is required to access the registry/repositories where you pushed the Confluent Docker images; set it to false otherwise.
- [3] If you set true in [2], set this to your Docker registry username.
- [4] If you set true in [2], set this to your Docker registry password.
If you set global.provider.registry.credential.required: true, take the additional steps to configure Kubernetes Secret as described in the following section.

Additional steps for Docker registry authentication¶

If you set global.provider.registry.credential.required to true, the username and password values will be used to create a Kubernetes Secret object named confluent-docker-registry when you deploy Operator, in the same namespace where you deployed Operator. You need to take one or more additional steps so that Kubernetes can successfully use this Secret to pull the necessary images when deploying the remaining Confluent Platform components. This section describes several common scenarios.

Scenario 1: You deploy all Confluent Platform components to the same namespace as Operator, and all Confluent Platform component pods will have the namespace’s default Service Account associated to it.

In this scenario, associate the Secret mentioned above to the default Service Account using the following command:
```
kubectl -n <namespace> patch serviceaccount default -p \
  '{"imagePullSecrets": [{"name": "confluent-docker-registry" }]}'
```
Scenario 2: You deploy all Confluent Platform components to the same namespace as Operator, and associate different Service Accounts with different Confluent Platform component clusters that you manage with Operator.

In this scenario, for each additional Service Account you plan to use, you need to associate it to the confluent-docker-registry Secret using the following command:
```
kubectl -n <namespace> patch serviceaccount <serviceAccountName> -p \
  '{"imagePullSecrets": [{"name": "confluent-docker-registry" }]}'
```
Scenario 3: You deploy Confluent Platform components to different namespaces than the namespace of Operator.

In this scenario, create a copy of the confluent-docker-registry Secret object in each of those namespaces using the following command:
```
kubectl get secret confluent-docker-registry -n <operatorNamespace> -o yaml | \
  kubectl apply -n <newNamespace> -f -
```
Then, associate the confluent-docker-registry Secret in the <newNamespace> to any Service Accounts in that namespace that you plan to associate to your Confluent Platform component clusters.

Custom registry for a Confluent Platform component¶

To have Operator deploy one specific Confluent Platform component image from a different registry, follow the steps in this section.

Pull the desired component image and the init container image (cp-init-container-operator) from the Confluent repository and push them to your target registry.

The component image and the init container image must be in the same repository.

Deploy the component image using the following command:

helm upgrade --install \
  <component-helm-release-name> \
  ./confluent-operator \
  --values $VALUES_FILE \
  --namespace <namespace> \
  --set <component-helm-release-name>.enabled=true \
  --set <component-helm-release-name>.name=connect \
  --set global.provider.registry.fqdn=<custom-registry-endpoint> \
  --set global.provider.registry.credential.required=<true or false> \
  --set global.provider.registry.credential.username=<value if required> \
  --set global.provider.registry.credential.password=<value if required> \
  --set connect.image.repository=<custom-registry>/<custom-repository> \
  --set connect.image.tag=<custom-tag> \
  --set global.initContainer.image.repository=<mypath>/<name> \
  --set global.initContainer.image.tag=<custom-tag>

Namespaced deployment¶

By default, Confluent Operator deploys Confluent Platform across all namespaces. If you want a Confluent Platform deployed to one namespace where it only reconciles the objects in that namespace, enable a namespaced deployment.

With a namespaced deployment, the Operator service can run without requiring access to cluster scoped Kubernetes resources. The Operator service only manages the resources within the namespace it is deployed to.

To enable a namespaced deployment of Confluent Operator, set the following in your configuration file ($VALUES_FILE):

operator:
  namespaced: true

Note

The previous step does not trigger Confluent Operator to automatically install the required cluster-level CustomResourceDefinitions (CRDs). You need to install the CRDs as a separate step prior to deploying Operator and Confluent Platform. See Grant permissions for namespaced deployment for instructions.

See Grant permissions for namespaced deployment for the permissions required for namespaced deployment of Operator and Confluent Platform.

Cluster-wide deployment with pre-created roles and role bindings¶

The following options are available to use ClusterRoleBinding with Confluent Operator:

Confluent Operator Helm charts create the required roles and role binding during the Operator install. This is the default behavior.
Kubernetes admin creates the ClusterRoles and ClusterRoleBinding, and the Confluent Platform admin uses those when deploying Operator.

To use the ClusterRoleBinding set up by Kubernetes admin:

Your Kubernetes cluster admin sets up ClusterRoleBinding.

Make sure that the roleRef in ClusterRoleBinding is the name of the ClusterRole.
Set the following in your Operator configuration file ($VALUES_FILE):
```
operator:
  installClusterResources: false
  namespaced: false
```

Modify default component settings¶

The global configuration file ($VALUES_FILE) contains the global configuration parameters. The values.yaml file contains additional component specific parameters you can add to your configuration. The values.yaml files also contain detailed comments that describe each configuration parameter and how to use it. The table below lists each of the values.yaml files and the location under the Confluent Operator home directory.

Component	Chart Name	values.yaml path
Operator	operator	helm/confluent-operator/charts/operator/values.yaml
Kafka	kafka	helm/confluent-operator/charts/kafka/values.yaml
ZooKeeper	zookeeper	helm/confluent-operator/charts/zookeeper/values.yaml
Connect	connect	helm/confluent-operator/charts/connect/values.yaml
Schema Registry	schemaregistry	helm/confluent-operator/charts/schemaregistry/values.yaml
Control Center	controlcenter	helm/confluent-operator/charts/controlcenter/values.yaml
Replicator	replicator	helm/confluent-operator/charts/replicator/values.yaml
ksqlDB	ksql	helm/confluent-operator/charts/ksql/values.yaml

Important

You should not modify a component values.yaml file. When you need to use or modify a component configuration parameter, add it to or change it in the the global configuration file ($VALUES_FILE). The global provider file overrides other values.yaml files when you install and when you upgrade a component configuration.

Complete the following steps to make component configuration changes:

Find the configuration parameter block in the values.yaml file that you want to use.
Copy the configuration parameter into the correct location in the global configuration file ($VALUES_FILE) and make the required changes.

Enter the following upgrade command:

helm upgrade --install \
  --values $VALUES_FILE \
  --set <component>.enabled=true \
  <component> \
  ./confluent-operator

For example, to change a Kafka configuration parameter, you enter the following upgrade command after saving your configuration changes in the $VALUES_FILE file.

helm upgrade --install \
  --values $VALUES_FILE \
  --set kafka.enabled=true \
  kafka \
  ./confluent-operator

Container memory and CPU settings¶

The default parameters in the the global configuration file ($VALUES_FILE) specifies pod resources needed. If you are testing Confluent Operator and Confluent Platform, your resource requirements may not be as great as the default values shown. However, ZooKeeper and Kafka must be installed on individual pods on individual nodes.

Important

At least three Kafka brokers are required for a fully functioning Confluent Platform deployment. A one- or two-broker configuration is not supported and should not be used for development testing or production.

Confluent Operator can define pod resource limits for all Confluent Platform components it deploys. You can define these settings using the requests and limits tags for each component in their values.yaml file.

The following example shows the default pod resource parameters in a global configuration file snippet for Kafka. See Managing Compute Resources for Containers for more details.

## Kafka Cluster
##
kafka:
  name: kafka
  replicas: 3
  resources:
    ## It is recommended to set both resource requests and limits.
    ## If not configured, kubernetes will set cpu/memory defaults.
    ## Reference: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
    requests:
      cpu: 200m
      memory: 1Gi
    limits: {}

Kafka and ZooKeeper configuration overrides¶

You can override default Kafka and ZooKeeper configuration parameters using the following keys in your configuration file ($VALUES_FILE).

zookeeper:
  configOverrides:
    server: []
    jvm: []
    log4j: []
kafka:
  configOverrides:
    server: []

You can only change settings for configuration parameters that are key/value pairs. Configuration parameters with only a key can’t be modified using the override. Refer to the following Confluent documents for configuration parameters used in Kafka and ZooKeeper:

Kafka: Broker Configurations
ZooKeeper: Configuration Options

The following example for Kafka changes:

The value for auto.create.topics.enable from false (default value) to true.
The log-level from INFO to DEBUG.

kafka:
  configOverrides:
    server:
      - auto.create.topics.enable=true
    jvm: []
    log4j:
      - log4j.rootLogger=DEBUG, stdout

Then, run the upgrade command. For example, if you are adding this override for Kafka you enter the following command:

helm upgrade --install \
  --values $VALUES_FILE \
  --set kafka.enabled=true \
  kafka \
  ./confluent-operator

Kafka inter broker version override¶

Operator sets the Kafka inter-broker protocol version to 2.3. This default version may not match the Kafka broker version of the corresponding Confluent Platform.

To use the Kafka features available in later versions of Kafka, such as Tiered Storage or Self-Balancing Clusters, use the Kafka config override to set the inter.broker.protocol.version and log.message.format.version properties to a higher version and roll the cluster.

For the inter-broker version to use for a specific version of Kafka, see Upgrade Confluent Platform.

The following snippet in the Operator configuration file ($VALUES_FILES) sets Kafka inter-broker protocol version to 2.4 that is required to enable Tiered Storage.

kafka:
  configOverrides:
    server:
      inter.broker.protocol.version=2.4
      log.message.format.version=2.4

Schema validation¶

You use a configuration override in Kafka to set up schema validation. The following example shows HTTP endpoint override for an example release name schemaregistry on namespace operator.

kafka:
  configOverrides:
    server:
      - confluent.schema.registry.url=http://schemaregistry.operator.svc.cluster.local:8081

If Schema Registry is deployed using a secure HTTPS endpoint, use the following configuration override:

kafka:
  configOverrides:
    server:
      - confluent.schema.registry.url=https://<domain>:8081

OR (using the same example names as above):

kafka:
  configOverrides:
    server:
      - confluent.schema.registry.url=http://schemaregistry.operator.svc.cluster.local:9081

Note

You can view the <domain> name by running helm status <release-name>.

See Schema Validation in the Schema Registry documentation for additional details.

Provide Service Account¶

Each pod running in Kubernetes has an associated Service Account. This applies to Confluent Platform component pods deployed by Confluent Operator as well.

Service Accounts serve several functions in the context of Confluent Operator deployments. For example, in order to deploy Confluent Platform Docker images from a private Docker image registry, you can associate Docker registry credentials to the Service Account associated with your Confluent Platform components.

You do not need to specify a Service Account when deploying Confluent Platform; if you do not specify a Service Account, Operator associates the default Service Account in the Kubernetes namespace where your component is being deployed.

To provide a custom Service Account, add serviceAccountName in the component section of the configuration file ($VALUES_FILE):

<component>:
  serviceAccountName:

Provide Mounted Secrets¶

When installing Confluent Platform components via Helm, the Helm chart securely stores sensitive credential configuration values you provide as Kubernetes Secrets. However, there are cases where you may wish to provide your own pre-existing Secrets to be used in the configuration of Confluent Platform, rather than having the Helm chart create the Secret object for you. The most common scenario for this is when providing custom configuration overrides, which the Helm chart treats as passthrough configuration and does not consider whether that data belongs in a Secret object or not.

To utilize your own Secret objects, you can specify a list of Secret objects to mount in-memory into the containers of a Confluent Platform component. Combined with the Externalizing Secrets capability of Confluent Platform and custom configuration overrides, you can configure Confluent Platform with your Secret data so that you store the information in an external file and pass the content of the a key file without exposing the sensitive information.

Note

Custom configuration overrides are currently only supported for ZooKeeper and Kafka.

To provide and use Mounted Secrets, add the following in the component (namely Kafka or ZooKeeper) section of the configuration file ($VALUES_FILE):

<component>:
  mountedSecrets:
  - secretRef:                                               ----- [1]
    keyItems:                                                ----- [2]
    - key:                                                   ----- [3]
      path:                                                  ----- [4]

  configOverrides:
    server:
      - some.property=${file:/mnt/secrets/<secretRef>:<key>} ----- [5]

[1] Set secretRef to the name of your Secret.
[2] Optional. If keyItems are not specified, Secrets are stored under /mnt/secrets/<secretRef> where <secretRef> is the value set in [1].

For using Secrets in Kubernetes, see Projection of Secret keys to specific paths.
[3] Set key to the Secret key.
[4] Set path to where the Secret key ([3]) is stored.

For example, if secretRef is set to my-secret-aws, key is set to credentials, and path is set to aws/creds, the credentials are stored in /mnt/secrets/my-secret-aws/aws/creds.
[5] Set the externalized file in the ${file:/mnt/secrets/<secretRef>:<key>} format.

Following is an example workflow of using Mounted Secrets.

Start with a file, my_file.txt, that contains your sensitive data:
```
username: user1
password: $ecret_passw0rd
```
Create a Secret object, example-secret, which stores the content of the above file in a key my_credentials:
```
kubectl create secret generic example-secret \
  --from-file=my_credentials=path/to/my_file.txt
```
The contents of my_file.txt will be Base64-encoded and stored under the my_credentials key of example-secret.

Set the configOverrides and mountedSecrets in the configuration file ($VALUES_FILE):

kafka:
  configOverrides:
    server:
      - kafka.some.user=${file:/mnt/secrets/example-secret/my_credentials:username}
      - kafka.some.password=${file:/mnt/secrets/example-secret/my_credentials:password}

  mountedSecrets:
    - secretRef: example-secret

Deploy or update your Kafka cluster:

helm upgrade --install kafka path/to/chart \
  -values $VALUES_FILE \
  --set kafka.enabled=true

Pod annotations¶

You can define custom annotations for Confluent Platform components deployed through Confluent Operator. Those annotations are applied to the Kubernetes pods created by the Operator.

In your configuration file ($VALUES_FILE), under each component, set podAnnotations.

podAnnotations must be a map with string keys and string values. The values cannot be any other types, such as numbers or booleans.

In general, annotation values must pass Kubernetes annotations validation. See the Kubernetes documentation on Annotations for details.

The following are example specifications of podAnnotations for several Confluent Platform components:

zookeeper:
  podAnnotations:
    string: "value"
    number: "1"
    boolean: "true"
    list: "[{\"labels\": {\"key\": \"value\"}},{\"key1\": \"value1\"}]"

kafka:
  podAnnotations:
    key1: "value1"
    key2: "value2"

controlcenter:  # ...
  podAnnotations:
    key1: "value3"
    key2: "value4"

Affinity¶

Operator supports Kubernetes affinity features for Confluent Platform clusters.

Note

The following settings used for the previous implementation of affinity are are still available, but they’re deprecated and will be removed in a future release of the Operator: disableHostPort, nodeAffinity, rack.

For all types of affinity, you can specify multiple rules of the same type, PREFERRED or REQUIRED, but you cannot specify both the PREFERRED and REQUIRED rules. PREFERRED is the default rule type.

Node affinity¶

Node affinity allows you to specify which Kubernetes nodes your Confluent Platform pod is eligible to be scheduled on. For example, if you have special hardware that you want to use to run one or more pods, use node affinity to pin those pods to the special hardware nodes.

The node affinity features are based on the labels on Kubernetes nodes. You can use Kubernetes built-in node labels or add your own node labels.

affinity:
  nodeAffinity:
    key:            ----- [1]
    values:
    -               ----- [2]
    rule:           ----- [3]

[1] The key of a node label.
[2] The values of the node label in [1].
[3] Rule type. Specify PREFERRED or REQUIRED. PREFERRED is the default.

The following example ensures that the cluster is in the uscentral1-a zone (the node label with the failure-domain.beta.kubernetes.io/zone key and the value of uscentral1-a).

affinity:
  nodeAffinity:
    key: failure-domain.beta.kubernetes.io/zone
    values:
      - uscentral1-a

Pod affinity and pod anti-affinity¶

Pod affinity allows you to specify the nodes that your Confluent Platform pod is eligible to be scheduled on, based on labels on pods that are already running on the node.

The affinity rules apply at the namespace level.

Specify the following in the configuration file ($VALUES_FILE):

affinity:
  podAntiAffinity:
    rule:           ----- [1]
    terms:
    - key:          ----- [2]
      values:       ----- [3]
      -
      -
      namespaces:
      -             ----- [4]
      topologyKey:
        key:        ----- [5]
      weight:
        weight:     ----- [6]

[1] Optional. Rule type. Specify PREFERRED or REQUIRED. PREFERRED is the default.
[2] A key of a pod label Operator injects. Use the pod labels that Operator provides:
[3] Values of the pod label in [2].

Following are the label keys that Operator provides and the example values you can use for each label key:
- namespace: operator
- type: zookeeper
[4] Optional. The namespace this rule applies to.
[5] Optional. If not provided, the default is kubernetes.io/hostname. For the description, see Affinity in Kubernetes.
[6] Optional. Valid values are from 1 to 100. For the description, see Affinity in Kubernetes.

The following snippet specifies a rule that this cluster is not scheduled on nodes which already host pods of the zookeeper label.

affinity:
  podAntiAffinity:
    terms:
      - key: type
        values:
        - zookeeper

One replica per node for ZooKeeper and Kafka¶

You can configure Operator to enforce only one Kafka or ZooKeeper replica to run on one Kuebernetes node. This rule applies at the namespace level and only to the replicas from the same cluster.

oneReplicaPerNode replaces disableHostPort at the namespace-level. Both oneReplicaPerNode and disableHostPort default to true.

Note

In 5.5.x and older versions, disableHostPort defaulted to false for ZooKeeper and Kafka.

Use oneReplicaPerNode in the Kafka and the ZooKeeper section of the configuration file ($VALUES_FILE) to enable or disable the feature.

The following example allows more than one ZooKeeper replica to run on one Kubernetes node:

zookeeper:
  oneReplicaPerNode: false

When oneReplicaPerNode is set to true, all pod anti-affinity rules are ignored. If you want to use pod anti-affinity rules, set oneReplicaPerNode to false.