Configure Confluent Operator and Confluent Platform

The following is the process of configuring Confluent Operator and Confluent Platform:

  1. Create the global configuration file.
  2. Configure the global infrastructure settings.
  3. Optionally configure the Confluent Platform component level settings.

The examples in this guide use the following assumptions:

  • $VALUES_FILE refers to the configuration file you set up in Create the global configuration file.

  • To present simple and clear examples in the Operator documentation, all the configuration parameters are specified in the config file ($VALUES_FILE). However, in your production deployments, use the --set or --set-file option when applying sensitive data with Helm. For example:

    helm upgrade --install kafka \
     --set kafka.services.mds.ldap.authentication.simple.principal=”cn=mds,dc=test,dc=com” \
     --set kafka.services.mds.ldap.authentication.simple.credentials=”Developer!” \
     --set kafka.enabled=true
    
  • operator is the namespace that Confluent Platform is deployed in.

  • All commands are executed in the helm directory under the directory Confluent Operator was downloaded to.

Configuration overview

During installation, Confluent Operator and Confluent Platform components are created based on parameters stored in multiple Helm Chart values.yaml files (one for Operator and one for each Confluent Platform component) and the global configuration file.

Do not modify parameters in the individual component values.yaml files. If you need to adjust capacity, add a parameter, or change a parameter for a component, you modify the component section in the global configuration file. You can also adjust configuration parameters after installation using the helm upgrade command.

The global configuration file is layered over the values.yaml files at installation and contains values that are specific to environments and that can be modified by you prior to installation.

After you download the Helm bundle you’ll see that:

  • The values.yaml file for each Confluent Platform component is stored in helm/confluent-operator/charts/<component>/.
  • The values.yaml file for Confluent Operator is stored in helm/confluent-operator/.
  • The <provider>.yaml file for each provider is stored in helm/providers/.

At installation, Helm reads the values files in the following layered order:

  1. The values.yaml for the Confluent Platform component is read.
  2. The values.yaml for Operator is read.
  3. The global configuration file is read.

Create the global configuration file

To customize the default configuration file:

  1. Go to the helm/providers directory under the directory where you downloaded the Confluent Operator bundle.

  2. Make a copy of the provider file corresponding to your provider environment. For example, copy gcp.yaml to my-values.yaml if your provider is Google Cloud.

  3. Set an environment variable pointing to your copy of the configuration file. For example:

    export VALUES_FILE="/path/to/my-values.yaml"
    

    The remainder of this topic uses $VALUES_FILE to refer to the global configuration file.

Configure the global settings

The following are the configuration changes necessary for the initial deployment of Confluent Operator and Confluent Platform. You specify the settings in the configuration file ($VALUES_FILE) you created above.

You can manually update the the configuration file after deployment if necessary. See Updating configuration for detail.

  1. Validate or change your region, and zone or zones (if your Kubernetes cluster spans multiple availability zones).
  2. Specify the storage class.
  3. Validate or change the Docker image registry endpoint.
  4. Configure DNS and load balancers for access to Confluent Platform from outside Kubernetes.
  5. Validate or configure network encryption, authentication, and authorization.
  6. Validate or configure your license key.

Add a license key

You can use Confluent Operator, Kafka, and Confluent Control Center for a 30-day trial period without a license key. After 30 days, Operator, Kafka, and Control Center require license keys.

Tip

For complete license information for Confluent Platform, see Confluent Platform Licenses.

To configure Confluent Platform with RBAC, Schema Registry requires a license key.

Add the license key to your configuration file ($VALUES_FILE) and run the helm upgrade command to activate the license. The examples below assume you are using your configuration file ($VALUES_FILE) to update your configuration.

Operator license

Add the following section to the Operator block in your configuration file ($VALUES_FILE) file and specify a license key:

operator:
  licenseKey:

Run the following command to activate the license:

helm upgrade --install <operator-release-name> \
  --values $VALUES_FILE \
  --set operator.enabled=true \
  ./confluent-operator

Kafka license

Add the following section to the Kafka block in your configuration file ($VALUES_FILE) file and specify a license key:

kafka:
  license:

Run the following command to activate the license:

helm upgrade --install <kafka-release-name> \
  --values $VALUES_FILE \
  --set kafka.enabled=true \
  ./confluent-operator

Confluent Control Center license

Add the following section to the Confluent Control Center block in your configuration file ($VALUES_FILE) file and specify a license key:

controlcenter:
  license:

Run the following command to activate the license:

helm upgrade --install <controlcenter-release-name> \
  --values $VALUES_FILE \
  --set controlcenter.enabled=true \
  ./confluent-operator

Schema Registry license

Add the following section to the Schema Registry block in your configuration file ($VALUES_FILE) file and specify a license key:

schemaregistry:
  license:

Run the following command to activate the license:

helm upgrade --install <schema-registry-release-name> \
  --values $VALUES_FILE \
  --set schemaregistry.enabled=true \
  ./confluent-operator

Custom Docker registry

The default Confluent Platform image registry is Docker Hub. If you are using a private image registry, specify the registry endpoint and the container image name in the configuration file.

The following example shows the default public image registry for container images. If you are installing from images downloaded from Docker Hub and then moved to a separate image registry, you must enter your image registry’s FQDN.

If the registry you use requires basic authentication, you need to change the credential parameter to required: true and enter a username and password.

## Docker registry endpoint where Confluent Images are available.
##
registry:
  fqdn: docker.io
  credential:
    required: false
    username:
    password:

Namespaced deployment

By default, Confluent Operator deploys Confluent Platform across all namespaces. If you want a Confluent Platform deployed to one namespace where it only reconciles the objects in that namespace, enable a namespaced deployment.

With a namespaced deployment, the Operator service can run without requiring access to cluster scoped Kubernetes resources. The Operator service only manages the resources within the namespace it is deployed to.

To enable a namespaced deployment of Confluent Operator, set the following in your configuration file ($VALUES_FILE):

operator:
  namespaced: true

The previous step does not trigger Confluent Operator to automatically install the required cluster-level CustomResourceDefinitions (CRDs). You need to install the CRDs as a separate step. See Install Custom Resource Definitions (CRDs) for instructions.

Cluster permissions for namespaced deployment

To view the required cluster permissions for a namespaced deployment, run the following command.

helm template <release-name> \
  <path-to-chart> \
  --values <path-to-values-file> \
  --namespace <namespace> \
  --set operator.enabled=true \
  --set operator.namespaced=true \
  --show-only charts/operator/templates/clusterrole.yaml

For example:

helm template operator \
  ./confluent-operator/ \
  --values $VALUES_FILE \
  --namespace operator \
  --set operator.enabled=true \
  --set operator.namespaced=true \
  --show-only charts/operator/templates/clusterrole.yaml

Cluster-wide deployment

By default, Confluent Operator deploys Confluent Platform cluster-wide, across all namespaces. If you want Confluent Operator to manage Confluent Platform components across all namespaces, but you don’t want the user who installs Confluent Operator to need permissions to manage cluster-level resources, you can create the ClusterRole and ClusterRoleBindings needed by Confluent Operator.

The following options are available to use ClusterRoleBinding with Confluent Operator:

  • Confluent Operator Helm charts create the required roles and role binding during the Operator install.
  • Kubernetes admin creates the ClusterRoles and ClusterRoleBinding, and the Confluent Platform admin then uses those when deploying Operator.

ClusterRoleBinding created by Confluent Operator Helm charts

To have Confluent Operator Helm charts create the cluster roles and cluster role bindings, set operator.installClusterResources and operator.namespaced as shown below in your Operator configuration file ($VALUES_FILE):

operator:
  installClusterResources: true
  namespaced: false

ClusterRoleBinding created by Kubernetes admin

To use the ClusterRoleBinding set up by Kubernetes admin:

  1. Your Kubernetes cluster admin sets up ClusterRoleBinding.

    Make sure that the roleRef in ClusterRoleBinding is the name of the ClusterRole.

  2. Set the following in your Operator configuration file ($VALUES_FILE):

    operator:
      installClusterResources: false
      namespaced: false
    

Modify default component settings

The global configuration file ($VALUES_FILE) contains the global configuration parameters. The values.yaml file contains additional component specific parameters you can add to your configuration. The values.yaml files also contain detailed comments that describe each configuration parameter and how to use it. The table below lists each of the values.yaml files and the location under the Confluent Operator home directory.

Component Chart Name values.yaml path
Operator operator helm/confluent-operator/charts/operator/values.yaml
Kafka kafka helm/confluent-operator/charts/kafka/values.yaml
ZooKeeper zookeeper helm/confluent-operator/charts/zookeeper/values.yaml
Connect connect helm/confluent-operator/charts/connect/values.yaml
Schema Registry schemaregistry helm/confluent-operator/charts/schemaregistry/values.yaml
Control Center controlcenter helm/confluent-operator/charts/controlcenter/values.yaml
Replicator replicator helm/confluent-operator/charts/replicator/values.yaml
ksqlDB ksql helm/confluent-operator/charts/ksql/values.yaml

Important

You should not modify a component values.yaml file. When you need to use or modify a component configuration parameter, add it to or change it in the the global configuration file ($VALUES_FILE). The global provider file overrides other values.yaml files when you install and when you upgrade a component configuration.

Complete the following steps to make component configuration changes:

  1. Find the configuration parameter block in the values.yaml file that you want to use.

  2. Copy the configuration parameter into the correct location in the global configuration file ($VALUES_FILE) and make the required changes.

  3. Enter the following upgrade command:

    helm upgrade --install \
      --values $VALUES_FILE \
      --set <component>.enabled=true \
      <component> \
      ./confluent-operator
    

    For example, to change a Kafka configuration parameter, you enter the following upgrade command after saving your configuration changes in the $VALUES_FILE file.

    helm upgrade --install \
      --values $VALUES_FILE \
      --set kafka.enabled=true \
      kafka \
      ./confluent-operator
    

Container memory and CPU settings

The default parameters in the the global configuration file ($VALUES_FILE) specifies pod resources needed. If you are testing Confluent Operator and Confluent Platform, your resource requirements may not be as great as the default values shown. However, ZooKeeper and Kafka must be installed on individual pods on individual nodes.

Important

At least three Kafka brokers are required for a fully functioning Confluent Platform deployment. A one- or two-broker configuration is not supported and should not be used for development testing or production.

Confluent Operator can define pod resource limits for all Confluent Platform components it deploys. You can define these settings using the requests and limits tags for each component in their values.yaml file.

The following example shows the default pod resource parameters in a global configuration file snippet for Kafka. See Managing Compute Resources for Containers for more details.

## Kafka Cluster
##
kafka:
  name: kafka
  replicas: 3
  resources:
    ## It is recommended to set both resource requests and limits.
    ## If not configured, kubernetes will set cpu/memory defaults.
    ## Reference: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
    requests:
      cpu: 200m
      memory: 1Gi
    limits: {}

Kafka and ZooKeeper configuration overrides

You can override default Kafka and ZooKeeper configuration parameters using the following keys in your configuration file ($VALUES_FILE).

zookeeper:
  configOverrides:
    server: []
    jvm: []
    log4j: []
kafka:
  configOverrides:
    server: []

You can only change settings for configuration parameters that are key/value pairs. Configuration parameters with only a key can’t be modified using the override. Refer to the following Confluent documents for configuration parameters used in Kafka and ZooKeeper:

The following example for Kafka changes:

  • The value for auto.create.topics.enable from false (default value) to true.
  • The log-level from INFO to DEBUG.
kafka:
  configOverrides:
    server:
      - auto.create.topics.enable=true
    jvm: []
    log4j:
      - log4j.rootLogger=DEBUG, stdout

Then, run the upgrade command. For example, if you are adding this override for Kafka you enter the following command:

helm upgrade --install \
  --values $VALUES_FILE \
  --set kafka.enabled=true \
  kafka \
  ./confluent-operator

Schema validation

You use a configuration override in Kafka to set up schema validation. The following example shows HTTP endpoint override for an example release name schemaregistry on namespace operator.

kafka:
  configOverrides:
    server:
      - confluent.schema.registry.url=http://schemaregistry.operator.svc.cluster.local:8081

If Schema Registry is deployed using a secure HTTPS endpoint, use the following configuration override:

kafka:
  configOverrides:
    server:
      - confluent.schema.registry.url=https://<domain>:8081

OR (using the same example names as above):

kafka:
  configOverrides:
    server:
      - confluent.schema.registry.url=http://schemaregistry.operator.svc.cluster.local:9081

Note

You can view the <domain> name by running helm status <release-name>.

See Schema Validation in the Schema Registry documentation for additional details.

Provide Service Account

Each pod running in Kubernetes has an associated Service Account. This applies to Confluent Platform component pods deployed by Confluent Operator as well.

Service Accounts serve several functions in the context of Confluent Operator deployments. For example, in order to deploy Confluent Platform Docker images from a private Docker image registry, you can associate Docker registry credentials to the Service Account associated with your Confluent Platform components.

You do not need to specify a Service Account when deploying Confluent Platform; if you do not specify a Service Account, Operator associates the default Service Account in the Kubernetes namespace where your component is being deployed.

To provide a custom Service Account, add serviceAccountName in the component section of the configuration file ($VALUES_FILE):

<component>:
  serviceAccountName:

Provide Mounted Secrets

When installing Confluent Platform components via Helm, the Helm chart securely stores sensitive credential configuration values you provide as Kubernetes Secrets. However, there are cases where you may wish to provide your own pre-existing Secrets to be used in the configuration of Confluent Platform, rather than having the Helm chart create the Secret object for you. The most common scenario for this is when providing custom configuration overrides, which the Helm chart treats as passthrough configuration and does not consider whether that data belongs in a Secret object or not.

To utilize your own Secret objects, you can specify a list of Secret objects to mount in-memory into the containers of a Confluent Platform component. Combined with the Externalizing Secrets capability of Confluent Platform and custom configuration overrides, you can configure Confluent Platform with your Secret data so that you store the information in an external file and pass the content of the a key file without exposing the sensitive information.

Note

Custom configuration overrides are currently only supported for ZooKeeper and Kafka.

To provide and use Mounted Secrets, add the following in the component (namely Kafka or ZooKeeper) section of the configuration file ($VALUES_FILE):

<component>:
  mountedSecrets:
  - secretRef:                                              ----- [1]
    keyItems:                                               ----- [2]
    - key:                                                  ----- [3]
      path:                                                 ----- [4]

  configOverrides:
    server:
      - some.property={file:/mnt/secrets/<secretRef>:<key>} ----- [5]
  • [1] Set secretRef to the name of your Secret.

  • [2] Optional. If keyItems are not specified, Secrets are stored under /mnt/secrets/<secretRef> where <secretRef> is the value set in [1].

    For using Secrets in Kubernetes, see Projection of Secret keys to specific paths.

  • [3] Set key to the Secret key.

  • [4] Set path to where the Secret key ([3]) is stored.

    For example, if secretRef is set to my-secret-aws, key is set to credentials, and path is set to aws/creds, the credentials are stored in /mnt/secrets/my-secret-aws/aws/creds.

  • [5] Set the externalized file in the {file:/mnt/secrets/<secretRef>:<key>} format.

Following is an example workflow of using Mounted Secrets.

  1. Start with a file, my_file.txt, that contains your sensitive data:

    username: user1
    password: $ecret_passw0rd
    
  2. Create a secret object, example-secret, which stores the content of the above file in a key my_credentials:

    kubectl create secret generic example-secret \
      --from-file=my_credentials=path/to/my_file.txt
    

    The contents of my_file.txt will be Base64-encoded and stored under the my_credentials key of example-secret.

  3. Set the configOverrides and mountedSecrets in the configuration file ($VALUES_FILE):

    kafka:
      configOverrides:
        server:
          - kafka.some.user={file:/mnt/secrets/example-secret/my_credentials:username}
          - kafka.some.password={file:/mnt/secrets/example-secret/my_credentials:password}
    
      mountedSecrets:
        - secretRef: example-secret
    
  4. Deploy or update your Kafka cluster:

    helm upgrade --install kafka path/to/chart \
      -values $VALUES_FILE \
      --set kafka.enabled=true
    

Pod annotations

You can define custom annotations for Confluent Platform components deployed through Confluent Operator. Those annotations are applied to the Kubernetes pods created by the Operator.

In your configuration file ($VALUES_FILE), under each component, set podAnnotations.

podAnnotations must be a map with string keys and string values. The values cannot be any other types, such as numbers or booleans.

In general, annotation values must pass Kubernetes annotations validation. See the Kubernetes documentation on Annotations for details.

The following are example specifications of podAnnotations for several Confluent Platform components:

zookeeper:
  podAnnotations:
    string: "value"
    number: "1"
    boolean: "true"
    list: "[{\"labels\": {\"key\": \"value\"}},{\"key1\": \"value1\"}]"

kafka:
  podAnnotations:
    key1: "value1"
    key2: "value2"

controlcenter:  # ...
  podAnnotations:
    key1: "value3"
    key2: "value4"

Affinity

Operator supports Kubernetes affinity features for Confluent Platform clusters.

Note

The following settings used for the previous implementation of affinity are are still available, but they’re deprecated and will be removed in a future release of the Operator: disableHostPort, nodeAffinity, rack.

For all types of affinity, you can specify multiple rules of the same type, PREFERRED or REQUIRED, but you cannot specify both the PREFERRED and REQUIRED rules. PREFERRED is the default rule type.

Node affinity

Node affinity allows you to specify which Kubernetes nodes your Confluent Platform pod is eligible to be scheduled on. For example, if you have special hardware that you want to use to run one or more pods, use node affinity to pin those pods to the special hardware nodes.

The node affinity features are based on the labels on Kubernetes nodes. You can use Kubernetes built-in node labels or add your own node labels.

affinity:
  nodeAffinity:
    key:            ----- [1]
    values:
    -               ----- [2]
    rule:           ----- [3]
  • [1] The key of a node label.
  • [2] The values of the node label in [1].
  • [3] Rule type. Specify PREFERRED or REQUIRED. PREFERRED is the default.

The following example ensures that the cluster is in the uscentral1-a zone (the node label with the failure-domain.beta.kubernetes.io/zone key and the value of uscentral1-a).

affinity:
  nodeAffinity:
    key: failure-domain.beta.kubernetes.io/zone
    values:
      - uscentral1-a

Pod affinity and pod anti-affinity

Pod affinity allows you to specify the nodes that your Confluent Platform pod is eligible to be scheduled on, based on labels on pods that are already running on the node.

The affinity rules apply at the namespace level.

Specify the following in the configuration file ($VALUES_FILE):

affinity:
  podAntiAffinity:
    rule:           ----- [1]
    terms:
    - key:          ----- [2]
      values:       ----- [3]
      -
      -
      namespaces:
      -             ----- [4]
      topologyKey:
        key:        ----- [5]
      weight:
        weight:     ----- [6]
  • [1] Optional. Rule type. Specify PREFERRED or REQUIRED. PREFERRED is the default.

  • [2] A key of a pod label Operator injects. Use the pod labels that Operator provides:

  • [3] Values of the pod label in [2].

    Following are the label keys that Operator provides and the example values you can use for each label key:

    • namespace: operator
    • type: zookeeper
  • [4] Optional. The namespace this rule applies to.

  • [5] Optional. If not provided, the default is kubernetes.io/hostname. For the description, see Affinity in Kubernetes.

  • [6] Optional. Valid values are from 1 to 100. For the description, see Affinity in Kubernetes.

The following snippet specifies a rule that this cluster is not scheduled on nodes which already host pods of the zookeeper label.

affinity:
  podAntiAffinity:
    terms:
      - key: type
        values:
        - zookeeper

One replica per node for ZooKeeper and Kafka

You can configure Operator to enforce only one Kafka or ZooKeeper replica to run on one Kuebernetes node. This rule applies at the namespace level and only to the replicas from the same cluster.

oneReplicaPerNode replaces disableHostPort at the namespace-level. Both oneReplicaPerNode and disableHostPort default to true.

Note

In 5.5.x and older versions, disableHostPort defaulted to false for ZooKeeper and Kafka.

Use oneReplicaPerNode in the Kafka and the ZooKeeper section of the configuration file ($VALUES_FILE) to enable or disable the feature.

The following example allows more than one ZooKeeper replica to run on one Kubernetes node:

zookeeper:
  oneReplicaPerNode: false