Install Confluent for Kubernetes Blueprints

A Confluent for Kubernetes (CFK) Blueprints deployment includes the following clusters:

  • Control Plane cluster: This is a Kubernetes cluster where you install the Orchestrator and Blueprints.

  • Data Plane cluster: This is a Kubernetes cluster where you install the Agent and Confluent Platform.

    The Orchestrator automatically creats the Confluent Platform cluster pods when you deploy Confluent component cluster custom resources (CRs) from the Control Plane.

    Data Plane can be deployed in the same Kubernetes cluster as Control Plane with a local Agent, or in a different cluster with a remote Agent.

You can have multiple Data Plane clusters registered with one Control Plane.

The high-level workflow to deploy Confluent Platform using CFK Blueprints is:

  1. Install the Orchestrator.
  2. Install the Agent and CFK, locally or remotely.
  3. Configure and deploy a Blueprint.
  4. Deploy Confluent Platform.

Install Orchestrator and CFK in Control Plane

As a platform administrator, you deploy the Orchestrator where you will add specific configurations for the Blueprints and Confluent Platform resources.

In the Kubernetes cluster you want to install the Orchestrator, take the following steps:

  1. Create the namespace for the CFK Blueprints system components. <cpc-system-namespace> will be used to denote this namespace in this document:

    kubectl create namespace <cpc-system-namespace>

  2. Install the Orchestrator Helm chart:

    helm repo add confluentinc https://packages.confluent.io/helm

    helm repo update

    helm upgrade --install confluent-orchestrator confluentinc/cfk-blueprint \
  --set orchestrator.enabled=true \
  --namespace <cpc-system-namespace>

  3. Install the CFK Helm chart to apply the CFK CRDs in the Control Plane cluster:

    helm upgrade --install confluent-operator confluentinc/cfk-blueprint \
  --set operator.enabled=true \
  --namespace <cpc-system-namespace>

Install local Agent in Control Plane

For the local deployment of Confluent Platform, install the Agent in the same Kubernetes cluster where the Orchestrator is installed:

  1. Install the Agent Helm chart in the Local mode:

    helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
 --set agent.mode=Local \
 --set agent.enabled=true \
 --namespace <cpc-system-namespace>

  2. Register the Agent Kubernetes cluster and the Health Check with the Orchestrator.

    1. Find the Kubernetes ID of the cluster:

      kubectl get namespace kube-system -oyaml | grep uid

    2. Create a KubernetesCluster CR and a HealthCheck CR, using the Kubernetes ID retrieved in the previous step.

      You can use this example file as a template and update the cluster id (k8sID) in the file.

      Every time when you add a new Agent, you need to create a unique set of KubernetesCluster CR and HealthCheck CR to register the new Agent.

      KubernetesCluster CR cannot be updated once created.

    3. Apply the changes to create the KubernetesCluster CR and CPHealthCheck CR, using the kubectl apply -f command.

Install remote Agent and CFK in Data Plane

To install Confluent Platform in a different Kubernetes cluster from the Orchestrator, install the Agent where you want to run Confluent Platform:

  1. Create the namespace for the CFK Blueprints system resources in the Data Plane. <cpc-system-namespace> will be used to denote this namespace in this document:

    kubectl create namespace <cpc-system-namespace>

  2. Generate a Kubeconfig and the Kubeconfig secret. The file is used for the Agents on the remote Data Plane cluster to get a token to connect and communicate to the Control Plane cluster.

    See CFK Blueprint example repo for an example script and example steps.

    You will use the Kubeconfig secret (<kubeconfig-secret>) in the next step.

  3. In the Data Plane, install the Agent Helm chart using the secret created in the previous step:

    helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
  --set agent.mode=Remote \
  --set agent.enabled=true \
  --set agent.remoteKubeConfig.secretRef=<kubeconfig-secret> \
  --kube-context <data-plane> \
  --namespace <cpc-system-namespace>

  4. In the Data Plane, install the CFK Helm chart in the cluster mode:

    helm upgrade --install confluent-operator confluentinc/cfk-blueprint \
  --kube-context <data-plane> \
  --set operator.enabled=true \
  --namespace <cpc-system-namespace>

  5. Register the Agent Kubernetes cluster and the Health Check with the Orchestrator.

    1. In the Data Plane cluster, get the Kubernetes ID:

      kubectl get namespace kube-system -oyaml --context <data-plane> | grep uid

    2. In the Control Plane, create a KubernetesCluster CR and a HealthCheck CR, using the Kubernetes ID retrieved in the previous step.

      You can use this example file as a template and update the cluster id (k8sID) in the file.

      Every time when you add a new Agent, you need to create a unique set of KubernetesCluster CR and HealthCheck CR to register the new Agent.

      KubernetesCluster CR cannot be updated once created.

    3. In the Control Plane, apply the changes to create the KubernetesCluster CR and CPHealthCheck CR, using the kubectl apply -f command.

Install customized CFK Blueprints

You can customize the configuration of CFK Blueprints when you install or update it. Using Helm, you can:

  • Set configuration values in a YAML file which you pass to helm commands via the --values or -f flag.
  • Set configuration values as key=value pairs directly in your helm commands using the --set, --set-string, and --set-file flags.
  • Combine the above two approaches, where you may set some configuration in a values file and others on the command line with the --set flags.

Refer to Helm documentation for more details on the Helm flags.

For customized installation, replace the standard installation helm upgrade commands in the above sections with the following helm upgrade commands, but refer to the above sections for the complete workflow.

Deploy customized CFK Blueprints using the values.yaml file

  1. Pull the CFK Blueprints Chart, and find the default values.yaml file:

    mkdir -p <CFK-home>

    helm pull confluentinc/cfk-blueprint \
  --untar \
  --untardir=<CFK-home> \
  --namespace <namespace>

    The values.yaml file is in the <CFK-home>/cfk-blueprint directory.

  2. Create a copy of the values.yaml file to customize CFK Blueprints configuration. Do not edit the default values.yaml file. Save your copy to any file location, <path-to-values-file>.

  3. Install CFK Blueprints using the customized configuration from the <CFK-home> directory:

    • For the customized Orchestrator:

      helm upgrade --install confluent-orchestrator \
  ./cfk-blueprint/charts/orchestrator \
  --values <path-to-values-file> \
  --namespace <namespace>

    • For the customized local Agent:

      helm upgrade --install confluent-orchestrator \
  ./cfk-blueprint/charts/agent \
  --values <path-to-values-file> \
  --set agent.mode=Local \
  --namespace <namespace>

    • For the customized remote Agent:

      helm upgrade --install confluent-orchestrator \
  ./cfk-blueprint/charts/agent \
  --values <path-to-values-file> \
  --set agent.mode=Remote \
  --namespace <namespace>

    • For the customized CFK:

      helm upgrade --install confluent-orchestrator \
  ./cfk-blueprint/charts/operator \
  --values <path-to-values-file> \
  --namespace <namespace>

Deploy customized CFK Blueprints using the helm upgrade command

Specify the configuration options using the --set flag:

  • For the customized Orchestrator:

    helm upgrade --install confluent-orchestrator confluentinc/cfk-blueprint \
  --set orchestrator.enabled=true \
  --set orchestrator.<setting1>=<value1> \
  --set orchestrator.<setting2>=<value2> \
  ...
  --namespace <namespace>

  • For the customized local Agent:

    helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
  --set agent.mode=Local \
  --set agent.enabled=true \
  --set agent.<setting1>=<value1> \
  --set agent.<setting2>=<value2> \
  ...
  --namespace <cpc-system-namespace>

  • For the customized remote Agent:

    helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
  --kube-context <data-plane> \
  --set agent.mode=Remote \
  --set agent.enabled=true \
  --set agent.remoteKubeConfig.secretRef=<kubeconfig-secret> \
  --set agent.<setting1>=<value1> \
  --set agent.<setting2>=<value2> \
  ...
  --namespace <cpc-system-namespace>

  • For the customized local CFK:

    helm upgrade --install confluent-operator confluentinc/cfk-blueprint \
  --set operator.enabled=true \
  --set operator.<setting1>=<value1> \
  --set operator.<setting2>=<value2> \
  ...
  --namespace <cpc-system-namespace>

  • For the customized remote CFK:

    helm upgrade --install confluent-operator confluentinc/cfk-blueprint \
  --kube-context <data-plane> \
  --set operator.enabled=true \
  --set operator.<setting1>=<value1> \
  --set operator.<setting2>=<value2> \
   ...
  --namespace <cpc-system-namespace>