Install Confluent for Kubernetes Blueprints

A Confluent for Kubernetes (CFK) Blueprints deployment includes the following clusters:

• Control Plane cluster: This is a Kubernetes cluster where you install the Orchestrator and Blueprints.

• Data Plane cluster: This is a Kubernetes cluster where you install the Agent and Confluent Platform.

  The Orchestrator automatically creats the Confluent Platform cluster pods when you deploy Confluent component cluster custom resources (CRs) from the Control Plane.

  Data Plane can be deployed in the same Kubernetes cluster as Control Plane with a local Agent, or in a different cluster with a remote Agent.

You can have multiple Data Plane clusters registered with one Control Plane.

The high-level workflow to deploy Confluent Platform using CFK Blueprints is:

1. Install the Orchestrator.
2. Install the Agent and CFK, locally or remotely.
3. Configure and deploy a Blueprint.
4. Deploy Confluent Platform.

Install Orchestrator in Control Plane

As a platform administrator, you deploy the Orchestrator where you will add specific configurations for the Blueprints and Confluent Platform resources.

In the Kubernetes cluster you want to install the Orchestrator, take the following steps:

1. Create the namespace for the CFK Blueprints system components. <cpc-system-namespace> will be used to denote this namespace in this document:

   kubectl create namespace <cpc-system-namespace>
   

2. Install the Orchestrator Helm chart:

   helm repo add confluentinc https://packages.confluent.io/helm
   

   helm repo update
   

   helm upgrade --install confluent-orchestrator confluentinc/cfk-blueprint \
     --set orchestrator.enabled=true \
     --namespace <cpc-system-namespace>
   

3. Install the CFK Helm chart to apply the CFK CRDs in the Control Plane cluster:

   helm upgrade --install confluent-operator confluentinc/cfk-blueprint \
     --set operator.enabled=true \
     --namespace <cpc-system-namespace>
   

Install local Agent in Control Plane

For the local deployment of Confluent Platform, install the Agent in the same Kubernetes cluster where the Orchestrator is installed:

1. Install the Agent Helm chart in the Local mode:

   helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
    --set agent.mode=Local \
    --set agent.enabled=true \
    --namespace <cpc-system-namespace>
   

2. Register the Agent Kubernetes cluster and the Health Check with the Orchestrator.

   1. Find the Kubernetes ID of the cluster:

      kubectl get namespace kube-system -oyaml | grep uid
      

   2. Create a KubernetesCluster CR and a HealthCheck CR, using the Kubernetes ID retrieved in the previous step.

      You can use this example file as a template and update the cluster id (k8sID) in the file.

      Every time when you add a new Agent, you need to create a unique set of KubernetesCluster CR and HealthCheck CR to register the new Agent.

      KubernetesCluster CR cannot be updated once created.

   3. Apply the changes to create the KubernetesCluster CR and CPHealthCheck CR, using the kubectl apply -f command.

Install remote Agent and CFK in Data Plane

To install Confluent Platform in a different Kubernetes cluster from the Orchestrator, install the Agent where you want to run Confluent Platform:

1. Create the namespace for the CFK Blueprints system resources in the Data Plane. <cpc-system-namespace> will be used to denote this namespace in this document:

   kubectl create namespace <cpc-system-namespace>
   

2. Generate a Kubeconfig and the Kubeconfig secret. The file is used for the Agents on the remote Data Plane cluster to get a token to connect and communicate to the Control Plane cluster.

   See CFK Blueprint example repo for an example script and example steps.

   You will use the Kubeconfig secret (<kubeconfig-secret>) in the next step.

3. In the Data Plane, install the Agent Helm chart using the secret created in the previous step:

   helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
     --set agent.mode=Remote \
     --set agent.enabled=true \
     --set agent.remoteKubeConfig.secretRef=<kubeconfig-secret> \
     --kube-context <data-plane> \
     --namespace <cpc-system-namespace>
   

4. In the Data Plane, install the CFK Helm chart in the cluster mode:

   helm upgrade --install confluent-operator confluentinc/cfk-blueprint \
     --kube-context <data-plane> \
     --set operator.enabled=true \
     --namespace <cpc-system-namespace>
   

5. Register the Agent Kubernetes cluster and the Health Check with the Orchestrator.

   1. In the Data Plane cluster, get the Kubernetes ID:

      kubectl get namespace kube-system -oyaml --context <data-plane> | grep uid
      

   2. In the Control Plane, create a KubernetesCluster CR and a HealthCheck CR, using the Kubernetes ID retrieved in the previous step.

      You can use this example file as a template and update the cluster id (k8sID) in the file.

      Every time when you add a new Agent, you need to create a unique set of KubernetesCluster CR and HealthCheck CR to register the new Agent.

      KubernetesCluster CR cannot be updated once created.

   3. In the Control Plane, apply the changes to create the KubernetesCluster CR and CPHealthCheck CR, using the kubectl apply -f command.