Confluent for Kubernetes Blueprints Quick Start

The quick start tutorials in this document describe how to deploy and run the basic configuration of Confluent for Kubernetes (CFK) Blueprints.

For more CFK Blueprints use cases, see the example GitHub repo.

Prepare

1. Install Helm 3 on your local machine.

2. Install kubectl command-line tool on your local machine.

3. Install cfssl on your local machine.

4. Have the Kubernetes clusters you want to use for the Control Plane and the Data Plane. Kubernetes versions 1.22+ are required.

5. Rename the Kubernetes contexts for easy identification:

   kubectl config rename-context <Kubernetes control plane context> control-plane
   

6. From the <CFK examples directory> on your local machine, clone the example repo:

   git clone git@github.com:confluentinc/confluent-kubernetes-examples.git
   

7. Set the tutorial directory for this tutorial:

   export TUTORIAL_HOME=<CFK examples directory>/blueprints/quickstart-deploy
   

Single-site Deployment using CFK Blueprints

This tutorial walks you through the configuration and deployment of CFK Blueprints in a single-site environment where the Control Plane and Data Plane run in the same cluster. For clarity, we will call out the Control Plane and Data Plane separately in this tutorial.

You will go through the following scenarios.

1. Deploy the Control Plane.
2. Deploy the local Data Plane in the same cluster as the Control Plane.
3. Deploy the Blueprint in the Control Plane.
4. Using the Blueprint, deploy Confluent Platform in the Data Plane.

Deploy Control Plane

In the Kubernetes cluster you want to install the Control Plane, take the following steps:

1. Create a namespace for the Blueprint system resources. These examples use: cpc-system

   kubectl create namespace cpc-system --context control-plane
   

2. Create a CA key pair to be used for the Blueprint and the Orchestrator:

   cat << EOF > openssl.cnf
   [req]
   distinguished_name=dn
   [ dn ]
   [ v3_ca ]
   basicConstraints = critical,CA:TRUE
   subjectKeyIdentifier = hash
   authorityKeyIdentifier = keyid:always,issuer:always
   EOF
   

   If /tmp does not exist in your local system, create the folder:

   mkdir /tmp
   

   Generate a CA key pair:

   openssl req -x509 -new -nodes -newkey rsa:4096 -keyout /tmp/cpc-ca-key.pem \
     -out /tmp/cpc-ca.pem \
     -subj "/C=US/ST=CA/L=MountainView/O=Confluent/OU=CPC/CN=CPC-CA" \
     -reqexts v3_ca \
     -config openssl.cnf
   

3. Install the Orchestrator Helm chart:

   helm repo add confluentinc https://packages.confluent.io/helm
   helm repo update
   

   helm upgrade --install confluent-orchestrator confluentinc/cfk-blueprint \
     --set orchestrator.enabled=true \
     --namespace cpc-system \
     --kube-context control-plane
   

Deploy Local Data Plane

For the local deployment, install the Data Plane in the same Kubernetes cluster where you installed the Control Plane.

1. Install the Agent Helm chart in Local mode:

   helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
    --set agent.mode=Local \
    --set agent.enabled=true \
    --namespace cpc-system \
    --kube-context control-plane
   

2. Register the Data Plane Kubernetes cluster.

   1. Get the Kubernetes ID:

      kubectl get namespace kube-system -oyaml | grep uid
      

   2. Edit $TUTORIAL_HOME/registration/control-plane-k8s.yaml and set spec.k8sID to the Kubernetes ID retrieved in the previous step.

   3. Create the KubernetesCluster custom resource (CR) and the HealthCheck CR in the Control Plane Kubernetes cluster:

      kubectl apply -f $TUTORIAL_HOME/registration/control-plane-k8s.yaml \
        --context control-plane --namespace cpc-system
      

   4. Verify that the Agent is up and running:

      kubectl get cpcHealthCheck \
        --context control-plane --namespace cpc-system
      

3. Install the CFK Helm chart in cluster mode (--set namespaced=false):

   helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes \
     --set namespaced="false" \
     --namespace cpc-system \
     --kube-context control-plane
   

Deploy Blueprint

Deploy the Blueprint and the Confluent cluster class CRs:

kubectl apply -f $TUTORIAL_HOME/deployment/confluentplatform_blueprint.yaml \
  --context control-plane --namespace cpc-system

Deploy Confluent Platform in Local Data Plane

1. Create a namespace for deploying the Confluent components. These examples use: org-confluent

   kubectl create namespace org-confluent --context control-plane
   

2. Deploy Confluent Platform:

   kubectl apply -f $TUTORIAL_HOME/deployment/control-plane/confluentplatform_prod.yaml \
     --namespace org-confluent \
     --context control-plane
   

3. Validate the deployment.

   1. Check that the Confluent components are up and running:

      kubectl get pods --namespace org-confluent --context control-plane -w
      

   2. Set up port forwarding to the Control Center web UI from the local machine:

      kubectl port-forward controlcenter-prod-0 9021:9021 --context control-plane --namespace org-confluent
      

   3. Navigate to Control Center in a browser and check the cluster:

      http://localhost:9021

4. Uninstall Confluent Platform:

   kubectl delete -f $TUTORIAL_HOME/deployment/control-plane/confluentplatform_prod.yaml \
     --namespace org-confluent \
     --context control-plane
   

Multi-site Deployment using CFK Blueprints

This tutorial walks you through the configuration and deployment of CFK Blueprints in a multi-cluster environment. You will go through the following scenarios:

1. Deploy the Control Plane.
2. Deploy the Data Plane in a separate cluster.
3. Deploy the Blueprint in the Control Plane.
4. Using the Blueprint, deploy Confluent Platform in the Data Plane.

Deploy Control Plane

In the Kubernetes cluster you want to install the Control Plane, take the following steps:

1. Set the current context to the Control Plane cluster:

   kubectl config use-context control-plane
   

2. Create a namespace for the Blueprint system resources. cpc-system is used in these examples:

   kubectl create namespace cpc-system --context control-plane
   

3. Create a CA key pair to be used for the Blueprint and the Orchestrator:

   cat << EOF > openssl.cnf
   [req]
   distinguished_name=dn
   [ dn ]
   [ v3_ca ]
   basicConstraints = critical,CA:TRUE
   subjectKeyIdentifier = hash
   authorityKeyIdentifier = keyid:always,issuer:always
   EOF
   

   If /tmp does not exist in your local system, create the folder:

   mkdir /tmp
   

   Generate a CA key pair:

   openssl req -x509 -new -nodes -newkey rsa:4096 -keyout /tmp/cpc-ca-key.pem \
     -out /tmp/cpc-ca.pem \
     -subj "/C=US/ST=CA/L=MountainView/O=Confluent/OU=CPC/CN=CPC-CA" \
     -reqexts v3_ca \
     -config openssl.cnf
   

4. Install the Orchestrator Helm chart:

   helm repo add confluentinc https://packages.confluent.io/helm
   helm repo update
   

   helm upgrade --install cpc-orchestrator confluentinc/cfk-blueprint \
     --set orchestrator.enabled=true \
     --namespace cpc-system \
     --kube-context control-plane
   

5. Install the CFK Helm chart in the Control Plane:

   helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes \
     --set namespaced="false" \
     --kube-context control-plane \
     --namespace cpc-system
   

Deploy Remote Data Plane

In the remote deployment mode, the Data Plane is installed in a different Kubernetes cluster from the Control Plane cluster.

1. Rename the Kubernetes context for easy identification:

   kubectl config rename-context <Kubernetes data plane context> data-plane
   

2. Create a namespace for the Blueprint system resources. cpc-system is used in these examples:

   kubectl create namespace cpc-system --context data-plane
   

3. In the Control Plane, generate the Kubeconfig for the Agent to communicate with the Orchestrator:

   kubectl config use-context control-plane
   

   $TUTORIAL_HOME/scripts/kubeconfig_generate.sh \
     --name control-plane-sa \
     --namespace cpc-system \
     --kube-output-dir /tmp \
     --regenerate
   

4. In the Data Plane, create the KubeConfig secret:

   kubectl create secret generic control-plane-kubeconfig \
     --from-file=kubeconfig=/tmp/kubeconfig \
     --context data-plane \
     --namespace cpc-system
   

5. In the Data Plane, install the Agent Helm chart in the Remote mode:

   helm upgrade --install confluent-agent confluentinc/cfk-blueprint \
     --set agent.mode=Remote \
     --set agent.enabled=true \
     --set agent.remoteKubeConfig.secretRef=control-plane-kubeconfig \
     --kube-context data-plane \
     --namespace cpc-system
   

6. In the Data Plane, install the CFK Helm chart in the cluster mode (--set namespaced=false):

   helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes \
     --set namespaced="false" \
     --kube-context data-plane \
     --namespace cpc-system
   

7. Register the Data Plane Kubernetes cluster with the Control Plane.

   1. In the Data Plane cluster, get the Kubernetes ID:

      kubectl get namespace kube-system -oyaml --context data-plane | grep uid
      

   2. Edit $TUTORIAL_HOME/registration/data-plane-k8s.yaml and set spec.k8sID to the Kubernetes ID from the previous step.

   3. In the Control Plane, register the Kubernetes cluster and the Health Check by creating the KubernetesCluster and the HealthCheck custom resources (CRs):

      kubectl apply -f $TUTORIAL_HOME/registration/data-plane-k8s.yaml \
        --context control-plane --namespace cpc-system
      

   4. Verify that the Agent is up and running:

      kubectl get cpcHealthCheck \
        --context control-plane --namespace cpc-system
      

Deploy Blueprint

Deploy the Blueprint and the Confluent cluster class CRs:

kubectl apply -f $TUTORIAL_HOME/deployment/confluentplatform_blueprint.yaml \
  --context control-plane --namespace cpc-system

Deploy Confluent Platform in Remote Data Plane

From the Control Plane cluster, deploy Confluent Platform.

1. Create the namespace org-confluent to deploy the Confluent Platform clusters CR into:

   kubectl create namespace org-confluent --context control-plane
   

2. Deploy Confluent Platform:

   kubectl create namespace confluent-dev --context data-plane
   

   kubectl apply -f $TUTORIAL_HOME/deployment/data-plane/confluentplatform_dev.yaml \
     --namespace org-confluent \
     --context control-plane
   

   The Confluent components are installed into the confluent-dev namespace in the Data Plane.

3. In the Data Plane, validate the deployment using Control Center.

   1. Check that the Confluent components are up and running:

      kubectl get pods --namespace confluent-dev --context data-plane -w
      

   2. Set up port forwarding to the Control Center web UI from the local machine:

      kubectl port-forward controlcenter-0 9021:9021 --context data-plane --namespace confluent-dev
      

   3. Navigate to Control Center in a browser and check the cluster:

      http://localhost:9021

4. In the Control Plane, uninstall Confluent Platform:

   kubectl delete -f $TUTORIAL_HOME/deployment/data-plane/confluentplatform_dev.yaml \
     --context control-plane --namespace org-confluent