Prepare Kubernetes Cluster for Confluent Platform

This topic describes the required tasks to prepare your Kubernetes cluster for Confluent Platform deployment.

There are sometimes up to three actors involved in the deployment process of CFK. Two are human users and one is a Service Account, as follows:

  • Kubernetes cluster admin: The Kubernetes administrator provisions a namespace and specific permissions for the CFK user and the CFK Service Account.
  • CFK user: The CFK user deploys CFK and Confluent Platform components.
  • CFK Service Account: The CFK Service Account allows the CFK control plane to access the Kubernetes API and create StatefulSets, Services, Secrets, and other Kubernetes resources needed to deploy Confluent Platform components and resources.

If you are just trying out CFK for the first time, working in a proof-of-concept or sandbox environment, you may be acting as both the Kubernetes cluster admin and the CFK user. If you are deploying to a production environment, especially in a highly regulated organization, the Kubernetes cluster admin and CFK user are typically two separate people or teams. Regardless of your particular scenario, the first step for preparing your Kubernetes cluster is to create a namespace:

Create a namespace for CFK

  1. Create a Kubernetes namespace to deploy the Confluent for Kubernetes (CFK) control plane into:

    kubectl create namespace <confluent-namespace>
    
  2. Set the new namespace as the current namespace.

    This step is not required, but is given here to simplify the example commands in the rest of the documents. The --namespace flag in the subsequent kubectl commands will be omitted as the commands assume the current namespace.

    kubectl config set-context --current --namespace=<confluent-namespace>
    

Configure Kubernetes RBAC and Custom Resource Definitions

If you are a Kubernetes cluster admin, you may want to restrict the level of access granted to the CFK user and CFK Service Account. Start by considering the following questions:

  • Do you want CFK to be able to create and delete Confluent Platform clusters within only one specific namespace, or within any and all namespaces?
  • Do you want the CFK Service Account to be able to create cluster-level resources such as Custom Resource Definitions (CRDs), or not? Allowing CFK to create such resources reduces the number of manual steps for the Kubernetes cluster admin, but requires more permissions to be granted to the CFK Service Account.
  • Should the CFK user manage Kubernetes RBAC permissions for the CFK Service Account, or not? The CFK Service Account RBAC permissions can be automatically bootstrapped along with the CFK installation, reducing the number of manual steps, but requires the CFK user to have permissions to not only deploy CFK but also manage some Kubernetes RBAC permissions.

Here are the preparation instructions for Kubernetes RBAC and CRDs in three common scenarios:

Quick, easy deployment and relaxed permissions

In this scenario, we are okay allowing CFK to deploy Confluent Platform components into any and all Kubernetes namespaces, and we are okay granting CFK cluster-level permissions to automatically install CRDs to avoid manual steps. Also, we are okay allowing the CFK user to manage the CFK Service Account permissions to simplify the overall installation workflow. In this kind of scenario, often the Kubernetes cluster admin and the CFK user are the same person.

In this scenario, no additional preparation is required, you can proceed straight to deploying CFK, see Deploy Confluent for Kubernetes. Be sure to set namespaced to false (see Configure CFK to manage Confluent Platform components across all namespaces).

Strict permissions and restricted namespace access

In this scenario, we will restrict CFK to deploying Confluent Platform components only into the namespace where the CFK control plane will be installed. We also want to constrain the CFK Service Account to not have any cluster-level permissions, so we won’t have it install CRDs automatically; these will need to be installed manually. Furthermore, we will restrict the CFK user to only installing and using CFK and Confluent Platform, we will not want the CFK user managing any sort of permissions for the CFK Service Account. All Kubernetes RBAC permission management will be done exclusively by the Kubernetes cluster admin.

In this scenario, the Kubernetes cluster admin must perform the following additional steps to prepare the Kubernetes cluster before CFK and Confluent Platform can be deployed.

  1. Pull the CFK Helm Chart and extract its contents to get the Confluent CRDs:

    mkdir -p <cfk-dir>
    
    helm pull confluentinc/confluent-for-kubernetes \
      --untar --untardir=<cfk-dir>
    
  2. Install the Confluent CRDs with the following command:

    kubectl apply -f <cfk-dir>/crds
    
  3. Create a Kubernetes Role and Role Binding for the CFK Service Account; see this example. Change any places where it says namespace: confluent by replacing confluent with your <confluent-namespace>. Save this in a file, say cfk-permissions.yaml, and apply it:

    kubectl apply -f cfk-permissions.yaml
    
  4. Deploy CFK with the rbac property set to false as described in Deploy Confluent for Kubernetes without creating roles and role bindings.

Strict permissions, but deployments spanning all namespaces

This scenario is similar to the previous one. In terms of permissions for the CFK Service Account and CFK user, it is very restricted. But this scenario allows for deploying Confluent Platform components across any namespace in your Kubernetes cluster.

In this scenario, the Kubernetes cluster admin must perform the following additional steps to prepare the Kubernetes cluster before CFK and Confluent Platform can be deployed.

  1. Pull the CFK Helm Chart and extract its contents to get the Confluent CRDs:

    mkdir -p <cfk-dir>
    
    helm pull confluentinc/confluent-for-kubernetes \
      --untar --untardir=<cfk-dir>
    
  2. Install the Confluent CRDs with the following command:

    kubectl apply -f <cfk-dir>/crds
    
  3. Create a Kubernetes Cluster Role and Cluster Role Binding for the CFK Service Account; see this example. Change where it says namespace: confluent by replacing confluent with your <confluent-namespace>. Save this in a file, say cfk-permissions.yaml, and apply it:

    kubectl apply -f cfk-permissions.yaml
    
  4. Deploy CFK with the rbac property set to false as described in Deploy Confluent for Kubernetes without creating roles and role bindings.