Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Plan for Confluent Operator Installation

This topic contains the prerequisites and recommendations you need to consider when you plan to install and deploy Confluent Platform using Confluent Operator.

The topic uses the Google Kubernetes Engine (GKE) as the example provider environment. Use this as a guide for deploying Operator and Confluent Platform in other supported provider environments.

General prerequisites

Review and address the following prerequisites before you start the installation process:

  • A Kubernetes cluster conforming to one of the supported environments.
  • Cluster size based on the Sizing Recommendations.
  • kubectl is installed, initialized, with the context set. You also must have the kubeconfig file configured for your cluster.
  • Helm 3 is installed.
  • Access to the Confluent Operator bundle.
  • Storage: StorageClass-based storage provisioner support. This is the default storage class used. You can use other user-provided storage classes. SSD or SSD-like disks are required for persistent storage.
  • Security: TLS certificates for each component required (if using TLS). Default SASL/PLAIN security is used in the example steps. See Configure Security with Confluent Operator for information about how to configure additional security.
  • DNS: DNS support on your platform environment is required for external access to Confluent Platform components after deployment. After deployment, you create DNS entries to enable external access to individual Confluent Platform components. See Configure external load balancers for additional information. If your organization does not allow external access for development testing, see No-DNS development access.
  • Kubernetes Load Balancing:
    • Layer 4 load balancing with passthrough support (terminating at the application) is required for Kafka brokers with external access enabled.
    • Layer 7 load balancing can be used for Operator and all other Confluent Platform components.

Sizing recommendations

Review the following sizing guidelines and recommendations before creating your Kubernetes cluster.

Kubernetes worker node number and compute resource recommendations

The number of nodes required in your cluster depends on whether you are deploying a development testing cluster or a production-ready cluster.

  • Test Cluster: Each node should typically have a minimum of 2 or 4 CPUs and 7 to 16 GB RAM. If you are testing a deployment of Operator and all Confluent Platform components, you can create a 10-node cluster with six nodes for Apache ZooKeeper™ and Apache Kafka® pods (three replicas each) and four nodes for all other components pods.

    Note

    Confluent recommends running ZooKeeper and Kafka on individual pods on individual Kubernetes nodes. You don’t have to do this, but we’ve found that this is the best way to run ZooKeeper and Kafka clusters.

    Tip

    Confluent recommends running ZooKeeper and Kafka on individual pods on individual nodes. You can bin pack other components. Bin packing places component tasks on nodes in the cluster that have the least remaining CPU and memory capacity. Bin packing maximizes node utilization and can reduce the number of nodes required. Each Confluent Platform component YAML file has the default entry disableHostPort: false. You can enable bin packing by adding this parameter to the component section in the <provider.yaml> file and setting it to true. Bin packing components is not recommended for production deployments. Also, note that when set to false, the default port used is 28130.

  • Production Cluster: Review the default capacity values in the global provider file, helm/providers/<provider>.yaml. Determine how these values affect your production application and build out your nodes accordingly. You can also use the on-premises System Requirements to determine what is required for your public or private cloud production environment. Note that the on-premises storage information provided is not applicable for cloud environments.

    Note

    If you need information to determine the number of nodes required for your application, contact Confluent Support .