Confluent Platform on OpenShift Container Platform on AWS

Confluent Operator allows you to deploy and manage Confluent Platform as a cloud-native, stateful container application on Kubernetes and OpenShift. OpenShift Container Platform (OCP) is the flagship product from the OpenShift product series, which is an on-premises platform as a service built around Docker containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux. It can be deployed in all the public clouds as well.

Prerequisites

This section will guide you through the following prerequisites for deploying Confluent Platform on OCP on AWS:

  • Access to an AWS account
  • Access to Red Hat Network (RHN)
  • Familiarity with OCP SDK (the oc command line utility) and OpenShift installer.
  • Confluent Operator

AWS cloud account

Whether you access AWS using root login (not recommended) or via SSO, you must create a user and then use the AWS Access Key ID and AWS Secret Access Key (with the aws configure command) to set up on the shell of the choice.

The AWS account must have appropriate roles to create virtual machines and access the Route53 service. In this tutorial, we will be using example.com, but it’s important that you have a real, working domain instead.

Following is a typical output from the aws configure command:

aws configure

AWS Access Key ID [****************OOPS]:
AWS Secret Access Key [****************NOOP]:
Default region name [None]:
Default output format [json]:

Red Hat Network account

A subscription to the Red Hat Network is required to access OCP. You can get access to it by logging in to the Red Hat website.

OCP SDK and Installer

Download and install the OCP SDK from the website. The link provided is for MacOS.

Add the installed directory to your $PATH.

After OCP SDK is installed, the OpenShift installer and OpenShift command line utility (oc) are ready for use.

Confluent Operator

Download and upzip the Confluent Operator bundle.

OCP cluster

Configure OCP

  1. Log onto the Red Hat website.

  2. After logging in, naviage to Clusters -> Create Cluster -> Red Hat OpenShift Container Platform.

  3. Select the cloud of your choice to deploy OCP. For the purpose of this tutorial, we will be deploying on AWS. Click the AWS tile.

  4. Click Installer-provisioned infrastructure as the recommended option for deploying OCP.

  5. Click Copy pull secret.

  6. Switch to the shell and run the installer.

    mkdir oc-aws-demo

openshift-install create install-config --dir=oc-aws-demo/

    You will be prompted with a set of options to select as shown in the following steps.

  7. Select aws as the platform and press Enter.

    ? Platform  [Use arrows to move, enter to select, type to filter, ? for more help]
> aws
  azure
  gcp
  openstack
  ovirt

  8. Choose the region for deploying the OCP cluster.

    ? Platform aws

INFO Credentials loaded from the "default" profile in file "<your home directory>/.aws/credentials"

? Region  [Use arrows to move, enter to select, type to filter, ? for more help]
  eu-west-3 (Paris)
  me-south-1 (Bahrain)
  sa-east-1 (São Paulo)
> us-east-1 (N. Virginia)
  us-east-2 (Ohio)
  us-west-1 (N. California)
  us-west-2 (Oregon)

  9. Choose a Base Domain that was created for this purpose or use an existing one, for example, example.com.

    ./openshift-install create install-config --dir=oc-aws-demo/

? Platform aws
INFO Credentials loaded from the "default" profile in file "<your home directory>/.aws/credentials"
? Region us-east-1

? Base Domain  [Use arrows to move, enter to select, type to filter, ? for more help]
> example.com

  10. Give a name to the OCP cluster, for example, ocp-us-east-1-k8s-demo-cluster.

    openshift-install create install-config --dir=oc-aws-demo/
? Platform aws
INFO Credentials loaded from the "default" profile in file "<your home directory>/.aws/credentials"
? Region us-east-1
? Base Domain example.com
? Cluster Name [? for help] ocp-us-east-1-k8s-demo-cluster

  11. Put the Pull Secret to finish creating the configuration.

    openshift-install create install-config --dir=oc-aws-demo/
? Platform aws
INFO Credentials loaded from the "default" profile in file "<your home directory>/.aws/credentials"
? Region us-east-1
? Base Domain example.com
? Cluster Name ocp-us-east-1-k8s-demo-cluster
? Pull Secret [? for help]
**********************************
**********************************
**********************************

  12. Now you should see the install-config.yaml file in the directory, oc-aws-demo, which was passed as an argument to the installer command.

  13. Update the install-config.yaml file to suit your deployment scenario.

    For the purposes of this tutorial, select low CPU and memory AWS instances:

    compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    aws:
      types: m5.xlarge
  replicas: 3

    To see more details about tuning install-config.yaml parameters, see here.

Deploy OCP cluster

  1. Deploy the OCP cluster.

    openshift-install create cluster --dir=oc-aws-demo/

    The deployment takes roughly 30-35 minutes. You should see output similar to the following:

    INFO Install complete!

INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=~/Documents/work/k8s/oc/oc-aws-demo/auth/kubeconfig'

INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp-us-east-1-k8s-demo-cluster.example.com

INFO Login to the console with user: kubeadmin, password: XXXXXXXxxxxXXXXxxxx

  2. If the OCP cluster is deployed successfully with the INFO Install complete! message, use the following commands to access the cluster:

    The commands are based on the assumption that you ran the command in the preceding step from the ~/Documents/work/k8s/oc/ directory.

    export KUBECONFIG=~/Documents/work/k8s/oc/oc-aws-demo/auth/kubeconfig

oc get nodes

    You should see an output similar to below:

    NAME                           STATUS    ROLES     AGE       VERSION
ip-10-0-135-239.ec2.internal   Ready     master    24h       v1.17.1
ip-10-0-140-27.ec2.internal    Ready     worker    23h       v1.17.1
ip-10-0-144-235.ec2.internal   Ready     worker    23h       v1.17.1
ip-10-0-151-53.ec2.internal    Ready     master    24h       v1.17.1
ip-10-0-166-231.ec2.internal   Ready     worker    23h       v1.17.1
ip-10-0-168-131.ec2.internal   Ready     master    24h       v1.17.1

The OCP installation is now complete. Proceed to the next step to deploy Confluent Operator.

Confluent Operator and Confluent Platform

The steps in the remaining sections are based on the following assumptions:

  • Helm version 3

    Run the following command to check your Helm version:

    helm version

  • The commands are issued from the following directory:

    ~/Documents/work/k8s/oc/550/helm

Create a new project

Create a project for this tutorial. OpenShift project is a Kubernetes namespace with additional annotations. When you run the helm commands, you will pass the project name for the --namespace flag.

oc new-project confluent

Deploy Confluent Operator

The sample oc-aws.yaml used in the current blog can be found here.

  1. Copy the sample file oc-aws.yaml to providers/oc-aws.yaml.

  2. Customize the providers/oc-aws.yaml file according to your configuration, for example, the domain name.

  3. Install Confluent Operator:

    helm install operator ./confluent-operator \
  --values providers/oc-aws.yaml \
  --namespace confluent \
  --set operator.enabled=true

Set OpenShift security context

Set the OpenShift security context by using the customUID file. This is a cluster-wide setting.

oc create -f scripts/openshift/customUID/scc.yaml

Deploy ZooKeeper

helm install zookeeper ./confluent-operator \
  --values providers/oc-aws.yaml \
  --namespace confluent  \
  --set zookeeper.enabled=true

Check that the appropriate pod security context from the previous step was correctly set:

  1. Start the ZooKeeper shell:

    oc exec -ti zookeeper-0 bash

  2. In the ZooKeeper shell, run the following command:

    id

    An output similar to below should return:

    uid=1002580000(1002580000) gid=0(root) groups=0(root),1002580000

Deploy Kafka brokers

helm install kafka-oc-demo ./confluent-operator \
  --values providers/oc-aws.yaml \
  --namespace confluent \
  --set kafka.enabled=true

Deploy Schema Registry

helm install schemaregistry ./confluent-operator \
  --values providers/oc-aws.yaml \
  --namespace confluent \
  --set schemaregistry.enabled=true

Deploy Control Center

helm install controlcenter ./confluent-operator \
  --values providers/oc-aws.yaml \
  --namespace confluent \
  --set controlcenter.enabled=true

Deploy Connect

helm install connectors ./confluent-operator \
  --values providers/oc-aws.yaml  \
  --namespace confluent \
  --set connect.enabled=true

Verify deployments

Verify the deployments using the following command:

oc get pods

The output should be similar to below:

NAME                        READY     STATUS    RESTARTS   AGE
cc-operator-fcb87457-7d6pn  1/1       Running   0          2d20h
connectors-0                1/1       Running   0          19m
controlcenter-0             1/1       Running   0          25h
kafka-oc-demo-0             1/1       Running   0          2d8h
kafka-oc-demo-1             1/1       Running   0          2d8h
kafka-oc-demo-2             1/1       Running   0          2d7h
schemaregistry-0            1/1       Running   0          25h
schemaregistry-1            1/1       Running   0          25h
zookeeper-0                 1/1       Running   0          2d19h
zookeepe                    1/1       Running   0          2d19h
zookeeper-2                 1/1       Running   0          2d19h

Access Confluent Control Center

You can access Control Center via its elastic load balancer (elb):

To get the Confluent Control Center elb address, run the following command:

oc get svc | grep control

The output will include the elb address as below:

controlcenter                ClusterIP      None             <none>                                                                    9021/TCP,7203/TCP,7777/TCP
controlcenter-0-internal     ClusterIP      172.30.213.238   <none>                                                                    9021/TCP,7203/TCP,7777/TCP
controlcenter-bootstrap-lb   LoadBalancer   172.30.250.158   ab7ece27b585c4f238616abd5e2c857c-1775407194.us-east-1.elb.amazonaws.com   80:32275/TCP

Access Confluent Control Center using the elb address:

ab7ece27b585c4f238616abd5e2c857c-1775407194.us-east-1.elb.amazonaws.com

The default username and password for Control Center login is: admin/Developer1

Test the deployment

You can run the command to get detailed instructions for getting external and internal client access Apache Kafka®:

helm status kafka-oc-demo

  1. Extract the client JAAS configuration:

    kubectl  -n confluent get kafka kafka-oc-demo -ojsonpath='{.status.internalClient}'

  2. Start a bash session on one of the Kafka pods:

    oc exec -ti kafka-oc-demo-0 bash

  3. On the pod, create the kafka.properties file and manually copy the content from Step #1 into the file. For example:

    cat << EOF > kafka.properties
bootstrap.servers=kafka:9071
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test123";
sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT
EOF

  4. Create a topic:

    kafka-topics --create \
  --topic demo_topic \
  --command-config kafka.properties \
  --partitions 6 \
  --replication-factor 3 \
  --bootstrap-server kafka-oc-demo:9071

  5. Produce data to the topic:

    kafka-producer-perf-test \
  --topic demo_topic \
  --producer.config kafka.properties \
  --record-size 4096 \
  --throughput -1 \
  --num-records 10000

  6. Consume the data:

    kafka-consumer-perf-test \
  --topic demo_topic \
  --broker-list kafka-oc-demo:9071 \
  --messages 10000 \
  --consumer.config kafka.properties

Cleanup

  1. Clean up all the assets that you created in the confluent project, including Confluent Operator, ZooKeeper, Kafka brokers, Confluent Control Center, etc.:

    oc delete project confluent

  2. Destroy the OCP cluster:

    openshift-install destroy cluster --dir=oc-aws-demo/