Scale Confluent Platform Clusters using Ansible

Scale up Confluent Platform

Ansible Playbooks for Confluent Platform supports scaling up Confluent clusters. This use-case scenario demonstrates how to scale up Kafka, ksqlDB, and Connect using Ansible playbooks.

Ansible Playbooks for Confluent Platform does not currently support scaling up ZooKeeper.

  1. The below example hosts.yaml file configures:

    • 3 ZooKeeper nodes
    • 3 Kafka broker nodes
    • 2 ksqlDB clusters, named ksql1 and ksql2, with each cluster having 1 node
    • 2 Connect clusters, named elastic and syslog, with each cluster having 1 node
    • 1 Control Center node
    zookeeper:
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
    ip-172-31-37-15.us-east-2.compute.internal:
    ip-172-31-34-231.us-east-2.compute.internal:

kafka_broker:
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
    ip-172-31-34-247.us-east-2.compute.internal:
    ip-172-31-34-248.us-east-2.compute.internal:

ksql:
  children:
    ksql1:
    ksql2:

ksql1:
  vars:
    ksql_service_id: ksql1_
  hosts:
    ip-172-31-34-15.us-east-2.compute.internal:

ksql2:
  vars:
    ksql_service_id: ksql2_
  hosts:
    ip-172-31-34-17.us-east-2.compute.internal:

kafka_connect:
  children:
    syslog:
    elastic:

syslog:
  vars:
    kafka_connect_group_id: connect_syslog
  hosts:
    ip-172-31-34-245.us-east-2.compute.internal:

elastic:
  vars:
    kafka_connect_group_id: connect-elastic
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:

control_center:
  hosts:
    ip-172-31-37-15.us-east-2.compute.internal:

  2. Update the hosts.yaml file to scale up the clusters:

    zookeeper:
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
    ip-172-31-37-15.us-east-2.compute.internal:
    ip-172-31-34-231.us-east-2.compute.internal:

kafka_broker:
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
    ip-172-31-34-247.us-east-2.compute.internal:
    ip-172-31-34-248.us-east-2.compute.internal:
    ip-172-31-34-249.us-east-2.compute.internal:  --- [1]
    ip-172-31-34-250.us-east-2.compute.internal:  --- [2]

ksql:
  children:
    ksql1:
    ksql2:

ksql1:
  vars:
    ksql_service_id: ksql1_
  hosts:
    ip-172-31-34-15.us-east-2.compute.internal:
    ip-172-31-37-16.us-east-2.compute.internal:   --- [3]

ksql2:
  vars:
    ksql_service_id: ksql2_
  hosts:
    ip-172-31-34-17.us-east-2.compute.internal:
    ip-172-31-37-18.us-east-2.compute.internal:   --- [4]

kafka_connect:
  children:
    syslog:
    elastic:

syslog:
  vars:
    kafka_connect_group_id: connect_syslog
  hosts:
    ip-172-31-34-245.us-east-2.compute.internal:

elastic:
  vars:
    kafka_connect_group_id: connect-elastic
  hosts:
    ip-172-31-34-246.us-east-2.compute.internal:
    ip-172-31-34-247.us-east-2.compute.internal:  --- [5]

control_center:
  hosts:
    ip-172-31-37-15.us-east-2.compute.internal:
    • [1] [2] Add 2 new nodes to the Kafka broker cluster.
    • [3] Add 1 new node to the ksql1 ksqlDB cluster.
    • [4] Add 1 new node to the ksql2 ksqlDB cluster.
    • [5] Add 1 new node to the elastic Connect cluster.

  3. Re-run the playbook to scale up the clusters:

    ansible-playbook -i hosts.yml confluent.platform.all

    The clusters are scaled up with:

    • 5 Kafka broker nodes (scaled up)
    • 2 ksqlDB clusters, ksql1 and ksql2 with 2 nodes each (scaled up)
    • The syslog Connect cluster with 1 node
    • The elastic Connect cluster with 2 nodes (scaled up)
    • 1 Control Center node

  4. To distribute existing loads to new nodes, make sure that you enable the Self-Balancing feature using the Ansible custom properties. For the list of configuration options, see Self-Balancing Configurations on the Brokers.