Scale Confluent Platform Clusters using Ansible

Scale up Confluent Platform

Ansible Playbooks for Confluent Platform supports scaling up Confluent clusters. This use-case scenario demonstrates how to scale up Kafka, ksqlDB, and Connect using Ansible playbooks.

  1. The below example hosts.yaml file configures:

    • 3 KRaft nodes

    • 3 Kafka broker nodes

    • 2 ksqlDB clusters, named ksql1 and ksql2, with each cluster having 1 node

    • 2 Connect clusters, named elastic and syslog, with each cluster having 1 node

    • 1 Control Center node

    kafka_controller:
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:
    ec2-18-237-72-224.us-west-2.compute.amazonaws.com:
    ec2-35-161-39-212.us-west-2.compute.amazonaws.com:

kafka_broker:
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:
    ec2-34-211-33-32.us-west-2.compute.amazonaws.com:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:

ksql:
  children:
    ksql1:
    ksql2:

ksql1:
  vars:
    ksql_service_id: ksql1_
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:

ksql2:
  vars:
    ksql_service_id: ksql2_
  hosts:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:

kafka_connect:
  children:
    syslog:
    elastic:

syslog:
  vars:
    kafka_connect_group_id: connect_syslog
  hosts:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:

elastic:
  vars:
    kafka_connect_group_id: connect-elastic
  hosts:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:

control_center_next_gen:
  hosts:
    ec2-35-163-80-4.us-west-2.compute.amazonaws.com:

  2. Update the hosts.yaml file to scale up the clusters:

    kafka_controller:
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:
    ec2-34-211-33-32.us-west-2.compute.amazonaws.com:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:

kafka_broker:
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:
    ec2-34-211-33-32.us-west-2.compute.amazonaws.com:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:
    ec2-35-89-77-113.us-west-2.compute.amazonaws.com:--- [1]
    ec2-35-89-77-114.us-west-2.compute.amazonaws.com:--- [2]

ksql:
  children:
    ksql1:
    ksql2:

ksql1:
  vars:
    ksql_service_id: ksql1_
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:
    ec2-35-89-77-116.us-west-2.compute.amazonaws.com:--- [3]

ksql2:
  vars:
    ksql_service_id: ksql2_
  hosts:
    ec2-34-219-110-48.us-west-2.compute.amazonaws.com:
    ec2-34-219-110-50.us-west-2.compute.amazonaws.com:--- [4]

kafka_connect:
  children:
    syslog:
    elastic:

syslog:
  vars:
    kafka_connect_group_id: connect_syslog
  hosts:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:

elastic:
  vars:
    kafka_connect_group_id: connect-elastic
  hosts:
    ec2-35-89-77-112.us-west-2.compute.amazonaws.com:
    ec2-35-89-77-113.us-west-2.compute.amazonaws.com:--- [5]

control_center_next_gen:
  hosts:
    ec2-35-163-80-4.us-west-2.compute.amazonaws.com:

    • [1] [2] Add 2 new nodes to the Kafka broker cluster.

    • [3] Add 1 new node to the ksql1 ksqlDB cluster.

    • [4] Add 1 new node to the ksql2 ksqlDB cluster.

    • [5] Add 1 new node to the elastic Connect cluster.

  3. Re-run the playbook to scale up the clusters:

    ansible-playbook -i hosts.yml confluent.platform.all

    The clusters are scaled up with:

    • 5 Kafka broker nodes (scaled up)

    • 2 ksqlDB clusters, ksql1 and ksql2 with 2 nodes each (scaled up)

    • The syslog Connect cluster with 1 node

    • The elastic Connect cluster with 2 nodes (scaled up)

    • 1 Control Center node

  4. To distribute existing loads to new nodes, make sure that you enable the Self-Balancing feature using the Ansible custom properties. For the list of configuration options, see Self-Balancing Configurations on the Brokers.