Advanced Confluent Platform Configurations with Ansible Playbooks

This section provides information about various deployment configurations for Confluent Platform using Ansible.

Configure Tiered Storage with GCS buckets

Configure Tiered Storage to use the Custom Properties and the Copy Files features of Ansible Playbooks for Confluent Platform.

  1. Get the GCP credentials JSON file on the Ansible control node.

  2. Set the following variables in the hosts.yml file:

    all:
      vars:
        kafka_broker_copy_files:
          - source_path: /tmp/gcloud-a5f9c87c81ae.json
            destination_path: /etc/security/google/creds.json
    
        kafka_broker_custom_properties:
          confluent.tier.feature: "true"
          confluent.tier.enable: "true"
          confluent.tier.backend: GCS
          confluent.tier.gcs.bucket: bucket-name
          confluent.tier.gcs.region: us-west2
          confluent.tier.gcs.cred.file.path: /etc/security/google/creds.json
    

    The credential file destination path must match the confluent.tier.gcs.cred.file.path custom property.

Configure Tiered Storage with S3 buckets

Configure Tiered Storage to use the Custom Properties and the Copy Files features of Ansible Playbooks for Confluent Platform.

  1. Get the AWS credentials file on the Ansible control node.

  2. Set the following variables in the hosts.yml file:

    all:
      vars:
        kafka_broker_copy_files:
          - source_path: /tmp/credentials
            destination_path: /etc/security/aws/credentials
    
        kafka_broker_custom_properties:
          confluent.tier.feature: "true"
          confluent.tier.enable: "true"
          confluent.tier.backend: S3
          confluent.tier.s3.bucket: bucket-name
          confluent.tier.s3.region: us-west-2
          confluent.tier.s3.cred.file.path: /etc/security/aws/credentials
    

    The credential file destination path must match the confluent.tier.s3.cred.file.path custom property.

Deploy Confluent Replicator

Starting in the 6.1.0 release, Ansible Playbooks for Confluent Platform supports deployment of Confluent Replicator.

Using Ansible, you can deploy Replicator with the following security mechanisms:

  • SASL PLAIN
  • SASL SCRAM
  • Kerberos
  • mTLS
  • Plaintext (which is no auth no encryption)

Note

Ansible currently does not support deploying Replicator with RBAC.

The general deployment model is to deploy Replicator after both the source and destination clusters have been deployed.

We recommend creating an inventory file specifically for the Replicator deployment, excluding other cluster deployment-related configuration. In this section, an example file, replicator-hosts.yml, is used.

There are two clusters in this example, the source cluster and the destination cluster.

Replicator has four client connections split across the two clusters:

The following are the configuration properties required in the Replicator inventory file.

The examples use:

  • SASL PLAIN with TLS on the source cluster
  • Kerberos with TLS on the destination cluster

After configuring the replicator, you deploy the replicator with the following command. The command uses the example inventory file, replicator-hosts.yml.

ansible-playbook -i replicator-hosts.yml all.yml

Configure Replicator configuration connection

  1. Define the kafka_connect_replicator group and hosts to deploy to.

    For example:

    kafka_connect_replicator:
      hosts:
        ip-172-31-34-246.us-east-2.compute.internal:
    
  2. Define the listener for Replicator configuration cluster.

    The following is an example of a listener with Kerberos authentication and TLS enabled:

    kafka_connect_replicator_listener:
      ssl_enabled: true
      ssl_mutual_auth_enabled: false
      sasl_protocol: kerberos
    
  3. Define the basic configuration for Replicator connection:

    kafka_connect_replicator_white_list: <a comma-separated list of topics to be replicated>
    kafka_connect_replicator_bootstrap_servers: <configuration cluster hostname:port>
    
  4. Define security configuration for the Replicator connection:

    kafka_connect_replicator_kerberos_principal: <Kafka principal primary>
    kafka_connect_replicator_kerberos_keytab_path: <path to your keytab>
    kafka_connect_replicator_ssl_ca_cert_path: <path to your CA certificate>
    kafka_connect_replicator_ssl_cert_path: <path to your signed certificate>
    kafka_connect_replicator_ssl_key_path: <path to your SSL key>
    kafka_connect_replicator_ssl_key_password: <SSL key password>
    

Configure consumer connection

  1. Define the configuration for the consumer listener on the source cluster.

    The following is an example with TLS and SASL PLAIN enabled:

    kafka_connect_replicator_consumer_listener:
      ssl_enabled: true
      ssl_mutual_auth_enabled: false
      sasl_protocol: plain
    
  2. Define the basic configuration for the consumer client connection:

    kafka_connect_replicator_consumer_bootstrap_servers: <source cluster hostname:port>
    
  3. Define the security configuration for the consumer client connection:

    kafka_connect_replicator_consumer_ssl_ca_cert_path: <path to your CA certificate>
    kafka_connect_replicator_consumer_ssl_cert_path: <path to your signed certificate>
    kafka_connect_replicator_consumer_ssl_key_path: <path to your SSL key>
    kafka_connect_replicator_consumer_ssl_key_password: <SSL key password>
    
  4. Define custom properties for each client connection:

    kafka_connect_replicator_consumer_custom_properties:
      <custom property: value>
    

Configure producer connection

  1. Define the listener configuration for the producer connection to the destination cluster.

    The following is an example with TLS and Kerberos for authentication enabled:

    kafka_connect_replicator_producer_listener:
      ssl_enabled: true
      ssl_mutual_auth_enabled: false
      sasl_protocol: kerberos
    
  2. Define the basic producer configuration:

    kafka_connect_replicator_producer_bootstrap_servers: <destination cluster hostname:port>
    
  3. Define the security configuration for the producer connection:

    kafka_connect_replicator_producer_kerberos_principal: <kafka principal primary>
    kafka_connect_replicator_producer_kerberos_keytab_path: <path to your keytab>
    kafka_connect_replicator_producer_ssl_ca_cert_path: <path to your CA cert>
    kafka_connect_replicator_producer_ssl_cert_path: <path to your signed cert>
    kafka_connect_replicator_producer_ssl_key_path: <path to your ssl key>
    kafka_connect_replicator_producer_ssl_key_password: <ssl key password>
    
  4. Define custom properties for each client connection:

    kafka_connect_replicator_producer_custom_properties:
      <custom property:value>
    

Configure monitoring connection

  1. Define the listener configuration for the monitoring interceptors:

    kafka_connect_replicator_monitoring_interceptor_listener:
      ssl_enabled: true
      ssl_mutual_auth_enabled: false
      sasl_protocol: kerberos
    
  2. Define the basic monitoring configuration:

    kafka_connect_replicator_monitoring_interceptor_bootstrap_servers: <monitoring cluster hostname:port>
    
  3. Define the security configuration for the monitoring connection.

    kafka_connect_replicator_monitoring_interceptor_kerberos_principal: <kafka principal primary>
    kafka_connect_replicator_monitoring_interceptor_kerberos_keytab_path: <path to your keytab>
    kafka_connect_replicator_monitoring_interceptor_ssl_ca_cert_path: <path to your CA cert>
    kafka_connect_replicator_monitoring_interceptor_ssl_cert_path: <path to your signed cert>
    kafka_connect_replicator_monitoring_interceptor_ssl_key_path: <path to your ssl key>
    kafka_connect_replicator_monitoring_interceptor_ssl_key_password: <ssl key password>
    

Deploy Confluent Platform across multiple regions

To configure multi region clusters, use the following properties in the hosts.yml inventory file:

  • Set replica.selector.class on kafka_broker group.
  • Set broker.rack uniquely on each Kafka broker host.

For example:

kafka_broker:
  vars:
    kafka_broker_custom_properties:
      replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector

  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      broker_id: 1
      kafka_broker_custom_properties:
        broker.rack: us-west-2a
    ip-192-24-5-30.us-west.compute.internal:
      broker_id: 2
      kafka_broker_custom_properties:
        broker.rack: us-west-2b
    ip-192-24-10-0.us-west.compute.internal:
      broker_id: 3
      kafka_broker_custom_properties:
        broker.rack: us-west-2a

You can apply the kafka_broker_custom_properties directly within the kafka_broker group as well.

Configure ksqlDB log streaming

To configure ksqlDB log stream, use the following properties in the hosts.yml inventory file.

  • Without RBAC enabled, set ksql_log_streaming_enabled to true:

    all:
      ksql_log_streaming_enabled: true
    
  • With RBAC and Kerberos enabled, set the ksql_log_streaming_enabled to true, and additionally provide the keytab location, ksql_kerberos_keytab_path, and keytab principal, ksql_kerberos_principal, for connecting to your internal listener:

    all:
      ksql_log_streaming_enabled: true
    ksql:
      hosts:
        ip-192-24-34-224.us-west.compute.internal:
          ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab
          ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
    

    When RBAC is enabled, you also need to add your keytab principal to your LDAP server in order for the client connection to authenticate.

    To configure ksqlDB log streaming with RBAC and MTLS enabled, add the Certificate’s CN, without any special formatting to your LDAP server for the client connection to authenticate. For example, LDAP Username should be specified as ksql1 instead of cn=ksql1.

Configure multiple ksqlDB clusters

To configure multiple ksqlDB clusters, create new groups for each cluster and set them as children of the ksqlDB group.

The Ansible groups cannot be named ksql.

The name of these groups determine how each cluster is named in Control Center.

Each ksqlDB cluster needs a unique value for the ksql_service_id property. By convention, the service ID should end with an underscore.

For example:

ksql:
  ksql1:
    vars:
      ksql_service_id: ksql1_
    hosts:
      ip-172-31-34-15.us-east-2.compute.internal:
      ip-172-31-37-16.us-east-2.compute.internal:

  ksql2:
    vars:
      ksql_service_id: ksql2_
    hosts:
      ip-172-31-34-17.us-east-2.compute.internal:
      ip-172-31-37-18.us-east-2.compute.internal:

  ksql:
    children:
      ksql1:
      ksql2:

To configure Control Center for multiple ksqlDB clusters, set the ksql_cluster_ansible_group_names property to a list of all ksqlDB children groups.

For example:

control_center:
  hosts:
    ip-172-31-37-15.us-east-2.compute.internal:

    vars:
      ksql_cluster_ansible_group_names:
        - ksql1
        - ksql2

Configure multiple Connect clusters

To configure multiple Connect clusters, create a new group for each cluster and set it as children of the kafka_connect group.

The Ansible groups cannot be named kafka_connect.

Each connect cluster needs a unique value for the kafka_connect_group_id property. The value of kafka_connect_group_id will be the name of the connect cluster within Control Center.

For example:

kafka_connect:
  syslog:
    vars:
      kafka_connect_group_id: connect_syslog
    hosts:
      ip-172-31-34-246.us-east-2.compute.internal:

  elastic:
    vars:
      kafka_connect_group_id: connect-elastic
    hosts:
      ip-172-31-34-247.us-east-2.compute.internal:

  kafka_connect:
    children:
      syslog:
      elastic:

To configure Control Center for multiple connect clusters, set the kafka_connect_cluster_ansible_group_names property to a list of all kafka_connect children groups.

For example:

control_center:
  hosts:
    ip-172-31-37-15.us-east-2.compute.internal:

  vars:
    kafka_connect_cluster_ansible_group_names:
      - syslog
      - elastic

Connect to Confluent Cloud

You can use Ansible Playbooks for Confluent Platform to configure and deploy on-premises Confluent Platform to connect to Kafka and Schema Registry running in Confluent Cloud.

See the sample inventory file at the following location for the required configuration settings:

https://github.com/confluentinc/cp-ansible/blob/6.1.0-post/sample_inventories/ccloud.yml