Configure Ansible Playbooks for Confluent Platform

This topic describes commonly used settings you can use to configure Ansible Playbooks for Confluent Platform.

Set Ansible host variables

Once you configure the hosts in your inventory file and verify the connections, you can set variables in the inventory which describe your desired Confluent Platform configuration.

Review the commented out variables with the example inventory file at:

https://github.com/confluentinc/cp-ansible/blob/7.2.13-post/docs/hosts_example.yml

For a full list of supported variables, see the Ansible variable file at:

https://github.com/confluentinc/cp-ansible/blob/7.2.13-post/docs/VARIABLES.md

You can apply variables to all hosts or to specific hosts.

In the below example, all hosts get the ssl_enabled=true variable set:

all:
  vars:
   ssl_enabled: true

In the example below, the broker_id variable is set to a Kafka broker host:

kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      broker_id: 0

We generally recommend applying variables in the all group so that each host is aware of how the Confluent Platform is configured as a whole.

You can also make use of group_vars and host_vars directories that are located next to the inventory file to pass variables. See Ansible Directory Layout.

Additionally, consider saving sensitive variables in their own variables file in the above structure and use Ansible Vault to encrypt the variable files.

The remainder of this document describes how to configure Confluent Platform using Ansible variables.

Set Confluent Platform software installation method

Ansible Playbooks for Confluent Platform supports the following methods for installing Confluent Platform software onto host machines.

Package installation, using the packages hosted on packages.confluent.io
This is the default option. It requires internet connectivity from all hosts to packages.confluent.io. No inventory variables are required to use this method.
Package installation, using the packages hosted on your own RPM or DEB package repository

This option works for hosts that do not have outside internet connectivity. It requires you to pull the Confluent Platform packages and put them on your repository.

Set the following in your inventory file to use this method.

For packages on an RHEL/Centos host:

all:
  vars:
    repository_configuration: custom
    custom_yum_repofile_filepath: /tmp/my-repo.repo

For packages on a Debian host:

all:
  vars:
    repository_configuration: custom
    custom_apt_repo_filepath: /tmp/my-source.list

For the end-to-end workflow of deploying Ansible Playbooks for Confluent Platform in an air-gapped environment, see Air-gapped Deployment of Ansible Playbooks for Confluent Platform.

Tar installation, using the tarball hosted on packages.confluent.io

It requires internet connectivity from all hosts to packages.confluent.io.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
Tar installation using the tarball hosted on your own web server

This does not require outside internet connectivity, but does require you to pull the tarball and host it on a web server.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
    confluent_archive_file_source: <web server url>/path/confluent-7.2.13.tar.gz

For the end-to-end workflow of deploying Ansible Playbooks for Confluent Platform in an air-gapped environment, see Air-gapped Deployment of Ansible Playbooks for Confluent Platform.

Tar installation using the tarball placed on Ansible control node

This does not require outside internet connectivity, but requires you to pull and copy the tarball to the control node.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
    confluent_archive_file_source: /path/to/confluent-7.2.13.tar.gz
    confluent_archive_file_remote: false

For the end-to-end workflow of deploying Ansible Playbooks for Confluent Platform in an air-gapped environment, see Air-gapped Deployment of Ansible Playbooks for Confluent Platform.

Set custom component properties

When a configuration setting is not directly supported by Ansible Playbooks for Confluent Platform, you can use the custom property feature to configure Confluent Platform components.

Before you set a custom property variable, first check the Ansible variable file at the following location for an existing variable:

https://github.com/confluentinc/cp-ansible/blob/7.2.13-post/docs/VARIABLES.md

If you find an existing variable that directly supports the setting, use the variable in the inventory file instead of using a config override.

Configure the custom properties in the Ansible inventory file, hosts.yml, using the following dictionaries:

  • zookeeper_custom_properties
  • kafka_broker_custom_properties
  • schema_registry_custom_properties
  • kafka_rest_custom_properties
  • kafka_connect_custom_properties
  • ksql_custom_properties
  • control_center_custom_properties
  • kafka_connect_replicator_custom_properties
  • kafka_connect_replicator_consumer_custom_properties
  • kafka_connect_replicator_producer_custom_properties
  • kafka_connect_replicator_monitoring_interceptor_custom_properties

In the example below:

  • The initLimit and syncLimit properties are set in the ZooKeeper properties file.
  • The num.io.threads property gets set in the Kafka properties file.
  • The confluent.controlcenter.ksql.default.advertised.url property gets set in the Confluent Control Center properties file.

Note that the default in the confluent.controlcenter.ksql.default.advertised.url property value is the name Control Center should use to identify the ksqlDB cluster.

all:
  vars:
    zookeeper_custom_properties:
      initLimit: 6
      syncLimit: 3
    kafka_broker_custom_properties:
      num.io.threads: 15
    control_center_custom_properties:
      confluent.controlcenter.ksql.default.advertised.url: http://ksql-external-dns:1234,http://ksql-external-dns:2345

Set custom properties on a specific host

You can configure a specific host with unique properties. Put the component properties block directly under the host.

In the example below, the broker.rack property is set to us-west-2a for the host, ip-192-24-10-207.us-west.compute.internal.

kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      kafka_broker_custom_properties:
        broker.rack: us-west-2a

Add Confluent license

To add a Confluent license key for Confluent Platform components, use a custom property for each Confluent Platform component in the hosts.yml file as following:

all:
  vars:
    kafka_broker_custom_properties:
     confluent.license: <license-key>
     kafka.rest.confluent.license.topic: "_confluent-command"
   schema_registry_custom_properties:
     confluent.license: <license-key>
   kafka_connect_custom_properties:
     confluent.license: <license-key>
   control_center_custom_properties:
     confluent.license: <license-key>
   kafka_rest_custom_properties:
     confluent.license: <license-key>
   ksql_custom_properties:
     confluent.license: <license-key>

Note that Confluent Server (Kafka broker) contains Kafka REST Server, and this component also requires a valid license configuration. Set the kafka.rest.confluent.license.topic property to the _confluent-command topic that stores the Confluent license.

To add license to a connector, use the following config in the hosts.yaml file:

all:
  vars:
    kafka_connect_connectors:
    - name: sample-connector
       config:
          confluent.license: <license-key>

The following example adds a license key for Kafka and Schema Registry. The example creates a variable for the license key and uses the variable in the custom properties.

vars:
  confluent_license: asdfkjkadslkfjaslkdf
  kafka_broker_custom_properties:
    confluent.license: "{{ confluent_license }}"
    kafka.rest.confluent.license.topic: "_confluent-command"
  schema_registry_custom_properties:
    confluent.license: "{{ confluent_license }}"

For additional license configuraion parameters you can set with the above custom properties, see License Configurations for Confluent Platform.

Enable JMX Exporter

JMX Exporter is disabled by default. When enabled, the JMX Exporter jar is pulled from the internet and enabled on all Confluent Platform components besides Confluent Control Center.

Enable JMX Exporter in hosts.yml as below:

all:
  vars:
    jmxexporter_enabled: true

For more information on how the JMX exporter works and how to monitoring Kafka cluster with the JMX data using Prometheus and Grafana, see Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana.

Enable Jolokia

Jolokia monitoring is disabled by default for Confluent Platform components when installed by Ansible Playbooks for Confluent Platform.

Enable Jolokia in hosts.yml as shown below:

all:
  vars:
    jolokia_enabled: true

Deploy Confluent Server or Kafka

Confluent Server is the default version deployed with Confluent Platform. To install Kafka instead, set the following property in the hosts.yml file.

all:
  vars:
    confluent_server_enabled: false

Configure Schema Validation

You can configure Schema ID Validation in your Kafka brokers when running Confluent Server. Set the following properties in the hosts.yml file.

all:
  vars:
    confluent_server_enabled: true
    kafka_broker_schema_validation_enabled: true

Copy files to hosts

To have Ansible copy files to your hosts, place the files on the Ansible control node and set the following variables:

all:
  vars:
    zookeeper_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_broker_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_rest_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_connect_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    schema_registry_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    ksql_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    control_center_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt

The files in each list will be copied to all hosts within each group, meaning you will distribute one file to all Kafka hosts.

Add custom Java arguments

To have Ansible add custom Java arguments to each component’s Java process, use the following variables in the inventory file:

all:
  vars:
    zookeeper_custom_java_args:
    kafka_broker_custom_java_args:
    kafka_rest_custom_java_args:
    kafka_connect_custom_java_args:
    schema_registry_custom_java_args:
    ksql_custom_java_args:
    control_center_custom_java_args:

The following example adds the -javaagent argument to ZooKeeper’s java process:

all:
  vars:
    zookeeper_custom_java_args: "-javaagent:/path/to/javaagent.jar"

Set environment variables

To have Ansible set the required environment variables to Confluent Platform component processes, for example, KAFKA_OPTS, use the following dictionary variables in the inventory file. Refer to the specific component documentation for the required environment variables.

all:
  vars:
    zookeeper_service_environment_overrides:
    kafka_broker_service_environment_overrides:
    kafka_rest_service_environment_overrides:
    kafka_connect_service_environment_overrides:
    kafka_connect_replicator_service_environment_overrides:
    schema_registry_service_environment_overrides:
    ksql_service_environment_overrides:
    control_center_service_environment_overrides:

For example, the following snippet sets the KAFKA_JMX_OPTS environment variable in the Kafka broker service:

all:
  vars:
    kafka_broker_service_environment_overrides:
      KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false"

Configure listeners

Ansible Playbooks for Confluent Platform configures two listeners on the broker:

  • An inter-broker listener on port 9091
  • A listener for the other Confluent Platform components and external clients on 9092

By default both of these listeners inherit the security settings you configure for ssl_enabled (encryption) and sasl_protocol (authentication).

If you only need a single listener, add the following variable in the hosts.yml file.

all:
  vars:
    kafka_broker_configure_multiple_listeners: false

You can customize the out-of-the-box listeners by adding the variable, kafka_broker_custom_listeners in the hosts.yml file.

In the example below, the broker, internal, and client listeners all have unique security settings. You can configure multiple additional client listeners, but do not change the dictionary key for the broker and internal listeners, broker and internal.

all:
  vars:
    kafka_broker_custom_listeners:
      broker:
        name: BROKER
        port: 9091
        ssl_enabled: false
        ssl_mutual_auth_enabled: false
        sasl_protocol: none
      internal:
        name: INTERNAL
        port: 9092
        ssl_enabled: true
        ssl_mutual_auth_enabled: false
        sasl_protocol: scram
      client_listener:
        name: CLIENT
        port: 9093
        ssl_enabled: true
        ssl_mutual_auth_enabled: false
        sasl_protocol: plain

Add advertised listener hostnames

When you have a complex networking setup with multiple network interfaces, you need to set up advertised listeners to the external address (host/IP) so that clients can correctly connect to.

To configure advertised listener hostnames on a specific listener, create an advertised listener ([1]) and set the variables on specific hosts ([2] and [3]) as shown in the following example:

all:
  vars:
    kafka_broker_custom_listeners:
      client_listener:                        -------------------------- [1]
        name: CLIENT
        port: 9093
kafka_broker:
  hosts:
    ip-172-31-43-14.us-west-2.compute.internal:
      kafka_broker_custom_listeners:
        client_listener:                      -------------------------- [1]
          hostname: ec2-34-209-19-18.us-west-2.compute.amazonaws.com --- [2]
    ip-172-31-43-15.us-west-2.compute.internal:
      kafka_broker_custom_listeners:          -------------------------- [1]
        client_listener:
          hostname: ec2-34-209-19-19.us-west-2.compute.amazonaws.com --- [3]

The above example sets the AWS external DNS hostnames ([2] and [3]) on the advertised listener ([1]) for clients to connect over the interface.

Configure secrets protection

Confluent Platform secrets allow you to securely store and manage sensitive information.

Secrets protection works on Confluent Platform components, namely Confluent Server, Schema Registry, Connect, ksqlDB, REST Proxy, Confluent Control Center.

Secrets protection is not supported for ZooKeeper or the community version of Kafka.

To use secrets protection on your component property files with Ansible Playbooks for Confluent Platform, set the following variable in your inventory file.

all:
  vars:
    secrets_protection_enabled: true

When secrets_protection_enabled is set to true, Ansible generates your master key and encrypts all properties containing password across all Confluent Platform components.

  • To have Ansible use your own masterkey and base secrets file that you generated ahead of time, add:

    all:
  vars:
    secrets_protection_enabled: true
    secrets_protection_masterkey: <masterkey>
    secrets_protection_security_file: <base secret file path>

    For example:

    all:
   vars:
     secrets_protection_enabled: true
     secrets_protection_masterkey: "UWQYODNQVqwbQeFgytYYoMr+FjK9Q6I0F6r16u6Y0EI="
     secrets_protection_security_file: "/tmp/security.properties"

  • To have more granular control over which properties get masked, use the <component>_secrets_protection_encrypt_passwords and <component>_secrets_protection_encrypt_properties variables.

    If <component>_secrets_protection_encrypt_passwords is set to false, then properties containing password will no longer get masked.

    Set <component>_secrets_protection_encrypt_properties to a list of variables to encrypt.

    For an example, to mask only the Kafka properties advertised.listeners and broker.id, set:

    all:
  vars:
    secrets_protection_enabled: true
    kafka_broker_secrets_protection_encrypt_passwords: false
    kafka_broker_secrets_protection_encrypt_properties: [advertised.listeners, broker.id]