Install using Ansible

Refer to the following sections for information about using Ansible playbooks provided by Confluent to install the Confluent Platform.

Important

Support: For Confluent Platform 5.3 (and later), the Ansible playbooks are fully supported for those with a Confluent Support contract. Any issues encountered can be reported to Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, issues can be reported using the cp-ansible GitHub repo and are addressed on a best-effort basis.

Feature recommendations: For those with a Confluent Support contract, future work and additional features should be filed by opening a case with Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, please review the Contributing document.

About the Ansible playbooks

Ansible provides a simple way to deploy, manage, and configure Confluent Platform. The cp-ansible repository provides the playbooks and templates that allow you to easily provision the Confluent Platform in your environment.

Note

This guide assumes basic knowledge of how Ansible playbooks work. If you want to learn more about Ansible playbooks, see these resources:

The Ansible playbooks provided by Confluent perform the following operations:

  • Installs Confluent Platform packages.
  • Starts services using systemd scripts.
  • Provides variables for configuring various security settings between Confluent Platform component services:
    • SSL (one way and two way) with self-signed certs or custom certs
    • SASL (Kerberos, Plain)

You can install the services listed below using the Ansible playbooks provided by Confluent:

  • Apache ZooKeeper™
  • Apache Kafka®
  • Confluent Schema Registry
  • REST Proxy
  • Confluent Control Center
  • Kafka Connect (distributed mode)
  • Confluent KSQL

Note

With the exception of Confluent Control Center, all Confluent Platform components are installed with Jolokia enabled and metrics exposed.

Installation Requirements

Refer to the following sections for installation requirements.

General Requirements

Supported Operating Systems

  • RHEL 7 or later
  • Ubuntu 16.04 or later
  • Debian 8 or later

Hardware Requirements

Refer to the Confluent Platform component on-premises system requirements.

Ports

The playbook configures Confluent Platform components to use the ports listed below.

Important

Make sure these ports are open.

Component Port
Kafka brokers: inter-broker communication 9091
Kafka brokers: client communication 9092
Confluent Control Center 9021
Connect REST API 8083
KSQL REST API 8088
REST Proxy 8082
Schema Registry REST API 8081
ZooKeeper 2181, 2888, 3888

Running the playbook

Using Ansible to install and configure Confluent Platform consists of a few steps. You create an inventory file, set up the variables you need, and then run the playbook. The following steps provide a high-level overview of the procedure.

  1. Get the source code.

    git clone https://github.com/confluentinc/cp-ansible

  2. Create a copy of the hosts_example.yml file.

    cd cp-ansible

cp hosts_example.yml hosts.yml

  3. Edit your inventory file and enter the correct hosts, SSH variables, and custom variables.

    vi hosts.yml

  4. Confirm Ansible can connect over SSH.

    ansible -i hosts.yml all -m ping

  5. Run the all.yml playbook.

    ansible-playbook -i hosts.yml all.yml

The following sections provide detailed information about generating your inventory file, setting custom variables, and configuring different options for security.

Generating your inventory file

Before running the Ansible playbook you need to generate an inventory file. The inventory file provisions the hosts used for the Confluent Platform components. See Inventory basics: hosts and groups for more information about the inventory file.

An example base inventory file is shown below:

zookeeper:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
    ip-192-24-5-30.us-west.compute.internal:
    ip-192-24-10-0.us-west.compute.internal:
kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      broker_id: 1
    ip-192-24-5-30.us-west.compute.internal:
      broker_id: 2
    ip-192-24-10-0.us-west.compute.internal:
      broker_id: 3
schema_registry:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
control_center:
  hosts:
    ip-192-24-12-225.us-west.compute.internal:
kafka_connect:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
kafka_rest:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
ksql:
  hosts:
    ip-192-24-5-30.us-west.compute.internal:

Save your inventory file onto the host running Ansible as hosts.yml. Enter the following command to verify that Ansible can reach each host:

ansible -i /path/to/hosts.yml all -m ping

This verifies that the Ansible bastion can reach each host over SSH. This is a requirement for running the playbook. If ping fails, you likely need to set up your ssh variables. See Connecting to hosts: behavioral inventory parameters for more information about setting up ssh variables.

Setting up custom variables

Confluent provides a sample inventory file with all supported customization variables. Use this file as a reference when setting your configuration properties and when setting up your TLS an SASL configuration variables.

You can set custom properties for each component in the inventory file using variables. An example is shown below:

all:
  vars:
    zookeeper:
      properties:
        initLimit: 6
        syncLimit: 3
    kafka_broker:
      properties:
        num.io.threads: 15

In the example above, the initLimit and syncLimit properties are set in the ZooKeeper properties file. The num.io.threads property gets set in the Kafka properties file.

JMS Exporter

Additionally, a variable is available that allows you to enable JMX Exporter on your Kafka brokers. The following example shows how to enable this property.

## JMX Exporter is disabled by default. When enabled, JMX Exporter jar will be pulled from the Internet and enabled on the broker *only*.
## To enable, uncomment this line:
jmxexporter_enabled: true

Configuring Security

The following sections provide instructions for configuring security.

TLS encryption

By default the components are installed using the PLAINTEXT protocol. This means there is no data encryption. You can enable TLS encryption with one of the following options:

  • Self Signed Certs: A Certificate Authority will be generated by the playbooks and used to sign the certs for each host.
  • Custom Certs: You can provide signed certs and keys for each host as well as the Certificate Authority Cert used to sign the certs.
  • Custom Keystores and Truststores: You can provide keystores and truststores for each host.

Note

By default the TLS connection is one-way TLS. Mutual TLS is also configurable.

TLS encryption variables should be added under the all group in the hosts.yml file. To turn on TLS encryption set the following parameter to true:

ssl_enabled: true

By default, components are configured with one-way TLS. To turn on TLS mutual authentication, set the following parameter to true:

ssl_mutual_auth_enabled: true

By default, the certs for this configuration are self-signed. To deploy custom certificates, there are two options available: Custom Certificates and Custom Keystores and Truststores.

Custom Certificates

To provide Custom Certificates for each host, you will need the Certificate Authority certificate, the signed certificates, and keys for each host on the Ansible host. Complete the following steps to use Custom Certificates.

  1. Set this variable to true.

    ssl_custom_certs: true

  2. Enter the path to the Certificate Authority Cert used to sign each host certificate.

    ssl_ca_cert_filepath: "/tmp/certs/ca.crt"

  3. Set the following variables for each host.

    ssl_signed_cert_filepath: "/tmp/certs/{{inventory_hostname}}-signed.crt"

ssl_key_filepath: "/tmp/certs/{{inventory_hostname}}-key.pem"

    The variable {{inventory_hostname}} in the example shows that Ansible can read the hostnames set in the inventory file. For this reason, you can keep the inventory file shorter if you put the hostname in the filename for each signed certificate and key file.

    As an alternative, you can set the variables directly under a host:

    schema_registry:
   hosts:
      ip-192-24-10-207.us-west.compute.internal:
         ssl_signed_cert_filepath: "/tmp/certs/192-24-10-207-signed.crt
         ssl_key_filepath: "/tmp/certs/192-24-10-207-key.pem

Custom Keystores and Truststores

To provide Custom Keystores and Truststores for each host, you will need to have keystores and truststores for each host on the Ansible host, as well as their passwords. Complete the following steps to use Custom Keystores and Truststores.

  1. Set this variable to true.

    provided_keystore_and_truststore: true

  2. Set the following variables for each host.

    ssl_keystore_filepath: "/tmp/certs/{{inventory_hostname}}-keystore.jks"
ssl_keystore_key_password: mystorepassword
ssl_keystore_store_password: mystorepassword
ssl_truststore_filepath: "/tmp/certs/truststore.jks"
ssl_truststore_password: truststorepass

By taking advantage of the {{inventory_hostname}} variable and setting the same password for each host, you can set these variable once under the all group in the hosts.yml file. As an alternative, you can set these variables under each host as shown in the alternative Custom Certificates example.

SASL Authentication

Confluent Ansible can configure SASL/PLAIN or SASL/GSSAPI (or Kerberos). By default, there is no SASL authentication.

SASL authentication options

By default the components are installed without any SASL authentication. You can enable the following SASL options:

  • PLAIN: SASL/PLAIN uses a simple username and password for authentication.
  • SCRAM: SASL/SCRAM uses usernames and password stored in ZooKeeper. Credentials get created during installation.
  • Kerberos: SASL with Kerberos provides authentication using your Kerberos or Active Directory server.

Note

Kerberos Key Distribution Center (KDC) and Active Directory KDC configurations are not currently configured by the Ansible playbook.

Configuring SASL/PLAIN

To configure SASL/PLAIN security, set the following option in the all group in the hosts.yml file.

sasl_protocol: plain

During installation, users are created for each component. This includes an admin user for the Kafka brokers and a client user for use by external components. To configure additional users, add the following section to the all group in the hosts.yml file. The following example shows an additional three users added.

sasl_plain_users:
  user1:
    principal: user1
    password: my-secret
  user2:
    principal: user2
    password: my-secret
  user3:
    principal: user3
    password: my-secret

Configuring SASL/SCRAM

To configure SASL/SCRAM security, set the following option in the all group in the hosts.yml file.

sasl_protocol: scram

During installation, users are created for each component. This includes an admin user for the Kafka brokers and a client user for use by external components. To configure additional users, add the following section to the all group in the hosts.yml file.

sasl_scram_users:
  user1:
    principal: user1
    password: my-secret

Configuring SASL with Kerberos

To configure SASL with Kerberos, you need to create principals within your organization’s KDC server for each component and for each host in each component. You also need to generate keytabs for these principals. The keytab files must be present on the Ansible host.

Note

You need to set up your own Key Distribution Center (KDC) independently of the playbook and provide your own keytabs.

The following example shows the Kerberos Configuration variables in the all section of the hosts.yaml file. This installs Kerberos packages and configures the /etc/krb5.conf file on each host.

all:
  vars:

  ...omitted

  #### Kerberos Configuration ####

  kerberos_kafka_broker_primary: <Name of the primary set on the kafka brokers principal (eg. kafka)>
  kerberos_configure: <Boolean for ansible to install kerberos packages and configure this file: /etc/krb5.conf, defaults to true>
  kerberos:
    realm: <KDC server realm eg. confluent.example.com>
    kdc_hostname: <hostname of machine with KDC running eg. ip-192-24-45-82.us-west.compute.internal>
    admin_hostname: <hostname of machine with KDC running eg. ip-192-24-45-82.us-west.compute.internal>

Note

If the hosts already have the /etc/krb5.conf file configured, make sure to set kerberos_configure to false.

The following examples show the variables that need to be added to the hosts.yml file for each component to use Kerberos for authentication. Each host must have these variables set in the hosts.yml file.

Zookeeper section

Use the following example to enter the zookeeper_kerberos_keytab_path and zookeeper_kerberos_principal variables. Each host needs to have these two variables. Three hosts are shown in the example.

zookeeper:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-37-15.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-34-224.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Kafka broker section

Use the following example to enter the kafka_broker_kerberos_keytab_path and kafka_broker_kerberos_principal variables. Each host needs to have these two variables. The example shows three hosts.

kafka_broker:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-37-15.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-34-224.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Schema Registry section

Use the following example to enter the schema_registry_kerberos_keytab_path and schema_registry_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

schema_registry:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      schema_registry_kerberos_keytab_path: /tmp/keytabs/schemaregistry-ip-192-24-34-224.us-west.compute.internal.keytab
      schema_registry_kerberos_principal: schemaregistry/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Kafka Connect section

Use the following example to enter the kafka_connect_kerberos_keytab_path and kafka_connect_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

kafka_connect:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      kafka_connect_kerberos_keytab_path: /tmp/keytabs/connect-ip-192-24-34-224.us-west.compute.internal.keytab
      kafka_connect_kerberos_principal: connect/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

REST proxy section

Use the following example to enter the kafka_rest_kerberos_keytab_path and kafka_rest_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

kafka_rest:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      kafka_rest_kerberos_keytab_path: /tmp/keytabs/restproxy-ip-192-24-34-224.us-west.compute.internal.keytab
      kafka_rest_kerberos_principal: restproxy/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

KSQL section

Use the following example to enter the ksql_kerberos_keytab_path and ksql_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

ksql:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab
      ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Control Center section

Use the following example to enter the control_center_kerberos_keytab_path and control_center_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

control_center:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      control_center_kerberos_keytab_path: /tmp/keytabs/controlcenter-ip-192-24-34-224.us-west.compute.internal.keytab
      control_center_kerberos_principal: controlcenter/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Security for Internal and External Listeners

Confluent Ansible configures two listeners on the broker: an internal listener for the broker to communicate with other brokers and an external listener for the other Confluent Platform components and external clients. By default both of these listeners inherit the security settings you configure for ssl_enabled and sasl_protocol.

If you only need a single listener, add the following variable to the all group in hosts.yml file.

kafka_broker_configure_additional_brokers: false

To configure different security settings for the internal and external listeners, add the following variables to the all group in the hosts.yml file.

kafka_broker_custom_listeners:
  internal:
    name: INTERNAL
    port: 9091
    ssl_enabled: false
    ssl_mutual_auth_enabled: false
    sasl_protocol: none
  external:
    name: EXTERNAL
    port: 9092
    ssl_enabled: true
    ssl_mutual_auth_enabled: false
    sasl_protocol: scram

You can create additional custom listeners within the kafka_broker_custom_listeners section shown in the example above. Simply add another listener block. The example below shows an additional custom client listener.

client_listener:
  name: CLIENT
  port: 9093
  ssl_enabled: true
  ssl_mutual_auth_enabled: true
  sasl_protocol: plain

Migrating to Confluent Server

Confluent Platform 5.3 (and later) includes a commercial component named confluent-server. Confluent Server is a Kafka broker with support for commercial features like Confluent Operator, the Kafka Java client, Kafka Streams, and Kafka Connect. Confluent Server also includes proprietary plugins for the security features Confluent LDAP Authorizer and Role-based access control.

Confluent Platform also includes a version of confluent-kafka, which is an open source Apache 2.0 licensed component. Both confluent-community and confluent-platform packages have a dependency on confluent-kafka. You can migrate from confluent-kafka to confluent-server using the instructions in the following two sections.

Red Hat Enterprise Linux

Complete the following steps to migrate to confluent-server when using Red Hat Enterprise Linux (RHEL).

  1. Install your cluster using cp-ansible. This installs confluent-kafka by default.

  2. Start your cluster and verify that it is working as expected.

  3. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka

  4. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  5. Use yum swap to change to confluent-server:

    yum swap confluent-kafka-2.12 confluent-server

  6. Copy your Kafka broker configuration backup to the broker configuration directory:

    sudo cp /tmp/server.properties /etc/kafka/

  7. Copy your override file using the following command:

    sudo cp /tmp/override.con /etc/systemd/system/confluent-server.service.d/

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  8. Copy your JAAS file to the broker configuration directory:

    sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/

  9. Start your Kafka brokers:

    sudo systemctl start confluent-server

Complete the following steps to change from confluent-server back to confluent-kafka.

  1. Log in to each Kafka broker host and stop each broker:

    sudo systemctl stop confluent-server

  2. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-server.service.d/override.conf /tmp

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  3. Use yum swap to change to confluent-kafka:

    yum swap confluent-kafka-2.12 confluent-kafka

  4. Copy your configuration files to the broker configuration directory:

    sudo cp /tmp/server.properties /etc/kafka/

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  5. Copy your JAAS file to the broker configuration directory:

    sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/

  6. Start your Kafka brokers:

    sudo systemctl start confluent-kafka

Ubuntu

Complete the following steps to migrate to confluent-server when using Ubuntu.

  1. Install your cluster using cp-ansible. This installs confluent-kafka by default.

  2. Start your cluster and verify that it is working as expected.

  3. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka

  4. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  5. Remove confluent-kafka from your Kafka broker hosts.

    sudo apt-get remove confluent-kafka-2.12

  6. Install confluent-server on your Kafka broker hosts.

    sudo apt-get install confluent-server

  7. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  8. Copy your override file using the following command:

    sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /etc/systemd/system/confluent-server.service.d

  9. Start your Kafka brokers:

    sudo systemctl start confluent-server

Complete the following steps to change from confluent-server back to confluent-kafka.

  1. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-server

  2. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp

    If running with Kerberos, you need to back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp

  3. Remove confluent-server from your Kafka broker hosts.

    sudo apt-get remove confluent-server

  4. Install confluent-kafka on your Kafka broker hosts.

    sudo apt-get install confluent-kafka-2.12

  5. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.

  6. Start your Kafka brokers.

    sudo systemctl start confluent-kafka

Additional information

This repository uses the systemd scripts provided with the Confluent Platform. Because of this dependency, there is an expected default user/service mapping that uses the prefix convention cp- followed by the service name. For example cp-kafka or cp-schema-registry. The one exception is that ZooKeeper runs as the cp-kafka user. This matches the systemd scripts as well.

Troubleshooting

The playbook may fail to run for a variety of reasons. Complete the following steps if the playbook fails:

  1. Append -vvv to the playbook run command and pipe it to a file.

    ansible-playbook -vvvv -i hosts.yml all.yml >failure.txt

  2. Open a support ticket with Confluent Support and provide the following:

    • Playbook name you are running.
    • The step where the playbook failed.
    • All changes you made to the playbook.
    • Attach the output from the failed test as a compressed text file.