Install using Ansible

Refer to the following sections for information about using Ansible playbooks provided by Confluent to install the Confluent Platform.

Important

Support: For Confluent Platform 5.3 (and later), the Ansible playbooks are fully supported for those with a Confluent Support contract. Any issues encountered can be reported to Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, issues can be reported using the cp-ansible GitHub repo and are addressed on a best-effort basis.

Feature recommendations: For those with a Confluent Support contract, future work and additional features should be filed by opening a case with Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, please review the Contributing document.

About the Ansible playbooks

Ansible provides a simple way to deploy, manage, and configure Confluent Platform. The cp-ansible repository provides the playbooks and templates that allow you to easily provision the Confluent Platform in your environment.

The Ansible playbooks provided by Confluent perform the following operations:

  • Installs Confluent Platform packages.
  • Starts services using systemd scripts.
  • Provides variables for configuring various security settings between Confluent Platform component services:
    • SSL (one way and two way) with self-signed certs or custom certs
    • SASL (Kerberos, Plain)

You can install the services listed below using the Ansible playbooks provided by Confluent:

  • Apache ZooKeeper™
  • Apache Kafka®
  • Confluent Schema Registry
  • REST Proxy
  • Confluent Control Center
  • Kafka Connect (distributed mode)

Note

With the exception of Confluent Control Center, all Confluent Platform components are installed with Jolokia enabled and metrics exposed.

Installation Requirements

General Requirements:

Operating System:

  • RHEL 7 or later
  • Ubuntu 16.04 or later
  • Debian 8 or later

Minimum Hardware:

  • 4 hosts
  • 8 CPU Cores per host
  • 32 GB of RAM per host

Running the playbook

Using Ansible to install and configure Confluent Platform consists of a few steps. You create an inventory file, set up the variables you need, and then run the playbook. The following steps provide a high-level overview of the procedure.

  1. Get the source code.

    git clone https://github.com/confluentinc/cp-ansible
    
  2. Create a copy of the hosts_example.yml file.

    cd cp-ansible
    
    cp hosts_example.yml hosts.yml
    
  3. Edit your inventory file and enter the correct hosts, SSH variables, and custom variables.

    vi hosts.yml
    
  4. Confirm Ansible can connect over SSH.

    ansible -i host.yml all -m ping
    
  5. Run the all.yml playbook.

    ansible-playbook -i host.yml all.yml
    

The following sections provide detailed information about generating your inventory file, setting custom variables, and configuring different options for security.

Generating your inventory file

Before running the Ansible playbook you need to generate an inventory file. The inventory file provisions the hosts used for the Confluent Platform components. See Inventory basics: hosts and groups for more information about the inventory file.

An example base inventory file is shown below:

zookeeper:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
    ip-192-24-5-30.us-west.compute.internal:
    ip-192-24-10-0.us-west.compute.internal:
kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      broker_id: 1
    ip-192-24-5-30.us-west.compute.internal:
      broker_id: 2
    ip-192-24-10-0.us-west.compute.internal:
      broker_id: 3
schema_registry:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
control_center:
  hosts:
    ip-192-24-12-225.us-west.compute.internal:
kafka_connect:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
kafka_rest:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
ksql:
  hosts:
    ip-192-24-5-30.us-west.compute.internal:

Save your inventory file onto the host running Ansible as hosts.yml. Enter the following command to verify that Ansible can reach each host:

ansible -i /path/to/hosts.yml all -m ping

This verifies that the Ansible bastion can reach each host over SSH. This is a requirement for running the playbook. If ping fails, you likely need to set up your ssh variables. See Connecting to hosts: behavioral inventory parameters for more information about setting up ssh variables.

Setting up custom variables

Confluent provides a sample inventory file with all supported customization variables. Use this file as a reference when setting your configuration properties and when setting up your TLS an SASL configuration variables.

You can set custom properties for each component in the inventory file using variables. An example is shown below:

all:
  vars:
    zookeeper:
      properties:
        initLimit: 6
        syncLimit: 3
    kafka_broker:
      properties:
        num.io.threads: 15

In the example above, the initLimit and syncLimit properties get set in the ZooKeeper properties file. The num.io.threads property gets set in the Kafka properties file.

JMS Exporter

Additionally, a variable is available that allows you to enable JMX Exporter on your Kafka brokers. The following example shows how to enable this property.

## JMX Exporter is disabled by default. When enabled, JMX Exporter jar will be pulled from the Internet and enabled on the broker *only*.
## To enable, uncomment this line:
jmxexporter_enabled: true

Configuring Security

The following sections provide instructions for configuring security.

Configuring TLS encryption

By default the components are installed using the PLAINTEXT protocol. This means there is no data encryption. You can enable TLS encryption with one of the following options:

  • Self Signed Certs: A Certificate Authority will be generated by the playbooks and used to sign the certs for each host.
  • Custom Certs: You can provide signed certs and keys for each host as well as the Certificate Authority Cert used to sign the certs.
  • Custom Keystores and Truststores: You can provide keystores and truststores for each host.

Note

By default the TLS connection is one-way TLS. Mutual TLS is also configurable.

TLS encryption variables should be added under the all group in the hosts.yml file. To turn on TLS encryption set the following parameter to true:

ssl_enabled: true

By default, components are configured with one-way TLS. To turn on TLS mutual authentication, set the following parameter to true:

ssl_mutual_auth_enabled: true

By default, the certs for this configuration are self-signed. To deploy custom certificates, there are two options available: Custom Certificates and Custom Keystores and Truststores.

Note

The ssl_enabled property applies globally to all Confluent Platform components except ZooKeeper. Starting from the 5.4 release of the Confluent Platform playbooks, this global setting will apply to ZooKeeper as well.

Custom Certificates

To provide Custom Certificates for each host, you will need the Certificate Authority certificate, the signed certificates, and keys for each host on the Ansible host. Complete the following steps to use Custom Certificates.

  1. Set this variable to true.

    ssl_custom_certs: true
    
  2. Enter the path to the Certificate Authority Cert used to sign each host certificate.

    ssl_ca_cert_filepath: "/tmp/certs/ca.crt"
    
  3. Set the following variables for each host.

    ssl_signed_cert_filepath: "/tmp/certs/{{inventory_hostname}}-signed.crt"
    
    ssl_key_filepath: "/tmp/certs/{{inventory_hostname}}-key.pem"
    

    The variable {{inventory_hostname}} in the example shows that Ansible can read the hostnames set in the inventory file. For this reason, you can keep the inventory file shorter if you put the hostname in the filename for each signed certificate and key file.

    As an alternative, you can set the variables directly under a host:

    schema_registry:
       hosts:
          ip-192-24-10-207.us-west.compute.internal:
             ssl_signed_cert_filepath: "/tmp/certs/192-24-10-207-signed.crt
             ssl_key_filepath: "/tmp/certs/192-24-10-207-key.pem
    

Custom Keystores and Truststores

To provide Custom Keystores and Truststores for each host, you will need to have keystores and truststores for each host on the Ansible host, as well as their passwords. Complete the following steps to use Custom Keystores and Truststores.

  1. Set this variable to true.

    provided_keystore_and_truststore: true
    
  2. Set the following variables for each host.

    ssl_keystore_filepath: "/tmp/certs/{{inventory_hostname}}-keystore.jks"
    ssl_keystore_key_password: mystorepassword
    ssl_keystore_store_password: mystorepassword
    ssl_truststore_filepath: "/tmp/certs/truststore.jks"
    ssl_truststore_password: truststorepass
    

By taking advantage of the {{inventory_hostname}} variable and setting the same password for each host, you can set these variable once under the all group in the hosts.yml file. As an alternative, you can set these variables under each host as shown in the alternative Custom Certificates example.

Configuring SASL Authentication

Confluent Ansible can configure SASL/PLAIN or SASL/GSSAPI (or Kerberos). By default, there is no SASL authentication.

SASL authentication options

By default the components are installed without any SASL authentication. You can enable the following SASL options:

  • PLAIN: SASL/PLAIN uses a simple username and password for authentication.
  • Kerberos: SASL with Kerberos provides authentication using your Kerberos or Active Directory server.

Note

Kerberos Key Distribution Center (KDC) and Active Directory KDC configurations are not currently configured by the Ansible playbook.

Configuring SASL/PLAIN

To configure SASL/PLAIN security, set the following option in the all group in the hosts.yml file.

sasl_protocol: plain

Configuring SASL with Kerberos

To configure SASL with Kerberos, you need to create principals within your organization’s KDC server for each component and for each host in each component. You also need to generate keytabs for these principals. The keytab files must be present on the Ansible host.

Note

You need to set up your own Key Distribution Center (KDC) independently of the playbook and provide your own keytabs.

The following example shows the Kerberos Configuration variables in the all section of the hosts.yaml file. This installs Kerberos packages and configures the /etc/krb5.conf file on each host.

all:
  vars:

  ...omitted

  #### Kerberos Configuration ####

  kerberos_kafka_broker_primary: <Name of the primary set on the kafka brokers principal (eg. kafka)>
  kerberos_configure: <Boolean for ansible to install kerberos packages and configure this file: /etc/krb5.conf, defaults to true>
  kerberos:
    realm: <KDC server realm eg. confluent.example.com>
    kdc_hostname: <hostname of machine with KDC running eg. ip-192-24-45-82.us-west.compute.internal>
    admin_hostname: <hostname of machine with KDC running eg. ip-192-24-45-82.us-west.compute.internal>

Note

If the hosts already have the /etc/krb5.conf file configured, make sure to set kerberos_configure to false.

The following examples show the variables that need to be added to the hosts.yml file for each component to use Kerberos for authentication. Each host must have these variables set in the hosts.yml file.

Zookeeper section

Use the following example to enter the zookeeper_kerberos_keytab_path and zookeeper_kerberos_principal variables. Each host needs to have these two variables. Three hosts are shown in the example.

zookeeper:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-37-15.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-34-224.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Kafka broker section

Use the following example to enter the kafka_broker_kerberos_keytab_path and kafka_broker_kerberos_principal variables. Each host needs to have these two variables. The example shows three hosts.

kafka_broker:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-37-15.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-34-224.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Schema Registry section

Use the following example to enter the schema_registry_kerberos_keytab_path and schema_registry_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

schema_registry:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      schema_registry_kerberos_keytab_path: /tmp/keytabs/schemaregistry-ip-192-24-34-224.us-west.compute.internal.keytab
      schema_registry_kerberos_principal: schemaregistry/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Kafka Connect section

Use the following example to enter the kafka_connect_kerberos_keytab_path and kafka_connect_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

kafka_connect:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      kafka_connect_kerberos_keytab_path: /tmp/keytabs/connect-ip-192-24-34-224.us-west.compute.internal.keytab
      kafka_connect_kerberos_principal: connect/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

REST proxy section

Use the following example to enter the kafka_rest_kerberos_keytab_path and kafka_rest_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

kafka_rest:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      kafka_rest_kerberos_keytab_path: /tmp/keytabs/restproxy-ip-192-24-34-224.us-west.compute.internal.keytab
      kafka_rest_kerberos_principal: restproxy/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

KSQL section

Use the following example to enter the ksql_kerberos_keytab_path and ksql_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

ksql:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab
      ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Control Center section

Use the following example to enter the control_center_kerberos_keytab_path and control_center_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

control_center:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      control_center_kerberos_keytab_path: /tmp/keytabs/controlcenter-ip-192-24-34-224.us-west.compute.internal.keytab
      control_center_kerberos_principal: controlcenter/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Migrating to Confluent Server

Confluent Platform 5.3 (and later) includes a commercial component named confluent-server. Confluent Server is a Kafka broker with support for commercial features like Confluent Operator, the Kafka Java client, Kafka Streams, and Kafka Connect. Confluent Server also includes proprietary plugins for the security features Confluent LDAP Authorizer and Role-based access control.

Confluent Platform also includes a version of confluent-kafka, which is an open source Apache 2.0 licensed component. Both confluent-community and confluent-platform packages have a dependency on confluent-kafka. You can migrate from confluent-kafka to confluent-server using the instructions in the following two sections.

Red Hat Enterprise Linux

Complete the following steps to migrate to confluent-server when using Red Hat Enterprise Linux (RHEL).

  1. Install your cluster using cp-ansible. This installs confluent-kafka by default.

  2. Start your cluster and verify that it is working as expected.

  3. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka
    
  4. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
    sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp
    

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
    
  5. Use yum swap to change to confluent-server:

    yum swap confluent-kafka-2.12 confluent-server
    
  6. Copy your Kafka broker configuration backup to the broker configuration directory:

    sudo cp /tmp/server.properties /etc/kafka/
    
  7. Copy your override file using the following command:

    sudo cp /tmp/override.con /etc/systemd/system/confluent-server.service.d/
    

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  8. Copy your JAAS file to the broker configuration directory:

    sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/
    
  9. Start your Kafka brokers:

    sudo systemctl start confluent-kafka
    

Complete the following steps to change from confluent-server back to confluent-kafka.

  1. Log in to each Kafka broker host and stop each broker:

    sudo systemctl stop confluent-kafka
    
  2. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
    sudo cp /etc/systemd/system/confluent-server.service.d/override.conf /tmp
    

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
    
  3. Use yum swap to change to confluent-kafka:

    yum swap confluent-kafka-2.12 confluent-kafka
    
  4. Copy your configuration files to the broker configuration directory:

    sudo cp /tmp/server.properties /etc/kafka/
    

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  5. Copy your JAAS file to the broker configuration directory:

    sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/
    
  6. Start your Kafka brokers:

    sudo systemctl start confluent-kafka
    

Ubuntu

Complete the following steps to migrate to confluent-server when using Ubuntu.

  1. Install your cluster using cp-ansible. This installs confluent-kafka by default.

  2. Start your cluster and verify that it is working as expected.

  3. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka
    
  4. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
    sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp
    

    If running with Kerberos, you must back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
    
  5. Remove confluent-kafka from your Kafka broker hosts.

    sudo apt-get remove confluent-kafka-2.12
    
  6. Install confluent-server on your Kafka broker hosts.

    sudo apt-get install confluent-server
    
  7. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.

    Note

    If you are running Kerberos, complete the next step. If not, skip the next step.

  8. Copy your override file using the following command:

    sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /etc/systemd/system/confluent-server.service.d
    
  9. Start your Kafka brokers:

    sudo systemctl start confluent-kafka
    

Complete the following steps to change from confluent-server back to confluent-kafka.

  1. Log in and stop each Kafka broker host.

    sudo systemctl stop confluent-kafka
    
  2. As a precaution, back up the following configuration files:

    sudo cp /etc/kafka/server.properties /tmp
    sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp
    

    If running with Kerberos, you need to back up your JAAS file:

    sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
    
  3. Remove confluent-server from your Kafka broker hosts.

    sudo apt-get remove confluent-server
    
  4. Install confluent-kafka on your Kafka broker hosts.

    sudo apt-get install confluent-kafka-2.12
    
  5. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.

  6. Start your Kafka brokers.

    sudo systemctl start confluent-kafka
    

Additional information

This repository makes use of the systemd scripts provided with the Confluent Platform. For this reason, there is an expected default user/service mapping that uses the prefix convention cp- followed by the service name. For example cp-kafka or cp-schema-registry. The one exception is that Apache ZooKeeper™ runs as the cp-kafka user. This matches the systemd scripts as well.

Troubleshooting

The playbook may fail to run for a variety of reasons. Complete the following steps if the playbook fails:

  1. Append -vvv to the playbook run command and pipe it to a file.

    ansible-playbook -vvvv -i hosts.yml all.yml >failure.txt
    
  2. Open a support ticket with Confluent Support and provide the following:

    • Playbook name you are running.
    • The step where the playbook failed.
    • All changes you made to the playbook.
    • Attach the output from the failed test as a compressed text file.